How To: Upgrade VMware vCloud End to End

Now that vCloud Director 1.5 has been released I wanted to let folks know about the upgrade process that I happened to try out in my lab for the entire stack.  This is meant to be only a high level overview, but covers all the major components of vCloud NOT just vCloud Director.  More detailed steps might be forthcoming however those were developed for internal use at this point in time.  That being said you can use the high level steps below to understand what order things should be done in.  It is fair to say that you can do things out of order, but you might have some issues on the vShield side.  I will add some screen shots later, but for the sake of time I wanted to get this out there.  It is important to note that VMware has upgrade guides for each of the products and those should still be used.

References:

Chapter 3 of the vCD install guide.
The vSphere 5 upgrade guide.
The upgrade section of the Chargeback release notes.

All of this is the official documentation released with each product as it is part of every VMware product release

Other Known Items I found in my lab in addition to what is listed in the Release Notes:

  • Migrating from Oracle to SQL will not be supported by VMware at this time
  • Upgrading from 1.0.1 to 1.5 updates the database tables therefore in multi-cell deployments the first cell that issues the Database upgrade commands will require all cells are disconnected from the database.  This implies downtime on the portal and API access, but only during the database upgrade
  • The vCloud 1.0.1 agent will not work with vSphere 5 so it is not recommended by me to upgrade vSphere first.  If you do vCloud Director will not function until itself is upgraded to vCD 1.5
  • Install directory changed to /opt/vmware/vcloud-director but your certs will most likely be in the old directories.  If you want to clean that up you can move the certificate store and response.properties files to the new directories and re-run the config script.
  • vCloud Connector 1.0 does not work with vCloud Director 1.5 for moving Virtual machines.  vCloud Connector 1.5 is currently in BETA.
  • vCenter ChargeBack 1.6.2 is ONLY Supported with vCloud Director 1.5 and should not be upgraded until after vCloud Director so stop the services before you being it will just not work otherwise.

Step #1 – Upgrade vCloud Director Cell(s)

This step will might require the use of the Cell Management Tool in order to quiesce the cells properly.  If you are using a load balancer there is instructions on upgrading the cells by creating separate load balancer pools.  However it is important to note from 1.0.1 to 1.5 there is a database upgrade that will be run from at least one cell, but you will need to ensure all cells are shut down before updating the database.  You can upgrade the binaries on all cells, but only issue the database upgrade from one.  You can time this as you see fit to minimize downtime, but some there will be a period of outage until one cell is brought back up on the new database tables.  You should also be able to see that the ESX hosts are still connected and available or “Prepared”.

If you are using NFS for multiple cells you will also note the directory structures changed to /opt/vmware/vcloud-director, so be sure to update your FSTAB mount to the new directory.  The old one will still be there but not a good idea to leave the old stuff behind.  For this reason I also did the reboot of the cell to re-map the mount point.  If you elected to move your certificate store you will need to re-run the config script to pick up the new location and as always keep your response.properties files.  Then you could clean up the old directories that were left behind.

Step #2 – Upgrade vShield Manager

In this previous post I addressed the fact you want to use the .TAR files to upgrade the vShield Manager.  This is absolutely the case and the last thing you want to do is remove your vShield Manager Virtual Machine.  However once upgraded you will notice there is an entirely new security model.  After upgrading the vShield Manager it will no longer be connected to vCenter.  You will need to use a service account to connect it back to vCenter and assign this user as an “Enterprise Administrator” in vShield Manager.  After you have ensured the vShield Manager is reconnected with the proper service account, you must also update the vCloud Director connection under vCenter properties to also use this account to talk to vShield Manager.  Previously most customer’s used “Admin” or other local account.  In my testing I always used service accounts but new local accounts may work.  I simply have not tried it yet in my lab.

Step #3 – Upgrade vShield Edge Devices

At this point you can choose to upgrade all the vShield Edge devices you have deployed.  However, per the Release Notes, you should wait at least 5 minutes before resetting so the vShield Manager can notify vCloud Director it has been updated.  You may want to ensure vCloud Director and vShield Manager are all working properly first by deploying a new NAT routed network and seeing that the right accounts are being used in vCenter and things are deployed properly.  This step is actually very easy and requires simply “Resetting” the NAT routed networks.  You will see a new vShield Edge get deployed with .updated in the name.  Eventually the old Virtual Machine will go away and the new one will be renamed to look like the original.  I have seen a small network connectivity loss to the vApps, but nothing significant.

Step #4 – Upgrade vCenter Chargeback to 1.6.2

This is the easiest of all the steps as the change to 1.6.2 is needed in order to support vCloud Director 1.5’s new database tables.  There is some other items in the release notes to take into consideration about what features like VPN Chargeback can and cannot track.  I did also learn that Chargeback 1.6.2 is not backward compatible with vCloud 1.0.1 or 1.0 as indicated in the release notes.  This is due to the database changes with vCloud Director 1.5.  So no matter when you update this component, you may want to stop the services until vCloud DIrector and the Database are upgraded.  Conversely you can reverse Step #1 and Step #2 if you choose.  Either way it is a rock and a hard-place in regard to Chargeback Data compatibility with the old and new vCloud Director Databases.

Step #5 – Upgrade vSphere vCenter

Now you can either decide to remain on vCenter 4.1 or move to vSphere 5 for all the added goodies.  This should be a standard update to vCenter as you have done in the past.  Nothing I found in my lab changed, however if vCloud Director is only configured for one vCenter, you will lose the ability to deploy and manage Virtual Machines during this portion of the upgrade.

Step #6 – Upgrade vSphere ESXi

Thank heaven for Update Manager right?  You can use Update Manager to update all your ESXi hosts, but be sure to read any warning messages that may popup on things like agents as third party agents may need to be removed.  Once the host is upgraded it should still show in vCloud Director as “Prepared” with the new vCloud DIrector Agent.  I did not have to un-prepare and re-prepare in my Lab.  You may also need to remove third party agents manually before upgrading in case Update Manager is not able to remove them properly for you.

Step #7 – Upgrade vSwitches, VMFS, Tools, Virtual Hardware, and finally set vCloud to use Hardware Version 8

At this point there is more tedious effort involved but this could be done by multiple people.  To take advantage of some new features you need vCloud Director to be set for a maximum hard version of 8.  However, you don’t want to do that until you have upgraded ALL the ESX hosts in a provider vDC, AND updated the tools and hardware on the currently deployed vApps.  In turn do not forget to update you’re vApp Templates.  This is SOP as with other major ESX upgrades.  You will want to upgrade the tools first, then update the “Virtual Hardware” to maintain IP addresses, but also remember vCloud Director sets the IP’s from pools for you so this should not be as big a deal as in standard vSphere.  Also upgrade you VMFS to version 5 as well as your vNetwork Distributed Switches.  This final change will be setting vCloud Director to use the highest hardware version available as 8.

This should get you going, and I will add some screen shots from my lab as I will run through this one last time with the final GA bits of all products, but I suspect it will go smoothly.  The real thing to remember is the bullets at the beginning about Database migrations and ESX compatibility.  As Always be sure to know what is actually SUPPORTED with vCloud Director 1.5, and that article I will also be updating.

 

About Chris Colotti

Chris is active on the VMUG and event speaking circuit and is available for many events if you want to reach out and ask. Previously to this he spent close to a decade working for VMware as a Principal Architect. Previous to his nine plus years at VMware, Chris was a System Administrator that evolved his career into a data center architect. Chris spends a lot of time mentoring co-workers and friends on the benefits of personal growth and professional development. Chris is also amongst the first VMware Certified Design Experts (VCDX#37), and author of multiple white papers. In his spare time he helps his wife Julie run her promotional products as the accountant, book keeper, and IT Support. Chris also believes in both a healthy body and healthy mind, and has become heavily involved with fitness as a Diamond Team Beachbody Coach using P90X and other Beachbody Programs. Although Technology is his day job, Chris is passionate about fitness after losing 60 pounds himself in the last few years.

9 comments

  1. Hi Chris,
    Good to meet you at VMworld…..!
    Great article btw. Do you have any tweaks for the process of updating a vCD 1.5 beta (single cell) lab deployment to vCD 1.5 GA?
    Cheers.
    Jeremy.

    • Any upgrades no matter how small from BETA to GA builds are never supported or suggested. There was some changes from the beta builds and I am not sure how much to the database tables alone. Personally I always rebuild myself so I have no shortcuts to offer there. Just as easy to rebuild it IMHO.

  2. Hi chris
    Great job
    You say to use VUM
    First time we used it, it delete all vCloud stuff in the hosts. It was a major pain ( decembe r2010). are you sure it is fixed right now ????

    Gus

    • I am not sure what you mean. VUM will remove any agents not supported with ESXi 5 and will give you an error message. If the host was 4.1 and vCD is upgraded forst as indicated the new agent is installed. I can check again next week as I will be doing this one last time myself with GA code, but if anything vCD should see the host is a new version with or without the agent and push down the vCD 1.5 agent after it is upgraded. ESXi HAS to be upgraded AFTER vCloud Director is moved up to 1.5 this is not an option as the vCD 1.0 agent will not load to ESXi5. The order is very specific for a lot of reasons and it is why ESXi is done last in sequence. Without knowing the order or the release versions you used I cannot say for sure what happened.

  3. Couple of gotcha’s to be aware of when upgrading.

    http://virtual-grind.com/tag/vcd-upgrade-fail/

    If you plan to stick with an Oracle DB, check the maximum concurrent processes parameter of your vCD database is set high enough BEFORE running the upgrade script. Failure to do so results in broken database.

    KB article here: http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&externalId=2006919

Leave a Reply

Your email address will not be published. Required fields are marked *