How To: Upgrade vCloud Director in Detail – Part 1

Assumptions

This section also assumes that customer’s will have multiple vCloud Director Cells front-ended by a load balancer in a production-like environment. This also assumes that the transfer folder location was mapped to an NFS share as part of the multi-Cell Requirement. It is also assumed before starting this upgrade that all functions of vCloud Director are working properly. Two key functions would be the ability to deploy a new vApp as well as deploy a NAT Routed network with a vShield Edge.

Personnel Resources Needed

• VMware vCloud Director System Administrator
  
• VMware vCenter Administrator – with access to console of vCenter
  
• VMware vShield Manager Administrator
  
• Linux Administrator with Root Access to the vCloud Director Cells and with passwords to cell consoles, cell root login, SCP to cells (to transfer vCD installer and cell management tool),
  
• Windows Administrator with access to the Chargeback server and service account
  
• Database Administrator – For backups, installers, installs, etc
  
• Active Directory Administrator
  
• Network/Load Balancer Administrator – To redirect load balancer to temporary page

Phase I Impact

This phase of the upgrade will cause downtime the following components will be affected by this step in the upgrade process.

• vCenter Chargeback – Version 1.6.2 is only be used with vCloud Director 1.5, although this is not stated in the release notes. Therefore if you upgrade before vCloud Director you will have issues talking to vCloud Director 1.0.1. It is recommended you simply stop the services and proceed to upgrade Chargeback after vCloud Director is fully upgraded.
  
• vShield Manager – vShield Manager usually requires a specific build to work with vCloud Director. Although older vShield Managers will continue to work, you will have to upgrade this as part of this step to enable the new features of vShield.
  
• vCloud Portal and API – The vCloud portal and API will not be available during this phase. It is difficult to determine specific downtime, as it will depend on the number of Cells and size of each customer’s database. This also means the VMRC will not be available to users.
  
• vCloud Director Database – The upgrade does change the schema, so a database backup is important.
  
• vCloud Connector 1.0 – This version will no longer work with vCloud Director 1.5, and possibly not with vSphere 5. The vSphere connection has not been tested, but there is a new completely different architecture for vCloud Connector 1.5 currently in BETA. Just be aware that version 1.0 of vCloud Connector will no longer work.
  
• Rollback – This would require a lot of effort because the database changes are irreversible. It is recommended to back up the database after stopping the vCloud Director Services before moving forward; however, after step 3.5.2 there is no easy path to roll back without major downtime and possible data loss. You should also back up the vShield Manager database using the UI FTP commands. This is the only method of even possibly restoring the vShield Manager for a re-deployment. It is also recommended to take a backup of the vCenter database at this time, as well as during future steps throughout this document as the vCloud Director services get started multiple times throughout the process and can affect the vCenter and vShield Manager databases.

Phase I Advantages

The following is a list of advantages of upgrading to vCloud Director 1.5; however, not all new features will be available upon completing just Phase I. This is not meant to be an exhaustive list, only major items that customers have expressed as their reason to upgrade to 1.5.

• Added vCloud Director Database Support – Microsoft SQL Server and newer versions of Oracle. See release notes for further information
  
• Updated API with Call-Outs
  
• Fast Provisioning – This will not function without HW8 support and the completion of Phase III, and possible redesign of existing vCloud Organizations and/or Allocation Models may be needed.
  
• Advanced vShield Management
  
• Enhanced vApp Properties

Upgrading VMware vCloud Director Cells

The basic upgrade instructions are located in the VMware vCloud Installation Guides and are as simple as copying over the installer file and running it. However, the upgrade to vCloud Director 1.5 does update the database schema. You will also need to use the Cell Management tool to quiesce all the cells in the installation prior to upgrading.

Back Out Plan for the vCloud Director Cells

At this point it could be possible to roll back the upgrade, but this has NOT been tested as of yet and would need to be done before ANY new items are deployed in vCloud Director as those objects would be in the new database table structure. If a decision is to be made to roll back it needs to be done at this point so the database backup could be restored, and vCloud Director 1.5 uninstalled and 1.0.1 re-installed. Once you continue to move forward it becomes increasingly difficult to roll back due the additional functions in vShield Edge and the separate vShield Manager database. Additionally once users are back on the system and new objects are deployed there is no point of return at all (without object loss and much manual cleanup of existing vCenter objects no longer recognized by vCD, which still aren’t guaranteed to function properly).

There appears to be a few possible options to assist in rolling back the upgrade:

• Snapshot the vCloud Director Cells – We have always recommended that the cells are virtual machines. Therefore the snapshot functions in vCenter can be used.
  
• Backup the entire vCloud Database – It is also a recommendation that you backup the entire database for vCloud Director.
  

If both the snapshot and the backup are taken, one should be able to restore the database and revert/delete the snapshot. If items are deployed between the time of the backup for validation and testing there could be some items to clean up as those objects would not have been in the original backup.

Important Considerations and Requirements:

• Take a Full Backup of the vCloud Director Database
  
• Stop the services completely on vCenter Chargeback as only version 1.6.2 is compatible with vCloud Director 1.5. The 1.6.2 version is also NOT backward compatible with vCloud Director 1.0.1 so it is just easier to stop the services all together and upgrade this component last.
  
• Run required Red Hat patch updates prior to running the vCloud Director installer. The package dependencies have updates that seem to be required by vCloud Director 1.5.
  
  • DO NOT update kernel or other packages that would essentially bring the system to an unsupported version of RHEL.
    
  • In most cases you may want to update just the pre-requisite packages needed by vCloud Director 1.5 from the Installation guides.
    
  • Refer to the documentation, but there are new versions of RHEL that are supported just be sure not to patch beyond those.
    
• Consider lowering the TTL (Time to Life) on the DNS for the Load Balanced VIP’s for HTTP and Console Proxy a day or two prior to upgrading
  
  • Lowering these will allow clients to update their DNS cache quicker when resolving the portal name. Since we are re-directing the DNS name to temporary Maintenance pages and back a lower TTL will prevent the need for users to manually flush their DNS cache for updates.
    
• Redirect the DNS prior to upgrading to a custom Maintenance Page on a completely separate web server outside of the vCloud Director Cells. Ensure all users are now redirected to the Maintenance Page before shutting down cells.
  
• Install the Cell Management tool and follow the KB instructions for gracefully shutting down a Cell also shown below. Username can be any SYSTEM level user including an LDAP account. Perform this on ALL vCloud Director Cells. Even though user connections have been redirected, the Cells can still be performing tasks.
  
  • Log in to the target cell as root.
  • Navigate to $VCLOUD_HOME/bin/ 
    • (typically /opt/vmware/cloud-director/bin/).
      
  • Suspend the scheduler by typing the following command:
    • ./cell-management-tool -u -p -q true
      
  • View running tasks by typing the following command:
    • ./cell-management-tool -u -p -t 
      
  • When the task count reaches zero, shut down the Cell by typing the following command:
    • ./cell-management-tool -u -p -s
      
• Once all Cells are shut down and there is no connections to the database and you should have a Database Administrator back up the database
  
• You can run the vCloud Director installer
  
  • Version 1.5 creates a NEW separate directory called /opt/vmware/vcloud-director, but maintains the old directory (/opt/vmware/cloud-director) where most likely certificates.ks is located. There is no need to remove the old directory
    
  • Do not start the services yet
    
  • You will also need to update /etc/fstab for the new NFS mount point as it will be referring to the OLD transfer location under /opt/cloud-director. It is important to note that the installer will appear to MOVE the mount point to the new directory location but if the server is rebooted the cell will not start because FSTAB is still mounting the old location on each system boot.
    
  • On the first Cell upgraded you can also run the Database Upgrade script located at /opt/vmware/vcloud-director/bin/upgrade
    
    • NOTE: This is NOT done on the first server installed ONLY on the subsequent servers.
      
• Repeat the installer on all other Cells
  
  • DO NOT run the database upgrade script again
    
• Start the services on each Cell one at a time by REBOOTING one cell at a time to ensure the NFS shares are mounted.
  
  • Validate with /opt/vmware/vcloud-director/logs/cell.log that the services are coming back up
    
  • Also validate each Cell by connecting to their respective host HTTP ports and log into the portal
    
• DO NOT REDIRECT THE LOAD BALANCER nor allow users back onto the system yet
  

vCloud Director Connectivity to Hosts

VMware vCloud Director should update the host agent on all connected hosts. You should check the connected hosts in vCloud Director to ensure they are still showing available and “Prepared”. The vCloud Director 1.5 agent is different than the previous version and once installed should remain when you get to the ESXi Upgrade steps

Validation Steps to Perform After Cells Are Upgraded

The following two simple steps should be performed to ensure that functions are still working before moving to the next major step. These are basic but will ensure that there is a baseline of functionality as you move forward.

1. Deploy a new vApp
   
2. Create a NAT routed network (either a routed Organization Network or Fence a vApp that is deployed)
   

If either of these steps fail, stop here and troubleshoot the issue before moving on.

1. Rolling Back the Upgrade

If you decide you want to roll back at this point you can restore the database backup and revert/delete the snapshot. It should be understood that as you progress you will get closer to the point of no return for Phase I.