I have seen it get disabled so many times to “Power on Just one more Machine”, but that excuse gets repeated 100 times over, without actually addressing the real problem which is usually cluster capacity.  Or worse yet HA admission control gets disabled on a cluster never to be re-enabled again, because the vSphere administrators got to busy and forgot.  Then months later when the cluster is falling down with no more capacity, OR a host actually fails and things don’t power on  they blame vSphere.  After you investigate you see that HA admission control was in fact disabled, most times without a proper change request.  Okay, can you tell I have seen this one too many times?

Now what about vCloud Director and HA admission control?  Well the same rules still apply, but it’s even more important in my mind.  This is because you don’t only have vSphere administrators adding virtual machines, you have consumers doing it.  Without HA admission control, vCloud Director has no idea that the cluster can or cannot support a failure.  That is a vSphere function to tell vCloud Director when the tank is full.  If you go under the covers and mess with HA admission control under vCloud Director, you add the potential to over subscribe your provider vDC cluster.

Don’t forget in reading Gabe’s article that if you knew it or not HA Admission Control uses per virtual machine reservations.  in vCloud director, there are TWO allocation models that assign per virtual machine reservations Specifically.  Pay as you go and Allocation pool, do this, and you can read more about that in this article.  Per mine and Duncan’s comments in Gabe’s article you also do not want to forget about overhead reservation with or without vCloud Director.

The real trick is to make sure you configure HA admission control policy for the best fit to your needs.  Many people still use N+1, but more people are starting to use percentage based.  Then be sure to add capacity to those clusters that are the provider vDC’s behind vCloud Director.  Simply disabling it will do nothing for you but get you in a situation that will be hard to level back out.  I treat HA admission control with vCloud Director like DRS…Don’t Disable DRS….or HA admission control.  If someone can provide me with some valid useful reasons why you would want to disable it I’m all ears.

There is a couple of things you need to know first about vCloud Director Fast Provisioning.

1. It is Enabled on a PER organization vDC level so it is either on or off.
2. ONLY System Administrators can consolidate Fast Provisioned virtual machines, Organization Administrators cannot
3. Once disabled existing machines will remain fast provisioned

The real key that I wanted to look at here was deploying items back and forth from the catalog with the feature enabled.  So what I setup was pretty basic.

• Master Organization with a published catalog and Fast Provisioning DISABLED
• Customer organization with local catalogs and Fast Provisioning ENABLED
• Both Organization vDC’s are Pay-As-You-Go for reference
• The template is CentOS 6.2 minimal exported/imported from vCenter.

The rest of this post will be various operations in certain orders to see what happens with and without fast provisioning enabled on a consumer organization.

The Shadow Virtual Machine

In some cases, in order for fast provisioning to work it will use the concept of a Shadow VM on each datastore where a linked clone will live, but in many cases it may not get created for some time.  This VMware KB has a lot of good information, and a couple of key points taken from it are as follows:

• The source template virtual machines are called primary virtual machines
• Shadow virtual machines are created on demand
• Subsequent copies to the same datastore are fast
• Org Admin/User only sees the ‘source’ virtual machine. Shadow virtual machines are an implementation detail that are only visible to vCloud Director administrators.
• Shadow virtual machines stored in System vDC
• Shadow virtual machines disk space billed to the service provider

Shadow virtual Machines are only created once a clone needs to be placed on a storage volume different from where the original one is located.  Until that needs to happen everything else is done on the same storage.

Initial vApp Template Import Into the Catalogs

The first thing I wanted to see was taking an OVF, and importing to each catalog to see what happens.  Obviously on the Master organization the catalog item will be imported as a thick copy, but I was not 100% sure on the Fast Provisioned vDC.  Interestingly, From what I saw both vApp templates were brought in as full copies into the catalog by the initial import from OVF.

However I actually tested COPYING the vApp template from the Master Catalog to the Org’s local catalog.  In this case the copy was actually a linked clone.  It created the initial snapshot and then made the local Org’s catalog version a linked clone to the original just as if it was deployed to the cloud itself.  I found this interesting as it leads me to something we will discuss later about updating and re-adding items to the catalog.

The Use Case

What we see is a pretty common use case for why I am testing this.  It has also been asked how does someone deal with patching and updates once these catalog’s are linked together.  The consumer wants to deploy the Guest Operating System from the provider’s published catalog, customize it, and save a copy to their local organization catalog.  The consumer’s organization is enabled for Fast Provisioning, but they may want to ensure their local catalog chain does not link back to the master catalog.

Deploying from vCloud Director Shared and Local Catalogs

Where this gets really interesting is on the deployment of each version of the vApp Catalog item.  Now we are going to only work off the Master Shared Catalog since that’s what most people would do.  The first time a deploy operation of a vApp Template to a Fast Provisioned vDC is requested, the original virtual machine is put into snapshot mode.  This means the actual virtual machine that is the catalog vApp in the master organization is now in snapshot mode indefinitely.  If there happens to be a Shadow VM required, that would be created, then a snapshot taken.  At this point forward anything deployed from that catalog virtual machine will be deployed from the base snapshot.

Now that we have a deployed vApp as a linked clone, we can update it, patch it, add new applications to it, whatever we want to treat it like a normal virtual machine.  Let’s say we want to save this to our LOCAL catalog at this point.  When we make that copy to the local catalog, we will get a full copy however there is a catch.  The full copy appears to be a full copy of the deployed virtual machine’s Delta Disk as you can see in the info below.  As we can see below the VMDK is still pointing back to the original catalog base disk.

Let’s take this one step further, and now deploy a vApp from this new local catalog virtual machine.  What we see here is that the newly deployed virtual machine from the local catalog is also linked back to the original master VMDK.  This means that as this consumer edits, saves, and re-deploys they are always saving Delta files and referencing back to the original disk.

Master Catalog VM (Orange)

# cat CentOSMinimal\ \(ff68332f-8a07-4739-9c75-4d945dbec22b\).vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=a01c48e3
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"
# Extent description
RW 33554432 VMFS "CentOSMinimal (ff68332f-8a07-4739-9c75-4d945dbec22b)-flat.vmdk"

First Consumer VM (Green):

# cat CentOSMinimal\ \(14775f16-7089-4a8e-8993-3045bed173e7\).vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=a01c48e3
parentCID=a01c48e3
isNativeSnapshot="no"
createType="vmfsSparse"
parentFileNameHint="/vmfs/volumes/4f316743-5041e641-2229-005056890004/CentOSMinimal (ff68332f-8a07-4739-9c75-4d945dbec22b)/CentOSMinimal (ff68332f-8a07-4739-9c75-4d945dbec22b).vmdk"
# Extent description
RW 33554432 VMFSSPARSE "CentOSMinimal (14775f16-7089-4a8e-8993-3045bed173e7)-delta.vmdk"

Consumer Local Catalog VM (Green):

# cat CentOSMinimal\ \(f308e901-60b2-406b-88a1-fb3bcc58540a\).vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=a01c48e3
parentCID=a01c48e3
isNativeSnapshot="no"
createType="vmfsSparse"
parentFileNameHint="/vmfs/volumes/4f316743-5041e641-2229-005056890004/CentOSMinimal (14775f16-7089-4a8e-8993-3045bed173e7)/CentOSMinimal (14775f16-7089-4a8e-8993-3045bed173e7).vmdk"
# Extent description
RW 33554432 VMFSSPARSE "CentOSMinimal (f308e901-60b2-406b-88a1-fb3bcc58540a)-delta.vmdk"

Consumer Re-Deployed Catalog VM (Purple):

# cat CentOSMinimal\ \(f5b612e7-512b-4730-aff8-ee9489012ffe\).vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=a01c48e3
parentCID=a01c48e3
isNativeSnapshot="no"
createType="vmfsSparse"
parentFileNameHint="/vmfs/volumes/4f316743-5041e641-2229-005056890004/CentOSMinimal (f308e901-60b2-406b-88a1-fb3bcc58540a)/CentOSMinimal (f308e901-60b2-406b-88a1-fb3bcc58540a).vmdk"
# Extent description
RW 33554432 VMFSSPARSE "CentOSMinimal (f5b612e7-512b-4730-aff8-ee9489012ffe)-delta.vmdk"

Something to Consider – Break The Chain

Based on now knowing how some of this works, you want to decide how you to handle fast provisioned Org vDC’s since once they are enabled you can see how everything from that point on will be based on linked clones.  Something many providers and consumer organizations alike may want to do is break the link chain of their local catalog so they are always deploying from a full copy within their local organization catalog.

This can in fact be done with a few steps.  Remember in the beginning when I stated that only a System Administrator can consolidate?  That comes into play here, as once a vApp is checked into the local catalog a system admin OR something with system administrator credentials, like vCenter Orchestrator or other scripting method, can consolidate the virtual machines in the consumer’s local catalogs.  Below you can see the vCloud Director interface option on the virtual machine in the consumer’s catalog.  Using a service to do this for you would certainly be a better way to go.

This will break the chain so that only the deployed virtual machines will have a link to the local catalog VMDK only instead of back to the master catalog’s VMDK.  This would isolate the local consumer’s deployed virtual machines and catalog chains from the Master Catalog.  However, this process will still exist anytime a virtual machine is deployed for updates then placed back into the catalog in the fashion described above.  The advantage to this is that the consumer’s can all deploy in this fashion and in THeory the provider can have nothing linked to their original templates allowing them to patch them and replace them as they need:

• Start with the provided templates
• Customize them
• Add them to their catalog
• Consolidate to break the chain
• Remove the linked customized vApp

To accomplish this of course you need some custom intervention and have your consumer’s deploying via custom tools that interface with things like vCenter Orchestrator.  Either way it can be done such that the original virtual machines have no links to them and the consumer’s all have the links to their own local copies.  Figuring out all the hooks to leverage the API’s to make this happen…..well….that’s for you to figure out.

It should be noted that vCloud Director is smart enough to know that if you delete the item from the catalog and there are virtual machines linked to it, the base disks will remain on storage.  They will not be removed from storage until all linked VMDK’s are eventually removed.  So you can deploy, patch, and remove a catalog virtual machine from vCloud Director, and the linked ones will still function.

The vCloud Director Agent is used for two things:

1. Communicating with the cross host fencing vmkernel module that is installed when the agent is pushed. This is needed when you use VCD-NI network pools.
2. Retrieving virtual machine thumbnails

I was pretty sure I knew about the VCD-NI aspect I just forgot about it.  The image thumbnails on the other hand was quite news to me.  Either way, now we know what the vCloud Director Agent does once it is installed as part of the host preparation.  For information on ways to manually remove the agent see the related posts below.

As part of the Disaster Recovery solution that Duncan Epping and Chris Colotti developed, Kris Boyd has been working on a comparative VMware View Solution using the same basic principles.  However, one thing both solutions came across is a unique situation on the vCenter virtual machines when trying to use Site Recovery Manager’s ability to change the IP Addresses on the Guest OS in the recovery site.  This situation only seems to affect the vCenter virtual machines and is a very specific condition.  It is unique because in most cases Site Recovery Manager is talking to an upper layer vCenter Server, and in our Disaster Recovery solutions there is also a vCenter Server being managed by another vCenter and SRM as depicted below.

Most of us know that SRM has the capability to change the IP on a virtual machine that is part of a recovery plan, and in most cases this works just fine.  There are a few situations, such as Virtual Appliances, that have some issues due to the version of VMware Tools.  Most recently we have seen a specific issue with a vCenter Server (non-appliance) virtual machine as we are doing in both DR solutions.  We wanted to take a few moments to describe the condition and the high level work around that is required to deal with the vCenter Server virtual machine.

Note:  This is not meant to provide the exact steps, but is intended to highlight the areas of consideration.  It also only applies when the recovery site is using a different IP range and addresses must be changed during the failover process.

As you can see from the diagram above, the entirety of infrastructure components were VM’s that were being failed over to the recovery site as managed by Site Recovery Manager.  Since vCenter was one of these machines, we tried, (unsuccessfully), to re-IP that VM along with all of the other servers with static IP’s using the feature in SRM.  As it turns out not only does SRM not re-IP the machine, but the SRM recovery test will fail if you tell it to re-IP vCenter.

What we learned when we dug into this is the real reason why this was failing.  This particular issue has to do with the vCenter services not successfully starting while waiting for the network to come up.  SRM needs the VMware tools running in order to change the IP address on a virtual machine.  in the case of vCenter Server, the vCenter services try to start before the tools are running.  When the vCenter service starts and cannot connect to the database, (due to the fact there is a different network), the service start just hangs.  The Guest sits at “Applying Computer Settings” indefinitely.  This prevents the VMware Tools from even starting, thus preventing SRM from changing the IP address.  It’s pretty much a rock and a hard-place situation.   SRM was successful in its attempt to re-IP all of the other servers, but vCenter required a set of manual steps to accomplish the same task.

These manual tasks also had their own set of challenges, because if you do things wrong you may notice that windows will never let you login to this server again for the same reason.  Without going into the gory details, here is what you need to take away from this:

1. Never assume anything when testing a disaster recovery plan
2. vCenter must have its IP updated manually during a fail over
3. Since other services are dependent on vCenter, you should include a wait step in the SRM recovery process to give you time to update the vCenter IP.

Last week I was going through some tweet history from my European colleagues, and one particular tweet stood out to me by Alan Renouf. He mentioned that “if you are a system admin of any kind, you *should* be using this tool”. That tool being vCheck (see HERE for info, download, & video on how to use). At the time, I think I may have heard of it, but since I had a few free minutes, I thought I’d check it out to see what the buzz was about.

DISCLAIMER: The vCheck script is from Alan Renouf as well as several other contributers. Any troubles you may have running the script in and of itself, you will need to follow-up with Alan on his site so he can assist you.

After downloading vCheck, going through Alan’s video, and ‘tweaking’ things to make it work, I began to think…”can I separate these plugins out so I can have a more ‘niched’ output for what I want to run?”.  Also, this script needs to be ‘configured’ upon initial run, and with all the plugins it has, entering information could take a while (albeit only on the first ‘run’ of the script). What my intentions were was to have a report for my Exchange, VMs, Hosts, vCenter, & virtualized Storage. I’m not really a scripter (one of my goals for this year is to become more proficient in PoSH/CLI), so after some frustrations in trying to do modifications myself, I tweeted out to the VMware/PoSH community for some guidance. ***PLUG: If you’re in Technology & you’re NOT part of some kind of technology community on Twitter, you’re severely hindering your potential career advancement and more importantly knowledge gaining. END PLUG***.  I received some help from one of the more profound experts in the Powershell community, Jonathan Medd, as well as a virualization colleague Todd Scalzott, proficient in PoSH in his own right. Thinking others may want this vCheck Reporting segregation, I thought I would share with everyone what I did…fairly simplistic really (though took some testing to get it to work correctly). I don’t have a blog myself, so thanks to Chris Colotti for allowing me to borrow his blog for this brief post to share with everyone.

So enough of the pre-cursers…let’s get started!

So, first thing to do is obviously download the vCheck files. They come in a zip file. Place them somewhere on your workstation. They will look like this:

If you go into the Plugins folder, you can see all the Plugins that you would potentially need to answer questions for in the initial run through (too many for me to capture on 1 screenshot, but there are roughly 70).

Ok, so let me stop here and say this. You’ve downloaded the files, so before continuing with this post below, for the next ‘step’, I recommend going to Alan Renouf’s site (link above) and watch his “How To” video. It shows the basics of how to run the script. And, it’s only 5-6mins long. Continue reading when done…

Alright, so now to the nitty-gritty. So, to simply have different “groups” of reports, what I did was first make my group folders:

 …then, within each folder, I coped all the files in Fig. 1 above EXCEPT the Plugins folder. I then created a Plugins folder then added the scripts I wanted to run for that particular ‘group’ (e.g. Exchange). Since I may have the potential of running the script as a Scheduled Task to have the report run weekly (or whatever you deem necessary for your environment), I renamed the .ps1 file to mirror the group I’m running it against so as not to confuse myself having several vCheck.ps1 files running a task (e.g. vCheckExchange.ps1 shown below):

Next, you need to add Plugins to your Plugins folder. NOTE: For Exchange, Alan has some Plugins available for download on his site. You can download them HERE. The Exchange Plugins download actually comes with all the vCheck files. You can keep them all or disregard (since you already copy/pasted them as suggested above) and just use the Plugins. Once downloaded, add the Exchange Plugins to your Plugins folder. You can change the name of the .ps1 file for intuitiveness or leave it as is. It’s your choice. If you choose to run this as a Scheduled Task in Windows (you can reference a post on Alan’s site on how to run a Scheduled Task), you can name the Task something descriptive and that should be sufficient. Now the moment of truth! Let’s run the script!

First, change to the directory where you created your ‘group’ folders, and more specifically, to the Exchange folder:

At the prompt, all you need to do is simply run your script by entering: .\vCheckExchange.ps1 (or whatever your .ps1 script name is in the Exchange folder). Your first run of the script, if you recall from above, will prompt you for some information (also, Alan’s video shares some of what is asked as well). Enter the information as you’re prompted. The information in brackets at the end of each question is the sample of what to put, OR it will be the default information if nothing is entered. After you enter information, press ENTER for the next question until completed. I will list a little of what I entered as guidance:

IP or Hostname of server to connect to: your Exchange server to run against, or your vCenter depending upon what ‘group’ you’re running

SMTP server: your Exchange server hostname

Email who will send the vCheck Report: your email address or address of user you have configured in your environment to send notifications from

Email who will receive the vCheck Report: your email address

Name of the report: your choice here

Display report in local browser: your choice; if you do, type $true; if not, type $false. I chose not to because it will be displayed directly in my email

Send email report…once completed: $true (but, your choice)

Add HTML attachment: your choice; not needed if answered $true to any of the other 2 above

You can then define the colors of the next 3 question items. I personally didn’t play with this and left them as default.

Keep default header

Drive details <=%: I wanted to know what drives/volumes on my Exchange server were running close to capacity. So, I defined what I wanted that to be (in percent) and entered my info here.

MAPI Latency >=%: your choice here. Best to use default (0) so you can see what it is.

Largest Mailbox: your choice here. This will show the largest mailbox sizes based off storage group (SG). The number you enter here will be the top 10 (default) or however many you specify

Per DB: I entered $True here because I wanted to see it on all 4 of my DBs (SGs), but your choice.

Top Dumpster Mailboxes: this displays mailboxes with the most Deleted Items. Your choice here (same as Largest Mailbox)

Per DB: same as above. Your choice though.

The remaining questions are either the same or self-explanatory (mostly). Alan explains some more info about other Plugins HERE and HERE. When you’ve answered the last question (“Enabled Plugins” for the Exchange ‘group’), the script will run and you will be sent an email and/or a web browser HTML page opened displaying the report, depending on how you answered the questions.

All subsequent runs will not prompt for information to be entered and will just run:

If you make a mistake in answering the questions, at this point, the only way I’ve found to re-do the script is to re-copy all the vCheck files to the directory. If I get an answer from Alan or Jonathan on how to make changes without having to go through the whole process again, I will post a comment, or submit an update to this post.

Ok, so the remaining ‘group’ folders will need some slightly more tweaking, but not much. And this is where I ran into troubles with my testing. So you should have all the vCheck files/folders in each of the ‘group’ folders (VMs, vCenter, etc.), but the Plugins folder is empty. What you want to do is, from the original vCheck Plugins folder CTRL+Click select & cut/paste only the .ps1 Plugin files that pertain to what group you want them to be in. So, for example, in Fig. 2 above, you would CTRL+Click and cut the .ps1 items you want from that list, then paste them in the Plugins folder within the appropriate ‘group’ folder (VMs, etc.) you want. There are a couple files that need to be in every ‘group’ Plugins folder (except the Exchange folder). They are:

00 Connection Plugin for vCenter.ps1

VeryLastPlugin Used to Disconnect.ps1

…so the script can connect to your vCenter Server and then disconnect. Copy the above 2 files in all your ‘group’ Plugins folder, then add the individual .ps1 plugins you want. In Powershell, CD to the directory, then run the name of your vCheck .ps1 file (e.g. for my VM ‘group’, I will run .\vCheckVM.ps1). You will need to answer some of the same questions as in the Exchange plugin example above, as well as additional information pertaining to some of the Plugins. When complete, you will receive a similar email (or HTML page or attachment) as above, again depending on how you answered to receive the output. All subsequent runs will not need information added.

So, there you go. I thought someone else may benefit from wanting a little more segregation as I did so was willing to do this guest post for those who may be interested.

 2012-03-26 09:43:10,470 | WARN     | consoleproxy              | ConnectionTracker              | Closing connection java.nio.channels.SocketChannel[closed] as it has not initialized in time |

We suspect we see this message because the initial connection has left vCloud Cell 1 with a time stamp, and by the time it arrives to vCloud Cell 2 that time stamp was more than 2 minutes old. Which is well above the 2 second allowed skew.  vCloud Cell 2 ignores the request because its more than the time allowed.  The result is just an inconsistent drop in connections on the Remote Console connections.

I had run into other issues with NTP and vCD listed in the Related Posts Section as well that were new tolerances between 1.0.1 and 1.5 if you want to take a look. It seems NTP is one of those things you always want to check first just to rule it out on the cells.  Be sure to read the other NTP article as well to see some other symptoms.

In the course of examining the vCloud Allocation Models and their deeper relationship with vSphere, we did discover a helpful tip on your catalog vApps.  Essentially you want to create a Pay As You Go Org vDC to house all these catalog items.  The reason is simple without going into too much gory detail.  I will save that for the white paper.

The nature of the The Pay As You Go vCloud Allocation model is to assign resource settings to the individual virtual machines.  The resource pool itself has no settings assigned for limits and reservations.  This means that the virtual machines will not consume the reservations assigned unless they are powered on.  Since you cannot directly power on virtual machines and vApps in the catalog without deploying them, they will never consume any reservations while sitting in the catalog.  They will consume storage, but their per VM settings will not be factored in by vSphere for resource allocation.  The other two models, Reservation Pool and Allocation Pool, will assign reservations on the pool levels removing those resources from the provider vDC immediately.  If your intent is to only use this vDC for published catalogs, you would essentially be removing resources from your provider vDC for no reason.

This makes the Pay As You Go the perfect vCloud Allocation model for provider published catalogs.  Of course if you deploy these vApps, you will then be using the assigned reservations on the virtual machines, but you can of course deploy them to a different org vDC if you choose.  If you want to be really safe also be sure to set the Org vDC hosting these catalogs to 0% reservations.  This way if you DO deploy them to the same Org vDC, there will only be the overhead reservation required to power them on.

You may want to take this into account as a design consideration when you are designing your Organization and vDC’s.  Of course this does not apply to the consumer catalogs in an organization as they may only have access to their org vDC.  The catalog virtual machines for them will also be powered off and not affect their available resources.  The idea here is to make sure the Provider is not consuming Provider vDC resources by using an Allocation Pool or Reservation Pool model for unused virtual machines since those will allocate the reserved memory for the resource pool.

New platform support (Huge!)

• vCloud Director 1.5.1 adds support for the following:
• vCenter Server 5.0 Update 1
• ESXi 5.0 Update 1
• vShield 5.0.1
• Red Hat Enterprise Linux 5 (64 bit), Update 7 (as a supported vCloud Director Server operating system)

Firewall rules can be configured with CIDR blocks, IP ranges, and port ranges
Added system notification for lost connection to AMQP host
Enhanced cell management tool and log collection script
Support for AES-256 encryption for VPN tunnels
Increased the retention maximum for vCenter Chargeback history

What is most interesting is many of the Resolved Issues which I have pulled out what I think are the most notable.

Cannot compose a vApp or add a virtual machine to a vApp after upgrading from vCloud Director 1.x to 1.5

Networking issues on virtual machines running certain guest operating systems

Cannot reset an organization network after upgrading from vShield Manager 4.1 to 5.0

vCloud Director Web interface displays erroneous entries when an LDAP-based Organization Administrator creates a catalog or catalog item

• This was actually a bug Harry Smith and I found together in the field and filed

A Note About Upgrading

Remember that vCloud is a group of products that need to be upgraded in sequence for the most part.  I posted some articles on upgrading from 1.0.1 to 1.5 which document the order.  If you are going from 1.5 to 1.5.1 the same basic order should apply for the smoothest upgrade.  You can see the order in the first article broken down by the four phases under Related Posts below

Many people who have been following me know that my only other hobby besides work is Health and Fitness.  About a month ago I started reaching out through Twitter and a new blog site, Virtual Fitness, to help others get fit if they wanted to change-up their routines.  The interesting thing I have found is that many people in the VMware community and IT space can relate to my story.  Many of us travel for a living, we eat out all the time, and generally we are just not healthy.  A few years ago I decided to turn all that around using P90X, and recently P90X2.  I decided I wanted to share and help others in our industry do the same because I know how hard it can be.  So I decided to become with is known as a Team Beachbody Coach.

The response has been great so far!  We have one group of folks from literally around the world, Tim Gleed is in Australia, working together to do Virtual Fitness.  We are already looking to get additional groups together, and there is even talk of some vFitness activities for VMworld with Wade Holmes and a few others.  I did want to thank all those that have been jumping in from around the Virtualization community and the world.  It has made me feel like I am making a difference in people’s live’s above and beyond the technical expertise I bring to the community.  It is really cool to be able to take my own hobby and passion and bring it to others.  I can also say these folks have kept me motivated to do my own workouts!

If there is anyone else out there that is interested in joining any of our challenge groups, please ping me on twitter or read more about the rules of a challenge here.  Most of the posts in our little FaceBook group have been really funny as well.  Guys seem to have a rough day of coding or dealing with work and have just been taking it out on some workouts.  I think many would be surprised to see the current folks challenging themselves from the community.  Some are pretty well-known in the blogosphere and Twitter space.  All I know is it’s pretty cool to hear these guys talk about how they are continuing to get better each week.

I encourage those of you interested in Disaster Recovery with vCloud Director to download and read the case study

Source - VMware vCloud Director Infrastructure Resiliency Case Study

Description - vCloud Director disaster recovery can be achieved through various scenarios and configurations. This case study focuses on a single scenario as a simple explanation of the concept, which can then easily be adapted and applied to other scenarios. In this case study it is shown how vSphere 5.0, vCloud Director 1.5 and Site Recovery Manager 5.0 can be implemented to enable recoverability after a disaster.

Downloads:
http://www.vmware.com/files/pdf/techpaper/vcloud-director-infrastructure-resiliency.pdf
http://www.vmware.com/files/pdf/techpaper/vcloud-director-infrastructure-resiliency.epub
http://www.vmware.com/files/pdf/techpaper/vcloud-director-infrastructure-resiliency.mobi

The Setup

Static IP Pools are just that.  A pool of IP addresses that vCloud Director assigns as a Static IP to the Operating system.  However, these are not something you can designate to any particualr VM.  It’s like a pool of phone numbers for that matter.  Have you ever gotten a new phone number and had calls from people looking for the previous number owner?  That’s because the number you got went BACK into the pool when the original owner gave it up.  Static IP pools work the same way.  The key is this can also affect how you create or handle operational aspects of the vShield Edge firewall rules.  Here is a simple Scenario:

Static IP Pool Range = 192.168.0.10 – 192.168.0.20

Virtual Machines are deployed that get assigned static IP’s from this pool in the following order:

VM1 = 192.168.0.10
VM2 = 192.168.0.11
VM3 = 192.168.0.12
VM4 = 192.168.0.13
VM5 = 192.168.0.14
VM6 = 192.168.0.15

Now, let’s say that VM3 has is a web server and has firewall rules for the inbound ports of 80,443,and 22 but the others in the list have no incoming rules.  We can also assume that VM3 has a NAT for an external address through vShield Edge.  Now, let’s pretend that VM3 is determined to need a rebuild and is removed from vCloud Director.

What do you think happens to its IP address?

If you answered “It goes back into the pool” you are correct!  Also, we must assume there is more than one user in this organization creating Virtual Machines.  Do you think before you can re-create your VM3 that the .12 address could be re-issued to a new VM and your re-deployment can get a new address of 192.168.0.16?  The answer is absolutely!  That also means that your original firewall rules are now open to the virtual machine that some other user created which they may not want or even know about.  You can see how this would also translate to a Disaster Recovery Scenario where IP addresses could change based on the recovery site’s different Static IP Pool configuration and why it came up in those discussions.

Possible Work Around

Everyone Out of the Pool…..

So how can we work around this?  Really the anser is to select “Static – Manual” for the IP addresses if your cloud Virtual Machines are going to be around for some time.  Now in a Public cloud you may not have this option especially on external IP’s you do not control directly.  I have seen this situation happen to me real-time in my public cloud as I was building some vApps.  I grabbed an external IP, but had to rebuild and when I did I got a new external IP.  I had to update my DNS and firewall rules accordingly.  Now I will never delete the vApp so I can keep the IP address it has since the provider does everything with pools.

Another possible way to deal with this is through orchestration.  I could see ways to leverage a custom portal and orchestration to assign the IP addresses based on the usage of the VM.  For example, pools are great for test/dev that is not going to need firewall rules, but having that question as part of the workflow might be useful before the VM is created.  Many providers have put the firewall rule capability into their custom portals today.

Summary

The point here is simple.  You need to know that anything pool based, like IP addresses could be re-assigned over time as virtual machines are created and destroyed.  In a private cloud however, you have the option of controlling this through the use of manually assigning an IP Address in vCloud Director.  Don’t get frustrated if you see your IP change when you delete and re-create a vApp.  It can and will happen, you just need to be aware of it and understand you need to deal with it operationally.  Play with this yourself and you will see IP Addresses get used and re-inserted back into the pool.  The best I can tell in my testing is that vCloud Director will always use the lowest IP available in the pool when it assigns one through a “Static – IP Pool”.

1. vCloud Director invokes an OVF export in vCenter Server
2. vCenter Server returns a lease which contains a bunch of https urls (basically disks, ovf descriptor, etc..) + an authorization token. The URLs point to the ESX host (it’s really opaque to vCloud Director). There could be multiple ESX hosts that are available (based on connection to the datastore). It is unknown to me what strategy is used to choose one of these ESX urls.  My guess is it will be the one that has the virtual machine registered to it.
3. vCloud Director downloads these files to its transfer directory, and a lease is associated with these files. The download happens through https as a transport protocol, and session management is done through NFC, VMware’s network file copy protocol).
4. vCloud Director exposes a https download link that the user can use to download the files, this is https served by Jetty web endpoint.
5. When the transfer lease expires, vCloud Director deletes the files from the transfer directory.

The key to this, and what I thought was true, is that vCloud Director export is always a two-step process, from ESXi host directly to the cell and then from the cell to the end user.  This is the case regardless if the export is invoked through the UI, API, PowerCLI, or vCenter Orchestrator.  The reason this came up is that the customer noticed that an export in vCenter was happening almost twice as fast, and that is in fact correct!  That process does not move the data to any “Transfer” space it is moved directly from Host to the end-user through the vCenter Client.  Therefore, vCloud Director exports will take longer, but there are places you can check to see if there is a way to increase the copy.

My original design considerations around interfaces and traffic flows should absolutely be considered since you are moving data from a host, to a shared folder, then out to a user or to another vCenter in the case of a copy between vDCs.  Understanding the underlying nuts and bolts about how operations are performed by both vCloud Director and vSphere is extremely important to not only design but troubleshooting possible bottlenecks.

It is worth noting that this is the process used by vCloud Connector on the initial Export as well only instead of the CLIENT as the endpoint the vCloud Connector Node becomes the end point for the second step.  Then this process is also repeated for an import back into another vCloud Director location.  The files are moved back into that cloud’s transfer directory, then into the catalog location, and finally into the cloud itself.

I wanted to take a moment to thank all the voters this year that took the time to vote on the top 25 vBlogs.  As always my good friends Duncan Epping, Frank Denneman, and Alan Renouf were tops on the list.  When I found out I was nominated I was humbled, knowing I made it into the top 25 was even more humbling.  To be in the midst of such talent is just pretty cool.  Most people did not know who I was before the end of 2010 when I started blogging.  I want to make a special thanks to Duncan and Frank.  Both those guys and I have become such good friends over 2011.  It was actually Duncan that convinced me to start blogging and getting more into the community.

I have said it before and I will say it agin, I am not the smartest guy….I went to public school.  However, I do believe that I have a knack for delivering some useful content both online, but really also in person.  Those who have seen our presentations know I just love presenting.  I know look forward to doing sessions at VMworld with Duncan, Frank, and David Hill.  Now that the results are out, it will be good to see all of you hopefully.

As Duncan stated in his thank you article, we are all approachable.  Sometimes we may not look like it, but we are.  Even being an honorary member of the Dutch vMaffia sometimes we may beat down the Monster VM, but we are still guys you can talk to.  I know I will probably never be number one, and maybe not even in the top 5, for me just making the list my first year out is a sweet feeling having been one of the few that

If you have not seen the video you can view it at VMware Videos dot com and see all the results on vSphereLand.com.  I want to also give a special note to John Troyer for his kind words about me, and David Davis for the special mention of my work trying to help some of our own community members get fit.  That’s not a one shot deal, I really want to keep that up since I do like to have more about me than just Virtualization and work.  Below are the stats for the voting for me that I thought we pretty cool.  I hope I can keep up all the voter’s expectations!

Total Votes = 119
Total #1 Votes = 28

Database Server

Microsoft SQL seemed to have no issues when changing the IP on the guest OS.  The services appear to bind to whatever IP is on the guest at the time the services start.  We did not see any issues here, but we also did not test using Oracle.  I would suggest checking with your DBA’s to ensure there are no issues, but I would suspect on Oracle you may need to make additional changes.

VMware vCenter Server

Changing the IP on this service is not specifically required when using the full install.  Simply updating the IP on the guest OS is enough since vCenter Server has no direct tie to the Guest IP.  This may be different with the vCenter Virtual Appliance, but we did not use that in our testing.

VMware vShield Manager

This is easy enough to change its IP on the appliance by re-running the setup command

1. Log into the appliance with admin/default
2. Enabled Admin mode with the command “en” and the same credentials
3. re-run “setup” script to change the IP address

VMware vCloud Director Cells

There is actually a KB article for this that explains it really well.  It is written for Oracle, however the steps are the same for MSSQL.  The bottom line is you need to do a couple of things:

1. Change the Guest OS IP and routing information
2. Stop the vCloud Director Cells
3. Update the “cells” table with the new IP addresses of the cells per the KB article
4. Edit the /opt/vmware/vcloud-director/global.properties entries
1. vcloud.cell.ip.primary – Change it to the new primary IP address
2. consoleproxy.host.https – Change it to the new console proxy IP address
3. vcloud.cell.ips – Change both the IPs in this field appropriately
5. Restart the vCloud Director Cell service

Once  you make these changes the cell should start and you will see it bind to the new IP addresses when you examine the cell.log file.

VMware vCenter ChargeBack

There is a little known script located on the chargeback server that is needed to change the IP address.  Essentially you need to re-run the configuration of the server without re-installing.

1. Start Menu
2. VMware
3. Vmware vCenter Chargeback
4. vCenter Chargeback Manager Tools
5. Reset IP address of the vCenter Chargeback Manager Server

VMware vCenter Orchestrator

This can be done in one of two places, but I recommend checking them both once you make the change to ensure it is set up correctly.  You can change the IP through the “Configure Network” option in the console or through the “Appliance Configuration” option, however you still need to make the change elsewhere in the “Orchestrator Configuration”

1. Edit the vCO network tab in side the configuration screens for new IP

• Check Installation Folder>\app-server\bin\boot.properties

1. Reboot is required for change to take effect

Summary

This covers the “Core” components of the vCloud management stack and changing the IP addresses on them through various means.  The key here is to remember that just by updating the networking in the Guest OS does not mean the various applications are properly updated.  You may need to check and double-check once the IP addresses are changed to ensure they are working before you continue on with your updates.