How To Troubleshoot vCloud VMRC Connections

I have seen a couple of times now posts in various places about connection issues with the vCloud Director remote console.  I wanted to write up a few short things that seem to be the most common things that folks have run into when configuring the remote console section of vCloud Director.  The first thing you should do is check out the diagrams by Hany Michael as they will also help you see the ports required.  No need in re-creating the wheel since those are out there.

Understanding the “Proxy” Connection

The vCloud Director Cell is in fact proxying the connection between the client and ESXi.  It is sitting in the middle as any proxy does.  This is to allow the client to open the console without directly connecting to an ESXi host in the provider.  For this to work, the Cell needs to have not only proper communication to and from the guest on the Console Proxy port, but also to the ESXi hosts.  It also needs proper DNS setup to find the ESXi hosts by their name listed in vCenter.  If that fails you will be chasing your tail.  Below is just a simple graphic showing that the cell is in fact in the middle of the process.

First and Foremost – Routing

You are building a Linux operating system with multiple interfaces.  What does this mean to you?  Well if you are savvy with networking you know that the fundamental requirements of TCP/IP will always remain true.  Most important that systems will have a single Default Gateway for routing.  Where we see the most trouble is that people use different networks for their various ports.  This is okay, it is not a bad thing…..as long as you configure all your routing properly.  For example, below are four common IP configurations for a vCloud Director Cell:

  • Linux Management – 192.168.1.10
  • vCloud HTTP – 192.168.2.10
  • vCloud  Console Proxy- 192.168.3.10
  • vCloud NAS Connection – 192.168.4.10
Question:  Which network has the Default Gateway?
Answer: That depends!  Which one did you configure as the Default Gateway

Any one of the networks above could use the default gateway, but which ones even make the most sense?  Which interface is going to be used the MOST often and need the easiest path back to clients?  In my opinion, that is usually the vCloud HTTP port.  Although some may argue the console proxy, or management port, I say the purpose of the cell’s existence is to provide portal or API access.  That all happens through the HTTP port.

Now then, what happens when you need to reach the other ports?  What do you need on top of the default gateway?  Anyone?  STATIC ROUTES!!  The other networks may OR may not need them.  I will say the ones that would require them for sure is the vCloud Console Proxy.  The diagram below illustrates a basic layout with the above network examples (although not as pretty as Hany’s).  Once you decide which interfaces will be routed and which ones will not be, you will need to set up the correct static routes.

The NAS connection for the transfer space MIGHT need it, but of the NAS share is on the same network you can leave that as layer 2 connections only.  Linux OS management and Console Proxy will always need to find the way out of their network.  

I hate to say it bluntly, but this has nothing to do with vCloud Director, this is system admin 101 stuff for having multiple interfaces and networks on a single system and understanding who needs to talk to what.  Let’s not forget the basics here.

Firewalls

I am not going to go over all the ports you need, you can get that from the documentation and Hany’s diagrams.  However if you are using a complex setup like above, there could very well be firewalls involved as well.  EIther hardware ones or software ones preventing the connection.  If you have validated all the above, then start looking into firewall settings.  Is IPTABLES or SELinux enabled on the Cell?  If I am not mistaken, one or both should remain off for the cell to function.

Load Balancer and External Address

I have covered this pretty clearly in past articles on configured a load balancer for vCloud Director, so if you have not checked that out there is an important thing to take away.  You simply cannot do SSL offload on the VMRC connection.  Read more about the setup here, but if you are trying to offload SSL for the console connection, it simply will not work.

About Chris Colotti

Chris is active on the VMUG and event speaking circuit and is available for many events if you want to reach out and ask. Previously to this he spent close to a decade working for VMware as a Principal Architect. Previous to his nine plus years at VMware, Chris was a System Administrator that evolved his career into a data center architect. Chris spends a lot of time mentoring co-workers and friends on the benefits of personal growth and professional development. Chris is also amongst the first VMware Certified Design Experts (VCDX#37), and author of multiple white papers. In his spare time he helps his wife Julie run her promotional products as the accountant, book keeper, and IT Support. Chris also believes in both a healthy body and healthy mind, and has become heavily involved with fitness as a Diamond Team Beachbody Coach using P90X and other Beachbody Programs. Although Technology is his day job, Chris is passionate about fitness after losing 60 pounds himself in the last few years.

5 comments

  1. I’m not sure how good static routes are for something like the console proxy port, if you have users on multiple networks, they may come into the console proxy but you’ll likely end up with asymmetric routing for return packets if they’re on other end user vlan’s etc.. and building/maintaining a large static routing table is usually not tops on anyone’s list of fun things to do.

    an alternative I used for our vCD cell was using ip2 routing to guarantee traffic coming in over a specific interface went back out over the same interface.. some might argue that it’s not all that important but for anyone who is concerned about such a problem then see this for example:

    the following script runs at network startup:
    #!/bin/bash

    # set up subnet 192.168.100 routing table for eth0 traffic – http
    /sbin/ip route flush table 100
    /sbin/ip route add table 100 to 192.168.100.0/24 dev eth0
    /sbin/ip route add table 100 to default via 192.168.100.1 dev eth0

    # set up subnet 192.168.200 routing table for eth1 traffic – console proxy.
    /sbin/ip route flush table 200
    /sbin/ip route add table 200 to 192.168.200.0/24 dev eth1
    /sbin/ip route add table 200 to default via 192.168.200.1 dev eth1

    # create rules to choose what table to use, choose based on source IP
    /sbin/ip rule add from 192.168.100.0/24 table 100 priority 100 
    /sbin/ip rule add from 192.168.200.0/24 table 200 priority 200

    and so on if you have seperate NAS etc networks..

    • Hey Jeff!  Thanks for adding this.  I am not a fan of static routes either, but in some cases it is the fastest way to determine what you’re connectivity issue might be.  The main theme I have seen is that people seem to simply forget that a system with multiple IP Addresses will not always have a path back to the user on all of them.  Specifically the Console port.  

      Nobody ever seems to have an issue with the HTTP port since 9 times out of ten that is the first one setup.  Also folks that have the HTTP and Console Proxy on the same subnet do not see the issue.  Personally I think the HTTP and Console proxy could be on the same subnet seeing as those two are always access by the same user.  You cannot open a console until you log into the portal, for instance.  Beyond that the other two are optional.  I did some testing on traffic in the clone wars series about how that can end up using the HTTP port for virtual machine data transfer if you do not segment it.  

      The goal here is always to simply think about the connectivity of the cell for sure.

  2.  Users should also make sure that the DNS names in vCD are set correctly for both portal and the proxy and that those names are resolving to the correct addresses.  I found this to cause issues with both the proxy and vCloud Connector portions

Leave a Reply

Your email address will not be published. Required fields are marked *