Interesting VMware vSwitch Advanced Setting

In recent weeks I have been working on configuring a workload as a virtual machine and I ran into a scenario where it required a specific vSwitch advanced setting to be used.  I will describe the basic setup then the resolution to the issue.  Below is a basic diagram of the setup for any vSphere virtual switch we have all seen before along with a simple workload in a port group.

VMware vSwitch

The Required vSwitch or DVS Port Configuration

Something to point out is when running this particular workload in a virtual machine the standard vSwitch Port Group or DVS Port Group needs to be place in promiscuous mode.  This is because when the virtual machine talks to the physical server, the traffic that goes out the virtual machine to the physical server is from the MAC address something else not the MAC address of the virtual machine given by vSphere.  You will notice we also have the vSwitch connected with redundant uplinks in a standard Active/Active setup as most implementations are done.  Because of this particular workload we see a condition similar to this:

  1. The virtual machine sends packets (multicast)
  2. The vSwitch sends it to ESXi-NIC1 to Switch-Left
  3. Switch-Left sends it to Switch Right
  4. Switch-Right sends it back to ESXi-NIC2
  5. ESXi-NIC2 sends it to virtual machine (since the port group is configured in promiscuous mode)

Ultimately the workload inside the virtual machine cannot resolve the traffic going out and coming back into itself.

How To Resolve The Conflict

Since we know that we must have Promiscuous Mode enabled for this workload to perform its function as a virtual machine, and we assume that people will use a separate port group on a Distributed vSwtich or standard vSwitch to maintain the security there is a way to fix this.  You actually have three options to resolve the conflict above:

  1. Use a Separate vSwitch with only ONE pNIC (Not ideal)
  2. Use a separate vSwtich and Enable LACP on the two pNIC’s
  3. Use a separate vSwtich and set the following advanced setting on each host:
    Net.ReversePathFwdCheckPromisc=1

In my testing I have found the third option to be the easiest and it was not a setting I have used before.  Essentially this prevents the vSwitch from checking the reverse path when in Promiscuous Mode, and works perfectly.  Once it is set you will see the workload functions normally inside the virtual machine.  The setting was actually provided by some folks on the VMware Engineering side as something to try to it worked.

I actually had a lab setup recently where the first option was easier to use for that setup, so any of the three are viable solutions.  You cannot just set one pNIC to “inactive” or standby either, that will not work and that was the first thing I tried.  I am actually in the process of doing some messing around with this workload with Mark Achtemichuk as well.

I just thought this was interesting and not something you would run into every day on a most virtual machine workloads.  I had never found the need for this setting before, but I guess you run into new things every day.

6 comments

  1. Rachit Srivastava

    Hi Chris,

    Excellent post about something I might never not run into.

    I am not a network guy so I am a bit puzzled about the setup and would need to help.

    Why are your vSwitch connected to each other ? Is is even possible ?

    I fail to understand is even if both pNIC are in active for a port group, virtual machine traffic should ideally flow through any one pNIC for a specific flow. ( Now I am not sure what load balancing policy is used)

    Secondly why cannot just setting one pNIC to “inactive” or standby either, will not work ?

    Thanks in anticipation.

    • Honestly this was so long ago I cannot remember all the details. I don’t actually know why passive or standby would not work I just recall trying it and it still persisted

      • Rachit Srivastava

        Thanks Chris ! I can understand missing a few pieces amid a lot of great technical you folks do 🙂

        I will try to replicate the setup and see if it works for me.

        However, using the advanced setting seems more inline with the best practice to me.

        cheers,
        Rachit.

        • Just to add (a bit late) to the discussion – the issue “does” persist regardless of whether additional pNICs are in standby or not – so VMs essentially see back-scatter even though a single pNIC is active. Obviously something related to the way the vSwitch works internally that is completely backwards from how we initially believe it should work.

          Also, a quick tidbit that if you change this advanced setting, existing port groups already set to allow promiscuous mode will not be changed “live”. You need to change the port group’s promiscuous mode setting back to Reject, then again back to Accept (essentially causing a refresh of the setting at the port group level) before it becomes active. This will prevent you from having to reboot the host.

  2. Rachit Srivastava

    Hi Chris,

    Excellent post about something I might never not run into.

    I am not a network guy so I am a bit puzzled about the setup and would need your help here.

    Why are your vSwitch connected to each other ? Is is even possible ?

    I fail to understand is even if both pNIC are in active – active for a port group, virtual machine traffic should ideally flow through any one pNIC for a specific flow. ( Now I am not sure what load balancing policy is used)

    Secondly why cannot just setting one pNIC to “inactive” or standby either, will not work ?

    Thanks in anticipation.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Scroll To Top
%d bloggers like this: