VMware vSphere — Why checking NUMA Configuration is so important!

What is NUMA

NUMA KPIs — CNIL Metrics and Logs
Source: http://frankdenneman.nl/2016/07/07/numa-deep-dive-part-1-uma-numa/

How does the VMware ESXi host use NUMA?

How to detect NUMA performance issues

esxtop NUMA view
  • esxtop
  • m (the letter)
  • f (to enable fields)
  • G (to enable NUMA information)
VMware ESXi NUMA Home Node View

The tricky configuration: Hot-Add CPU for large VMs

Key takeaways: VMware configuration for NUMA optimization

  • Don´t enable Node interleaving in your server BIOS!
  • Order or configure the physical server hardware to have equal memory amounts for each NUMA node.
  • Assign less or equal amount of vCPU´s to VMs than the total number of physical cores of a single CPU Socket (stay within 1 NUMA Node). Don´t count Hyperthreading!
  • Check your virtual infrastructure, in general, to work optimized with the physical NUMA node limits of your servers. Watch for Monster-VMs!
  • Avoid single VM consuming more vCPUs than the single NUMA node or it might be scheduled across multiple NUMA nodes causing memory access degradation.
  • Avoid single or multiple VMs consuming more RAM than a single NUMA node, because it will cause the VMkernel to span a percentage of the memory content in the remote NUMA node resulting in reduced performance.
  • vNUMA (virtual NUMA) is enabled by default for VMs with 8 or more vCPUs. Caution! When enabling “hot add CPU/memory” or configuring CPU affinity, it automatically deactivates vNUMA.
  • VMkernel NUMA re-balancing takes place every 2 seconds.
  • vNUMA is not re-balancing! The vNUMA architecture is set, when the VM is powered on for the first time. Be aware of vMotion (DRS), when the VM is migrated to ESXi hosts with different NUMA configurations.

What can potentially cause NUMA misplacement

  • Configuring VM memory limits lower than the configured memory
  • Configuring more vCPUs than physical cores per CPU socket
  • Configuring more memory than the NUMA node has available

Fullstack Monitoring, Analytics, Security: VMware, Docker, Kubernetes, Applications

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to install Python on Windows 10, and print “Hello world” on the Python shell.

How to measure your ERP project ROI

How To Get All Stock Symbols

What exactly is Team Velocity and Team Capacity

How to manage your cloud space efficiently

C# Delegates and how to use them

Return values in Ruby

Get Your Cloud Migration Strategy Right the First Time Around

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Dennis Zimmer

Dennis Zimmer

Fullstack Monitoring, Analytics, Security: VMware, Docker, Kubernetes, Applications

More from Medium

Software Version

Time to rethink your digital architecture design

Steps to improve AEM site performance

Installing many JDK on macOS using Homebrew ad openjdk