Before going any further, if you don’t understand how memory management is done by the OS, I strongly advise you to have a read at the Resource Management – Fundamentals blog.

Within vSphere environment, an extra layer is added so that, the Guest OS memory management is now done within the actual Virtual Machine. Though as far as the ESXi server is concerned, the VM behaves just like any other application running inside a physical server – they hypervisor:

memory-mgmt-esxi

  1. There are two Virtual Machines running within the ESXi Hypervisor OS – Virtual Machine Yellow & Virtual Machine Orange
  2. Host Physical Memory is the total amount of memory available to the ESXi/OS and Virtual Machines running within; white squares represent free memory
  3. Guest Application Memory represents a *contiguous* memory space given to applications, within the Virtual Machine. Here the ESXi will intercept all memory allocation requests and back it with actual physical memory within the host. It does this by using three data structures:
      1. Guest OS Page-Tables – maps guest virtual memory to guest physical memory
      2. PMAP – maps guest physical to host physical memory; one such structure exists for each VM
      3. Shadow Page-Tables – using the two structures above is able to map guest virtual to host physical memory
  4. Guest Physical Memory – this is the memory presented to the guest OS
  5. Swapping device is represented by the orange Hard-Disk icon – as you should know, once specific memory utilisation thresholds are reached, the OS will start copying idle memory content to disk in order to free up physical RAM to other, more active, applications – we call this OS Level Swapping
  6. Host Level Swapping – host swapping is one of the memory reclamation techniques ESXi uses – more details later
  7. Notice how, in order to define the different memory allocation scopes I have simply added the word Guest (i.e. within the VM’s guest OS)
  8. I think we can agree that it’s ultimately the Shadow Page-Tables that actually matter – they link the memory request at the guest OS level to the actual memory allocation within the ESXi host. In order to speed-up access to the shadow page-tables, a dedicated cache is used for this purpose – this is the Translation Lookaside Buffer
  9. Once Host Physical Memory is allocated, the host has no visibility of which application within the guest OS is using which blocks of memory. As far as ESXi host is concerned, those blocks will be allocated to the VM and not to a specific application running inside the VM (hence the color code – orange and yellow only!)

THE PROBLEM

When memory is allocated inside a VM, the ESXi will immediately try to back that request with physical memory. The guest OS will try to allocate a previously used memory block which has been freed-up in the meantime; otherwise, a new memory block will be used. 

Here is an illustration:

memory-alloc

Using the diagram above for reference, if at some stage Application Blue frees up MemBlock-2, the guest OS will free up the respective memory block from the Guest Physical Memory – Memblock-13. In the host physical memory, the respective memory blockMemBlock-8 – remains allocated. 

This is because the ESXi host has no visibility of the memory free-list within the Guest OS – see Memory Allocation section

Upon new memory allocation requests, the guest OS would either …

    1. Assign a previously used Guest Physical memory block – in which case, no work is done by the ESXi host since the mapping is still there
    2. Alternatively, a new memory block is allocated on the Guest Physical Memory – in which case the new allocation must be backed by a new allocation on the Host Physical memory

The idea is that between the Guest Physical Memory and the Host Physical Memory, memory blocks will be re-used whenever possible. However, due to new memory allocations the host will eventually run out of physical memory

VMware uses four techniques in order to delay the host running out of memory, as much as possible. I will be presenting these in a another blog – Resource Management – Memory – Part II.

Go back to Index


Thank you,
Signature
View Rafael A Couto Cabral's profile on LinkedIn



Comments are closed.