About a month ago I had a friend contact me asking about some issues he had while buidling a low cost ESX server for his home lab. He was having difficulty with regards to one small piece of hardware, and he didn’t know about the hardware compatibility list for ESX. While it’s a bit late for my friend Dave, here is some great instructions on how to build a $500 ESX Server. Also check out the VM Help site for an unofficial whitebox HCL list; this is not hardware offially supported by VMware but apparently tested by the internet VMware community…your milage may vary.
Last week, before the end of VMworld, we had a session with one of my customer’s to discuss ESX performance. This discussion was lead by one of VMware’s performance gurus Scott Drummonds. Scott works as the Manager of VMware’s Performance Marketing team, working with the VMware field teams and customers to provide and advise on product performance issues.
During this conversation, I wrote down some key quotes from Scott that I wanted to share. For those that have been around VMTN for a while, you might already know some or all of these performance related suggestions. However, they are key enough that I wanted to draw attention to them.
- “Administration work on the LUN has an impact on performance more than the number of hosts.”
When admin work is done on the LUN is the only time that SCSI reservation locking is used. So placing more VMDKs on the same LUN doesn’t directly impact the performance. See Scalable Storage Performance with VMware ESX Server 3.5 Vroom! blog post for additional details.
- “Put Windows VMs and Linux VMs in sperate clusters because they can share memory more efficiently…”
Part of the memory optimization doen within the ESX hyperadvisor is to share common memory used by multiple VMs. So if you are running 10 Windows VM, the memory used to store the majority of the Windows OS is shared amonst all 10 VMs since the memory contents are the same and do not change. This is what enables memory overcommitting. If you start to mix and match different OS VMs on the same host, this advantage can be minimized.
- “High storage latencies is the largest source of performance problems that I see…”
You can monitor the storage latencies within VC by changing the stats level. See Scott’s Understanding VirtualCenter Performance Statistics performance community doc for more details.
- “Once your swapping, you’re in trouble…”
Seeing swapping within your VM means that you are not allocating enough memory to the VM for the applications that are running. Watch this closely and add memory or migrate VMs when and where needed.
- “RHEL5 does 1000 interrupts per second for “greater precision” (versus most OS’ which only do 100) which can add up to un-wanted overhead in the OS…”
This is an issue with the Linux Timer Rates that Scott talks about. There is a small configuration change that you can make in RHEL5 SMP that will provide an across the board increase in system performance.
Check out all of Scott’s VMTN articles, there is a wealth of information in them.
One item that I have seen come up again and again recently with my customers is a mis-understanding of ESX and VM Server. Particularly the performance aspects of both. These are two entirely different products with entirely different goals that need to be kept in mind when comparing the two.
VM Server is the free server virtualization product from VMware. It’s purpose was to help seed the market and get people more confortable and familiar with virtualization concepts on the server. What better way to do that than give something away for free for people to play with! VM Server is similar to VM Workstation/Fusion in that it runs on top of an OS; so from the ground up you have:
server hardware > operating system > VM Server > virtual machine.
This means VM Server inherets the same performance issues that the OS has. It also means that it has to rely on the OS for a lot for the management of resources. Net result: the performance of VM Server will be impacted by the OS you are running it on top of.
ESX is the data center grade hyperadvisor from VMware. It’s purpose is to provide virtualization for business critical data centers. ESX essentially doesn’t run on top of an OS. You can think of it as a data center OS as it is the base layer for the Virtual Infrastructure set of products. Similarily, from the ground up you have:
server hardware > ESX > virtual machines
You can see immediately that ESX has eliminated the operating system level, this is the source of the performance gain that ESX provides. It is running on top of the raw hardware. It is truely a specialized OS with the sole pupose of virtualization.
One thing that VM Server and ESX share, is the engine to run a virtual machine. This engine has lots of tricks built into it to manage the physical resources of the box and to distribute them efficiently to the virutal machines that need them. Of couse, you still suffer from that extra piece of the base operating system under VM Server. This is why you can have VM Server running with no VMs and still see it utilizing resources. It’s doing the work of interfacing with the base OS for resource management even when no one is using the resources.
So, if you hear somone complain about their VMs are not performing very well ask them which product they are using. If they are using VM Server, point them here. If they are running ESX, then it’s time to dive into the typical operating system and application stack tuning/debugging routine.
(Disclosure: I currently work at VMware as a Solutions Consultant)