3. How to connect our XenServer to iSCSI storage

Given we’d purchased a NAS device that provides iSCSI connectivity we have three  possibilities of connecting to it.

1. Software iSCSI initiator

XenServer uses the built-in Linux Kernel software iSCSI initiator to allow easy connection to iSCSI targets.

Advantages

• It’s free, and built into the XenServer OS.
• No additional hardware required

Disadvantages

• Uses host CPU resources for SCSI disc traffic.
• IOPS limited by cpu resources available
• Sends disc traffic over the main production NICs which is not best practice.
• You can’t boot from an iSCSI disc using the software initiator

2. Software iSCSI using TCP offload NIC

This method uses the Software iSCSI adaptor built into XenServer, but utilises the TCP offload capability of certain NICs. The effect is that TCP traffic takes up less CPU resources as it is handled by the processor on the NIC.

Advantages

• Reduces the increase in CPU load cause by software iSCSI disc traffic by offloading processing to NIC
• Requires no additional HBA hardware – uses existing NICs

Disadvantages

• Requires both NIC chipset and the XenServer/Linux NIC drivers to support TCP offload capability
• TCP offload is not compatible with many kinds of application (e.g. provisioning services)
• Does not separate traffic unless additional physical NICs with offload capability are added

3. Hardware Host Bus Adaptor

A dedicated iSCSI host bust adaptor can be used to offload SCSI disc traffic processing away from the main CPU and NICs.

Advantages

• Offloads SCSI command processing to the adaptor processor freeing up CPU resources for the virtualised guests
• Separates disc traffic from other networking and management traffic
• Allows you to boot the server from an iSCSI disc if required

Disadvantages

• HBA adaptors are expensive
• Takes up an expansion slot on the server
• Range of HBA’s supported on XenServer (i.e. on the HCL) is very limited

The diagram below shows the three methods and how they handle the disc traffic differently.

And the winner is…

One of the objectives of the lab was to be as “real world” as possible. In the real word, you would not combine management, VM and storage traffic on the same bonded NIC pair, and as our host server only has two on-board NICs available, we would need to add an additional NIC adaptor to separate out our traffic types.

The onboard Broadcom NIC chipset does support TCP offload (which can be used to reduce CPU overhead on all network traffic) however this is not supported by XenServer. It may be possible to compile the latest Broadcom drivers using the XenServer SDK and add offload capability but that’s a task for another day and blog post.

So, do we go for a standard Gigabit Ethernet card to provide additional physical interfaces, or a dedicated storage HBA to connect to our iSCSI back end?

I quite liked the appeal of having dedicated hardware performing the communications to the NAS server and this represented as close to “real world” as possible without going down the fibre channel route. Examining the XenServer HCL, the only iSCSI HBA adaptors that are listed are QLogic. I wanted to be able to “simulate” multi-pathing (even though we only have a single switch), so needed a card with multiple ports. Given the server had two free PCI Express slots, this narrow the selection down to one – QLogic QLE4062C dual-port Gigabit iSCSI HBA. Added to the shopping list.

We plan to do some performance comparisons between the above three solutions once the lab is up and running.

Resources

Citrix KB – iSCSI start-up best practices

Citrix KB – QLogic iSCSI configuration

Citrix KB – Storage Multipath support on XenServer

Citrix KB – XenServer storage overview

Citrix KB – Storage Deep Dive

Open-iSCSI documentation