We’ve now got a XenApp 6.5 server available in the community lab.

Please contact us if you would like access to take a look, please comment on this article providing a few details on what you’re interested in trying.

Update : 28/06/11

We’ve received quite alot of interest in the XenApp 6.5 lab and there’s quite a queue of requests for access. I’d normally process these the same day, however this past week the lab has been utilised by participants (myself included) of the Item Development Workshop at Citrix who are writing the next XenApp 6.5 CCA, CCAA and CCEE exams. As such, I wanted to restrict changes to the environment and those who can access it.

The exam development workshops finish in two weeks, so after that I’ll process all the outstanding requests for access.

In the meantime, please see my latest blog post for an overview of some of the major, and lesser publicised changes.

Thanks for your patience.

We’ve got our lab Netscaler VPX appliance running (on top of the lab XenServer of course) and it’s now sitting in front of this site, and also the UK Citrix User group website.

This will reduce the load on the Linux VPS hosting the site, compress HTML data to speed up page rendering, and provide protection from SQL injection and similar attacks via the Application Firewall.

How did we do it? Read on…

Netscaler configuration

Add Load Balancing Server

Navigate to the Load Balancing section in the Netscaler Admin console and select “Servers”

Menu item for adding a server

Add the IP address of the Linux VM that is hosting the website.

Adding a server resource to the Netscaler

We added the IP address of the Linux Virtual server as a server. This defines it as a resource that can added to other objects in the Netscaler such as services and virtual servers.

Create the service

We now need to define the kind of service that this server is providing. This can take many forms, but in our case it’s a website, so the service is HTTP.

Navigate to the services branch.

Menu item for adding services

Choose the server added above from the drop-down list, then choose HTTP as the service.

Adding the HTTP service

We added a http monitor the check that our service was up.

Create the LB Virtual Server

From the Virtual Servers tree:

Menu for creating Load Balancing virtual servers

We choose Add new Virtual server, and selected the service we created above.

Creating the Virtual Server

We also untick “Directly addressable” so the VS will be internal to the Netscaler and not accessible externally. This then greys out the IP address field:

The Virtual Server will be internal to the Netscaler

Why internal?

We want this virtual server to front multiple websites being hosted on the Linux VM, and creating an external Virtual Server at this point would not allow this. We need to create the virtual server internal only, then use a Content Switching Virtual server to forward all incoming HTTP traffic to the LB Virtual Server.

Create the Content Switching VS

Content Switching menu

We select the “Virtual Servers” option under “Content Switching” and Add a new VS.

Configuring the CS Virtual Server

We allocated a unique public-facing IP address for the CS Virtual Server, and add a single (default) CSW policy that forwards all incoming traffic to the LB Virtual server we created earlier. We could at this point define more complex rules for forwarding traffic, based on HTTP headers, URLs or client details, but we just want a simple one-to-one rule, so our CS VS is complete. The Netscaler is now listening on the Internet using this IP address, and forwarding all traffic to our back-end Linux VM.

Viewing the complete configuration using the built-in Visualiser we see how they all link together:

Visualisation of the complete configuration

DNS changes

The original www.citrixcommunitylab.com and www.citrixug.org.uk DNS records were CNAMES pointing directly to the Linux Virtual server that hosts the web sites.

We created a new CNAME that pointed to the VIP of the Content Switching Virtual Server above and changed the www. CNAMEs of the two domains to point to it.

Once the DNS TTL cache expired on the clients (about 24 hours) then the content automatically switched to being delivered from the Netscaler.

The XenServer host server has been racked into it’s host data centre, connected to the iSCSI NAS storage and had XenServer 5.6 FP1 installed. We’ve got a range of IP addresses allocated for things like XenServer management, NAS management, server DRAC, Switch management etc. Most of the lab however will be internal-only, with access to the internet via a Netscaler VPX acting as a Layer3 router.

We’ve been busy testing everything and setting up the lab networking environment and domain controller, DHCP and DNS server over the past week, and are pleased to announce that this very website is now running on a Centos 5 VM on the lab hardware.

Next steps is to get Lab Manager installed to allow the simple creation of LabVMs for people to use.

There are two ways of configuring the HBA.

1. BIOS

2. Command line configuration utility

XenServer 5.5 and later ships with the QLogic configuration tools build in. The iSCSI version is:

/opt/QLogic_Corporation/SANsurferiCLI/iscli

We used the BIOS configuration utility as we found the iscli to crash frequently.

Configuring QLogic HBA via BIOS

The Fast!Util BIOS configuration is invoked by entering CTRL+Q during the POST.

The initial screen shows the adaptors present. Select the adaptor you wish to configure by highlighting it and pressing enter.

With the adaptor you wish to configure chosen, you now have the main configuration menu. We’ll choose the first option “Configuration settings” to configure the IP address.

The configuration settings menu. We’ll again choose the first item “Host Adapter Settings”

The screen shows the current host adaptor settings including the initiator (that’s us) iSCSI name (think of this like a FQDN or hostname of the adaptor). We want to configure the IP address, so choose “Initiator IP settings”

If there is already an IP address configured it will be displayed here. Otherwise, it will be 0.0.0.0. Highlight the IPv4 (or IPv6 if you are using v6) and hit enter to set the IP address. Do the same for the subnet mask, and Gateway address (if required). If your HBA is on the same layer 2 switch as your iSCSI SAN then no gateway is required.

Once you have set the IP address, escape back to the previous menu. We’ll now configure the iSCSI target that we want to connect to. Note this process is usually used when configuring the server to boot from the iSCSI device, but we won’t enable the boot device support, only configure the targets so that XenServer will detect our iSCSI targets and allow us to add the LUNs as storage repositories later.

Choose “iSCSI Boot Settings”

The QLogic BIOS allows you to setup a primary and secondary boot device (so you can boot your server in the event of a SAN failure). We only need to configure the primary, so choose “Primary Boot Device Settings”

The “Security Settings” option is used to configure a CHAP userid and password. In this example we’ll skip straight to configuring a target IP address.

Enter the IP address of your target iSCSI SAN. It’s usual to leave the port as default (unless you’ve change the iSCSI port on your SAN for some reason). You can choose to enter the iSCSI name manually here, but as they are usually quite long and complex, we will scan for available hosts instead.

Hit escape to return to the previous menu. You will be prompted to save the settings you have configured so far.

Before we scan for available targets, lets test networking connectivity by using the built in PING utility from the main menu.

Enter the IP address of your iSCSI target and choose “PING target”. If you have networking connectivity, you should see the below message:

Right, so know our HBA is configured and can talk to our iSCSI SAN. Lets scan for some targets. Choose “Scan iSCSI devices” and after a few minutes you should be presented with a list with the available targets listed.

As screen real estate is limited, and iSCSI names tend to be quite long, press F1 whilst highlighting one of the IDs and it will display the full iSCSI name

If a target hosts multiple LUNs, then you can choose of the target and get presented with a list as below:

We don’t need to go any further in the BIOS configuration unless we want to boot the XenServer itself from the iSCSI LUN, so exit from the BIOS config, saving any settings that you are prompted for.

Adding the new iSCSI Storage Repository

Using XenCentre, add a new Storage Repository to your server.

Choose “Hardware HBA” as the storage type

The detected LUNs will now be displayed below the type of HBA. The first HBA below is the local SAS storage (which is also presented as a HBA by the RAID controller, but is of course already attached). Our new iSCSI LUNs are shown under “QNAP”.

Type the description and choose the LUN you wish to use.

Accept the warning that adding the LUN will erase any existing data on it.

The Storage Repository will then be formatted and added to your XenServer. You may then start using it.

Please comment below, or share with your contact if you’ve found this post helpful.

1. Software iSCSI initiator

XenServer uses the built-in Linux Kernel software iSCSI initiator to allow easy connection to iSCSI targets.

Advantages

• It’s free, and built into the XenServer OS.
• No additional hardware required

Disadvantages

• Uses host CPU resources for SCSI disc traffic.
• IOPS limited by cpu resources available
• Sends disc traffic over the main production NICs which is not best practice.
• You can’t boot from an iSCSI disc using the software initiator

2. Software iSCSI using TCP offload NIC

This method uses the Software iSCSI adaptor built into XenServer, but utilises the TCP offload capability of certain NICs. The effect is that TCP traffic takes up less CPU resources as it is handled by the processor on the NIC.

Advantages

• Reduces the increase in CPU load cause by software iSCSI disc traffic by offloading processing to NIC
• Requires no additional HBA hardware – uses existing NICs

Disadvantages

• Requires both NIC chipset and the XenServer/Linux NIC drivers to support TCP offload capability
• TCP offload is not compatible with many kinds of application (e.g. provisioning services)
• Does not separate traffic unless additional physical NICs with offload capability are added

3. Hardware Host Bus Adaptor

A dedicated iSCSI host bust adaptor can be used to offload SCSI disc traffic processing away from the main CPU and NICs.

Advantages

• Offloads SCSI command processing to the adaptor processor freeing up CPU resources for the virtualised guests
• Separates disc traffic from other networking and management traffic
• Allows you to boot the server from an iSCSI disc if required

Disadvantages

• HBA adaptors are expensive
• Takes up an expansion slot on the server
• Range of HBA’s supported on XenServer (i.e. on the HCL) is very limited

The diagram below shows the three methods and how they handle the disc traffic differently.

And the winner is…

One of the objectives of the lab was to be as “real world” as possible. In the real word, you would not combine management, VM and storage traffic on the same bonded NIC pair, and as our host server only has two on-board NICs available, we would need to add an additional NIC adaptor to separate out our traffic types.

The onboard Broadcom NIC chipset does support TCP offload (which can be used to reduce CPU overhead on all network traffic) however this is not supported by XenServer. It may be possible to compile the latest Broadcom drivers using the XenServer SDK and add offload capability but that’s a task for another day and blog post.

So, do we go for a standard Gigabit Ethernet card to provide additional physical interfaces, or a dedicated storage HBA to connect to our iSCSI back end?

I quite liked the appeal of having dedicated hardware performing the communications to the NAS server and this represented as close to “real world” as possible without going down the fibre channel route. Examining the XenServer HCL, the only iSCSI HBA adaptors that are listed are QLogic. I wanted to be able to “simulate” multi-pathing (even though we only have a single switch), so needed a card with multiple ports. Given the server had two free PCI Express slots, this narrow the selection down to one – QLogic QLE4062C dual-port Gigabit iSCSI HBA. Added to the shopping list.

We plan to do some performance comparisons between the above three solutions once the lab is up and running.

Resources

Citrix KB – iSCSI start-up best practices

Citrix KB – QLogic iSCSI configuration

Citrix KB – Storage Multipath support on XenServer

Citrix KB – XenServer storage overview

Citrix KB – Storage Deep Dive

Open-iSCSI documentation

As we’d decided not to add any capacity to the local storage in our host server, we now needed to find some external storage that we could use to store the VM guests and ISO library on. Our criteria was:

Selection Criteria

Low cost

We don’t have budget to splash out on an expensive EMC/HP SAN or fibre switches and HBAs.

Small

Data centre space is charged per-U required, so we need something that will be 1 or 2U in height to keep colocation costs down.

Low power

Again, the other main factor in colocation charges is the amount of power consumed. Plus we want to be as green as possible.

On the XenServer HCL

We want whatever storage we choose to be fully supported when used with XenServer, so something listed on the HCL would be ideal.

NFS and iSCSI support

We would need NFS and iSCSI connection options available to allow a single storage solution to be used as a XenServer repository and NFS ISO host. We would not need fibre due to keeping costs down.

Short-list

After working our way through the HCL, and rejecting the multitude of £10,000+ SAN options, non-rack-mounted options, and fibre solutions, we selected three possible contenders that met the above criteria.

1U QNAP TS-459U (HCL)
2U QNAP TS-809U (HCL)
2U QNAP TS-859U (HCL)

And the winner is…

So, to best meet our “low cost” and “small” criteria, we opted for the 1U TS-459U which could host 4 x 2Tb SATA drives giving 6Tb RAID5 storage. So, onto the shopping list it goes.

Rejected contender

When researching the storage we also liked the Thecus 1U4600 as it had a good feature set but was not on the XenServer HCL for some reason. It was also a little more expensive than the QNAP whilst offering comparable features.

Drive selection

Now that we’d chosen the storage array, we now know we will need four 3.5″ SATA drives.

Consumer vs Enterprise

SATA hard drives are available in two types – consumer and enterprise. The former generally have larger capacities but lower MTBF figures. Enterprise drives are designed to be used in RAID arrays, be continuously spinning and higher reliability figures but are of smaller capacity (the capacity being achieved by combining them into RAID or SAN arrays)

Given our lab would be data-centre based, utilising a RAID5 array in our NAS we wanted enterprise drives. The largest enterprise drives currently available are 2Tb (compared to 3Tb in the consumer space).

SSD vs Magnetic

SSD obviously provide huge amounts of IOPS and very fast read access, but are also massively expensive in large capacities.

And the winner is…

Consulting the QNAP compatibility list, we chose 4 x Western Digital 2Tb RE4-GP to provide a good balance between power consumption, performance and capacity. Given the advisory warning on the QNAP compatibility list about issues with early firmwares, we ensured we obtained a version of the RE4 from our supplier that did not require any firmware upgrades.

Another addition to the shopping list.

1. Rent a server

The quickest and easiest way of getting this up and running, there are hundreds of vendors who will rent you a server. The problem is they are primarily targeted at the web hosting industry, so most servers with any kind of sensible price range are only 1 or 2 cpu core with 2Gb of RAM. Nowhere near powerful enough for a virtualisation host. To rent the kind of server you’d need to run multiple VM’s you’re talking about £300+ per month – a cost I doubt many people would want to absorb just to have a ready-built lab environment (and I certainly couldn’t afford to subsidise it!)

2. Buy a server

Plan B was to buy my own server. This way I’d get to choose the exact hardware specs, and also being a fixed asset the depreciation would be tax deductible   A bigger initial outlay, but would probably be cheaper overall.

The end result…

Thanks to the unfortunate liquidation of A.Nother company, I acquired a second-hand Dell PowerEdge 1950 III 1U server at auction that will provide most of the compute resource. It’s not new kit (budget’s don’t stretch that far!) but with a memory upgrade it’ll form a pretty decent XenServer host. It’s also on the XenServer HCL, which will be important if we discover any issues in building the lab.

Initial specification

2 x 4-core 2.3GHz Intel Xeon CPUs
Dual redundant power supplies
8Gb RAM
2 x Built-in Broadcom Gbit NICs
DRAC remote controller board
73Gb 15k SAS  hard drive + RAID controller

What to upgrade?

To make a decent server virtualisation host we’ll need to add a few components. Let’s take the base specification step-by-step and drill down further on what upgrades we need to consider.

CPU

2 x 4-core 2.3GHz Intel Xeon CPUs

8 CPU cores is pretty good considering the server is a few years old. It’s not the 12-cores-per-socket that we see in the latest AMD processors, but for a low power, lab server, 8 cores will be sufficient.

PSU

Dual redundant power supplies

This is good, as it will provide failover in the case of a PSU failure, or failure of one of the data centre PDU’s. The cost of providing dual-power to the server could become an issue, as all colocation facilities charge more for dual-power provision.

RAM

8Gb RAM

Whilst 8Gb would be sufficient to run two or three VMs, we’re potentially looking at VMs in double figures, so we’re going to need a RAM upgrade. This is likely to be one of the main expenses as Buffered matched pairs of ECC server RAM aren’t cheap!

The server has eight DIMM slots available, four of which are populated by the existing 8Gb. Each DIMM slot can take a maximum of 8Gb, and they have to be installed in matched pairs. So, without discarding any of the installed memory, the largest upgrade available to us is 4 x 8Gb (32Gb) giving a total memory size of 40Gb. If we chose to discard the installed 8Gb, this could be increased in the future to a maximum of 64Gb.

So, 2 x 16Gb of DDR2, PC2-5300 Fully Buffered ECC DRAM added to the shopping list.

NICs

2 x Built-in Broadcom Gbit NICs

Most of the network traffic will be kept internal to the lab environment, however having only two NICs available would mean external traffic to/from the lab, and XenServer management and storage traffic would all pass over the same NICs, which is not best practice. The server has two PCI-X expansion slots available, so to provide some additional networking capacity and flexibility, an additional NIC card of some kind will need to be added.

Server management

DRAC remote controller board

The DRAC board provides KVM over IP and console/BIOS access via it’s own Ethernet port. This will be essential for management the hardware once it’s located in the remote data centre facility. No further changes needed here, apart from a firmware upgrade to the onboard management software.

Storage

73Gb 15k SAS  hard drive + RAID controller

The server as purchased only contained a single 15k 73Gb SAS drive. Whilst this is nice and fast (Hardware RAID + SAS + 15k spindles) there is no resiliencey for a drive failure, and 73Gb provides very little capacity for storing VM images and ISOs. As there is a hardware RAID controller installed, but only two drive slots available (this is a 1U server remember) we only have a couple of options:

1. Replace the single installed SAS drive with two larger capacity (but slower) SATA drives and create a mirror
2. Add a second fast SAS drive of the same (limited) capacity and create a mirror, and use some external storage

We looked at the largest available SATA drives (currently 3Gb) but these are only 5400rpm, a third slower than the currently installed 15k SAS drive and they are also consumer-orientated so don’t have the MTBF figures of the enterprise-class hard drives.

Because option #1 would of reduced performance of the OS installation, and would not of allowed us to add a second server utilising the same storage if required in the future, we opted for option #2. This would give us a resilient and fast partition for the OS installation, but will require some additional storage for the VMs. This configuration is also typical of enterprise deployments, where VMs would usually be stored on a SAN or NAS so we’re keeping the configuration close to “real world” as possible.

Building the shared storage platform will have a post to itself.

End Specification

2 x 4-core 2.3GHz Intel Xeon CPUs
Dual redundant power supplies
40Gb RAM
2 x Built-in Broadcom Gbit NICs
2 x Additional Gbit NICs from PCI-X card
DRAC remote controller board
2 x 73Gb 15k SAS  hard drive + RAID controller