Note: Please note that whilst this document is complete enough to get you most of the way through building a single node it isn't as complete as I'd like due to the fact that the licences for the various components stop you from assembling a full VMWare "data centre" with multiple hosts. As a result it gets to install vCenter on Linux and then ends abruptly.

At this point I'll probably be going to look at OpenVZ with heartbeat and DRBD as a solution.

A walk-through guide to setting up a home VWare Lab.
This is still a very alpha quality document and clearly a work in progress, even when it is finished I can provide no warranty or guarantee as to the correctness, safety, validity or reproducibility of the documentation below.

Index

1) Introduction
1.1) Background
1.2) Requirements
1.3) Conventions
1.3.1) Machines and host names
1.3.2) Prompts
1.3.3) Metasyntatic Variables
3) Process
3.1) Hardware Preparation
3.2) Initial Software Installation
3.3) Boot Strapping VSphere into the ESX Host
3.4) vSphere CLI Install
3.5) Install vCenter for Linux
4) Upgrade Path
4.1) RAM, Local Storage and CPUs
4.2) Networking
4.3) Solid State Drives
5) Todo

Introduction

Background

I consider myself lucky in that what I do for a living at the moment is pretty close to what I'd choose for a hobby and it's probably fairly clear, even if you've only read this one page that it's engineering computer systems of one flavour or another. Primarily open source, web based systems but I see no reason to limit myself in choice of tools so I'll go with whatever does the job best really. Anyway, working and playing in this space dictates that I have a fairly beefy home system, I think the "lowest spec" machine I've ever bought (in 1999?) was a dual P3 733 MHz box that I took to Uni.
My current box, which is getting a bit long in the tooth is a dual Xeon 2.8GHz beasty with ~4G of RAM and when I came to study for my RHCT the other day I realised it was hugely overloaded acting as my workstation, home web server, media and TV. So I broke out that old dual P3 full tower box and stuck Red Hat on it, having passed I now want to do my RHCE and realised that my next "upgrade" isn't really for raw compute speed or disk space it's more about redundancy and flexibility. As a "reward" (perhaps a bit OTT as the RHCT wasn't that difficult) I decided that a new box to help me create some practice virtual machines would be a good idea.

Requirements

After arriving at the decision to buy a new "system" I came to the following general requirements...
1) Flexibility and separation via virtualization, I don't particularly play games and I'm not intending to simulate any large database or the like so I'm not overly interested in physical brute force. I'm interested in being able to knock out maybe a dozen simple servers and a couple of "service" machines.
2) Redundancy both of hardware and storage, I already have an effective off-site backup but local RAID redundancy would be very good.
3) Expandability, I don't want to have to blow any more than £500 upfront but later on when prices have dropped I do want to expand everything out. My tendency has been to buy the best base components and fit them with the best "non-premium priced" technology available at the time, upgrading to the ex-bleeding edge once the price has dropped.

After browsing around I decided to go with the HP DC7900.

I'm familiar with the well thought-out HP desktop kit from ${work}, several people have listed them as fully compatible with ESXi 4.0 update 1, they'll take up to 16GB of RAM, appear on eBay for just over £300 and come in Ultra Small, Small and Convertible Mini Tower Form Factor.

I went with the SFF format as the convertible mini-tower is a bit too big to fit on some shelves I have and the USF compromises the motherboard design considerably. The Core Duo 2, 3.0GHz processor is also the cheapest VPro version which means you get some added bonuses.

Hence the initial "ingredients" list is...

* HP DC7900, 3GHz Core Duo, 2GB of RAM, 160GB HDD, approximately £300 off of ebay. Preferably with it's Vista install disks and license, mine also came as originally boxed with keyboard and mouse (no monitor obviously).
* An existing Linux box capable of running VMPlayer.

Conventions

Machines and Host Names

I started with two physical servers in this guide...
1) My existing Linux server this is simply known as "giga" and will act as an initial management box.
2) The DC7900 which will act as the ESXi host.

Following on from my deliberations about naming conventions I have decided to extend the logical progression around my existing "giga" server and the other SI prefixes and this is what I've arrived at...

...using these prefixes outside the context of SI-measurements leaves them open to interpretation whilst still retaining both a distinction of class (less than 1 being virtual servers and greater than being physical) and also of natural order. In combination with the other conventions suggested in my naming convention leave me with this...

{Purpose}{pool ID}-{Hardware Type}.{Network Identifier}-{ISO 3166-2 Regional alphanum 1-3 code}{location ID}-{ISO 3166-1 Alpha-2 country code}.philipmather.me.uk

...this produces the following names for the machines I already have or will be building initially...

* lin01-giga.man-wok1-gb.philipmather.me.uk; My existing Linux Server.
* esx01-tera.man-wok1-gb.philipmather.me.uk; The new DC7900 ESXi Host.
* vsp01-micro.man-wok1-gb.philipmather.me.uk; A virtual VSphere management workstation, created in VMPlayer and migrated to esx01-tera.
* vcn01-micro.man-wok1-gb.philipmather.me.uk; A virtual VCenter server.
* spw01-centi.man-wok1-gb.philipmather.me.uk; A Virtual RedHat Spacewalk server with local PostgreSQL database.

...this probably seems hugely verbose and I agree but from here on in I'll only be using the first section (i.e. "lin01-giga") which given the context is enough for you to work out which machine I'm talking about. I'll cover that second section later on.

Prompts

There will be two types of prompt symbol used, the first is a "$" and denotes the the use of a non-root account on your existing linux server. The second prompt symbol is "#" and denotes the super user prompt on the ESXi host that is accessed by enabling SSH access later on. Any variation from this will be clearly explained at the time.

Metasyntatic Variables

See the wikipedia article on metasyntatic variables if you have no idea what I'm on about...
1) aaa.bbb.ccc.ddd
2) www.xxx.yyy.zzz

Process

Hardware Preparation

Lets start with the DC7900 and some background information from HP's official overview page. Some key features of this machine are...
1) It's almost "out-of-the-box" ready for VMWare except for a bug in the BIOS which might relate to this HP forum discussion "VMware / EM64T / VTd // HP DC7900".
2) The on-board RAID will not work with 3.5 apparently and possibly even 4.0.0 but I've not tested, will do at some point and report back about update 1.
3) ESX and EXi 4.0 compatibility notes can be found at www.vm-help.com and at ultimatewhitebox.com
4) Further discussions can be found on the VM-help forums, specifically HP DC7600 SFF (1),
HP DC7900 SFF (2),
VMWare ESX 3.5 on HP DC7900 and FAQ: Downloading the latest CommunityUnifiedDriverPack.

So start by getting Vista installed on the box for a moment, if you're lucky it'll still be in it's post-install state just waiting for you to plug a username in and do some final configuration to it. As we've seen above you want the latest BIOS (or at least version 1.23) to make sure the VT-d will play nicely and it's also good practice, the Management Engine that comes as part of the Intel vPro also has a fairly important security fix so update that straight after doing the BIOS. The links above assume you have 32bit Vista Business installed by the way and when I first installed it didn't seem to recognise the network card so I had to use a USB stick to transfer the driver over from my Linux box. How on Earth can Windows not recognise a fairly standard Intel NIC?

With the BIOS and ME Firmware upgrades complete now is a good time to reboot the computer and check all of your BIOS seetings, specifically I'd suggest enabling Security -> System Security -> options 1 - 5 and 7 as well to "Enabled". This enabled VT-d (which you obviously need), some extra security features and the prompt for the Intel AMT Management Engine. With those enabled, exit the BIOS and hit "Ctrl-P" to gain access to the ME interface. Remember those extra vPro goodies I talked about earlier, this is where they come in, take a quick read of the first link to see if you want this and if so follow the instructions in the second link, I'll cover the contents of the third link later...

* Intel® Active Management Technology
* Hints on setting up the Intel ME BIOS Extension that apply equally well to the DC7900.
* Intel AMT Open-source Tools and Drivers on sourceforge.

...website OOB interface.

Initial Software Installation

Once you've done all that we want to get VMWare ESXi 4.0 Update 1 with VMware vSphere, at this point you'll want to register an account if you haven't already got one. You'll be using this account quite a lot and VMWare don't seem to spam people so I'd use your real details. At the "Choose a hypervisor to download" stage go with the second option "ESXi 4.0 Update 1 Installable (CD ISO)", don't bother with the HP specific ISO. Unfortunately this is for server kit and it seems to expect things like iLO which your desktop hasn't got and you don't need because you bought the 3GHz vPro version didn't you? When I tried the HP specific image it PSOD'ed pretty quickly. If it does PSOD give it a moment to write out a dump file and then it offers you a recovery console which is the sort of thing you want to play with before you find it in production, usually at 03:00AM, when the website is down. Been there, done that.

Anyway this will download you "VMware-VMvisor-Installer-4.0.0.Update01-208167.x86_64.iso" which you can burn to a CD in you nice new DC7900, you can even put a nice Light Scribe label thingy on it if you want.
Slap your new bootable CD in and follow the prompts to install VMWare ESXi, the defaults are almost always utterly acceptable, configure your networking as required and set a root password, it'll tell you the IP address it's been given so go and have a look at it, from here on in I'll refer to this IP with the meta-syntactic variable of "vvv.www.xxx.yyy", hence stick this URL in your browser "https://vvv.www.xxx.yyy/" and accept the certificate warning.

Next thing to do is acquaint yourself with the hidden command line access so that you can get SSH access to your new host. Might as well do something useful with this right now, so try the following...

$ ssh root@www.xxx.yyy.zzz "esxcfg-vmknic -l"
root@www.xxx.yyy.zzz's password: 
Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type    
vmk0       Management Network  IPv4      www.xxx.yyy.zzz                            255.255.255.0   www.xxx.yyy.255   00:25:b3:19:9d:e9 1500    65535     true    DHCP    

...what's so useful about that then? Well first off you've just learnt how use one of the "esxcfg-*" family of commands to list out the network cards the vmkernel knows about and secondly you've gotten hold of the MAC address of your host (00:25:b3:19:9d:e9 in my case, from here on in I'm going to refer to this by another meta-syntactic variable aa:bb:cc:dd:ee:ff) which means you can turn off your new host when you aren't using it and automagically restart it from your other Linux box using the wake-on-LAN feature, "sudo etherwake aa:bb:cc:dd:ee:ff". Magic eh?

Boot Strapping VSphere into the ESX Host

Install VMWare player on your existing Linux machine.
* http://www.vmware.com/pdf/vmware_player300.pdf
* http://downloads.vmware.com/d/info/desktop_downloads/vmware_player/3_0
* http://forums.fedoraforum.org/archive/index.php/t-234928.html
* sudo yum install gcc kernel-devel
* sudo sh ./VMware-Player-3.0.1-227600.i386.bundle
* sudo vmware-modconfig --console --install-all
* sudo mv /usr/lib/vmware/resources/mozilla-root-certs.crt /usr/lib/vmware/resources/mozilla-root-certs.crt.old

Build a Vista based guest in VMPlayer and install vsphere to connect to new host, 25G disk, 1G RAM and bridged network. Set the screen resolution to 800x600 for now and make sure you attach a CD-ROM so that once it's in ESXi you can install vmware-tools.
Remember that once you've built this machine it's a pain to adjust it because the only thing you can adjust it with is either the command line or a cloned version of itself, this makes it all very fiddly. Enable rdesktop under control panel, system, remote settings and just double check that your admin user has remote desktop rights. You may as well do all the Windows upgrades at this point as well, next install VSphere...

instructions and URL
Screen shot
Connect to the host, create an admin user and find data store.

...clean up the disk (I managed it with 20G but it literally said 20G used at one point so I must have had megabytes left, best stick with 25G), use the "disk tidy" app in vista having also disabled indexing and then shut it down because you don't want the disk being written to whilst the next bit happens. For the sake of neatness use the defrag and compress disk utilities in vmplayer to shrink the disk a bit more. Close vmplayer completely and transfer the VSphere guest's disk file (.vmdk) over to the host via scp and clone it into a format that ESXi can handle.

$ ssh root@192.168.1.65 "mkdir -p /vmfs/volumes/datastore2/vsphere01-micro"
root@192.168.1.65's password: 
$ scp /home/philip.mather/vmware/vsphere01-micro/vsphere01-micro.vmdk root@192.168.1.65:/vmfs/volumes/datastore2/vsphere01-micro/vsphere01-micro-tmp.vmdk
vsphere01-micro.vmdk                      100%   18GB  9.8MB/s   32:26
$ ssh root@192.168.1.65 
root@192.168.1.65's password: 
You have activated Tech Support Mode.
The time and date of this activation have been sent to the system logs.
 
Tech Support Mode is not supported unless used in consultation
with VMware Tech Support.
 
VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/ sysadmintools for details.
 
Tech Support Mode may be disabled by an administrative user.
Disabling requires a reboot of the system.  Please consult the ESXi
Configuration Guide for additional important information.
 
# cd /vmfs/volumes/datastore2/vsphere01-micro/
# vmkfstools -i vsphere01-micro-tmp.vmdk -d thin vsphere01-micro.vmdk
Destination disk format: VMFS thin-provisioned
Cloning disk 'vsphere01-micro-tmp.vmdk'...
Clone: 100% done.
# exit

...now fire your vmplayer vshpere guest back up, connect to the ESXi host and create a new custom machine using the disk you've just imported. New cdrom, floopy, 1 processor, 1G of RAM, pretty basic. Power it up and also set it to start up automagically on reboot.

(screen shot)

You should now be able to close the vmplayer vsphere server, install rdesktop on your linux machine and...

$ rdesktop 192.168.1.66 -g 1270x1000 -u vsphere-admin

...into your vsphere server that is running as a guest on the host it will be managing. The real proof of the pudding it to shutdown the host, sit on your other linux server and etherwake it, that process should also start your vsphere server so that you can rdesktop into a few minutes later. Neat huh? Obviously it's a very good idea to keep a copy of your vmplayer version around in case anything bad every happens to the guest one running inside the host, otherwise you'll end up in a chicken and egg situation of needing the guest vsphere server to fix the host.

Warning: From what I can understand of Microsoft's licensing I am within the terms of the license in moving my virtual Vista Business machine from VMPlayer to the ESXi host, I guess it could be argued that by using one copy to "turn on" the other then for a brief moment I'm using/have installed two copies but given that I could shutdown the VMPlayer version, move it over to the ESXi host, delete the old copy from VMplayer, SSH into the ESXi Host convert and attach the disks then power it on all from the command line without using Windows then I am at least acting within the spirit of the agreement by not continuing or needing to continue using two copies at once as it were. I'm certainly not depriving Microsoft of revenue but at the end of the day I'm neither a lawyer nor a licensing expert so be it on your own head and hence use your own discretion in deciding if this is a legitimate process.
What you do need to be aware of is that when it's powered up on the ESXi host it evidently wants to be re-licensed, this is a real world process that involves phoning a free, automated help line that asks for a very long number and then gives you back a different, very long number. Fiddly but not really that difficult as instructions on screen are pretty helpful and straightforward and it's not an instant requirement.

vSphere CLI Install

* sudo yum install perl-XML-LibXML perl-Crypt-SSLeay perl-Class-MethodMaker (http://fedoraforum.org/forum/showthread.php?t=188639)
* http://communities.vmware.com/community/vmtn/vsphere/automationtools/vsphere_cli
* VMware vSphere Command-Line Interface - 64-bit Linux Installer
* VMware-vSphere-CLI-4.0.0-198790.i386.tar.gz
* untar it into /var/tmp/ and then run sudo ./vmware-install.pl

Install vCenter for Linux

Now on to vcenter, as we only have a single vista license and VCenter won't run on Vista anyway we'll have to use the linux technology preview which brings us to a problem in that the Linux Technology preview won't work with the trial version of ESX. So yes, a this point the following steps are semi-pointless, you'll get installed point your VSphere Client at it, download some extra bits and then find that you can't add your Host into a new data centre because "The host does not support clustering facillities" or some such. Perhaps at some point VMWare will allow a test version that works with trial version as the windows one does.

Regardless, this time we'll deploy the OVF via our new vsphere server, so fire up IE and download the zipped OVF template from http://communities.vmware.com/community/beta/vcserver_linux, follow the "Download VMware vCenter Server 2.5 on Linux (Tech Preview)." link and select the "VMware vCenter 2.5 for Linux Virtual Appliance (Open Virtual Machine Format)". Once you've got the zip file extract the contents to the desktop and start the vsphere client, click on "File" -> "Deploy OVF Template" -> "Deploy From File" and select the OVF file from the desktop, click next, next, accept, next and give it a name, I called my vcn01-micro, finally select next again and ensure that "Network 1" maps to "VM Network" and hit next and then finish.

The OVF will then run through it's deployment process, once complete power it up, decline the the use of NIS unless you really want it and you'll be left at the following screen...

From your existing linux server visit http://www.oracle.com/technology/software/tech/oci/instantclient/htdocs/linuxsoft.html and download...

* instantclient-basic-linux32-10.2.0.3-20061115.zip
* instantclient-odbc-linux32-10.2.0.3-20061115.zip

...transfer these using scp or similar into "/root/" (the default password for root is "vmware") of the deployed template and reboot it. The reboot process will find the two zip files and automagically install them for you and then prompt you details of the database server to connect to. As we will be needing another Oracle XE instance later for Spacewalk we may as well create a small virtual database server now to act as a common VMWare internal management database for both VCenter and Spacewalk, so build a basic CentOS server and then download "oracle-xe-univ-10.2.0.1-1.0.i386.rpm" from... "http://www.oracle.com/technology/software/products/database/xe/htdocs/102xelinsoft.html" ...into a new directory "/root/rebuild/". Insert the following into a file, I tend to put such things into a special directory "/root/rebuild/" so that I know what's required to rebuild a machine again later.

#!/bin/bash
/usr/sbin/groupadd -r dba
/usr/sbin/useradd -r -M -g dba -d /usr/lib/oracle/xe -s /bin/bash oracle
 
/usr/bin/yum -y install bc libc.so.6 libaio.so.1
/usr/bin/yum -y localinstall --nogpgcheck oracle-xe-univ-10.2.0.1-1.0.i386.rpm
 
cat >> /root/rebuild/oracle-xe-install.answers << 'EOF'
8080
1521
changeme
changeme
y
EOF
 
/etc/init.d/oracle-xe configure < /root/rebuild/oracle-xe-install.answers
 
cat >> /etc/tnsnames.ora << 'EOF'
XE =
    (DESCRIPTION =
       (ADDRESS_LIST =
          (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
       )
       (CONNECT_DATA =
          (SERVICE_NAME = xe)
       )
    )
EOF
 
mkdir /usr/lib/oracle/xe/oradata/vcdb/
chown oracle:dba /usr/lib/oracle/xe/oradata/vcdb/
export ORACLE_HOME=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server
 
cat > /root/rebuild/vcenter-post-install.sql << 'EOF'
CREATE SMALLFILE TABLESPACE "VPX" DATAFILE '/usr/lib/oracle/xe/oradata/vcdb/vpx01.dbf' SIZE 100M AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITED LOGGING EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO;
 
CREATE USER "VPXADMIN" PROFILE "DEFAULT" IDENTIFIED BY "changeme" DEFAULT TABLESPACE "VPX" ACCOUNT UNLOCK;
 
GRANT connect TO VPXADMIN;
GRANT resource TO VPXADMIN;
GRANT CREATE VIEW TO VPXADMIN;
GRANT CREATE any sequence TO VPXADMIN;
GRANT CREATE any TABLE TO VPXADMIN;
GRANT execute ON dbms_job TO VPXADMIN;
GRANT execute ON dbms_lock TO VPXADMIN;
GRANT unlimited tablespace TO VPXADMIN;
quit
 
/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/bin/sqlplus 'sys/changeme@xe as sysdba' @/root/rebuild/vcenter-post-install.sql

...and execute it. You should wind up with a little machine running Oracle XE which you can now use. Leave it running and return to your original VCenter server that is probably still prompting you for the following...

Once it's all set-up and connected you'll find a web interface at https://iii.jjj.kkk.lll:5480/#Core.Login (username and password is still root/vmware) but what you really want to do is hop back to your Vista VSphere client machine (vsp01-micro) and point the VSphere client at this IP address from where it will download some extra components...

...I found saving the installer to be the best option. Once logged in (root/vmware again) VMWare will guide you through creating a new Data Centre and you'll hit the problem outlined at the start of this stage regarding trial licenses.

* http://communities.vmware.com/servlet/JiveServlet/previewBody/9580-102-1-6866/vclinux_install_guide.pdf
* http://www.vcritical.com/2009/02/installing-vcenter-on-linux-technical-preview/
* http://blog.vmpros.nl/2009/03/11/vmware-howto-setup-vcenter-server-25-on-linux-connecting-to-oracle-xe-10g-release-2-part-i/
* http://virtualfuture.info/2009/03/virtualcenter-on-linux/

vCenter Server 2.5 on Linux
* https://vvv.www.xxx.yyy/folder?dcPath=ha-datacenter
* vifs --server 192.168.1.65 --username root --listds
* vifs --server 192.168.1.65 --username root --mkdir "[datastore2] vmware-vpxd"
* vifs --server vvv.www.xxx.yyy --username root --listds (http://www.vm-help.com/esx/esx3i/esx_3i_rcli/vifs.php)
* vifs --server vvv.www.xxx.yyy --username root --put ./vmware-vpxd/system-s001.vmdk "[datastore2] vmware-vpxd/system-s001.vmdk"
* vifs --server vvv.www.xxx.yyy --username root --put ./vmware-vpxd/system-s002.vmdk "[datastore2] vmware-vpxd/system-s002.vmdk"
* vifs --server vvv.www.xxx.yyy --username root --put ./vmware-vpxd/system.vmdk "[datastore2] vmware-vpxd/system.vmdk"
* vifs --server vvv.www.xxx.yyy --username root --put ./vmware-vpxd/VpxdOnLinux-VirtualAppliance.vmx "[datastore2] vmware-vpxd/VpxdOnLinux-VirtualAppliance.vmx"
* vmware-cmd --server vvv.www.xxx.yyy --username root -s register "[datastore2] vmware-vpxd/VpxdOnLinux-VirtualAppliance.vmx"
* vmware-cmd --server vvv.www.xxx.yyy --username root -l

Todo:
Seperate vsphere server/guest and IPs for clarity.
Insert screenshot of VMWare installation
Insert screenshot of bios security config page
Try out RAID
Cover expansion options, 16G of RAM, identical hard drive, network adapter and switching gear. SSD?
Intel Core 2 Quad Q9650 Socket 775 3GHz 1333FSB 12MB Cache Retail Boxed Processor as cheap as £230.

The following RAM is NOT compatible with the DC7900, Geil 8GB 2 x 4GB DDR2 PC2-6400 800Mhz Black Dragon Dual Channel Memory Kit - GB28GB6400C6DC, at least not one single pack of two pieces nor a single stick on it's own. The HP manual is slightly unclear, it might support a combination of all four slots populated but I can't out for sure and the cost was prohibitive that month so I had to send them back. At this point plug AWD-IT who immediately refunded the £200 odd pounds worth after a single quick phone call. Excellent service.

Get second host and boot via kickstart from VM in first host, turn into DC with vcenter.

Later
Install a management server.
Install Big IP LTM VEs
Install Free NAS server
Install 3 web servers, 3 PHP servers and 3 Postgresql nodes to run a three tier drupal site.

Further notes...
3.6) ESX Host Configuration
3.6.1) Auto VM Start-up
3.6.2) Virtual Networking
4) Deploying a Linux Stack
4.1) Overview
4.2) Spacewalk Management Server
4.2) LigHTTPd Web Servers
4.3) F5 Big IP LTM VE
4.4) PostgreSQL Database Servers
4.6) Drupal Admin Servers
Vist it here to deploy it
* http://blogs.vmware.com/vapp/2009/07/commandline-ovf-deployments-.html