Apache Cloudstack 4.3 Installation - Part 1

Installing a cloud orchestration engine is not like installing a new piece of desktop software.  It requires that the installer understands exactly what is being installed and where.  In order to achieve success the first time through the installers needs to carefully plan the project.  Cloudstack is probably one of the easier products to install but it still requires detailed planning of the end result state before diving into its installation.

Preparation

  • Background reading  - understand what it is you are about to install.  This will allow you to answer installation questions that will greatly affect the outcome.  The best source of information on how to install Cloudstack is located here, on the Apache website.  The document covers the installation process for 4.2 but that should not affect my installation
  • Overall solution design - it is vital that you know 'where you are going' during the installation process.  Designing your network, addressing of component infrastructure will help the installation go a little smoother 
  • Supporting Infrastructure - reading will reveal that there is a minimum amount of 'supporting infrastructure that should be in place before attempting an installation
  • Minimum hardware requirements - make sure that you have the minimum hardware requirements available before commencing installation
  • Keep it simple - Cloudstack can support many different flavours of installation profiles; different networking models, different hypervisors, different storage options.  For a first installation it is recommended that you do not complicate your life and choose the least complicated or most widely used options
  • Success criteria - define a setup of 'installation validation' tests that will prove a successful installation.  This will also help you to learn about Cloudstack in more detail and doubtless start you off on troubleshooting Cloudstack

Platform Background

I will perform this installation on top of a single, suitably powerful, server, running VMWare ESX 5.1.  By performing the installation entirely within a single virtualisation host gives me full control of the whole environment, including the network infrastructure.  It has the affect of reducing the possible locations for errors and thus greatly speeds up the process.

The hardware of this base system is mad up of two components - a 'home brew' server and a HP SAN/NAS device.  These devices are connected over a 1GbE network:

Server

  • 4th gen i7 hyperthreaded processor
  • 32GB of RAM
  • Single SSD containing the host hypervisor only

HP NAS/SAN

  • HP ProLiant G7 MicroServer AMD Turion II N54L
  • 2GB RAM
  • 4 X 500GB disks, configured in software RAID 5
  • Server runs Centos 6.5 x86 64 bit

Environment

The following virtual networks were created

  • Cloud-Management - this 'physical' network will contain the cloud management (untagged) and storage (tagged) traffic
  • Cloud-Public - this 'physical network will contain traffic to and from the client virtual machines (hosted on the virtualised XenServer instances) (untagged) and the intra-tenant virtual machine traffic on their dedicated VLANs (tagged)

All networks were setup with the VLAN setting as 'All (4095)'.  Neither of these networks connect to a physical NIC on the host server. 'Home-net exists for that purpose.

The following virtual servers have were pre-prepared and verified as working prior to starting the Cloudstack installation.  They all have access to the Internet:

  • DNS server
  • NTP server
  • 2 X NFS servers
  • Lightweight firewalls
  • 2 XenServer 6.2 hypervisors
  • Windows client
  • Linux client
  • MySQL database server
  • Standard Centos 6.5 x86 64bit server build ready to receive Cloudstack installation (we are installing a single instance of the management server)

Centos 6.5 x86 64bit was used for the DNS, NTP, NFS, MySQL and Linux clients. The specification of the Centos servers is:

  • 16GB HDD
  • 2GB RAM
  • Centos 6.4 installed from the 'minimal install' ISO CD image (which installs approximately 200 RPM packages)
  • Single NIC on 'Cloud-Management' network

M0n0wall was used for the basic firewalls.  It is very reliable, does what it says on the tin and has a tiny footprint (<1GB disk space and < 256MB RAM)

IP Details

  • The home network, to which the ESX host server connects, has a CIDR of 192.168.1.0/24 and connects to the Internet via the default gateway at 192.168.1.1
  • The Cloud-Public untagged CIDR, which will be used by Cloudstack to assign 'external' IP addresses to the client VMs is 10.200.10.0/24, DG is 10.200.10.1
  • The Cloud-Management untagged CIDR, which will be used for management and storage traffic is 192.168.4.0/24, DG is 192.168.4.1

The diagram above shows the installation infrastructure and Cloudstack components.

Pre-Installation Hypervisor Builds

I have not provided details above for the building of all of the prerequisite servers but I will give details of the actual XenServer 6.2 hypervisor installs.  These servers need to be built as standalone hypervisors with no virtual machines running on them.  Additionally, a small number of 'tweaks' need to be performed on them prior to adding them to Cloudstack.

As per the diagram above, the XenServers will contain 2 'physical' NICs.  

Specification is as follows:

  • 40GB local disk
  • 6GB RAM
  • Single processor, single core
  • First interface is placed on the 'Cloud-Management' virtual VLAN
  • ​Second interface is placed on the 'Cloud-Public' virtual VLAN

​BEFORE building the hypervisors but after adding the virtual machine container, the <server name>.vmk configuration file needs to be altered in order to allow the virtualised XenServer to see hardware virtualisation assist support (see this excellent article for further details about pass through of Intel-VT+EPT or AMD-V+RVI).  Here is what needs to be done:

  • On the ESX host, allow remote management via ssh
  • ​Connect to the management NIC IP address of the ESX host
  • Locate the .VMX file of the XenServer
  • Add the following lines to the end
  • vhv.enable = "true"
  • Save and exit

​If you forget to perform these changes you will see the following message during the installation process of XenServer:

During the XenServer build process, make sure that the correct NIC is selected as the management interface.  If you added the first NIC to 'Cloud-Management' virtual VLAN then 'eth0' will be the appropriate management interface.  This can be confirmed by checking the assigned MAC address in the vSphere UI.   

​Post Installation Configuration

Once the hypervisors have built they need to be 'prepped' ready to join Cloudstack.  The following steps need to be performed:

  • Set XenServer to use OpenVSwitch.  Run the following command at the console of the XenServer:
xe-switch-network-backend openvswitch
  •  Name the interfaces according to their planned names in Cloudstack.  During the Cloudstack build process you will have to tell Cloudstack to direct XenServer to use particular interfaces for certain traffic types.  It will do this via interface names.  The names that you use now in this post-installation step of XenServer MUST match what you use in Cloudstack (see later):
    • Connect to XenServer using XenCenter via the Windows client built earlier and find the 'Networking' tab
    • Select 'Network 0' and edit the properties
    • Name it 'Cloud-Management' and save the changes
    • Select 'Network 1' and edit the properties
    • Name it 'Cloud-Public' and save the changes

That's it.  All the prerequisites are complete.  We are ready to install Apache Cloudstack 4.3 on the earlier prepared Centos 6.4 server

Database Preparation

As covered elsewhere if you are installing to a separate database server it important that you do the following

  • Allow remote connections by root
  • Open port 3306 in iptables for inbound traffic

The Cloudstack documentation specifies some recommended database configuration parameters.  Edit /etc/my.cnf and add the following under [mysqld]:

innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'

Apache Cloudstack 4.3 Management Server Installation

Thankfully a RPM is available for the Apache Cloudstack installation.  This makes the tasks a lot simpler.  Let's just recap on how it will be installed here:

  • A separate MySQL 5.1 instance, running on Centos 6.5 server exists at 192.168.4.7.  Iptables is setup to allow 3306 TCP traffic inbound.  Remote root connection to mysql has NOT been disabled (as would happen if you accepted all the defaults in the 'mysql_secure_configuration' script.  In case the full MySQL lockdown script has been run, execute the following command on MySQL server:
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'password' with GRANT option;

SET PASSWORD FOR 'root'@'%' = PASSWORD('<password>');
  •  A pre-built Centos server, with IP address of 192.168.4.4 exists.  SELinux has been set to permissive
  • Remote access to the MySQL server has been confirmed (a port scan using 'nmap' would confirm 3306 open)

Let's begin:

Management Server Download and Install of Cloudstack and Dependencies

  • Add the RPM repository to your Cloudstack management server.  Specific instruction can be found here
  • In case content move, in essence, all that is needed is a /etc/yum.repos.d/cloudstack.repo file with the following content:
[cloudstack]
name=cloudstack
baseurl=http://cloudstack.apt-get.eu/rhel/4.3/
enabled=1
gpgcheck=0
  • On the Cloudstack management server run the following:
yum install cloudstack-management

On my minimal install Centos machine there is a list of 130 packages to install, with a total install size of 616MB.  I am so glad I have FTTH!  Occasionally, however you may see a timeout issue on the Cloudstack repository.  Simply run the command again.  The process will auto recover nicely.

Keep an eye on the installation process and right at the end you'll probably see this, a little message informing you to download vhdutils.

In order to control XenServer via Cloudstack the vhd-util must be installed on all Cloudstack management servers.  Here is an explanation.

  • From the management server, perform the following:
cd /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver

​wget http://download.cloud.com.s3.amazonaws.com/tools/vhd-util

Database Installation

  • From the Cloudstack management server, run the following command:
cloudstack-setup-databases cloud:<password>@192.168.4.7 --deploy-as=root:<sql_root_password> -i 192.168.4.4

If you haven't installed the mysql client (yum install mysql) you will see the following error:

The script outputs progress quite clearly, with colour coding to indicate outcomes.  When finished, you should output like this:

In order to prevent any problems from Cloudstack using virtualised XenServer hypervisors ( more info ) execute the following in mysql console on database server:

INSERT INTO `cloud`.`configuration` (`category`, `instance`, `component`, `name`,`value`, `description`) VALUES ('Advanced', 'DEFAULT', 'management-server', 'xen.check.hvm', 'false', 'Shoud we allow only the XenServers support HVM');

Configure Management Server

Execute the following command on the Cloudstack management server:

cloudstack-setup-management

This short script just sets up the firewall ports, sudoers etc.

Download The Hypervisor Specific System Templates

As of writing, there is no online documentation update for Cloudstack 4.3.  The Cloudstack 4.2 guide points to the old template.  After doing a little digging on the Internet, I found this github file that references the new file location.  It's a documentation patch.  The file referenced in the above link has been included below:

  • Execute the following commands on the Cloudstack management server:
mkdir /mnt/secondary_storage
mount -t nfs 192.168.4.6:/data/secondary_storage /mnt/secondary_storage
cd /mnt/secondary_storage
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /mnt/secondary_storage -u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-xen.vhd.bz2 -h xenserver -F
  • Download process of the system template should now start (looks distinctly like a 'wget')

Start The Management Server

The system templates should now be in place under /mnt/secondary_storage/template/

It should now be possible to start the Cloudstack management server and set it to start on server start:

service cloudstack-management start
chkconfig cloudstack-management on
chkconfig --list
  • Logon To Cloudstack
  • From a client computer (with a browser) that can reach the Cloudstack management server IP address, navigate to http://192.168.4.4:8080/client/
  • A logon screen should appear as follows:

​ 

  • Logon credentials:
    • Account: admin
    • Password: password

Usage Server Installation

The last thing we are going to do is install the usage tracking server.  This data could be used by a billing engine to calculate invoice quantities for tenants.  Unfortunately the documentation for 4.2 is out of date.  It suggests that you should run 'install.sh' but that script no longer exists.  Here is the link to the (currently) incorrect documentation.

From the management server, execute the following commands.  Remember that the Cloudstack management server must be running when these commands are run and that it must be installed on the same server as the management server (in our case 192.168.4.4):

yum install cloudstack-usage
​service cloudstack-usage start
chkconfig cloudstack-usage on

Next Steps

The installation is complete but without configuring the system (including adding at least one hypervisor) it is not at all useful.  The next part of this howto (Part 2) will cover

  • Choose a networking model (we will use advanced)
  • Create a zone
  • Create a pod
  • Create a cluster
  • Add primary and secondary storage
  • Launch the zone
  • Add XenServer 6.2 hypervisor

​Once all that is complete we will validate the installation with a series of basic tests.  Those tests will include (but not be limited to!):

  • Upload an ISO image
  • Create a virtual machine
  • Take a snapshot of the virtual machine
  • Add another hypervisor to the cluster
  • Create a virtual machine on second hypervisor and attempt to communicate between the two VMs via the tenant specific VLAN

UPDATE - Apache Cloudstack 4.3 Documentation Released

I have just noticed that the latest published documentation for Apache Cloudstack (Installation Guide, Administration Guide) is for version 4.3.   Links are in the credits