OpenStack RDO Installation on Centos 6.5 - Part 1

RDO

OpenStackThis article uses as a starting point a presentation located on the RDO homepage in which Lars Kellogg-Stedman runs through an OpenStack installation using this tool set.  It is a great video but I thought I would fill in some of the details for people starting out and lay it out as a step-by-step process.  Hope other find this useful.

Firstly, watch the video.  It gives a great overview of what we are about to do.  Secondly, flick through the slides quickly in order to review some of the most important bits from the video.

Preparation - High Level Info

Here is a diagram of the lab setup in order to understand the later instructions.  In order to use the step-by-step instructions in this article it is necessary to configure the servers with the same configuration as outlined below.

OpenStack Test Installation ​ ​ ​

The diagram shows the entire test environment is virtualised.  In this case it is ESXi 5.1.  The PackStack installation distributes the services/agents in the follow manner across the hosts (which is controlled by a configuration file that is passed to PackStack at point of installation.

OpenStack Test Installation ​ ​

Note the difference in in AMQP message broker as listed in Lars' original presentation.  The configuration file defaults have altered as RDO moves to Icehouse.  It defaults to Rabbitmq and not qpid. 

Preparation - The process

Virtual Networks

  • Create two isolated virtual networks (portgroups) were setup on the same virtual switch as follows:
Name Description Comments
Management Isolated portgroup, VLAN ID: All (4095) This network is used for PackStack interactions with all hosts.  It will be also be used for API interactions between all hosts
Private Isolated portgroup, VLAN ID: All (4095) This network will be used for GRE tunnels.
Public Isolated portgroup, VLAN ID: All (4095) This network will be used for accessing the Internet from the guest VMs.  It will also be used for inbound communication by 'users' to their virtual guest VMs via ssh, http etc (not via vnc proxy within OpenStack)

The isolated networks are connected to the Internet (simulated) by a pair of M0n0wall firewalls.  This router was selected due to the ease of setup and the very low resource requirements.  There are many other options in this space including Smoothwall, ClearOS.  All work fine in VMs.  The setup of the firewalls is not covered in this article.  If I get a chance I will add it soon.  It is important to state the following points about the overall function of the firewalls:

There are no egress limitation in place.  I.e. all egress ports are open

Routing between private IP ranges is enabled.  This allows the M0n0wall firewall to route traffic from the test environment to devices on the 192.168.1.0/24 network (my home 'production' network)

Virtual Machines

Configure the virtual machines that form part of this OpenStack test as per the following table:

NOte: ​BEFORE building the hypervisors but after adding the virtual machine container, the <server name>.vmk configuration file needs to be altered in order to allow the virtualised KVM to see hardware virtualisation assist support (see this excellent article for further details about pass through of Intel-VT+EPT or AMD-V+RVI).  Here is what needs to be done:

  • On the ESX host, allow remote management via ssh
  • ​Connect to the management NIC IP address of the ESX host
  • Locate the .VMX file of the XenServer
  • Add the following lines to the end
  • vhv.enable = "true"
  • Save and exit
  • As an additional step, upgrade the version of the virtual hardware to 9 (if using ESXi 5.1)
Name Processors RAM Disk Size Networks
PackStack Master 1 4GB 40GB

Management

HomeNet (connects to the 192.168.1.0/24 physical world)

Controller 1 4GB 60GB Management
Network 1 4GB 20GB

Management

Private

Public

Compute1 1 6GB 160GB

Management

Private

Compute2 1 6GB 160GB

Management

Private

 

Preparation - System Base Layer Builds

  • Build all 5 servers using Centos 6.5 minimal install ISO, configuring the network interfaces as per this table
Name Networking Comments
PackStack Master

eth0 - 192.168.10.2

eth1 - 192.168.1.247

Centos 6.5.  It is from here that PackStack is run.  This machine is connected to the 192.168.1.0/24 network so that an operator can hop onto any of the lab computers
Controller

eth0 - 192.168.10.3

Contains OpenStack components except for Neutron network and hypervisors
Network

eth0 - 192.168.10.4

eth1 - 172.16.0.4

eth2 - not configured

Network host, contains Neutron networking components
Compute1

eth0 - 192.168.10.10

eth1 - 172.16.0.10

Hypervisor 1, contains nova compute components
Compute2

eth0 - 192.168.10.11

eth1 - 172.16.0.11

Hypervisor 2, contains nova compute components
  • Ensure that all systems are fully patched
yum update -y

Preparation - System Base Layer Services

NTP

  • Install and start NTP
​yum install -y ntp
​chkconfig ntpd on
service ntpd start

Name resolution

  • Configure name resolution ensuring that all hosts can contact each other via FQDN and that 'hostname --fqdn' returns the FQDN of the server.  Not strictly 'enterprise grade' best-practice, add the following text to each hosts' /etc/hosts file:
192.168.10.2    master.cloud.local
192.168.10.3    controller.cloud.local
192.168.10.4    network.cloud.local
192.168.10.10   compute1.cloud.local
192.168.10.11   compute2.cloud.local

Certificate Based Logon From Master to OpenStack Hosts

  • On the master server, logged in as root, create the certificates:
cd ~
mkdir .ssh
cd .ssh
ssh-keygen -t rsa -b 2048 -N ''
  • Set permissions on the private key on the master server
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa 

Copy certificate from master to the other 4 servers

scp -r ./.ssh root@192.168.10.4:~
  • Login to each of the four OpenStack servers and perform the following
cp ~/.ssh/id_rsa.pub ~.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
restorecon -Rv ~/.ssh

(the last step is important and ensures correct SELinux contexts are set)

Most of the above came from this very helpful Centos Wiki page on securing SSH

​No configuration changes are necessary in /etc/ssh/sshd_config as this is just lab testing and there is no need to prevent password based logons, for example.

SNAPHOTS (Completely Optional)

  • Shutdown all servers and take a snaphot of their current states (not necessary but you may like to do this so you can go back to a clean setup, should the need arise)

Installation

All preparatory steps are now complete

Install PackStack

  • PackStack must be installed on the master server.  To do that the RDO repositories must be added first
yum install -y http://rdo.fedorapeople.org/rdo-release.rpm
  • Now install PackStack
yum install openstack-packstack

PackStack Configuration

  • The first step in configuring PackStack to install OpenStack on the servers is to get PackStack to create an installation configuration file
packstack --gen-answer-file packstack_answer_file.txt
  • Alter and save the PackStack configuration file to suit the multi-node environment.  In order to assist in this I have attached my original configuration file (see below) as a text file and a PDF version of it with the highlighted changes that I made.  These are the changes you need to make:
CONFIG_SSH_KEY=/root/.ssh/id_rsa.pub
CONFIG_SWIFT_INSTALL=n
CONFIG_CEILOMETER_INSTALL=n
CONFIG_HEAT_INSTALL=n
CONFIG_NAGIOS_INSTALL=n
CONFIG_MYSQL_HOST=192.168.10.3
CONFIG_AMQP_HOST=192.168.10.3
CONFIG_KEYSTONE_HOST=192.168.10.3
CONFIG_GLANCE_HOST=192.168.10.3
CONFIG_CINDER_HOST=192.168.10.3
CONFIG_NOVA_API_HOST=192.168.10.3
CONFIG_NOVA_CERT_HOST=192.168.10.3
CONFIG_NOVA_VNCPROXY_HOST=192.168.10.3
CONFIG_NOVA_COMPUTE_HOSTS=192.168.10.10,192.168.10.11
CONFIG_NOVA_CONDUCTOR_HOST=192.168.10.3
CONFIG_NOVA_SCHED_HOST=192.168.10.3
CONFIG_NOVA_COMPUTE_PRIVIF=eth0
CONFIG_NOVA_NETWORK_HOSTS=192.168.10.3
CONFIG_NOVA_NETWORK_PUBIF=eth1
CONFIG_NOVA_NETWORK_PRIVIF=eth0
CONFIG_NEUTRON_SERVER_HOST=192.168.10.4
CONFIG_NEUTRON_L3_HOSTS=192.168.10.4
CONFIG_NEUTRON_DHCP_HOSTS=192.168.10.4
CONFIG_NEUTRON_LBAAS_HOSTS=192.168.10.4
CONFIG_NEUTRON_METADATA_HOSTS=192.168.10.4
CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=gre
CONFIG_NEUTRON_OVS_TUNNEL_RANGES=1000:3000
CONFIG_NEUTRON_OVS_TUNNEL_IF=eth1
CONFIG_OSCLIENT_HOST=192.168.10.3
CONFIG_HORIZON_HOST=192.168.10.3
CONFIG_SWIFT_PROXY_HOSTS=192.168.10.3
CONFIG_SWIFT_STORAGE_HOSTS=192.168.10.3
CONFIG_PROVISION_DEMO=n
CONFIG_HEAT_HOST=192.168.10.3
CONFIG_HEAT_CLOUDWATCH_HOST=192.168.10.3
CONFIG_HEAT_CFN_HOST=192.168.10.3
CONFIG_CEILOMETER_HOST=192.168.10.3
CONFIG_MONGODB_HOST=192.168.10.3
CONFIG_NAGIOS_HOST=192.168.10.3

These changes were done by examining the file that Lars originally provided here.   

PackStack Execution

  • All that remain now is to run the PackStack command:
packstack --answer-file packstack_answer_file.txt

As Lars says in his video.  Time to get a coffee.  While you are away, your screen will look something like this:

PackStack Installation - Commencement

Once the process has finished there a number of post installation steps that will be detailed in part 2 of this article

Installation Errors - Symptoms and 'Fixes'

Error 1

Just in case the problems that I experienced during PackStack execution help others, I have included them here.

During my initial execution of PackStack, I experienced the following error:

stderr: Warning: Permanently added '192.168.10.11' (RSA) to the list of known hosts.
+ trap t ERR
+ rpm -q --whatprovides puppet
+ yum install -y puppet
http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/epel-6/repodata/repomd. xml: [Errno 14] PYCURL ERROR 7 - "Failed to connect to 2610:28:3090:3001:5054:ff:fedb:7f5a: Network is unreachable"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: openstack-icehouse. Please verify its path and try again
++ t
++ exit 1

PackStack Installation - First Error ​​​

I checked name resolution from the 'Compute 2' node and could see no problem.  I could see no issues with the default gateway, route to the Internet, name resolution within the OpenStack test servers.  Without a restart, I re-ran the PackStack command line and no such error occurred.

Error 2

This error occurred during the subsequent second execution of PackStack.  The error message is:

192.168.10.11_nova.pp:                            [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 192.168.10.11_nova.pp
Error: Command exceeded timeout
You will find full trace in log /var/tmp/packstack/20140508-180619-YqpjkL/manifests/192.168.10.11_nova.pp.log
Please check log file /var/tmp/packstack/20140508-180619-YqpjkL/openstack-setup.log for more information

PackStack Installation - Second Error

The trace log and setup logs are in the attachments section below.  I could not really get a clear understanding of what had failed from the Nova log except that perhaps an installation process of an application dependency had failed.  The yum log showed no such issue.  The main error however indicated a timer threshold had been exceeded.  I elected to revert the snapshots and try again.

3rd Time Lucky

On this third run through PackStack executed without error.  Here is what the final output should look like:

PackStack Installation - Success