Archive for March, 2013

In this guide I explain how to install Openstack in a single physical node. I install the nova controller and a compute node in this node. The aim of this article is to get you started with Openstack IaaS with minimum effort in a short period of time.

What you need

The steps below can be followed using one physical node. The node should posses two network interfaces. One of them could be a virtual one. I have tested this on Ubuntu 12.04 LTS 64 bit server. The memory and storage requirements of the node depend on how much virtual machines you run on Openstack once it is ready. For example if you plan to run 10 virtual machines with 256Mb memory and 5Gb HD each, then you need at least 3G memory and 60Gb hard disk for the node. You also need an internet connection to download the necessary Openstack software.
Note the installation described in this document is in no way production ready. You may need to do lot of enhancements, feature additions to make it such.

Installation Steps

Step1:Insatll Ubuntu server

Install Ubuntu server as you do any normal installation. Please refer to good Ubuntu documentation for this. During the installation steps do the following.
– Create a user account on the host machine(say nova).
– Install openssh.
– Assign hostname(say openstack). Assign domain name(say demo.com)
– Assign static ip(say
– Give gateway to access internet (say I assume here you have a wired connection to the internet. Insead if you have a wireless connection you can let it connect to internet using dhcp.
You can do the above steps once the Ubuntu installation is finished as well like below

– Create user account(say nova)

$ sudo /usr/sbin/adduser nova

– Install openssh

$ sudo apt-get install openssh-server(to ssh into instance)

– Assign static ip by editing /etc/network/interfaces file

auto eth0
iface eth0 inet static
auto eth1
iface eth1 inet manual
up ifconfig eth1 up


$ sudo ifup eth0
$ sudo ifup eth1

– Assign hostname and domain name by putting an entry in /etc/hosts file as in    openstack.demo.com    openstack


Log in using nova account you created.
$ sudo apt-get update


Checkout the Installation Scripts
$ sudo apt-get -y install git
$ git clone https://github.com/damitha23/openstack.git
$ cd openstack
$ unzip OpenStackInstaller.zip

Note that content of OpenStackInstaller folder has scripts I took from https://github.com/uksysadmin/OpenStackInstaller.git maintained by Kevin Jackson <kevin@linuxservices.co.uk> https://twitter.com/#!/itarchitectkevirc.freenode.org: uksysadmin

Step4: Installing Openstack

$ cd /home/nova/OpenStackInstaller

Modify oscontrollerinstall.sh as per your requirements and execute. It will take couple of minutes to install Openstack.
Also modify the OSinstall.sh to add following configuration that would go into nova.conf

--rpc_response_timeout=<new timeout in seconds>

Give a sufficient response timeout to avoid timeout errors.
Example oscontrollerinstall.sh

./OSinstall.sh -T all -C openstack.demo.com -F -f -s 126 -P eth0
 -p eth1 -t demo -v kvm

Important: The virtualization type here I used is kvm.
Note that I use -T all options since I install in this server both controller and a compute node.
With -C parameter we give the hostname of the node. You should have an entry in the /etc/hosts file for this as following.    openstack.demo.com    openstack

If your node ip regulary change it is good idea to have following kind of entry in /etc/rc.local file so that it will automatically add that entry when node bootup

ip=`/sbin/ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'`
echo $ip openstack.demo.com openstack >> /etc/hosts

Note that here ip is taken from eth0 interface. You may need adjustments.

With -F parameter we give the floating ip range for the project.
With -f parameter we give the fixed ip range for the project.
With -s parameter we give number of nodes in the private network.

I use eth1 as private interface. eth0 as public interface. For the public ips(floating ips) we should give an valid range from the network where the host machine took IP. So a valid floating ip subnet would be You can calculate such an range from the subnet calculator in link [1] or [2]
A valid fixed ip subnet would be Note that if the floating ip’s are exhausted, then there will be errors and instance would not be created. To avoid this situation, make sure that you allocate as many as floating ips, at least, as the fixed ips. Now you can access Openstack UI from http://openstack.demo.com using


You may need to add an host entry in the node where your browser reside when giving the above url as in    openstack.demo.com    openstack

Now you can manage your Openstack environment from the UI interface.

If one of your interface is a virtual interface(This could be the case when are installing on a laptop) your install command could be like following

./OSinstall.sh -T all -C openstack.demo.com -F -f -s 126 -P eth0
 -p eth0:0 -t demo -v kvm

Make sure eth0:0 is defined as following

auto eth0:0
iface eth0:0 inet manual

And make sure it is up by using
$ ifup eth0:0

Step5: Upload an Image

From this step on you can execute the commands as normal user. I upload an ubuntu image to glance. For kvm virtual machine download a base ubuntu image precise-server-cloudimg-amd64-disk1.img from http://cloud-images.ubuntu.com/precise/current/ and create a folder called /home/nova/upload folder and copy the image into it.

Modify /home/nova/OpenStackInstaller/uploadimage.sh and execute to upload the image.

An example uploadimage.sh would be

./imageupload.sh -a admin -p openstack -t demo -C openstack.demo.com -x amd64 -y ubuntu -w 12.04 
 -z /root/upload/precise-server-cloudimg-amd64-disk1.img -n cloudimg-ubuntu-12.04

Here openstack.demo.com is the hostname of the openstack controller.

$ cd OpenStackInstaller
$ source ./demorc
$ nova image-list

command to see whether your newly uploaded image appear in the image list.

Step6: Testing the Controller

$ cd OpenStackInstaller
$ source ./demorc

Now add a keypair. It is highly recommended that you use your own keypair when creating
instances. For example suppose you create an instance as normal user, using a keypair owned by root user. You may succeed in creating your instance. But you will get permission denied exception when trying to ssh to that instance.

$ nova keypair-add wso2 > wso2.pem

Set permission for the private key

$ chmod 0600 wso2.pem

You can see the created key listed

$ nova keypair-list

Allow needed ports for the default security group.

$ nova secgroup-add-rule default icmp -1 -1
$ nova secgroup-add-rule default tcp 22 22
$ nova secgroup-add-rule default tcp 80 80
$ nova secgroup-add-rule default tcp 443 443
$ nova secgroup-add-rule default tcp 3306 3306
$ nova secgroup-add-rule default tcp 8080 8080

Now list the images and select an image id to create an instance from it

$ nova image-list
$ nova boot –key_name=nova-key –flavor=1 –image=<image id> <instance name>

Instead of the above command you can use the following command if you need to pass some user data into the instance you want to create.

$ nova boot –key_name=nova-key –flavor=1 –image=<image id> –user_data=/root/client/payload.zip <instance name>

Now see whether your instance is up and running. Look for the running instances ip.

$ nova list
$ ssh -i wso2.pem ubuntu@ipaddress

If you can access the virtual machine instance then you have successfully created a controller with a compute node in it. Log into the nova mysql database running in the controller machine and observe that there is a compute node entry in the compute_nodes table.

$ mysql -uroot -popenstack

Note that mysql password is defined in the OpenStackInstaller/OSinstall.sh file.

mysql>use nova
mysql>select id, created_at from compute_nodes;

Your should see one compute node entry in the table. Now from your Openstack node you can start playing with creating/deleting your new instances. You can monitor the /var/log/nova/nova-compute.log to see the status of creating the nodes. You can create more and more instances and verify that in both compute nodes until you see a short, undescriptive message that basically say your quota has exceeded.

Some useful settings in the Openstack environment

In the following sections, some useful settings on Openstack Nova environment is explained.

Adding a new VM resource type

You can add new resource types by

$ nova-manage flavor create –name=m1.wso2 –memory=128 –cpu=1 –root_gb=2 –ephemeral_gb=0 –flavor=6 –swap=0 –rxtx_factor=1

User data injection

From openstack nova essex that ship with Ubuntu 12.04 LTS the instances created from cloud images are ready to get information such as user-data, public ip, keys etc from the metadata service. User data data can be passed to the instance at startup like

$ nova boot –key_name=nova-key –flavor=1 –image=<image id> –user_data=/root/client/payload.zip <instance name>

At instance startup, nova copy the zip file into the instance as /var/lib/cloud/instance/user-data.txt.

Accessing Metadata information from within instances

We can get the public ip from the metadata server

$ wget

Now public-iv4 file contain the public ip

Adding floating ip to instances

We can add floating ip’s to the instances automatically when spawned or later. For automatically assiginint ip when instance spawn, add the following line into /etc/nova.conf and restart nova services


To add a floating ip first allocate one using the following command

$ nova floating-ip-create
$ nova add-floating-ip <instance id> <floating ip>
$ nova remove-floating-ip <instance id> <floating ip>

$ nova floating-ip-delete <floating ip>

To list the floating ips

$ nova floating-ip-list

Monitoring Openstack

To see how much memory an lxc container is using

$ cat /sys/fs/cgroup/memory/libvirt/lxc/instance-0000002d/memory.stat

and look at rss entries
$ cat /sys/fs/cgroup/memory/libvirt/lxc/instance-0000002d/memory.usage_in_bytes

In /sys/fs/cgroup/memory/libvirt/lxc/instance-0000002d/ folder you can see several other memory related files

Some of the other folders that contain files regarding resources are



Cannot ping to the instance created

Make sure you have enabled icmp using the nova command-line tool:

$ nova secgroup-add-rule default icmp -1 -1 -s

Cannot ssh to the instance

Make sure you have enabled tcp port

Using the nova command-line tool:

$ nova secgroup-add-rule default tcp 22 22 -s

If you still cannot ping or SSH your instances after issuing the nova secgroup-add-rule commands, look at the number of dnsmasq processes that are running. If you have a running instance, check to see that TWO dnsmasq processes are running. If not, perform the following

as root:

$ sudo killall dnsmasq
$ sudo service nova-network restart

When installing nova essex into a new box dpkg error occur and then mysql configuration take a long time and fail

This happen when you forget to do an apt-get update before starting to install nova essex. This could not be corrected until doing a fresh installation again.

Your applications deployed in instances cannot be accessed

Make sure you have enabled your application port.

Using the nova command-line tool:

$ nova secgroup-add-rule default tcp 8080 8080 -s

Note that you need to replace 8080 above with the port your application is running.

Cannot shutdown the instance

Sometimes even after terminate command is executed on an instance it is not terminated but go to shutoff state. At such moments try restarting nova services.

Error returned when creating the very first instance

Make sure that you public and private interfaces are up

Eg: sudo ifconfig eth1 up

Timeout: Timeout while waiting on RPC response

Sometimes when creating instances you get the response timeout error. The default request
timeout for nova is 60seconds. To increase this add following entry to /etc/nova.conf and restart nova services

--rpc_response_timeout=<new timeout in seconds>

Successfully added compute node but cannot create instances in that node

When instances are created in that node the instance state is in ERROR. In the compute node log we have

libvirtError: Unable to read from monitor: Connection reset by peer

To avoid this make sure that you have commented out the following three entries in the compute nodes /etc/nova.conf


If not comment them out and restart nova services in the compute node.

Instances are not created

– Check whether both interfaces of the controller is up and all compute node interfaces are up. If not make them up and then restart nova services.

Disaster Recovery

Nova instances can be rebooted using

$ nova reboot <instance id>

I notices that when node is restarted while some vm’s are running I could not ping to those vm’s when node restarted. Then rebooting the vm as above solved it. But now I could ping to the instance but connection is refused when ssh to it. Then I cd to OpenStackInstaller and executed

$ sudo restartservices.sh

You may need to run this command twice if you see any warning/error first time. Then that problem is solved too.


[1] http://www.subnet-calculator.com/subnet.php?net_class=C


Read Full Post »