Archive for February, 2014

As promised in my previous blog post  How to setup Openstack Havana with Docker driver  here I would like to share some of my experience working with the Havana/Docker setup. I’ll basically explain how to run secure access containers in Openstack/Docker and accessing the user-data passed to the containers overcoming the technical difficulties in the versions used in our setup.

We will create a  Ubuntu image in docker local repository using a Dockerfile and then transfer that image into the glance repository. The image we create fix the following issues that we find in the selected version of Docker.

– does not allow to pass user data at container start up.

– User cannot  pass a public key at instance boot up and access using it.

– User cannot change /etc/hosts file

So we will fix these issues which are critical when using Openstack/Docker as IaaS for Stratos. We will create a 64 bit Ubuntu image fixing above issues, which can be used as a base image for  creating cartridge images for Stratos.

You can download all the scripts and other stuff used in this blog from[2]. Download Dockerfile, metadata_svc_bugfix.sh, file_edit_patch.sh, run_scripts.sh and ubuntu64-docker-ssh.tar.gz from [2].

Let’s start with the snapshot of our set up saved earlier. Create a virtualbox VM with this snapshot.

Then you need to rejoin the Openstack session using
cd devstack
. openrc

Or instead of running rejoin-stack.sh you can run stack.sh. But in this case you will lose your previous data including images stored in glance repository and previously run instances.

Now open another terminal to the virtual machine.

Upload the 64bit Ubuntu image you downloaded above, into the docker repository. We will use this image as the base image of the images we create in the Docker repository.

docker import - ubuntu64base < ./ubuntu64-docker-ssh.tar.gz

create a new folder and name it say stratosbase
cd stratosbase

Create the file below and name it as Dockerfile

# stratosbase
# VERSION 0.0.1
FROM ubuntu64base
MAINTAINER Damitha Kumarage "damitha23@gmail.com"
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update

RUN apt-get install -y openssh-server
RUN echo 'root:g' |chpasswd

RUN apt-get install -q -y zip
RUN apt-get install -q -y unzip
RUN apt-get install -q -y curl

ADD metadata_svc_bugfix.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/metadata_svc_bugfix.sh
ADD file_edit_patch.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/file_edit_patch.sh
ADD run_scripts.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/run_scripts.sh
ENTRYPOINT /usr/local/bin/run_scripts.sh | /usr/sbin/sshd -D

What this Dockerfile do is self descriptive. Note that I run the sshd daemon as an ENTRYPOINT instead of CMD. Reason is Docker driver will override “/usr/sbin/sshd -D” with “sh” if I use CMD and consequently sshd daemon will not be run. We have also set ssh password for root as ‘g’ .

There is a problem in docker containers as of the current versions where it does not allow to download user data. The reason is described in the Openstack bug report[1]. We will fix this problem by using a patch script called  metadata_svc_bugfix.sh. In this patch we also retrieve ssh public key of the user passed to the instance when booting up. There is limitation in Docker containers where it does not allow to edit /etc/hosts file. We will circumvent this issue by adding another patch file called file_edit_patch.sh.

We introduce another script called run_script.sh which will be executed at startup of the docker container and this script just contain execution codes for the above two patch scripts.

Following are the scripts mentioned.


NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
echo $NOVA_NIC
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
echo "[INFO] Create default route for $NOVA_NIC. Gateway"
ip r r default via dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0

sleep 5
#Get public keys from meta-data server
if [ ! -d /root/.ssh ]; then
mkdir -p /root/.ssh
chmod 700 /root/.ssh
# Fetch public key using HTTP
if [ ! -f /root/.ssh/authorized_keys ]; then
wget -O /tmp/metadata-key -o /var/log/metadata_svc_bugfix.log
if [ $? -eq 0 ]; then
cat /tmp/metadata-key >> /root/.ssh/authorized_keys
chmod 0600 /root/.ssh/authorized_keys
#restorecon /root/.ssh/authorized_keys
rm -f /tmp/metadata-key
echo "Successfully retrieved public key from instance metadata" >> /var/log/metadata_svc_bugfix.log


mkdir p - /root/lib
cp -f /lib/x86_64-linux-gnu/libnss_files.so.2 /root/lib
perl -pi -e 's:/etc/hosts:/tmp/hosts:g' /root/lib/libnss_files.so.2
perl -pi -e 's:/etc/resolv.conf:/tmp/resolv.conf:g' /root/lib/libnss_files.so.2
cp -f /etc/hosts /tmp/hosts
cp -f /etc/resolv.conf /tmp/resolv.conf



Copy the above three scripts(metadata_svc_bugfix.sh, file_edit_patch.sh, run_scripts.sh) into stratosbase folder. Now create the image in Docker local repository
docker build -t stratosbase .
Note the dot at the end of the command. Note that we tag the image as stratosbase. Now to see the image created in local Docker repo execute
docker images
You will see an image named stratosbase is created there.
Now you will tag this image and push it to the glance repository.
docker tag stratosbase
docker push
where is the ip of your Virtualbox VM. Your image is exported to the glance repository in Docker format. In fact you can push this image to any Docker repository you choose to and it is a good idea that Apache Stratos community keep a public Docker repository where they can share cartridge images. Then any one interested in the shared cartridges can pull it from public repository and use it with Stratos.

Now to see the image in Glance repository.
glance image-list
Now nova compute can spawn Docker containers from this image.
Log into to Horizon UI and create an instance using this image.
Note: you will log into Horizon web UI using the admin or demo user. The password for it is set in devstack/localrc file we created earlier in my previous blog.
Make sure that using Horizon Access & Security under Project tab you add rules for tcp port 22 and icmp for the security group from which you create containers(by default this is default group)

Now you should be able to access the spawned container using
ssh root@private_ip_of_container

or if you passwd your public key when creating your instance
ssh -i root@<private_ip_of_container>
Now when creating the container try passing some user data script using the Post Creation tab in the launch screen.
In the Customization Script box type

X1=1, X2=2

Now when the container is spawned you should be able to log in and retrieve the passed data by

And retrieve the public key buy


My aim of building this Openstack/Docker set up is to use this as a testing and developer environment for Apache Stratos PaaS Cloud environment. My next blog post Apache Stratos on Openstack/Docker-Part One will deal with how to set up Apache Stratos on the same Virtualbox VM we set up Openstack/Docker. We will use the stratosbase image we created as base for creating Stratos cartridge images in Openstack/Docker IaaS environment.

[1] https://bugs.launchpad.net/nova/+bug/1259267


Read Full Post »