Feeds:
Posts
Comments

Posts Tagged ‘docker’

As promised in my previous blog post  How to setup Openstack Havana with Docker driver  here I would like to share some of my experience working with the Havana/Docker setup. I’ll basically explain how to run secure access containers in Openstack/Docker and accessing the user-data passed to the containers overcoming the technical difficulties in the versions used in our setup.

We will create a  Ubuntu image in docker local repository using a Dockerfile and then transfer that image into the glance repository. The image we create fix the following issues that we find in the selected version of Docker.

– does not allow to pass user data at container start up.

– User cannot  pass a public key at instance boot up and access using it.

– User cannot change /etc/hosts file

So we will fix these issues which are critical when using Openstack/Docker as IaaS for Stratos. We will create a 64 bit Ubuntu image fixing above issues, which can be used as a base image for  creating cartridge images for Stratos.

You can download all the scripts and other stuff used in this blog from[2]. Download Dockerfile, metadata_svc_bugfix.sh, file_edit_patch.sh, run_scripts.sh and ubuntu64-docker-ssh.tar.gz from [2].

Let’s start with the snapshot of our set up saved earlier. Create a virtualbox VM with this snapshot.

Then you need to rejoin the Openstack session using
cd devstack
. openrc
rejoin-stack.sh

Or instead of running rejoin-stack.sh you can run stack.sh. But in this case you will lose your previous data including images stored in glance repository and previously run instances.

Now open another terminal to the virtual machine.

Upload the 64bit Ubuntu image you downloaded above, into the docker repository. We will use this image as the base image of the images we create in the Docker repository.

docker import - ubuntu64base < ./ubuntu64-docker-ssh.tar.gz

create a new folder and name it say stratosbase
cd stratosbase

Create the file below and name it as Dockerfile


# stratosbase
# VERSION 0.0.1
FROM ubuntu64base
MAINTAINER Damitha Kumarage "damitha23@gmail.com"
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update


RUN apt-get install -y openssh-server
RUN echo 'root:g' |chpasswd


RUN apt-get install -q -y zip
RUN apt-get install -q -y unzip
RUN apt-get install -q -y curl


ADD metadata_svc_bugfix.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/metadata_svc_bugfix.sh
ADD file_edit_patch.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/file_edit_patch.sh
ADD run_scripts.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/run_scripts.sh
ENV LD_LIBRARY_PATH /root/lib
EXPOSE 22
ENTRYPOINT /usr/local/bin/run_scripts.sh | /usr/sbin/sshd -D

What this Dockerfile do is self descriptive. Note that I run the sshd daemon as an ENTRYPOINT instead of CMD. Reason is Docker driver will override “/usr/sbin/sshd -D” with “sh” if I use CMD and consequently sshd daemon will not be run. We have also set ssh password for root as ‘g’ .

There is a problem in docker containers as of the current versions where it does not allow to download user data. The reason is described in the Openstack bug report[1]. We will fix this problem by using a patch script called  metadata_svc_bugfix.sh. In this patch we also retrieve ssh public key of the user passed to the instance when booting up. There is limitation in Docker containers where it does not allow to edit /etc/hosts file. We will circumvent this issue by adding another patch file called file_edit_patch.sh.

We introduce another script called run_script.sh which will be executed at startup of the docker container and this script just contain execution codes for the above two patch scripts.

Following are the scripts mentioned.

metadata_svc_bugfix.sh

#!/bin/bash
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
while [ "$NOVA_NIC" == "" ] ; do
echo "Find nova NIC..."
sleep 1
NOVA_NIC=$(ip a | grep pvnet | head -n 1 | cut -d: -f2)
done
echo $NOVA_NIC
echo "Device $NOVA_NIC found. Wait until ready."
sleep 3
# Setup a network route to insure we use the nova network.
#
echo "[INFO] Create default route for $NOVA_NIC. Gateway 10.11.12.1"
ip r r default via 10.11.12.1 dev $NOVA_NIC
# Shutdown eth0 since icps will fetch enabled enterface for streaming.
ip l set down dev eth0


sleep 5
#Get public keys from meta-data server
if [ ! -d /root/.ssh ]; then
mkdir -p /root/.ssh
chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
ATTEMPTS=30
FAILED=0
if [ ! -f /root/.ssh/authorized_keys ]; then
wget http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key -O /tmp/metadata-key -o /var/log/metadata_svc_bugfix.log
if [ $? -eq 0 ]; then
cat /tmp/metadata-key >> /root/.ssh/authorized_keys
chmod 0600 /root/.ssh/authorized_keys
#restorecon /root/.ssh/authorized_keys
rm -f /tmp/metadata-key
echo "Successfully retrieved public key from instance metadata" >> /var/log/metadata_svc_bugfix.log
fi
fi

And
file_edit_patch.sh

#!/bin/bash
mkdir p - /root/lib
cp -f /lib/x86_64-linux-gnu/libnss_files.so.2 /root/lib
perl -pi -e 's:/etc/hosts:/tmp/hosts:g' /root/lib/libnss_files.so.2
perl -pi -e 's:/etc/resolv.conf:/tmp/resolv.conf:g' /root/lib/libnss_files.so.2
cp -f /etc/hosts /tmp/hosts
cp -f /etc/resolv.conf /tmp/resolv.conf

 

Finally
run_scripts.sh
#!/bin/bash
/usr/local/bin/metadata_svc_bugfix.sh
/usr/local/bin/file_edit_patch.sh

Copy the above three scripts(metadata_svc_bugfix.sh, file_edit_patch.sh, run_scripts.sh) into stratosbase folder. Now create the image in Docker local repository
docker build -t stratosbase .
Note the dot at the end of the command. Note that we tag the image as stratosbase. Now to see the image created in local Docker repo execute
docker images
You will see an image named stratosbase is created there.
Now you will tag this image and push it to the glance repository.
docker tag stratosbase 192.168.57.30:5042/stratosbase
docker push 192.168.57.30:5042/stratosbase
where 192.168.57.30 is the ip of your Virtualbox VM. Your image is exported to the glance repository in Docker format. In fact you can push this image to any Docker repository you choose to and it is a good idea that Apache Stratos community keep a public Docker repository where they can share cartridge images. Then any one interested in the shared cartridges can pull it from public repository and use it with Stratos.

Now to see the image in Glance repository.
glance image-list
Now nova compute can spawn Docker containers from this image.
Log into to Horizon UI and create an instance using this image.
Note: you will log into Horizon web UI using the admin or demo user. The password for it is set in devstack/localrc file we created earlier in my previous blog.
Make sure that using Horizon Access & Security under Project tab you add rules for tcp port 22 and icmp for the security group from which you create containers(by default this is default group)

Now you should be able to access the spawned container using
ssh root@private_ip_of_container

or if you passwd your public key when creating your instance
ssh -i root@<private_ip_of_container>
Now when creating the container try passing some user data script using the Post Creation tab in the launch screen.
In the Customization Script box type

X1=1, X2=2

Now when the container is spawned you should be able to log in and retrieve the passed data by
wget http://169.254.169.254/2009-04-04/user-data

And retrieve the public key buy

wget http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key

My aim of building this Openstack/Docker set up is to use this as a testing and developer environment for Apache Stratos PaaS Cloud environment. My next blog post Apache Stratos on Openstack/Docker-Part One will deal with how to set up Apache Stratos on the same Virtualbox VM we set up Openstack/Docker. We will use the stratosbase image we created as base for creating Stratos cartridge images in Openstack/Docker IaaS environment.

[1] https://bugs.launchpad.net/nova/+bug/1259267
[2]https://www.dropbox.com/sh/dmmey60kvdihc31/3sbBkX7ns3

Read Full Post »

I will share my experience on the subject in detail. The guide will definitely work if you follow this with the versions of the software specified.

Software: 64bit server of Ubuntu 13.04.
Openstack Havana/stable branch
Docker version 0.7.6
Oracle virtualbox version 4.1.12_ubuntu

This worked for me with ubuntu running on virtualbox. The vertualbox version need not be exact. But if you need this definetely work follow the exact Ubuntu and Docker versions because I have not tested on other versions. But I believe these instructions will work with prior Ubuntu and Docker versions as well with slight changes.

I use the devstack setup to install Havana and Docker. It is difficult to maintain our own scripts with the fast growing development of Openstack with new technologies, hence the idea of following devstack scripts. My previous choice of lxc driver is changed to dokcer since I feel that lxc/libvirt driver soupport in Openstack commnunity is somewhat lagging and Docker community show promising growth. Besides Docker is based on lxc with better isolation and features. Most attracting idea of docker for me is the concept of portable containers.

In this setup all nova services run in a single virtual machine. This setup is mainly used to test my Apache Stratos PaaS environment where the Openstack/Docker is used as a IaaS layer.

Scripts which appear within this article can be downloaded from [1]
It is good habit to take virtualbox snapshots at every important step of the process. This way if something goes wrong, you can re-start from the previously saved state. I strongly recommend to follow the instructions exactly as indicated in the article. Once you achieved the article goal, you can do your own experiments beginning from various snapshots saved. Later you can delete snapshots to save disk space.

If you are too eager to get the setup running follow the Quick Steps below. Quick Steps guide will assume you are familiar with virtualbox environment and Openstack devstack setup. If you run into problems or need detailed steps I recommend you follow the whole blog entry as a tutorial. Also for quick steps download the scripts and other stuff from [1].

Quick Instructions

Download:

Download interfaces, hypervisor-docker, install_docker0.sh, install_docker1.sh, localrc and driver.py files from [1].

Setup Virtualbox:

Install Ubuntu 13.04 64 bit version in virtualbox with at least 40G dyanmically growing hard disk. Add hostonly interface eth1 with gateway 192.168.92.1. Add hostonly interface eth2 with gateway 192.168.57.1. Log in and create a user/password called wso2/g. Replace /etc/network/interfaces file with downloaded interface file. Reboot vm, open a terminal and ssh into instance

ssh wso2@192.168.57.30

sudo apt-get update

sudo apt-get install linux-image-3.8.0-26-generic

Reboot

Setup Docker:

sudo apt-get install git
git clone https://github.com/openstack-dev/devstack.git
cd devstack
git checkout stable/havana

Replace devstack/lib/nova_plugins/hypervisor-docker with the downloaded hypervisor-docker file.
Copy install_docker0.sh and install_docker1.sh into /devstack/tools/docker folder.
cd devstack
./tools/docker/install_docker0.sh
sudo usermod -a -G docker wso2
sudo chown wso2:docker /var/run/docker.sock
./tools/docker/install_docker1.sh
sudo service docker restart
cd files
curl -OR http://get.docker.io/images/openstack/docker-ut.tar.gz
docker import - docker-busybox < ./docker-ut.tar.gz
If permission denied error occur execute the following command again.
sudo chown wso2:docker /var/run/docker.sock
url -OR http://get.docker.io/images/openstack/docker-registry.tar.gz
ocker import - docker-registry < ./docker-registry.tar.gz
Set ipv4 forwarding in /etc/sysctl.conf
net.ipv4.ip_forward = 1
sudo apt-get install lxc wget bsdtar curl
sudo apt-get install linux-image-extra-3.8.0-26-generic
sudo modprobe aufs

Add following three lines to /etc/rc.local
chown wso2:docker /var/run/docker.sock
modprobe aufs
sudo killall dnsmasq

Setup Openstack:

Copy localrc file to devstack folder
cd devstack
./stack.sh
After stack.sh finished successfully execute docker images to see docker registry is still there. If there are no images do following again
cd devstack/files
docker import - docker-registry < ./docker-registry.tar.gz
Replace /opt/stack/nova/nova/virt/docker/driver.py with downlaoded driver.py
Reboot vm.
cd devstack
./stack.sh
Test:
Log into Horizon, add icmp and ssh rules to security group and create an instance of busybox image.

Detailed  Instructions

First install Ubuntu 13.04 in a Virtualbox VM. Add a host-only network adaptor to it. In the ipv4 Address field put 192.168.92.1 and ipv4 Network Mask field put 255.255.255.0 . Add another host-only network adaptor. In the ipv4 Address field put 192.168.57.1 and ipv4 Network Mask field put 255.255.255.0 . Make sure to give at least 40G dynamically growing hard disk. Now boot up the VM and follow the steps below. Connect using the terminal ui provided by virtualbox and create a user/password called wso2/g.

Change /etc/network/interfaces as following

auto eth0
iface eth0 inet dhcp


auto eth1
iface eth1 inet static
address 192.168.92.30
network 192.168.92.0
netmask 255.255.255.0
broadcast 192.168.92.255


auto eth2
iface eth2 inet manual
up ifconfig eth2 192.168.57.30 up

Now reboot and you can connect to the VM from a terminal using username wso2 and password g.

ssh wso2@192.168.57.30
Now from within this terminal exeute

sudo apt-get update

Now in order for Openstack/Docker to work correctly we need a linux kernel upgrade for ubuntu

sudo apt-get install linux-image-3.8.0-26-generic

Now restart the VM node.

sudo apt-get install git

git clone https://github.com/openstack-dev/devstack.git

cd devstack

git checkout stable/havana

Now we need to apply the following patch for devstack scripts

Apply patch

The first one is in file “devstack/tools/docker/install_docker.sh”, line 41:
install_package --force-yes lxc-docker=${DOCKER_PACKAGE_VERSION} socat
should be:
install_package --force-yes lxc-docker-${DOCKER_PACKAGE_VERSION} socat

The second one is in file “devstack/lib/nova_plugins/hypervisor-docker”, line 75:
if ! is_package_installed lxc-docker; then
should be:
if ! is_package_installed lxc-docker-${DOCKER_PACKAGE_VERSION}; then

Also add the following line in devstack/lib/nova_plugins/hypervisor-docker under the entry called # Defaults

DOCKER_PACKAGE_VERSION=0.7.6

Now we are supposed to execute ./tools/docker/install_docker.sh. But don’t do it. In my case I got permission error for /var/run/docker.sock and a curl download fail for docker registry image when executed it. So I solved those two problems by following steps.

Break the installer script into two called install_docker0.sh and install_docker1.sh.

My install_docker0.sh file can be downloaded from [1]

My install_docke1.sh can be downloaded from[1]

Now run the first script
./tools/docker/install_docker0.sh

Then add wso2 user to docker group. Here username wso2 is the name you have given for your ubuntu account user.
sudo usermod -a -G docker wso2
Then change permission of the /var/run/docker.sock

sudo chown wso2:docker /var/run/docker.sock
Important:Each time you restart the virbualbox VM make sure that above permission for /var/run/docker.sock set correctly. If it is changed execute the above command and change the permission before doing anything.

Now run the second script
./tools/docker/install_docker1.sh
and
sudo service docker restart

cd files
curl -OR http://get.docker.io/images/openstack/docker-ut.tar.gz
docker import - docker-busybox < ./docker-ut.tar.gz
If permission denied error occur execute the following command again.
sudo chown wso2:docker /var/run/docker.sock
Now
curl -OR http://get.docker.io/images/openstack/docker-registry.tar.gz (Take about 20 mins in 120k per second connection)
If file transfer failed continue with
curl -C - -o docker-registry.tar.gz 'http://get.docker.io/images/openstack/docker-registry.tar.gz'
Now import
docker import - docker-registry < ./docker-registry.tar.gz
 

So by now your Docker installation should be a success. Now we need to run stack.sh script to setup Openstack. But before that let’s do the following.
Set ipv4 forwarding in /etc/sysctl.conf
net.ipv4.ip_forward = 1
To setup aufs file system which is necessary for docker driver
sudo apt-get install lxc wget bsdtar curl
sudo apt-get install linux-image-extra-3.8.0-26-generic

sudo modprobe aufs

Add following three lines to /etc/rc.local
chown wso2:docker /var/run/docker.sock
modprobe aufs
sudo killall dnsmasq

Now create a file called localrc in devstack folder and add the following content

FLOATING_RANGE=192.168.92.0/27
FIXED_RANGE=10.11.12.0/24
FIXED_NETWORK_SIZE=256
FLAT_INTERFACE=eth1
ADMIN_PASSWORD=g
MYSQL_PASSWORD=g
RABBIT_PASSWORD=g
SERVICE_PASSWORD=g
SERVICE_TOKEN=g
SCHEDULER=nova.scheduler.filter_scheduler.FilterScheduler
VIRT_DRIVER=docker
SCREEN_LOGDIR=$DEST/logs/screen

Now execute stack.sh

./stack.sh
(Take about 1.5hours in 120k per secon connection)

After stack.sh finished successfully execute docker images to see our docker registry is still there(I remember I once lost it by this time). If there are no images
cd devstack/files
docker import - docker-registry < ./docker-registry.tar.gz

Now you need to patch /opt/stack/nova/nova/virt/docker/driver.py line 317
replace
destroy_disks=True):
with
destroy_disks=True, context=None):
Now restart node.

Again
cd devstack
stack.sh

If you have followed the steps correctly you should have a working state of Openstack installation with Docker driver. Log into horizon UI(http://192.168.57.30) using admin or demo user. Paasword for those users is ‘g‘(as we set in our devstack/localrc file above) and create instances from docker-busybox image that is uploaded in the default installation.
Don’t forget to add icmp and ssh rules for the security group you use(by default this is default group). Take a snapshot of this working state before you do any further playing with your setup.

If you restart the node
cd devstack
rejoin-stack.sh
To run the nova services. Or if you need a clean Openstack environment after restarting the node instead of running rejoin-stack.sh, run stack.sh. This time it won’t take long time as the first time, only few seconds.

Nova service logs are in /opt/stack/logs/screen folder.

If you run rejoin-stack.sh you can see each nova service log in the rejoin screen. To see each service log ctrl+A and press " then select the service log you need by moving up|down arrows and then click. You can scroll up and down the rejoin screen by ctrl+A and press Esc and then use up|down or pgup|pgdown keys to scroll.

Note: for some reason eth1(flat interface) does not show the ip when rejoin-stack.sh is run. But that does not prevent connecting to the virtualbox vm. But sometimes  problem occur and thats why you add second interface eth2.

My next blog Docker Driver for Openstack Havana will be on playing around this setup like creating new customized images and secure access(ssh) the containers. I’ll also deal with a bug fix on accessing metadata serivces.

[1]https://www.dropbox.com/sh/dmmey60kvdihc31/3sbBkX7ns3

Read Full Post »