Build an Application

on an OpenStack Cloud

Everett Toews @everett_toews
Dana Bauer @geography76

rack.to/app-on-os

Goal

To give you the foundation of building and deploying an application on OpenStack and to give you the confidence to use tools of a higher level of abstraction in the future.

Agenda

  1. The Cloud
  2. OpenStack
  3. Access
  4. Cloud Control
  5. Deployment
  6. Break

Agenda

  1. Application
  2. Architecture
  3. Database
  4. Object Storage
  5. Queues
  6. API

Agenda

  1. Workers
  2. App
  3. Run
  4. Deployment
  5. Next

Do Something

The Cloud

IaaS

Private

Public

Infrastructure

with an API

OpenStack

"To produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private clouds regardless of size, by being simple to implement and massively scalable."

Identity

(Keystone)

Compute

(Nova)

Databases

(Trove)

Networks

(Neutron)

Block Storage

(Cinder)

Queues

(Zaqar)

Images

(Glance)

Object Storage

(Swift)

Orchestration

(Heat)

Dashboard

(Horizon)

HTTP API

Rackspace

"To be recognized as one of the world's greatest service companies."

And more:

  • Managed Cloud
  • Load Balancers
  • Auto Scale
  • Monitoring
  • Backup
  • Metrics
  • Feeds
  • OnMetal (Ironic)
  • Cloud DNS (Designate)
  • Cloud CDN (Poppy)
  • Cloud Keep (Barbican)

Access

developer+
developer+ Advice

Temp Account

Cloud Control

HTTP API

SDK

CLI

OpenStack Resources for Application Development

Web UI

Rackspace Cloud Control Panel
  1. Servers > Cloud Servers
  2. Create Server
  1. Server Name: <my-username>-deploy-server
  2. Region: IAD
  3. Image: Linux > Ubuntu > 15.04
  4. Flavor: 1 GB General Purpose v1
  5. Create Server (copy the password)

Login, Update, and Utils


ssh root@my.deploy-server.ipv4.address

apt-get -y update; apt-get -y upgrade
apt-get -y install vim git jq
          

Utils

  • vim
  • git
  • jq

Setup Python


apt-get -y install python-dev python-pip libssl-dev libffi-dev
pip install virtualenv virtualenvwrapper

echo "export WORKON_HOME=$HOME/.virtualenvs" >> $HOME/.profile
echo "source /usr/local/bin/virtualenvwrapper.sh" >> $HOME/.profile

source $HOME/.profile
          

See: Virtual Environments

Install OpenStack Client


mkvirtualenv osc
pip install python-openstackclient==1.3.0

openstack --version
openstack complete > .osc.bash_completion

echo "source $HOME/.osc.bash_completion" >> $HOME/.profile
source $HOME/.profile
          

Configure Environment


cat <<EOF > $HOME/os_env.rc
export OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/
export OS_REGION_NAME=IAD
export OS_USERNAME=my-account-username
export OS_PROJECT_NAME=my-account-number
export OS_PASSWORD=my-account-password
EOF

echo "source $HOME/os_env.rc" >> $HOME/.profile
source $HOME/.profile
env | grep OS_
          

Note: Watch out for &s in the password.

Run OpenStack Client


workon osc

openstack image list
openstack --debug server list
openstack server list --format json
          

SSH Key for Deploys


openstack keypair create $OS_USERNAME-qcon > $HOME/.ssh/id_rsa.qcon_deploy

chmod 400 $HOME/.ssh/id_rsa.qcon_deploy
          

Deployment

Orchestration

(Heat)

Salt

Terraform

Ansible

Chef

Puppet

Docker

Kubernetes

CLI

and Ansible

Break

The Application

Watermark

Clone the Code


git clone https://github.com/everett-toews/app-on-openstack.git
          

The Architecture

Database

Note: Stateful

Cloud Databases

(Trove)

MySQL

Install Databases Client


workon osc

pip install python-troveclient==1.2.0
trove --version
          

Configure Environment


cat <<EOF >> $HOME/os_env.rc
export OS_TENANT_ID=$OS_PROJECT_NAME
export TROVE_SERVICE_TYPE=rax:database
EOF

source $HOME/.profile
more $HOME/os_env.rc
          

Run Databases Client


workon osc
trove flavor-list

trove create \
  --size 2 \
  --databases watermark \
  --users wmadmin:password \
  --datastore MySQL \
  $OS_USERNAME-watermark 2

trove list 2>/dev/null | grep $OS_USERNAME
trove --json list 2>/dev/null | jq .

export WM_DB_HOSTNAME=$(trove --json list 2>/dev/null | \
  jq -r '.[] | select(.name == "'"$OS_USERNAME-watermark"'").hostname')
echo $WM_DB_HOSTNAME
          

Configure DB Server Environment


cat <<EOF > $HOME/db_env.rc
export WM_DB_ENGINE_DRIVER=mysql+pymysql
export WM_DB_USERNAME=wmadmin
export WM_DB_PASSWORD=password
export WM_DB_HOSTNAME=$WM_DB_HOSTNAME
export WM_DB_DATABASE=watermark
EOF

echo "source $HOME/db_env.rc" >> $HOME/.profile
source $HOME/.profile
env | grep WM_
          

Initialize Database


mkvirtualenv api
pip install -r $HOME/app-on-openstack/code/api/requirements.txt
python $HOME/app-on-openstack/code/api/manage.py create_all
          

Object Storage

Cloud Files

(Swift)

Note: Stateful

Create a Container


workon osc

openstack container create $OS_USERNAME-watermark
          

Make a Container Public


# Scroll to the bottom!

TOKEN=`curl -s -X POST https://identity.api.rackspacecloud.com/v2.0/tokens \
  -d '{"auth": {"passwordCredentials": {"username":"'"$OS_USERNAME"'", "password":"'"$OS_PASSWORD"'"}}}' \
  -H "Content-type: application/json" | \
  jq -r .access.token.id`

CF_CDN_ENDPOINT=`curl -s -X POST https://identity.api.rackspacecloud.com/v2.0/tokens \
  -d '{"auth": {"passwordCredentials": {"username":"'"$OS_USERNAME"'", \
                                        "password":"'"$OS_PASSWORD"'"}}}' \
  -H "Content-type: application/json" | \
  jq -r '.access.serviceCatalog[] | select(.name == "cloudFilesCDN").endpoints[] | select(.region == "'"$OS_REGION_NAME"'").publicURL'`

WM_CONTAINER_CDN_URL=`curl -s -i -X PUT $CF_CDN_ENDPOINT/$OS_USERNAME-watermark \
  -H "X-Auth-Token: $TOKEN" \
  -H "X-Cdn-Enabled: True" |\
  grep X-Cdn-Uri | \
  awk '{print $2}'`

echo "export WM_CONTAINER_CDN_URL=$WM_CONTAINER_CDN_URL" >> $HOME/api_env.rc
more $HOME/api_env.rc
          

Note: Here you would use Swift's Container Read instead

Queues

Cloud Queues

(Zaqar)

Note: Stateful ("temporarily")

Create Queue


mkvirtualenv worker

pip install -r $HOME/app-on-openstack/code/worker/requirements.txt
$HOME/app-on-openstack/code/worker/deploy.py
          

deploy.py

API

Cloud Servers

(Nova)

Note: Stateless

Pick an Image


workon osc
openstack image list

export IMAGE_NAME="Ubuntu 15.04 (Vivid Vervet) (PVHVM)"
echo 'export IMAGE_NAME="'$IMAGE_NAME'"' >> $HOME/.profile
          

Pick an Flavor


openstack flavor list

export FLAVOR_ID=performance1-1
echo "export FLAVOR_ID=$FLAVOR_ID" >> $HOME/.profile
          

Configure API Server Environment


cat <<EOF >> $HOME/api_env.rc
export WM_CONFIG_ENV=testing
EOF

more $HOME/api_env.rc
          

Create API Servers


openstack server create \
  --image "$IMAGE_NAME" \
  --flavor $FLAVOR_ID \
  --key-name $OS_USERNAME-qcon \
  --user-data $HOME/app-on-openstack/code/api/deploy.sh \
  --config-drive true \
  --file /root/db_env.rc=$HOME/db_env.rc \
  --file /root/api_env.rc=$HOME/api_env.rc \
  --file /root/os_env.rc=$HOME/os_env.rc \
  $OS_USERNAME-api

!!
          

Note: Hostname "api"

requirements.txt

deploy.sh

Capture API Server IPs


openstack server list --format json | jq .

openstack server list --format json | \
  jq -r '.[] | select(.Name == "'"$OS_USERNAME-api"'").Networks'

openstack server list --format json | \
  jq -r '.[] | select(.Name == "'"$OS_USERNAME-api"'").Networks' | \
  sed "s/.*private=\(.*\)/\1/"

cat <<EOF > $HOME/api_ips.rc
export WM_IPS=(
$(openstack server list --format json | \
  jq -r '.[] | select(.Name == "'"$OS_USERNAME-api"'").Networks' | \
  sed "s/.*private=\(.*\)/\1/")
)
EOF

more $HOME/api_ips.rc
          

images.py

Check any API Server


ssh -i $HOME/.ssh/id_rsa.qcon_deploy root@any.api.server.ip

tail -f /var/log/cloud-init-output.log
tail -f /var/log/syslog
more $HOME/app-on-openstack/code/api/wm_api.log

exit

curl any.api.server.ip
          

Create an API Load Balancer Server


openstack server create \
  --image "$IMAGE_NAME" \
  --flavor $FLAVOR_ID \
  --key-name $OS_USERNAME-qcon \
  --user-data $HOME/app-on-openstack/code/lb/deploy.sh \
  --config-drive true \
  --file /root/ips.rc=$HOME/api_ips.rc \
  $OS_USERNAME-api-lb
          

Note: Hostname "api-lb"

Note: Stateless

deploy.sh

Capture API Load Balancer IP


cat <<EOF > $HOME/api_lb_ip.rc
export WM_API_LB_IP=$(openstack server list --format json | \
  jq -r '.[] | select(.Name == "'"$OS_USERNAME-api-lb"'").Networks' | \
  sed "s/.*private=\(.*\)/\1/")
EOF

source $HOME/api_lb_ip.rc
more $HOME/api_lb_ip.rc
          

Test the API


curl $WM_API_LB_IP
          

Note: Try it in your browser (but be patient)

Workers

Cloud Servers

(Nova)

Note: Stateless

Configure Worker Server Environment


cat <<EOF > $HOME/worker_env.rc
export WM_API_SCHEME=http
export WM_API_ENDPOINT=$WM_API_LB_IP
export WM_CONFIG_ENV=testing
EOF

more $HOME/worker_env.rc
          

Create Worker Servers


openstack server create \
  --image "$IMAGE_NAME" \
  --flavor $FLAVOR_ID \
  --key-name $OS_USERNAME-qcon \
  --user-data $HOME/app-on-openstack/code/worker/deploy.sh \
  --config-drive true \
  --file /root/os_env.rc=$HOME/os_env.rc \
  --file /root/worker_env.rc=$HOME/worker_env.rc \
  $OS_USERNAME-worker

!!
          

Note: Hostname "worker"

requirements.txt

deploy.sh

worker.py

Check any Worker Server


ssh -i $HOME/.ssh/id_rsa.qcon_deploy root@any.worker.server.ip

tail -f /var/log/cloud-init-output.log
tail -f /var/log/syslog
more $HOME/app-on-openstack/code/worker/wm_worker.log

exit
          

App

Cloud Servers

(Nova)

Note: Stateless

Configure App Server Environment


source $HOME/api_lb_ip.rc

cat <<EOF > $HOME/app_env.rc
export WM_API_SCHEME=http
export WM_API_ENDPOINT=$WM_API_LB_IP
export WM_CONFIG_ENV=testing
EOF

more $HOME/app_env.rc
          

Create App Servers


openstack server create \
  --image "$IMAGE_NAME" \
  --flavor $FLAVOR_ID \
  --key-name $OS_USERNAME-qcon \
  --user-data $HOME/app-on-openstack/code/app/deploy.sh \
  --config-drive true \
  --file /root/app_env.rc=$HOME/app_env.rc \
  --file /root/os_env.rc=$HOME/os_env.rc \
  $OS_USERNAME-app

!!
          

Note: Hostname "app"

requirements.txt

deploy.sh

Check any App Server


ssh -i $HOME/.ssh/id_rsa.qcon_deploy root@any.app.server.ip

tail -f /var/log/cloud-init-output.log
tail -f /var/log/syslog
more $HOME/app-on-openstack/code/app/wm_app.log

exit

curl any.app.server.ip
          

Capture App Server IPs


cat <<EOF > $HOME/app_ips.rc
export WM_IPS=(
$(openstack server list --format json | \
  jq -r '.[] | select(.Name == "'"$OS_USERNAME-app"'").Networks' | \
  sed "s/.*private=\(.*\)/\1/")
)
EOF

more $HOME/app_ips.rc
          

views.py

Create an App Load Balancer Server


openstack server create \
  --image "$IMAGE_NAME" \
  --flavor $FLAVOR_ID \
  --key-name $OS_USERNAME-qcon \
  --user-data $HOME/app-on-openstack/code/lb/deploy.sh \
  --config-drive true \
  --file /root/ips.rc=$HOME/app_ips.rc \
  $OS_USERNAME-app-lb
          

Note: Hostname "app-lb"

Note: Stateless

deploy.sh

Test the App


openstack server list | grep $OS_USERNAME
curl my.app.lb.server
          

Note: Try it in your browser (but be patient)

Run

Note: Stateful

Upload

Deployment Revisited

Next

  • OpenStack NY Meetup
  • Next Next

    Next Next Next

    • Networks (Neutron)
    • SSL (Let's Encrypt)
    • Service discovery
    • DNS
    • Load testing
    • Scaling
    • CI/CD

    Thanks!

    Everett Toews @everett_toews
    Dana Bauer @geography76
    Security Advice

    developer+ Advice

    $50/month credit

    for 12 months

    Over $50/month

    you get charged

    Meter starts running

    on resource creation

    1 GB General Purpose

    Server for 1 month

    < $25

    rackspace.com/calculator

    (ignore Service Level)

    Delete unused

    resources!