Commit d9253bf8 authored by Thiago Paiva's avatar Thiago Paiva
Browse files

Setup of CI

parent 9b13956a
......@@ -57,7 +57,12 @@
echo "Triggered by: https://review.openstack.org/$ZUUL_CHANGE patchset $ZUUL_PATCHSET"
# TODO: Update this link to point to your log server
echo "Detailed logs: http://<your-log-server>/$LOG_PATH/"
#
#
echo "Detailed logs: http://10.4.4.193/$LOG_PATH/"
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#^^^^^ DNS FOR THE LOG SERVER HERE ONCE DONE ^^^^^
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- publisher:
name: console-log
......
Using diskimage-builder to build devstack-gate nodes
====================================================
In addition to being able to just download and consume images that are the
same as what run devstack-gate, it's easy to make your own for local dev or
testing - or just for fun.
Install diskimage-builder
-------------------------
Install the dependencies:
::
sudo apt-get install kpartx qemu-utils curl python-yaml
Install diskimage-builder:
::
sudo -H pip install diskimage-builder
Build an image
--------------
Building an image is simple, we have a script!
::
DISTRO="ubuntu" bash tools/build-image.sh
See the script for environment variables to set distribution, etc.
You should be left with a .qcow2 image file of your selected
distribution.
It is a good idea to set ``TMP_DIR`` to somewhere with plenty of space
to avoid the disappointment of a full-disk mid-way through the script
run.
While testing, consider exporting DIB_OFFLINE=true, to skip updating the cache.
Mounting the image
------------------
If you would like to examine the contents of the image, you can mount it on
a loopback device using qemu-nbd.
::
sudo apt-get install qemu-utils
sudo modprobe nbd max_part=16
sudo mkdir -p /tmp/newimage
sudo qemu-nbd -c /dev/nbd1 /path/to/devstack-gate-precise.qcow2
sudo mount /dev/nbd1p1 /tmp/newimage
or use the scripts
::
sudo apt-get install qemu-utils
sudo modprobe nbd max_part=16
sudo tools/mount-image.sh devstack-gate-precise.qcow2
sudo tools/umount-image.sh
Other things
------------
It's a qcow2 image, so you can do tons of things with it. You can upload it
to glance, you can boot it using kvm, and you can even copy it to a cloud
server, replace the contents of the server with it and kexec the new kernel.
Pre-cache all of the things bindep can identify in our repos
#!/bin/bash
# Copyright (C) 2015 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
#
# Install bindep into a virtualenv
# This is in /usr instead of /usr/local due to this bug on precise:
# https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/839588
# dib-lint: disable=setu setpipefail
if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
set -x
fi
set -e
sudo -H virtualenv /usr/bindep-env
sudo -H /usr/bindep-env/bin/pip install bindep
#!/bin/bash
# Copyright (C) 2011-2015 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
#
# See the License for the specific language governing permissions and
# limitations under the License.
# dib-lint: disable=setu setpipefail
if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
set -x
fi
set -e
if [[ ${YUM:-yum} == "dnf" ]]; then
echo "*** caching disabled for dnf"
echo "*** see https://etherpad.openstack.org/p/infra-f22"
exit 0
fi
# Build up a list of packages to install by concatenating bindep lists
FALLBACKREPO=/opt/git/openstack-infra/project-config
FALLBACKFILE=jenkins/data/bindep-fallback.txt
PACKAGES=`git --git-dir=$FALLBACKREPO/.git show master:$FALLBACKFILE`
REQSFILE=other-requirements.txt
for REPO in /opt/git/*/* ; do
BRANCHES=`git --git-dir=$REPO/.git branch -r | grep '^ origin/[^H]'`
for BRANCH in $BRANCHES ; do
REQS=`git --git-dir=$REPO/.git show $BRANCH:$REQSFILE 2>/dev/null \
|| true`
if [ -n "$REQS" ] ; then
PACKAGES=`echo -e "$PACKAGES\n$REQS" | sort -u`
fi
done
done
# TODO(fungi) once https://review.openstack.org/195201 appears in a bindep
# release, replace the next four lines with:
# PACKAGELIST=`echo "$PACKAGES" | /usr/bindep-env/bin/bindep -b -f - || true`
PACKAGESFILE=`mktemp`
echo "$PACKAGES" > $PACKAGESFILE
PACKAGELIST=`/usr/bindep-env/bin/bindep -b -f $PACKAGESFILE || true`
rm $PACKAGESFILE
if grep "^CentOS release 6" /etc/redhat-release ; then
# --downloadonly is provided by the yum-plugin-downloadonly package on
# CentOS 6.x
CENTOS6=1
yum install -y yum-plugin-downloadonly
else
CENTOS6=0
fi
if [ -f /usr/bin/apt-get ] ; then
# --ignore-missing conflicts with set -e, so force it to be ok
apt-get -y --ignore-missing -d install $PACKAGELIST || true
else
if [ $CENTOS6 -eq 1 ] ; then
# some packages may depend on python-setuptools, which is not
# installed and cannot be reinstalled on CentOS 6.x once yum
# has erased them, so use --skip-broken to avoid aborting; also
# on this platform --downloadonly causes yum to return nonzero
# even when it succeeds, so ignore its exit code
yum install -y --downloadonly --skip-broken $PACKAGELIST || true
else
yum install -y --downloadonly $PACKAGELIST
fi
fi
Pre-cache all of the things devstack might need
#!/bin/bash
# Copyright (c) 2012-2014 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
#
# See the License for the specific language governing permissions and
# limitations under the License.
# We need a copy of the devstack repo so that we can read it so that
# we can generate the list of repos that we need to get. We'd like to
# use the normal repo fetching and caching routines for that, so use
# a modified version.
# The bulk of this code is copied from
# elements/source-repositories/extra-data.d/98-source-repositories
# Most of it should be removed when we can source and call the
# functions directly
# dib-lint: disable=setpipefail
if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
set -x
fi
set -eu
GIT_BASE=${GIT_BASE:-git://git.openstack.org}
# If the old cache exists, move it to the new name
function make_new_cache(){
local OLD_CACHE_BASE=$1
local CACHE_BASE=$2
# If the old cache name exists, move it to the new cache name
if [ -e "$OLD_CACHE_BASE" ] ; then
if [ ! -e "$CACHE_BASE" ] ; then
mv -n $OLD_CACHE_BASE $CACHE_BASE
else
echo "Not replacing new cache location with old cache"
fi
fi
}
# Gets repositories or individual files listed in the a repository file
# and places them in the specified destination path.
# The format of the repository file is one or more lines matching
# <name> <type> <destination> <location> [<ref>]
function get_repos_for_element(){
local REPO_SOURCES=$1
local CACHE_URL=$TMP_HOOKS_PATH/bin/cache-url
local REGEX="^([^ ]+) (git|tar|file|package) ?(/[^ ]+)? ?([^ ]+)? ?([^ ]*)$"
while read line; do
# expand variables
line=$(eval echo $line)
# ignore blank lines and lines beginning in '#'
[[ "$line" == \#* ]] || [[ -z "$line" ]] && continue
if [[ "$line" =~ $REGEX ]] ; then
local REPONAME=${BASH_REMATCH[1]}
local REPOTYPE=${BASH_REMATCH[2]}
local REPOPATH=${BASH_REMATCH[3]}
local REPOLOCATION=${BASH_REMATCH[4]}
local REPO_ORIG_LOCATION=$REPOLOCATION
local REPOREF=${BASH_REMATCH[5]:-master}
local REPO_DEST=$TMP_MOUNT_PATH$REPOPATH
local REPO_SUB_DIRECTORY=$(dirname $REPO_DEST)
# REPOTYPE can be overridden with DIB_REPOTYPE_{name}
local REPOTYPE_OVERRIDE=DIB_REPOTYPE_${REPONAME//[^A-Za-z0-9]/_}
REPOTYPE=${!REPOTYPE_OVERRIDE:-$REPOTYPE}
# REPOLOCATION can be overridden with DIB_REPOLOCATION_{name}
local REPOLOCATION_OVERRIDE=DIB_REPOLOCATION_${REPONAME//[^A-Za-z0-9]/_}
REPOLOCATION=${!REPOLOCATION_OVERRIDE:-$REPOLOCATION}
# REPOREF can be overridden with DIB_REPOREF_{name}
local REPOREF_OVERRIDE=DIB_REPOREF_${REPONAME//[^A-Za-z0-9]/_}
REPOREF=${!REPOREF_OVERRIDE:-$REPOREF}
# Determine a unique cache path for this repo
CACHE_NAME=$(echo "${REPOTYPE}_${REPOLOCATION}" | sha1sum | awk '{ print $1 }' )
OLD_CACHE_PATH=${CACHE_BASE}/${CACHE_NAME}
# Add the repo name to the sha1sum for readability
CACHE_NAME=${REPONAME//[^A-Za-z0-9]/_}_${CACHE_NAME}
CACHE_PATH=${CACHE_BASE}/$CACHE_NAME
make_new_cache $OLD_CACHE_PATH $CACHE_PATH
# Return if install type is not source
local INSTALL_TYPE_VAR=DIB_INSTALLTYPE_${REPONAME//[^A-Za-z0-9]/_}
local INSTALL_TYPE=${!INSTALL_TYPE_VAR:-source}
if [ ! $INSTALL_TYPE = "source" ]; then
echo "$REPONAME install type not set to source"
continue
fi
case $REPOTYPE in
git)
if [ -z "${!REPOLOCATION_OVERRIDE:-""}" -a -n "${DIB_GITREPOBASE:-""}" ] ; then
# Transform the current repo base to the new one
local NEW_REPOLOCATION=$(echo $REPOLOCATION |\
sed "s,^[^:]\+://[^/]\+/\(~[^/]\+\)\?\(.*\)$,${DIB_GITREPOBASE}\2,g")
echo "Transformed ${REPOLOCATION} to ${NEW_REPOLOCATION}"
REPOLOCATION=$NEW_REPOLOCATION
# Also update the cache location
CACHE_NAME=$(echo "${REPOTYPE}_${REPOLOCATION}" | sha1sum | awk '{ print $1 }' )
CACHE_PATH=~/.cache/image-create/repository-sources/$CACHE_NAME
fi
sudo mkdir -p $REPO_SUB_DIRECTORY
if [ ! -e "$CACHE_PATH" ] ; then
echo "Caching $REPONAME from $REPOLOCATION in $CACHE_PATH"
git clone $REPOLOCATION $CACHE_PATH.tmp
mv ${CACHE_PATH}{.tmp,}
fi
HAS_REF=$(git --git-dir=$CACHE_PATH/.git name-rev $REPOREF 2>/dev/null || true)
if [ -z "$DIB_OFFLINE" -o -z "$HAS_REF" ] ; then
echo "Updating cache of $REPOLOCATION in $CACHE_PATH with ref $REPOREF"
git --git-dir=$CACHE_PATH/.git fetch --update-head-ok $REPOLOCATION +refs/heads/*:refs/heads/*
fi
echo "Cloning from $REPONAME cache and applying ref $REPOREF"
sudo git clone $CACHE_PATH $REPO_DEST
pushd $REPO_DEST
sudo git fetch $CACHE_PATH $REPOREF
sudo git reset --hard FETCH_HEAD
# Get the reference in use
git_ref=$(git rev-parse FETCH_HEAD)
popd
# Write the reference being used into the source-repositories manifest
echo "$REPONAME git $REPOPATH $REPOLOCATION $git_ref" >> $GIT_MANIFEST
;;
tar)
# The top level directory of the tarball mightn't have a fixed name i.e.
# it could contain version numbers etc... so we write it to a tmpdir
# the then move the contents into the directory we want it in, this does
# assume the tarball only contains a single top level directory
local tmpdir=$(mktemp --tmpdir=$TMP_MOUNT_PATH/tmp -d)
if [ -n "$CACHE_PATH" ] ; then
echo "Caching $REPONAME tarball from $REPOLOCATION in $CACHE_PATH"
if [ ! -f "$CACHE_PATH" -o -z "$DIB_OFFLINE" ] ; then
$CACHE_URL $REPOLOCATION $CACHE_PATH
fi
tar -C $tmpdir -xzf $CACHE_PATH
else
echo "Fetching $REPONAME tarball from $REPOLOCATION"
curl $REPOLOCATION | tar -C $tmpdir -xzf -
fi
sudo mkdir -p $REPO_DEST
sudo mv $tmpdir/*/* $REPO_DEST
rm -rf $tmpdir
;;
file)
sudo mkdir -p $REPO_SUB_DIRECTORY
if [ -n "$CACHE_PATH" ] ; then
echo "Caching $REPONAME file from $REPOLOCATION in $CACHE_PATH"
if [ ! -f "$CACHE_PATH" -o -z "$DIB_OFFLINE" ] ; then
$CACHE_URL $REPOLOCATION $CACHE_PATH
fi
sudo cp $CACHE_PATH $REPO_DEST
else
echo "Fetching $REPONAME file from $REPOLOCATION"
sudo curl $REPOLOCATION -o $REPO_DEST
fi
;;
*)
echo "Unsupported repository type: $REPOTYPE"
return 1
;;
esac
# Capture the in-instance repository path for later review / other
# elements (like a pypi dependency cache).
echo "$REPOPATH" | sudo dd of=$TMP_MOUNT_PATH/etc/dib-source-repositories oflag=append conv=notrunc
else
echo "Couldn't parse '$line' as a source repository"
return 1
fi
done < $REPO_SOURCES
}
CACHE_BASE=$DIB_IMAGE_CACHE/source-repositories
OLD_CACHE_BASE=$DIB_IMAGE_CACHE/repository-sources
make_new_cache $OLD_CACHE_BASE $CACHE_BASE
mkdir -p $CACHE_BASE
# Use the IMAGE_NAME from the calling script, and make it unique with the temporary path
GIT_MANIFEST_NAME=dib-manifest-git-$(basename ${IMAGE_NAME})
GIT_MANIFEST_CACHE_NAME=${GIT_MANIFEST_NAME}_$(dirname ${TMP_MOUNT_PATH##*.})
GIT_MANIFEST=$CACHE_BASE/${GIT_MANIFEST_CACHE_NAME}
rm -f $GIT_MANIFEST
# Get source repositories for the target
echo "devstack git /opt/git/openstack-dev/devstack $GIT_BASE/openstack-dev/devstack.git" > $TMP_HOOKS_PATH/early-source-repository-config
for _SOURCEREPO in $(find $TMP_HOOKS_PATH -maxdepth 1 -name "early-source-repository-*" -not -name '*~'); do
get_repos_for_element $_SOURCEREPO
done
# Copy the manifest into the image if it exists (there may be no git repositories used)
if [ -e "$GIT_MANIFEST" ] ; then
sudo cp $GIT_MANIFEST $TMP_MOUNT_PATH/${DIB_MANIFEST_IMAGE_DIR}/$GIT_MANIFEST_NAME
fi
#!/usr/bin/env python
# Copyright (C) 2011-2013 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
#
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import subprocess
import sys
TMP_MOUNT_PATH = os.environ['TMP_MOUNT_PATH']
TMP_HOOKS_PATH = os.environ['TMP_HOOKS_PATH']
# this is a bit weird; we want to get the filter to use on the
# devstack lib/[apt|rpm] files to pre-install them. So we have to
# swizzle things around
RELEASE=None
if 'DIB_RELEASE' in os.environ:
RELEASE=os.environ['DIB_RELEASE']
try:
if open(os.path.join(
TMP_MOUNT_PATH,
'etc/redhat-release')).read().startswith('CentOS release 6'):
# not actually used in devstack, but for consistency and because this
# script will error if run on a platform where RELEASE isn't detected
# as something
RELEASE='rhel6'
except IOError:
pass
if 'DISTRO_NAME' in os.environ:
if os.environ['DISTRO_NAME'] == 'centos7':
# centos7 matches as rhel7 in devstack
RELEASE='rhel7'
if not RELEASE:
print "Can not determine RELEASE"
sys.exit(1)
DEVSTACK = os.path.join(TMP_MOUNT_PATH, 'opt/git/openstack-dev/devstack')
CACHEDIR = os.path.join(TMP_MOUNT_PATH, 'tmp')
IMAGES=os.path.join(TMP_HOOKS_PATH, 'source-repository-images')
def run_local(cmd, status=False, cwd='.', env={}):
print "Running:", cmd
newenv = os.environ
newenv.update(env)
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, cwd=cwd,
stderr=subprocess.STDOUT, env=newenv)
(out, nothing) = p.communicate()
if status:
return (p.returncode, out.strip())
return out.strip()
def git_branches():
branches = []
for branch in run_local(['git', 'branch', '-a'], cwd=DEVSTACK).split("\n"):
branch = branch.strip()
if not branch.startswith('remotes/origin'):
continue
branches.append(branch)
return branches
def tokenize(fn, tokens, distribution, comment=None):
for line in open(fn):
if 'dist:' in line and ('dist:%s' % distribution not in line):
continue
if 'qpid' in line:
continue # TODO: explain why this is here
if comment and comment in line:
line = line[:line.rfind(comment)]
line = line.strip()
if line and line not in tokens:
tokens.append(line)
def _legacy_find_images(basedir):
"""Divine what images we should use based on parsing stackrc."""
images = []
for line in open(os.path.join(basedir, 'stackrc')):
line = line.strip()
if line.startswith('IMAGE_URLS'):
if '#' in line:
line = line[:line.rfind('#')]
if line.endswith(';;'):
line = line[:-2]
line = line.split('=', 1)[1].strip()
if line.startswith('${IMAGE_URLS:-'):
line = line[len('${IMAGE_URLS:-'):]
if line.endswith('}'):
line = line[:-1]
if not line:
continue
if line[0] == line[-1] == '"':
line = line[1:-1]
# Add image to the list to be downloaded, but
# skip downloading giant vmware images
images += [x.strip() for x in line.split(',')
if not x.strip().endswith('vmdk')]
return images
def _find_images(basedir):
images = []
image_tool = os.path.join(basedir, 'tools', 'image_list.sh')
if os.path.exists(image_tool):
returncode, out = run_local(image_tool, status=True)
if returncode:
print "%s failed" % image_tool
print "Exit: %s, Output: %s" % (returncode, out)
# reset images so we'll fall back
images = []
else:
images = out.split('\n')
return images
def local_prep(distribution):
branches = []
for branch in git_branches():
# Ignore branches of the form 'somestring -> someotherstring'
# as this denotes a symbolic reference and the entire string
# as is cannot be checked out. We can do this safely as the
# reference will refer to one of the other branches returned
# by git_branches.
if ' -> ' in branch:
continue
branch_data = {'name': branch}
print 'Branch: ', branch
run_local(['sudo', 'git', 'checkout', branch], cwd=DEVSTACK)
run_local(['sudo', 'git', 'pull', '--ff-only', 'origin'], cwd=DEVSTACK)
if os.path.exists(os.path.join(TMP_MOUNT_PATH, 'usr/bin/apt-get')):
debs = []
debdir = os.path.join(DEVSTACK, 'files', 'debs')
if not os.path.exists(debdir):
debdir = os.path.join(DEVSTACK, 'files', 'apts')
for fn in os.listdir(debdir):
fn = os.path.join(debdir, fn)
tokenize(fn, debs, distribution, comment='#')
branch_data['debs'] = debs
if os.path.exists(os.path.join(TMP_MOUNT_PATH, 'usr/bin/yum')):
rpms = []
rpmdir = os.path.join(DEVSTACK, 'files', 'rpms')
for fn in os.listdir(rpmdir):
fn = os.path.join(rpmdir, fn)
tokenize(fn, rpms, distribution, comment='#')
branch_data['rpms'] = rpms
images = _find_images(DEVSTACK)
if not images:
images = _legacy_find_images(DEVSTACK)
branch_data['images'] = images
branches.append(branch_data)
return branches