Advanced Git and

Gitlab

Tony Wildish!
Dan Udwary
May 30, 2017

- 1 -
Advanced Gitlab
•  Prerequisites
•  Branching and Tagging
•  Building mul<ple containers
•  Pushing images to mul1ple repositories
•  Using metadata in containers
•  Deploying runners on NERSC hosts
•  Best prac<ces & recommenda<ons

•  => Get the code for this tutorial:

–  Fork the tutorial repository, then clone your fork to your laptop
–  h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 2 -
Prerequisites

•  Familiarity with git, docker, gitlab

–  Git, version 2.11 or higher
–  Docker, version 1.12.3 or higher
–  An account on gitlab.com

•  Earlier tutorials:
–  h@ps://www.nersc.gov/assets/Uploads/Git+Docker-Tutorial-Dec01-2016.pdf
•  Do exercises 4 and 5
–  h@ps://www.nersc.gov/assets/Uploads/2017-02-06-Gitlab-CI.pdf
•  Do the first exercise

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 3 -
Bonus gitlab tip: Notification emails

Go to your account seXngs,

No1fica1ons, Custom

- 4 -
 Bonus gitlab tip: Notification emails

Choose any/all fields,

can set per-project too

- 5 -
Branching and tagging

•  Branches
–  Allow parallel development in a single repository
–  Create branches as needed, delete when obsolete
–  Can merge branches if you like, or keep forever
•  Bugfix branches: merge, delete the branch “Pro Git”, by
Sco@ Chacon,
•  Feature branches: keep forever. Chapter 3
–  Can merge back & forth to control divergence
Master

Feature1

Bugfix1

- 6 -
Branching and tagging

•  Tags
–  Sta1c label, iden1fies a par1cular commit
–  Easily recover par1cular version at any 1me in future
–  Once pushed, tags shouldn’t be deleted or moved!

Tag1 Tag2

Master

Feature1

Tag3 Tag4
Bugfix1

- 7 -
Branching and tagging

•  Tags and branches in gitlab

–  Can be used to iden1fy build products, label images etc
•  If there’s a tag, use that
•  If not, use the branch name
•  ‘master’ branch -> ‘latest’ docker version (by conven1on)

–  Let’s do exercise 01!

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 8 -
Working with forked repositories

•  How do you keep a forked repository up to date?

–  Add the original source as another ‘remote’ repository
> git clone git@bitbucket.org:TWildish/jgi-lapinpy.git
Cloning into 'jgi-lapinpy'...
[…]

> cd jgi-lapinpy/
> git remote add upstream git@bitbucket.org:berkeleylab/jgi-lapinpy.git
> git remote -v show
origin git@bitbucket.org:TWildish/jgi-lapinpy.git (fetch)
origin git@bitbucket.org:TWildish/jgi-lapinpy.git (push)
upstream git@bitbucket.org:berkeleylab/jgi-lapinpy.git (fetch) “Pro Git”, by
upstream git@bitbucket.org:berkeleylab/jgi-lapinpy.git (push) Sco@ Chacon,

Sec1on 2.5
> git pull upstream master
From bitbucket.org:berkeleylab/jgi-lapinpy
* branch master -> FETCH_HEAD
Upda1ng a3f5e1e..03943c8
Fast-forward

- 9 -
Working with forked repositories

•  How do you keep a forked repo up to date?

–  Add the original source as another ‘remote’ repository
> git clone git@bitbucket.org:TWildish/jgi-lapinpy.git
Cloning into 'jgi-lapinpy'...
[…]

> cd jgi-lapinpy/
> git remote add upstream git@bitbucket.org:berkeleylab/jgi-lapinpy.git
> git remote -v show
origin git@bitbucket.org:TWildish/jgi-lapinpy.git (fetch)
origin git@bitbucket.org:TWildish/jgi-lapinpy.git (push)
upstream git@bitbucket.org:berkeleylab/jgi-lapinpy.git (fetch) “Pro Git”, by
upstream git@bitbucket.org:berkeleylab/jgi-lapinpy.git (push) Sco@ Chacon,

Sec1on 2.5
> git pull upstream master
From bitbucket.org:berkeleylab/jgi-lapinpy
* branch master -> FETCH_HEAD
Upda1ng a3f5e1e..03943c8
Fast-forward

- 10 -
Building multiple containers

•  Suppose you have a par<cular package with:

–  A few core dependencies, very small total
–  Several op1onal extras that add hundreds of MB

•  How do you build an op<mal container?

–  Include everything -> baggage that not all users need
–  Leave stuff out -> don’t sa1sfy all users

•  Solu<on:
–  Build two containers (or more) in the same repository

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 11 -
Building multiple containers

•  Gitlab supports building Docker images with names

other than the repository name
–  Default Docker name structure
•  $REGISTRY_USER/$APPLICATION:$RELEASE_TAG
–  Extended syntax:
•  $REGISTRY_USER/$APPLICATION/real-name:$RELEASE_TAG
–  Use extended syntax repeatedly in .gitlab-ci.yml, with
different ‘real-name’s
–  “myapp-lite” & “myapp”, or “myapp” & “myapp-full”

–  See exercise 02!

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 12 -
Pushing images to multiple repositories

GITLAB and SHIFTER

variables point to
different registry hosts

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 13 -
Pushing images to multiple repositories

Build and push

to GITLAB

Re-tag, push to
SHIFTER

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 14 -
Pushing images to multiple repositories
•  Caveat: Security!
–  Gitlab hands you a login-token for every build
–  For shiser, once you’re inside the firewall, there’s no
authen1ca1on needed, so no token
–  Anywhere else, you probably need a token or password,
but where do you store it?
•  Can’t be in the repository, is too visible
•  Has to be in the runner run1me environment somehow
•  Can do this in SPIN, though not very securely at the moment
•  Can do it on your laptops
•  Want to do it elsewhere? come for a chat

–  Exercise 03, in your own 1me J

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 15 -
 Using metadata in containers

•  Pass informa<on from the build environment

–  To the image, or to the user at run1me Thank
s Mich
ael, A
lex
•  Tell the user anything they might want to know:
–  What run1me environment the sosware needs
–  What level of tes1ng, cer1fica1on has been performed
–  Pointers to documenta1on, source code, maintainers…
–  Run1me details:
•  where the container looks for input
•  where it expects to be able to put output…

h@p://docs.master.dockerproject.org/v1.5/userguide/labels-custom-metadata/
h@ps://speakerdeck.com/garethr/shipping-manifests-bill-of-lading-and-docker-metadata-and-container

- 16 -
Using metadata in containers

Development environment docker build … --build-args XYZ=123

How metadata goes from the build

Build context (docker daemon) environment to the image, and to
(ARG XYZ=$XYZ) the running container

See Dockerfile.metadata in the repo

Docker image
(LABEL XYZ=$XYZ) docker inspect … | grep XYZ

Run1me environment (container) docker run… echo $XYZ

(ENV XYZ=$XYZ)

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 17 -
Using metadata in containers

•  How can we use metadata?

–  E.g. defining a proper ontology
–  Automa1ng pipelines, tes1ng, discovery…
•  Working group(?) to inves<gate this
–  Probably later in the year aser the migra1on
–  Volunteers/sugges1ons gratefully accepted!

–  Exercise 04!

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 18 -
Deploying runners on NERSC hosts
•  A runner at NERSC with write-access to $HOME etc?
•  You can do this, but there are serious risks involved!
–  Don’t share the runner registra1on token with anyone
•  ~= giving them your NERSC password
–  Don’t give other users master-level access to your repository
–  Consider alterna1ves:
•  Use a Docker image, with your custom build environment, on SPIN
•  Use a VM somewhere…
–  Talk to a consultant before a@emp1ng this!
–  Some of these risks are gitlab-specific
–  Some are inherent in running any internet-enabled services
h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 19 -
Deploying runners on NERSC hosts

•  Basic recipe
–  Download the binary for a gitlab runner
–  Register it, give it a host-specific config file
–  Give it specific tags when registering, to iden1fy it
–  Use those tags in your .gitlab-ci.yml file
–  Your pipeline can roam over the en1re filesystem if you
want, but it’s up to you then to ensure the directories you
use are clean

–  See exercise 05 for details – we won’t do this today!

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 20 -
Other gitlab features
•  API, programmable interface to Gitlab
–  h@ps://docs.gitlab.com/ee/api/
•  See JGI/gitlab-cli-tools repo for some basic tools, contribu1ons welcome!

•  Build hooks
–  Trigger ac1ons on external services other than gitlab
•  Similar capabili1es on github, bitbucket
–  Trigger ac1ons in gitlab from external service
•  E.g. nightly build, regardless of commits

•  Mirroring repositories
–  Master repository in bitbucket/github?
–  Can mirror to gitlab, automa1cally, transparently

•  Issue-tracking, wiki…
–  Other goodies come for free with gitlab, as with other hos1ng services

- 21 -
Best practices, recommendations

•  Git:
–  Use the fork/pull-request model instead of gran1ng
people direct-commit access to your repository

–  Use branches to experiment, try out bugfixes etc

•  Merge long-lived branches frequently to control divergence

–  Use tags to iden1fy stable versions, releases etc

–  Don’t delete or move tags once they’re pushed to the

master repository

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 22 -
Best practices, recommendations

•  Gitlab:
–  Build mul1ple Docker images if you have different use-
cases to serve from the same code-base
–  Pushing to mul1ple registries lets users access your images
from many places, easily
–  Use metadata in your containers!
•  Help us establish standards for JGI container metadata
–  Control access to your repositories
•  Don’t give out the runner-registra1on token
•  Avoid giving others admin/developer-access to the project
•  Think twice before deploying runners on NERSC resources

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 23 -
Finally…
•  You’re all experts now, so update your resumes!
–  “experience building and op1mizing Docker images for
bioinforma1c sosware”
–  “experience configuring and using con1nuous-integra1on
pla~orms, such as gitlab, to automate building and deploying
sosware”
–  “in-depth understanding of best-prac1ces for sosware
management, such as version control with git and use of
metadata to describe Docker images”
–  “understanding of git workflow models for teams, including the
use of branches, tags, and developer access-control”

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

- 24 -
National Energy Research Scientific Computing Center

- 25 -