diff --git a/.gitignore b/.gitignore index 54df527..c0d0ba5 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,10 @@ inputs base* *.qcow2 sigs +target-bin/bootstrap-fixup +.vagrant +docker +*.Dockerfile +cache +target-* +!target-bin/ diff --git a/CODEOWNERS b/CODEOWNERS new file mode 100644 index 0000000..68438f5 --- /dev/null +++ b/CODEOWNERS @@ -0,0 +1 @@ +* @devrandom diff --git a/README.md b/README.md index b43f486..ee0f84d 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,7 @@ +# MAINTENANCE MODE + +Due to the move of Bitcoin Core to [Guix](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#building), this repository is switching to maintenance mode. Only serious bugs (including security issues) will be considered going forward. + # Gitian Read about the project goals at the [project home page](https://gitian.org/). @@ -10,6 +14,30 @@ This performs a build inside a VM, with deterministic inputs and outputs. If th ## Prerequisites: +### Arch: + + sudo pacman -S python2-cheetah qemu rsync + sudo pacman -S lxc libvirt bridge-utils # for lxc mode + +From AUR: + +* [apt-cacher-ng](https://aur.archlinux.org/packages/apt-cacher-ng/) (you may have to play with permissions (chown to apt-cacher-ng) on files to get apt-cacher-ng to start) +* [debootstrap](https://aur.archlinux.org/packages/debootstrap-git/) +* [dpkg](https://aur.archlinux.org/packages/dpkg/) +* [gnupg1](https://aur.archlinux.org/packages/gnupg1/) +* [multipath-tools](https://aur.archlinux.org/packages/multipath-tools/) (for kpartx) + +Non-AUR packages: + +* [debian-archive-keyring](https://packages.debian.org/jessie/debian-archive-keyring) (for making Debian guests) +* [ubuntu-keyring](https://packages.ubuntu.com/search?keywords=ubuntu-keyring) (for making Ubuntu guests) + +From newroco on GitHub: + +* [vmbuilder](https://github.com/newroco/vmbuilder) + +Also, I had to modify the default /etc/sudoers file to uncomment the `secure_path` line, because vmbuilder isn't found otherwise when the `env -i ... sudo vmbuilder ...` line is executed (because the i flag resets the environment variables including the PATH). + ### Gentoo: layman -a luke-jr # needed for vmbuilder @@ -19,19 +47,54 @@ This performs a build inside a VM, with deterministic inputs and outputs. If th ### Ubuntu: +This pulls in all pre-requisites for KVM building on Ubuntu: + sudo apt-get install git apache2 apt-cacher-ng python-vm-builder ruby qemu-utils - sudo apt-get install qemu-kvm # for KVM mode - sudo apt-get install debootstrap lxc # for LXC mode + +If you'd like to use LXC mode instead, install it as follows: + + sudo apt-get install lxc + +If you'd like to use docker mode instead, install it as follows: + + sudo apt-get install docker-ce + +### Debian: + +See Ubuntu, and also run the following on Debian Jessie or newer: + + sudo apt-get install ubuntu-archive-keyring + +On Debian Wheezy you run the same command, but you must first add backports to your system, because the package is only available in wheezy-backports. ### OSX with MacPorts: sudo port install ruby coreutils export PATH=$PATH:/opt/local/libexec/gnubin # Needed for sha256sum + +### OSX with Homebrew: + + brew install ruby coreutils + export PATH=$PATH:/opt/local/libexec/gnubin #### VirtualBox: Install virtualbox from http://www.virtualbox.org, and make sure `VBoxManage` is in your `$PATH`. +## Debian Guests + +Gitian supports Debian guests in addition to Ubuntu guests. Note that this doesn't mean you can allow the builders to choose to use either Debian or Ubuntu guests. The person creating the Gitian descriptor will need to choose a particular distro and suite for the guest and all builders must use that particular distro and suite, otherwise the software won't reproduce for everyone. + +To create a Debian guest: + + bin/make-base-vm --distro debian --suite jessie + +There is currently no support for LXC Debian guests. There is just KVM support. LXC support for Debian guests is planned to be added soon. + +Only Debian Jessie guests have been tested with Gitian. If you have success (or trouble) with other versions of Debian, please let us know. + +If you are creating a Gitian descriptor, you can now specify a distro. If no distro is provided, the default is to assume Ubuntu. Since Ubuntu is assumed, older Gitian descriptors that don't specify a distro will still work as they always have. + ## Create the base VM for use in further builds **NOTE:** requires `sudo`, please review the script @@ -49,6 +112,15 @@ Set the `USE_LXC` environment variable to use `LXC` instead of `KVM`: export USE_LXC=1 +### Docker + + bin/make-base-vm --docker + bin/make-base-vm --docker --arch i386 + +Set the `USE_DOCKER` environment variable to use `DOCKER` instead of `KVM`: + + export USE_DOCKER=1 + ### VirtualBox Command-line `VBoxManage` must be in your `$PATH`. @@ -57,18 +129,18 @@ Command-line `VBoxManage` must be in your `$PATH`. `make-base-vm` cannot yet make VirtualBox virtual machines ( _patches welcome_, it should be possible to use `VBoxManage`, boot-from-network Linux images and PXE booting to do it). So you must either get or manually create VirtualBox machines that: -1. Are named `Gitian--` -- e.g. Gitian-lucid-i386 for a 32-bit, Ubuntu 10 machine. +1. Are named `Gitian--` -- e.g. Gitian-xenial-i386 for a 32-bit, Ubuntu 16 machine. 2. Have a booted-up snapshot named `Gitian-Clean` . The build script resets the VM to that snapshot to get reproducible builds. 3. Has the VM's NAT networking setup to forward port `localhost:2223` on the host machine to port `22` of the VM; e.g.: ``` - VBoxManage modifyvm Gitian-lucid-i386 --natpf1 "guestssh,tcp,,2223,,22" + VBoxManage modifyvm Gitian-xenial-i386 --natpf1 "guestssh,tcp,,2223,,22" ``` The final setup needed is to create an `ssh` key that will be used to login to the virtual machine: - ssh-keygen -t dsa -f var/id_dsa -N "" - ssh -p 2223 ubuntu@localhost 'mkdir -p .ssh && chmod 700 .ssh && cat >> .ssh/authorized_keys' < var/id_dsa.pub + ssh-keygen -t rsa -f var/id_rsa -N "" + ssh -p 2223 ubuntu@localhost 'mkdir -p .ssh && chmod 700 .ssh && cat >> .ssh/authorized_keys' < var/id_rsa.pub Then log into the vm and copy the `ssh` keys to root's `authorized_keys` file. @@ -86,13 +158,17 @@ Set the `USE_VBOX` environment variable to use `VBOX` instead of `KVM`: If you have everything set-up properly, you should be able to: PATH=$PATH:$(pwd)/libexec - make-clean-vm --suite lucid --arch i386 + make-clean-vm --suite xenial --arch i386 + + # on-target needs $DISTRO to be set to debian if using a Debian guest + # (when running gbuild, $DISTRO is set based on the descriptor, so this line isn't needed) + DISTRO=debian # For LXC: - LXC_ARCH=i386 LXC_SUITE=lucid on-target ls -la + LXC_ARCH=i386 LXC_SUITE=xenial on-target ls -la # For KVM: - start-target 32 lucid-i386 & + start-target 32 xenial-i386 & # wait a few seconds for VM to start on-target ls -la stop-target @@ -127,8 +203,8 @@ After you've merged everybody's signatures, verify them: * Log files are captured to the _var_ directory * You can run the utilities in libexec by running `PATH="libexec:$PATH"` -* To start the target VM run `start-target 32 lucid-i386` or `start-target 64 lucid-amd64` -* To ssh into the target run `on-target` or `on-target -u root` +* To start the target VM run `start-target 32 xenial-i386` or `start-target 64 xenial-amd64` +* To ssh into the target run `on-target` (after setting $DISTRO to debian if using a Debian guest) or `on-target -u root` * On the target, the _build_ directory contains the code as it is compiled and _install_ contains intermediate libraries * By convention, the script in `.yml` starts with any environment setup you would need to manually compile things on the target @@ -147,7 +223,7 @@ Right now `lxc-start` is the default, but you can force `lxc-execute` (useful fo export LXC_EXECUTE=lxc-execute -Recent distributions allow lxc-execute / lxc-start to be run by non-priviledged users, so you might be able to rip-out the `sudo` calls in `libexec/*`. +Recent distributions allow lxc-execute / lxc-start to be run by non-privileged users, so you might be able to rip-out the `sudo` calls in `libexec/*`. If you have a runaway `lxc-start` command, just use `kill -9` on it. diff --git a/RELEASE_NOTES b/RELEASE_NOTES index 514cfdc..10762ed 100644 --- a/RELEASE_NOTES +++ b/RELEASE_NOTES @@ -1,3 +1,48 @@ +2019-05-23 +---------- + +- Add `--skip-fetch` argument that skips fetching the latest remote source + +2019-05-13 +---------- + +- No longer fetch repository from remote when the directory already exists + +2017-02-14 +---------- + +- VirtualBox launches are now headless. You can use the VirtualBox Manager to open the console UI if needed. +- Debian on VirtualBox is supported via Vagrant Cloud images +- Note that Debian on kvm is currently not supported because vmbuilder fails in the grub install stage +- git submodule support - any submodules are cloned and checked out +- Note that lxc-execute in Ubuntu 17.10 has a showstopper bug in stdin handling + +2015-12-12 +---------- + +Since OpenSSH removed support for DSS, Gitian now uses RSA keys instead. Base images should be regenerated. + +2015-05-16 +---------- + +LXC support has been revamped: + +* debootstrap is now used directly, so that no kernel or grub packages are installed +* an attempt has been made to eliminate cases where an update of a package can fail because the container is missing a real init/upstart process + +2015-03-23 +---------- + +Now ensuring that `apt-get dist-upgrade` occurs at least once before package manifest +is computed. This is because distributions usually don't store old versions of packages +in the repos, so we can't download the exact package version we have installed if it's +out of date. + +gbuild now has a --upgrade flag that forces an upgrade after the first one. + +Other Notes +=========== + Important: We are planning on switching from using lxc-start to using lxc-execute. lxc-execute requires lxc-init (or init.lxc) to be available on the guest at one of the expected places. You might have to manually install lxc on your base VM image or recreate the image. diff --git a/Vagrantfile b/Vagrantfile index 47f6dc7..817d401 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -5,34 +5,52 @@ set -eu sudo apt-get update -y sudo apt-get upgrade -y -sudo apt-get install -y autoconf2.13 automake build-essential bsdmainutils faketime g++ g++-mingw-w64 git-core libqt4-dev libtool libz-dev mingw-w64 nsis pciutils pkg-config psmisc subversion unzip zip +sudo apt-get install -y autoconf2.13 automake build-essential bsdmainutils faketime g++ g++-mingw-w64 git libqt4-dev libtool libz-dev mingw-w64 nsis pciutils pkg-config psmisc subversion unzip zip echo "ok" SCRIPT archs = ["amd64", "i386"] -suites = ["precise", "quantal", "raring", "saucy", "trusty"] +ubuntu_suites = ["precise", "quantal", "raring", "saucy", "trusty", "xenial", "bionic"] +debian_suites = ["jessie", "stretch"] if ARGV[0] == "up" and ARGV.length == 1 puts "Specify a name of the form 'suite-architecture'" - puts " suites: " + suites.join(', ') + puts " ubuntu suites: " + ubuntu_suites.join(', ') + puts " debian suites (x86_64 only): " + debian_suites.join(', ') puts " architectures: " + archs.join(', ') Process.exit 1 end -Vagrant.configure("2") do |config| +# vagrant 1.9.1 (Ubuntu 17.10) compat +if Vagrant::DEFAULT_SERVER_URL =~ /hashicorp/ + Vagrant::DEFAULT_SERVER_URL.replace('https://vagrantcloud.com') +end +Vagrant.configure("2") do |config| config.vm.provision "shell", inline: $script config.vm.network :forwarded_port, id: "ssh", guest: 22, host: 2223 - suites.each do |suite| + debian_suites.each do |suite| + name = "#{suite}-amd64" + box = "debian/#{suite}64" + + config.vm.define name do |config| + config.vm.box = box + config.vm.provider :virtualbox do |vb| + vb.name = "Gitian-#{name}" + end + end + end + + ubuntu_suites.each do |suite| archs.each do |arch| name = "#{suite}-#{arch}" config.vm.define name do |config| config.vm.box = name - config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/#{suite}/current/#{suite}-server-cloudimg-#{arch}-vagrant-disk1.box" + config.vm.box_url = "https://cloud-images.ubuntu.com/#{suite}/current/#{suite}-server-cloudimg-#{arch}-vagrant.box" config.vm.provider :virtualbox do |vb| vb.name = "Gitian-#{name}" end diff --git a/bin/gbuild b/bin/gbuild index 0e9ca58..50e9618 100755 --- a/bin/gbuild +++ b/bin/gbuild @@ -1,4 +1,4 @@ -#!/usr/bin/ruby +#!/usr/bin/env ruby require 'optparse' require 'yaml' @@ -10,11 +10,13 @@ require 'pathname' @bitness = { 'i386' => 32, 'amd64' => 64, + 'linux64' => 64, } @arches = { 'i386' => 'i386', 'amd64' => 'x86_64', + 'linux64' => 'linux64', } def system!(cmd) @@ -35,12 +37,13 @@ def info(str) puts str unless @options[:quiet] end -def build_one_configuration(suite, arch, build_desc, reference_datetime) +def build_one_configuration(suite, arch, build_desc) + FileUtils.rm_f("var/install.log") FileUtils.rm_f("var/build.log") bits = @bitness[arch] or raise "unknown architecture ${arch}" - if ENV["USE_LXC"] + if ENV["USE_LXC"] == "1" ENV["LXC_ARCH"] = arch ENV["LXC_SUITE"] = suite end @@ -72,6 +75,10 @@ def build_one_configuration(suite, arch, build_desc, reference_datetime) system! "on-target true" + system! "on-target -u root tee -a /etc/sudoers.d/#{ENV['DISTRO'] || 'ubuntu'} > /dev/null << EOF +%#{ENV['DISTRO'] || 'ubuntu'} ALL=(ALL) NOPASSWD: ALL +EOF" if build_desc["sudo"] and @options[:allow_sudo] + info "Preparing build environment" system! "on-target setarch #{@arches[arch]} bash < target-bin/init-build.sh" @@ -90,13 +97,47 @@ def build_one_configuration(suite, arch, build_desc, reference_datetime) end end + if build_desc["multiarch"] + info "Adding multiarch support (log in var/install.log)" + for a in build_desc["multiarch"] + system! "on-target -u root dpkg --add-architecture #{a} >> var/install.log 2>&1" + end + end + + if build_desc["repositories"] + info "Adding repositories to the sources list (log in var/install.log)" + for r in build_desc["repositories"] + system! "on-target -u root tee -a /etc/apt/sources.list >> var/install.log 2>&1 << EOF +#{r["source"]} +EOF" + end + end + info "Updating apt-get repository (log in var/install.log)" - system! "on-target -u root apt-get update > var/install.log 2>&1" + system! "on-target -u root apt-get update >> var/install.log 2>&1" info "Installing additional packages (log in var/install.log)" - system! "on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install #{build_desc["packages"].join(" ")} > var/install.log 2>&1" + system! "on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install #{build_desc["packages"].join(" ")} >> var/install.log 2>&1" - info "Grabbing package manifest" + if build_desc["repositories"] + for r in build_desc["repositories"] + info "Installing additional packages from repository #{r["distribution"]} (log in var/install.log)" + system! "on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -t #{r["distribution"]} --no-install-recommends -y install #{r["packages"].join(" ")} >> var/install.log 2>&1" + end + end + + if build_desc["alternatives"] + info "Set alternatives (log in var/install.log)" + for a in build_desc["alternatives"] + system! "on-target -u root update-alternatives --set #{a["package"]} #{a["path"]} >> var/install.log 2>&1" + end + end + + if @options[:upgrade] || system("on-target -u root '[ ! -e /var/cache/gitian/initial-upgrade ]'") + info "Upgrading system, may take a while (log in var/install.log)" + system! "on-target -u root bash < target-bin/upgrade-system.sh >> var/install.log 2>&1" + end + info "Creating package manifest" system! "on-target -u root bash < target-bin/grab-packages.sh > var/base-#{suitearch}.manifest" info "Creating build script (var/build-script)" @@ -109,23 +150,42 @@ def build_one_configuration(suite, arch, build_desc, reference_datetime) script.puts "umask 002" script.puts "export OUTDIR=$HOME/out" script.puts "GBUILD_BITS=#{bits}" + if build_desc["enable_cache"] + script.puts "GBUILD_CACHE_ENABLED=1" + script.puts "GBUILD_PACKAGE_CACHE=$HOME/cache/#{build_desc["name"]}" + script.puts "GBUILD_COMMON_CACHE=$HOME/cache/common" + end script.puts "MAKEOPTS=(-j#{@options[:num_procs]})" - (ref_date, ref_time) = reference_datetime.split - script.puts "REFERENCE_DATETIME='#{reference_datetime}'" - script.puts "REFERENCE_DATE='#{ref_date}'" - script.puts "REFERENCE_TIME='#{ref_time}'" + script.puts "NUM_PROCS=#{@options[:num_procs]}" + script.puts "NUM_MEM=#{@options[:memory]}" script.puts + author_date = nil build_desc["remotes"].each do |remote| dir = sanitize(remote["dir"], remote["dir"]) + + author_date = `cd inputs/#{dir} && TZ=UTC git log --date='format-local:%F %T' --format="%ad" -1`.strip + raise "error looking up author date in #{dir}" unless $?.exitstatus == 0 + system! "copy-to-target #{@quiet_flag} inputs/#{dir} build/" script.puts "(cd build/#{dir} && git reset -q --hard && git clean -q -f -d)" end + script.puts + ref_datetime = build_desc["reference_datetime"] || author_date + (ref_date, ref_time) = ref_datetime.split + script.puts "REFERENCE_DATETIME='#{ref_datetime}'" + script.puts "REFERENCE_DATE='#{ref_date}'" + script.puts "REFERENCE_TIME='#{ref_time}'" + script.puts script.puts "cd build" script.puts build_desc["script"] end info "Running build script (log in var/build.log)" - system! "on-target setarch #{@arches[arch]} bash -x < var/build-script > var/build.log 2>&1" + if ENV["CI"] + system! "on-target setarch #{@arches[arch]} bash -x < var/build-script 2>&1 | tee var/build.log" + else + system! "on-target setarch #{@arches[arch]} bash -x < var/build-script > var/build.log 2>&1" + end end ################################ @@ -133,9 +193,15 @@ end OptionParser.new do |opts| opts.banner = "Usage: build [options] .yml" + opts.on("--allow-sudo", "override SECURITY on the target VM and allow the use of sudo with no password for the default user") do |v| + @options[:allow_sudo] = v + end opts.on("-i", "--skip-image", "reuse current target image") do |v| @options[:skip_image] = v end + opts.on("--upgrade", "upgrade guest with latest packages") do |v| + @options[:upgrade] = v + end opts.on("-q", "--quiet", "be quiet") do |v| @options[:quiet] = v end @@ -151,9 +217,24 @@ OptionParser.new do |opts| opts.on("-u PAIRS", "--url PAIRS", "comma separated list of DIRECTORY=URL pairs") do |v| @options[:url] = v end + opts.on("-o", "--cache-read-only", "only use existing cache files, do not update them") do |v| + @options[:cache_ro] = v + end + opts.on("--skip-fetch", "skip fetching the latest git objects and refs from the remote source") do |v| + @options[:skip_fetch] = v + end + opts.on("--fetch-branches", "fetch branches from the remote source") do |v| + @options[:fetch_branches] = v + end + opts.on("--fetch-tags", "fetch tags from the remote source") do |v| + @options[:fetch_tags] = v + end + opts.on("--skip-cleanup", "skip cleaning up the target VM. this may be useful for copying additional files from the target after the build") do |v| + @options[:skip_cleanup] = v + end end.parse! -if !ENV["USE_LXC"] and !File.exist?("/dev/kvm") +if ENV["USE_LXC"] != "1" and ENV["USE_DOCKER"] != "1" and ENV["USE_VBOX"] != "1" and !File.exist?("/dev/kvm") $stderr.puts "\n************* WARNING: kvm not loaded, this will probably not work out\n\n" end @@ -189,9 +270,20 @@ if enable_cache FileUtils.mkdir_p(File.join(cache_dir, package_name)) end +distro = build_desc["distro"] || "ubuntu" suites = build_desc["suites"] or raise "must supply suites" archs = build_desc["architectures"] or raise "must supply architectures" -reference_datetime = build_desc["reference_datetime"] or raise "must supply reference_datetime" +build_desc["reference_datetime"] or build_desc["remotes"].size > 0 or raise "must supply `reference_datetime` or `remotes`" +docker_image_digests = build_desc["docker_image_digests"] || [] + +# if docker_image_digests are supplied, it must be the same length as suites +if docker_image_digests.size > 0 and suites.size != docker_image_digests.size + raise "`suites` and `docker_image_digests` must both be the same size if both are supplied" +elsif ENV["USE_DOCKER"] == "1" and docker_image_digests.size > 0 and suites.size == docker_image_digests.size + suites = docker_image_digests +end + +ENV['DISTRO'] = distro desc_sum = `sha256sum #{build_desc_file}` desc_sum = desc_sum.sub(build_desc_file, "#{package_name}-desc.yml") @@ -229,14 +321,24 @@ build_desc["remotes"].each do |remote| remote["url"] = urls[remote["dir"]] end dir = sanitize(remote["dir"], remote["dir"]) + commit = sanitize(remote["commit"], remote["commit"]) unless File.exist?("inputs/#{dir}") - system!("git init inputs/#{dir}") + system!("git init inputs/#{dir}") end - system!("cd inputs/#{dir} && git fetch --update-head-ok #{sanitize_path(remote["url"], remote["url"])} +refs/tags/*:refs/tags/* +refs/heads/*:refs/heads/*") - commit = sanitize(remote["commit"], remote["commit"]) - commit = `cd inputs/#{dir} && git log --format=%H -1 #{commit}`.strip - raise "error looking up commit for tag #{remote["commit"]}" unless $?.exitstatus == 0 - system!("cd inputs/#{dir} && git checkout -q #{commit}") + if !@options[:skip_fetch] + if @options[:fetch_branches] + system!("cd inputs/#{dir} && git fetch -f --update-head-ok #{sanitize_path(remote["url"], remote["url"])} +refs/heads/*:refs/heads/*") + end + if @options[:fetch_tags] + system!("cd inputs/#{dir} && git fetch -f --update-head-ok #{sanitize_path(remote["url"], remote["url"])} +refs/tags/*:refs/tags/*") + end + system!("cd inputs/#{dir} && git fetch -f --update-head-ok #{sanitize_path(remote["url"], remote["url"])} #{commit}") + system!("cd inputs/#{dir} && git checkout -q FETCH_HEAD") + else + system!("cd inputs/#{dir} && git checkout -q #{commit}") + end + system!("cd inputs/#{dir} && git submodule update --init --recursive --force") + commit = `cd inputs/#{dir} && git log --format=%H -1`.strip in_sums << "git:#{commit} #{dir}" end @@ -249,13 +351,13 @@ suites.each do |suite| arch = sanitize(arch, "architecture") # Build! - build_one_configuration(suite, arch, build_desc, reference_datetime) + build_one_configuration(suite, arch, build_desc) - info "Grabbing results" + info "Grabbing results from target" system! "copy-from-target #{@quiet_flag} out #{build_dir}" - if enable_cache - info "Grabbing cache" + if enable_cache && !@options[:cache_ro] + info "Grabbing cache from target" system! "copy-from-target #{@quiet_flag} cache/#{package_name}/ #{cache_dir}" system! "copy-from-target #{@quiet_flag} cache/common/ #{cache_dir}" end @@ -264,6 +366,11 @@ suites.each do |suite| end end +unless @options[:skip_cleanup] + info "Cleaning up target" + system "stop-target" +end + out_dir = File.join(build_dir, "out") out_sums = {} cache_common_dir = File.join(cache_dir, "common") diff --git a/bin/gsign b/bin/gsign index 665e8ac..e79a0db 100755 --- a/bin/gsign +++ b/bin/gsign @@ -1,4 +1,4 @@ -#!/usr/bin/ruby +#!/usr/bin/env ruby require 'optparse' require 'yaml' @@ -45,6 +45,10 @@ OptionParser.new do |opts| opts.on("-d DEST", "--destination DEST", "directory to place signature in") do |v| @options[:destination] = v end + + opts.on("-p PROG", "--signing_program PROG", "specify signing program to use") do |v| + @options[:program] = v + end end.parse! base_dir = Pathname.new(__FILE__).expand_path.dirname.parent @@ -76,6 +80,7 @@ result['type'] = 'build' result['optionals'] = optionals signer = @options[:signer] or raise "must supply signer with --signer" +program = @options[:program] || "gpg --detach-sign" FileUtils.mkdir_p(destination) @@ -85,4 +90,4 @@ assert_path = File.join(release_path, "#{package_name}-build.assert") File.open(assert_path, "w") do |io| io.write result.to_yaml end -system!("gpg --detach-sign -u \"#{signer}\" \"#{assert_path}\"") +system!("#{program} -u \"#{signer}\" \"#{assert_path}\"") diff --git a/bin/gverify b/bin/gverify index 5eab37b..ff0e56c 100755 --- a/bin/gverify +++ b/bin/gverify @@ -1,10 +1,12 @@ -#!/usr/bin/ruby +#!/usr/bin/env ruby require 'optparse' require 'yaml' require 'fileutils' require 'pathname' +bold = ["\033[0m", "\033[1m"] + @options = {} def system!(cmd) @@ -17,7 +19,7 @@ def sanitize(str, where) end def sanitize_path(str, where) - raise "unsanitary string in #{where}" if (str =~ /[^@\w\/.:+-]/) + raise "unsanitary string in #{where}" if (str =~ /[^@\w\\ '\/.:+-]/) str end @@ -33,6 +35,11 @@ OptionParser.new do |opts| opts.on("-v", "--verbose", "be more verbose") do |v| @options[:verbose] = v end + @options[:markup] = true + opts.on("-m", "--[no-]markup", "markup the output using ANSI escape codes") do |m| + @options[:markup] = m + end + opts.on("-r REL", "--release REL", "release name") do |v| @options[:release] = v end @@ -43,6 +50,10 @@ OptionParser.new do |opts| opts.on("-c SIGNER", "--compare-to SIGNER", "compare other manifests to SIGNER's, if not given pick first") do |v| @options[:compareto] = v end + + opts.on("-p PROG", "--verify-program PROG", "specify verification program to use (default is gpg)") do |v| + @options[:program] = v + end end.parse! base_dir = Pathname.new(__FILE__).expand_path.dirname.parent @@ -62,6 +73,9 @@ destination = @options[:destination] || File.join(base_dir, "sigs", package_name release = @options[:release] || "current" release = sanitize(release, "release") verbose = @options[:verbose] +bold = ['', ''] unless @options[:markup] + +program = @options[:program] || "gpg" release_path = File.join(destination, release) @@ -99,8 +113,8 @@ Dir.foreach(release_path) do |signer_dir| end result = YAML.load_file(result_path) - system("gpg --keyserver pgp.mit.edu --recv-keys `gpg --quiet --batch --verify \"#{File.join(signer_path, 'signature.pgp')}\" \"#{result_path}\" 2>&1 | head -n1 | grep \"key ID\" | awk '{ print $15 }'` > /dev/null 2>&1") - out = `gpg --quiet --batch --verify \"#{sig_path}\" \"#{result_path}\" 2>&1` + system("#{program} --keyserver pgp.mit.edu --recv-keys `#{program} --quiet --batch --verify \"#{File.join(signer_path, 'signature.pgp')}\" \"#{result_path}\" 2>&1 | head -n1 | grep \"key ID\" | awk '{ print $15 }'` > /dev/null 2>&1") + out = `#{program} --quiet --batch --verify \"#{sig_path}\" \"#{result_path}\" 2>&1` if $? != 0 out.each_line do |line| if line =~ /^gpg: Signature made/ @@ -109,7 +123,8 @@ Dir.foreach(release_path) do |signer_dir| puts line end end - puts "#{signer_dir}: BAD SIGNATURE" + puts "#{bold[1]}#{signer_dir}: BAD SIGNATURE#{bold[0]}" + puts did_fail = true elsif current_manifest and (result['out_manifest'] != current_manifest or result['release'] != release or result['name'] != package_name) out.each_line do |line| @@ -123,7 +138,8 @@ Dir.foreach(release_path) do |signer_dir| puts line end end - puts "#{signer_dir}: MISMATCH" + puts "#{bold[1]}#{signer_dir}: MISMATCH#{bold[0]}" + puts if verbose lines1 = current_manifest.each_line lines2 = result['out_manifest'].each_line @@ -147,7 +163,8 @@ Dir.foreach(release_path) do |signer_dir| puts line end end - puts "#{signer_dir}: OK" + puts "#{bold[1]}#{signer_dir}: OK#{bold[0]}" + puts end if !current_manifest # take first manifest as 'current' to compare against diff --git a/bin/make-base-vm b/bin/make-base-vm index 161e197..30e4fbf 100755 --- a/bin/make-base-vm +++ b/bin/make-base-vm @@ -1,29 +1,55 @@ #!/bin/sh set -e -SUITE=lucid +DISTRO=ubuntu +SUITE=xenial ARCH=amd64 -MIRROR=http://${MIRROR_HOST:-127.0.0.1}:3142/archive.ubuntu.com/ubuntu -SECURITY_MIRROR=http://${MIRROR_HOST:-127.0.0.1}:3142/security.ubuntu.com/ubuntu +DISKSIZE=12287 LXC=0 VBOX=0 +DOCKER=0 +DOCKER_IMAGE_HASH="" usage() { echo "Usage: ${0##*/} [OPTION]..." echo "Make a base client." echo cat << EOF - --help display this help and exit - --suite U build suite U instead of lucid - --arch A build architecture A (e.g. i386) instead of amd64 - --lxc use lxc instead of kvm - --vbox use VirtualBox instead of kvm + --help display this help and exit + --distro D build distro D (e.g. debian) instead of ubuntu + --suite U build suite U instead of xenial + --arch A build architecture A (e.g. i386) instead of amd64 + --disksize S disk/image size S in MB (default 12287) + --lxc use lxc instead of kvm + --vbox use VirtualBox instead of kvm + --docker use docker instead of kvm + --docker-image-hash D digest of the docker image to build from The MIRROR_HOST environment variable can be used to change the apt-cacher host. It should be something that both the host and the target VM can reach. It may be set to 127.0.0.1, in which case it will be changed to 10.0.2.2 on the guest (or GITIAN_HOST_IP if it is defined) 10.0.2.2 is the host IP as visible from the guest under qemu networking. + + The DEBOOTSTRAP_DIR (but also GITIAN_SUDO_USE_DEBOOTSTRAP_DIR, see below!) + environment variable can be set to select a directory + that will contain data like in "/usr/share/debootstrap/". This allows user to + make a copy of this files to some local dir and modify them locally: + e.g. set env variable "DEBOOTSTRAP_DIR=./mydeboot/", then copy or link + system's version of files there, and modify them there + (e.g. copy your debootstrap-script file "xenial" to "./mydeboot/scripts/"). + + Set env GITIAN_SUDO_USE_DEBOOTSTRAP_DIR="yes" to allow sudo for debootstrap + to use flags like --preserve-env that are required for DEBOOTSTRAP_DIR to work. + It must be equal string "yes". + This is done as separate variable to make it clear that we modify sudo + behaviour here regarding security (though anyway env is cleared with + whitelist so should be perfectly safe). + + The --docker-image-hash option can be used to specify the hash of a particular + base image to use. These hashes can be found under the "RepoDigests" field of + "docker image inspect ". They will be reported in the form "sha256:"; + only need the part is needed EOF } @@ -34,6 +60,10 @@ if [ $# != 0 ] ; then usage exit 0 ;; + --distro|-d) + DISTRO="$2" + shift 2 + ;; --suite|-s) SUITE="$2" shift 2 @@ -42,6 +72,10 @@ if [ $# != 0 ] ; then ARCH="$2" shift 2 ;; + --disksize) + DISKSIZE="$2" + shift 2 + ;; --lxc) LXC=1 shift 1 @@ -50,6 +84,14 @@ if [ $# != 0 ] ; then VBOX=1 shift 1 ;; + --docker) + DOCKER=1 + shift 1 + ;; + --docker-image-digest) + DOCKER_IMAGE_HASH="$2" + shift 2 + ;; --*) echo "unrecognized option $1" exit 1 @@ -61,10 +103,27 @@ if [ $# != 0 ] ; then done fi +if [ $DOCKER = "1" ]; then + MIRROR_DEFAULT=172.17.0.1 +else + MIRROR_DEFAULT=127.0.0.1 +fi +MIRROR_BASE=http://${MIRROR_HOST:-$MIRROR_DEFAULT}:3142 + +if [ $DISTRO = "ubuntu" ]; then + MIRROR=$MIRROR_BASE/archive.ubuntu.com/ubuntu + SECURITY_MIRROR=$MIRROR_BASE/security.ubuntu.com/ubuntu + components=main,universe +elif [ $DISTRO = "debian" ]; then + MIRROR=$MIRROR_BASE/ftp.debian.org/debian + SECURITY_MIRROR=$MIRROR_BASE/security.debian.org/ + components=main,contrib +fi + mkdir -p var -if [ ! -e var/id_dsa ]; then - ssh-keygen -t dsa -f var/id_dsa -N "" +if [ ! -e var/id_rsa ]; then + ssh-keygen -t rsa -f var/id_rsa -N "" fi OUT=base-$SUITE-$ARCH @@ -74,11 +133,86 @@ if [ $ARCH = "amd64" -a $SUITE = "hardy" ]; then FLAVOUR=server fi -addpkg=openssh-server,pciutils,build-essential,git-core,subversion,lxc +if [ $DISTRO = "debian" -a $ARCH = "amd64" ]; then + FLAVOUR=amd64 +elif [ $DISTRO = "debian" -a $ARCH = "i386" -a \($SUITE = "squeeze" -o $SUITE = "lenny" -o $SUITE = "etch" -o $SUITE = "sarge" -o $SUITE = "woody" -o $SUITE = "potato" -o $SUITE = "slink" -o $SUITE = "hamm" -o $SUITE = "bo" -o $SUITE = "rex" -o $SUITE = "buzz"\) ]; then + FLAVOUR=686 +elif [ $DISTRO = "debian" ]; then + FLAVOUR=686-pae +fi + + +LOCALE_PKG=language-pack-en +if [ $DISTRO = "debian" ]; then + LOCALE_PKG=locales +fi + +addpkg=pciutils,build-essential,git,subversion,$LOCALE_PKG,wget,lsb-release,sudo + +if [ $DISTRO = "ubuntu" ]; then + # Need comma at end to work around an issue with apt for Debian <= Wheezy regarding empty strings + # + # If we left the comma down below when adding KERNEL_PKG to addpkg, the fact that KERNEL_PKG is undefined + # if DISTRO is debian would result in two commas in a row (,,), which is interpreted by apt-get as the + # package with the name empty string (""). This triggers a bug with apt versions < 1.0.3. So by adding the + # comma to the end of KERNEL_PKG, we are including that comma if the distro is ubuntu (and therefore we do + # have a kernel package that needs to be installed). If KERNEL_PKG is not set (i.e. we have Debian as the + # distro), then we don't add that extra comma and therefore, we don't end up with two commas in a row. + # + # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=744940 + # http://anonscm.debian.org/cgit/apt/apt.git/commit/?h=1.0.3&id=d99854cac4065bc7b337815fb2116269d58dab73 + KERNEL_PKG=linux-image-generic, +fi + +GRUB_PKG=grub +if [ $DISTRO = "ubuntu" ]; then + GRUB_PKG=grub-pc +fi + +if [ $LXC = "1" ]; then + addpkg=$addpkg,lxc +else + # Lack of comma after KERNEL_PKG is not a typo + addpkg=$addpkg,${KERNEL_PKG}${GRUB_PKG},openssh-server +fi # Remove cron to work around vmbuilder issue when umounting /dev on target removepkg=cron +if [ $DOCKER = "1" ]; then + + addpkg=`echo $addpkg | tr ',' ' '` + + mkdir -p docker + cd docker + + if [ -n "$DOCKER_IMAGE_HASH" ]; then + base_image="$DISTRO@sha256:$DOCKER_IMAGE_HASH" + OUT=base-$DOCKER_IMAGE_HASH-$ARCH + else + base_image="$DISTRO:$SUITE" + fi + + # Generate the dockerfile + cat << EOF > $OUT.Dockerfile +FROM $base_image + +ENV DEBIAN_FRONTEND=noninteractive +RUN echo 'Acquire::http { Proxy "$MIRROR_BASE"; };' > /etc/apt/apt.conf.d/50cacher +RUN apt-get update && apt-get --no-install-recommends -y install $addpkg + +RUN useradd -ms /bin/bash -U $DISTRO +USER $DISTRO:$DISTRO +WORKDIR /home/$DISTRO + +CMD ["sleep", "infinity"] +EOF + + docker build --pull -f $OUT.Dockerfile -t $OUT . + + exit 0 +fi + if [ $VBOX = "1" ]; then NAME="$SUITE-$ARCH" if ! vagrant status | grep "$NAME" | grep "not created" > /dev/null; then @@ -86,12 +220,21 @@ if [ $VBOX = "1" ]; then exit 1 fi + DISTRO_USER_CREATE=0 + if [ $DISTRO = "debian" ]; then + # we use a vagrant provider + DISTRO_USER_CREATE=1 + fi + vagrant up "$NAME" + if [ $DISTRO_USER_CREATE = "1" ]; then + vagrant ssh "$NAME" -c "sudo useradd -m -s /bin/bash $DISTRO" + fi vagrant ssh "$NAME" -c "sudo mkdir -p /root/.ssh && sudo chmod 700 /root/.ssh" - vagrant ssh "$NAME" -c "sudo sh -c 'cat >> /root/.ssh/authorized_keys'" < var/id_dsa.pub - vagrant ssh "$NAME" -c "sudo -u ubuntu mkdir -p /home/ubuntu/.ssh && sudo -u ubuntu chmod 700 /home/ubuntu/.ssh" - vagrant ssh "$NAME" -c "sudo sh -c 'cat >> /home/ubuntu/.ssh/authorized_keys'" < var/id_dsa.pub + vagrant ssh "$NAME" -c "sudo sh -c 'cat >> /root/.ssh/authorized_keys'" < var/id_rsa.pub + vagrant ssh "$NAME" -c "sudo -u $DISTRO mkdir -p /home/$DISTRO/.ssh && sudo -u $DISTRO chmod 700 /home/$DISTRO/.ssh" + vagrant ssh "$NAME" -c "sudo sh -c 'cat >> /home/$DISTRO/.ssh/authorized_keys'" < var/id_rsa.pub VBoxManage snapshot "Gitian-$NAME" take "Gitian-Clean" vagrant suspend "$NAME" @@ -99,26 +242,60 @@ if [ $VBOX = "1" ]; then exit 0 fi -if [ -e $OUT.qcow2 ]; then - echo $OUT.qcow2 already exists, please remove it first - exit 1 -fi +if [ $LXC = "1" ]; then + if [ -e $OUT ]; then + echo $OUT already exists, please remove it first + exit 1 + fi + sudo rm -rf $OUT-bootstrap + # Need universe for lxc in lucid -libexec/config-bootstrap-fixup -rm -rf $OUT -env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 sudo vmbuilder kvm ubuntu --rootsize 10240 --arch=$ARCH --suite=$SUITE --addpkg=$addpkg --removepkg=$removepkg --ssh-key=var/id_dsa.pub --ssh-user-key=var/id_dsa.pub --mirror=$MIRROR --security-mirror=$SECURITY_MIRROR --dest=$OUT --flavour=$FLAVOUR --firstboot=`pwd`/target-bin/bootstrap-fixup -mv $OUT/*.qcow2 $OUT.qcow2 -rm -rf $OUT + unset preserve_env + if [ "$GITIAN_SUDO_USE_DEBOOTSTRAP_DIR" = "yes" ]; then + echo "sudo will preserve (some) env flags" + preserve_env=yes # if you would want to set false then unset this variable + fi + env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 DEBOOTSTRAP_DIR="$DEBOOTSTRAP_DIR" sudo ${preserve_env+--preserve-env} debootstrap --arch=$ARCH --include=$addpkg --exclude=$removepkg --components=$components $SUITE $OUT-bootstrap $MIRROR + # Fix lxc issue + if [ -f $OUT-bootstrap/usr/lib/lxc/lxc-init ] + then + sudo cp $OUT-bootstrap/usr/lib/lxc/lxc-init $OUT-bootstrap/usr/sbin/init.lxc + else + if [ $ARCH = "amd64" ] + then + if [ -f $OUT-bootstrap/usr/lib/x86_64-linux-gnu/lxc/lxc-init ] + then + sudo cp $OUT-bootstrap/usr/lib/x86_64-linux-gnu/lxc/lxc-init $OUT-bootstrap/usr/sbin/init.lxc + fi + else + if [ -f $OUT-bootstrap/usr/lib/i386-linux-gnu/lxc/lxc-init ] + then + sudo cp $OUT-bootstrap/usr/lib/i386-linux-gnu/lxc/lxc-init $OUT-bootstrap/usr/sbin/init.lxc + fi + fi + fi + dd if=/dev/zero of=$OUT-lxc bs=1M count=1 seek=$DISKSIZE + /sbin/mkfs.ext4 -F $OUT-lxc + t=`mktemp -d gitian.XXXXXXXX` + sudo mount $OUT-lxc $t + sudo cp -a $OUT-bootstrap/* $t + sudo umount $t + rmdir $t -if [ $LXC = "1" ]; then - #sudo debootstrap --include=$addpkg --arch=$ARCH $SUITE $OUT-root $MIRROR - echo Extracting partition for lxc - qemu-img convert $OUT.qcow2 $OUT.raw - loop=`sudo kpartx -av $OUT.raw|sed -n '/loop.p1/{s/.*loop\(.\)p1.*/\1/;p}'` - sudo cp --sparse=always /dev/mapper/loop${loop}p1 $OUT - sudo chown $USER $OUT - sudo kpartx -d /dev/loop$loop - rm -f $OUT.raw - # bootstrap-fixup is done in libexec/make-clean-vm + sudo rm -rf $OUT-bootstrap + mv $OUT-lxc $OUT + # bootstrap-fixup is done in libexec/make-clean-vm +else + if [ -e $OUT.qcow2 ]; then + echo $OUT.qcow2 already exists, please remove it first + exit 1 + fi + libexec/config-bootstrap-fixup + + rm -rf $OUT + env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 sudo vmbuilder kvm $DISTRO --rootsize $DISKSIZE --arch=$ARCH --suite=$SUITE --addpkg=$addpkg --removepkg=$removepkg --ssh-key=var/id_rsa.pub --ssh-user-key=var/id_rsa.pub --mirror=$MIRROR --security-mirror=$SECURITY_MIRROR --dest=$OUT --flavour=$FLAVOUR --firstboot=`pwd`/target-bin/bootstrap-fixup + mv $OUT/*.qcow2 $OUT.qcow2 + rm -rf $OUT + # bootstrap-fixup is done on first boot fi diff --git a/contrib/README.md b/contrib/README.md new file mode 100644 index 0000000..b313986 --- /dev/null +++ b/contrib/README.md @@ -0,0 +1,9 @@ +Repository Tools +--------------------- + +### [Developer tools](/contrib/devtools) ### +Specific tools for developers working on this repository. +Contains the script `github-merge.py` for merging github pull requests securely and signing them using GPG. + +### [Verify-Commits](/contrib/verify-commits) ### +Tool to verify that every merge commit was signed by a developer using the above `github-merge.py` script. diff --git a/contrib/devtools/README.md b/contrib/devtools/README.md new file mode 100644 index 0000000..1af476c --- /dev/null +++ b/contrib/devtools/README.md @@ -0,0 +1,37 @@ +Contents +======== +This directory contains tools for developers working on this repository. + +github-merge.py +=============== + +A small script to automate merging pull-requests securely and sign them with GPG. + +For example: + + ./github-merge.py 3077 + +(in any git repository) will help you merge pull request #3077 for the +devrandom/gitian-builder repository. + +What it does: +* Fetch master and the pull request. +* Locally construct a merge commit. +* Show the diff that merge results in. +* Ask you to verify the resulting source tree (so you can do a make +check or whatever). +* Ask you whether to GPG sign the merge commit. +* Ask you whether to push the result upstream. + +This means that there are no potential race conditions (where a +pullreq gets updated while you're reviewing it, but before you click +merge), and when using GPG signatures, that even a compromised github +couldn't mess with the sources. + +Setup +--------- +Configuring the github-merge tool for this repository is done in the following way: + + git config githubmerge.repository devrandom/gitian-builder + git config githubmerge.testcmd "make -j4 check" (adapt to whatever you want to use for testing) + git config --global user.signingkey mykeyid (if you want to GPG sign) diff --git a/contrib/devtools/github-merge.py b/contrib/devtools/github-merge.py new file mode 100755 index 0000000..f82362f --- /dev/null +++ b/contrib/devtools/github-merge.py @@ -0,0 +1,251 @@ +#!/usr/bin/env python3 +# Copyright (c) 2016 Bitcoin Core Developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + +# This script will locally construct a merge commit for a pull request on a +# github repository, inspect it, sign it and optionally push it. + +# The following temporary branches are created/overwritten and deleted: +# * pull/$PULL/base (the current master we're merging onto) +# * pull/$PULL/head (the current state of the remote pull request) +# * pull/$PULL/merge (github's merge) +# * pull/$PULL/local-merge (our merge) + +# In case of a clean merge that is accepted by the user, the local branch with +# name $BRANCH is overwritten with the merged result, and optionally pushed. +from __future__ import division,print_function,unicode_literals +import os,sys +from sys import stdin,stdout,stderr +import argparse +import subprocess +import json,codecs +try: + from urllib.request import Request,urlopen +except: + from urllib2 import Request,urlopen + +# External tools (can be overridden using environment) +GIT = os.getenv('GIT','git') +BASH = os.getenv('BASH','bash') + +# OS specific configuration for terminal attributes +ATTR_RESET = '' +ATTR_PR = '' +COMMIT_FORMAT = '%h %s (%an)%d' +if os.name == 'posix': # if posix, assume we can use basic terminal escapes + ATTR_RESET = '\033[0m' + ATTR_PR = '\033[1;36m' + COMMIT_FORMAT = '%C(bold blue)%h%Creset %s %C(cyan)(%an)%Creset%C(green)%d%Creset' + +def git_config_get(option, default=None): + ''' + Get named configuration option from git repository. + ''' + try: + return subprocess.check_output([GIT,'config','--get',option]).rstrip().decode('utf-8') + except subprocess.CalledProcessError as e: + return default + +def retrieve_pr_info(repo,pull): + ''' + Retrieve pull request information from github. + Return None if no title can be found, or an error happens. + ''' + try: + req = Request("https://api.github.com/repos/"+repo+"/pulls/"+pull) + result = urlopen(req) + reader = codecs.getreader('utf-8') + obj = json.load(reader(result)) + return obj + except Exception as e: + print('Warning: unable to retrieve pull information from github: %s' % e) + return None + +def ask_prompt(text): + print(text,end=" ",file=stderr) + stderr.flush() + reply = stdin.readline().rstrip() + print("",file=stderr) + return reply + +def parse_arguments(): + epilog = ''' + In addition, you can set the following git configuration variables: + githubmerge.repository (mandatory), + user.signingkey (mandatory), + githubmerge.host (default: git@github.com), + githubmerge.branch (no default), + githubmerge.testcmd (default: none). + ''' + parser = argparse.ArgumentParser(description='Utility to merge, sign and push github pull requests', + epilog=epilog) + parser.add_argument('pull', metavar='PULL', type=int, nargs=1, + help='Pull request ID to merge') + parser.add_argument('branch', metavar='BRANCH', type=str, nargs='?', + default=None, help='Branch to merge against (default: githubmerge.branch setting, or base branch for pull, or \'master\')') + return parser.parse_args() + +def main(): + # Extract settings from git repo + repo = git_config_get('githubmerge.repository') + host = git_config_get('githubmerge.host','git@github.com') + opt_branch = git_config_get('githubmerge.branch',None) + testcmd = git_config_get('githubmerge.testcmd') + signingkey = git_config_get('user.signingkey') + if repo is None: + print("ERROR: No repository configured. Use this command to set:", file=stderr) + print("git config githubmerge.repository /", file=stderr) + exit(1) + if signingkey is None: + print("ERROR: No GPG signing key set. Set one using:",file=stderr) + print("git config --global user.signingkey ",file=stderr) + exit(1) + + host_repo = host+":"+repo # shortcut for push/pull target + + # Extract settings from command line + args = parse_arguments() + pull = str(args.pull[0]) + + # Receive pull information from github + info = retrieve_pr_info(repo,pull) + if info is None: + exit(1) + title = info['title'] + # precedence order for destination branch argument: + # - command line argument + # - githubmerge.branch setting + # - base branch for pull (as retrieved from github) + # - 'master' + branch = args.branch or opt_branch or info['base']['ref'] or 'master' + + # Initialize source branches + head_branch = 'pull/'+pull+'/head' + base_branch = 'pull/'+pull+'/base' + merge_branch = 'pull/'+pull+'/merge' + local_merge_branch = 'pull/'+pull+'/local-merge' + + devnull = open(os.devnull,'w') + try: + subprocess.check_call([GIT,'checkout','-q',branch]) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot check out branch %s." % (branch), file=stderr) + exit(3) + try: + subprocess.check_call([GIT,'fetch','-q',host_repo,'+refs/pull/'+pull+'/*:refs/heads/pull/'+pull+'/*']) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot find pull request #%s on %s." % (pull,host_repo), file=stderr) + exit(3) + try: + subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+head_branch], stdout=devnull, stderr=stdout) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot find head of pull request #%s on %s." % (pull,host_repo), file=stderr) + exit(3) + try: + subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+merge_branch], stdout=devnull, stderr=stdout) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot find merge of pull request #%s on %s." % (pull,host_repo), file=stderr) + exit(3) + try: + subprocess.check_call([GIT,'fetch','-q',host_repo,'+refs/heads/'+branch+':refs/heads/'+base_branch]) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot find branch %s on %s." % (branch,host_repo), file=stderr) + exit(3) + subprocess.check_call([GIT,'checkout','-q',base_branch]) + subprocess.call([GIT,'branch','-q','-D',local_merge_branch], stderr=devnull) + subprocess.check_call([GIT,'checkout','-q','-b',local_merge_branch]) + + try: + # Create unsigned merge commit. + if title: + firstline = 'Merge #%s: %s' % (pull,title) + else: + firstline = 'Merge #%s' % (pull,) + message = firstline + '\n\n' + message += subprocess.check_output([GIT,'log','--no-merges','--topo-order','--pretty=format:%h %s (%an)',base_branch+'..'+head_branch]).decode('utf-8') + try: + subprocess.check_call([GIT,'merge','-q','--commit','--no-edit','--no-ff','-m',message.encode('utf-8'),head_branch]) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot be merged cleanly.",file=stderr) + subprocess.check_call([GIT,'merge','--abort']) + exit(4) + logmsg = subprocess.check_output([GIT,'log','--pretty=format:%s','-n','1']).decode('utf-8') + if logmsg.rstrip() != firstline.rstrip(): + print("ERROR: Creating merge failed (already merged?).",file=stderr) + exit(4) + + print('%s#%s%s %s %sinto %s%s' % (ATTR_RESET+ATTR_PR,pull,ATTR_RESET,title,ATTR_RESET+ATTR_PR,branch,ATTR_RESET)) + subprocess.check_call([GIT,'log','--graph','--topo-order','--pretty=format:'+COMMIT_FORMAT,base_branch+'..'+head_branch]) + print() + # Run test command if configured. + if testcmd: + # Go up to the repository's root. + toplevel = subprocess.check_output([GIT,'rev-parse','--show-toplevel']).strip() + os.chdir(toplevel) + if subprocess.call(testcmd,shell=True): + print("ERROR: Running %s failed." % testcmd,file=stderr) + exit(5) + + # Show the created merge. + diff = subprocess.check_output([GIT,'diff',merge_branch+'..'+local_merge_branch]) + subprocess.check_call([GIT,'diff',base_branch+'..'+local_merge_branch]) + if diff: + print("WARNING: merge differs from github!",file=stderr) + reply = ask_prompt("Type 'ignore' to continue.") + if reply.lower() == 'ignore': + print("Difference with github ignored.",file=stderr) + else: + exit(6) + reply = ask_prompt("Press 'd' to accept the diff.") + if reply.lower() == 'd': + print("Diff accepted.",file=stderr) + else: + print("ERROR: Diff rejected.",file=stderr) + exit(6) + else: + # Verify the result manually. + print("Dropping you on a shell so you can try building/testing the merged source.",file=stderr) + print("Run 'git diff HEAD~' to show the changes being merged.",file=stderr) + print("Type 'exit' when done.",file=stderr) + if os.path.isfile('/etc/debian_version'): # Show pull number on Debian default prompt + os.putenv('debian_chroot',pull) + subprocess.call([BASH,'-i']) + reply = ask_prompt("Type 'm' to accept the merge.") + if reply.lower() == 'm': + print("Merge accepted.",file=stderr) + else: + print("ERROR: Merge rejected.",file=stderr) + exit(7) + + # Sign the merge commit. + reply = ask_prompt("Type 's' to sign off on the merge.") + if reply == 's': + try: + subprocess.check_call([GIT,'commit','-q','--gpg-sign','--amend','--no-edit']) + except subprocess.CalledProcessError as e: + print("Error signing, exiting.",file=stderr) + exit(1) + else: + print("Not signing off on merge, exiting.",file=stderr) + exit(1) + + # Put the result in branch. + subprocess.check_call([GIT,'checkout','-q',branch]) + subprocess.check_call([GIT,'reset','-q','--hard',local_merge_branch]) + finally: + # Clean up temporary branches. + subprocess.call([GIT,'checkout','-q',branch]) + subprocess.call([GIT,'branch','-q','-D',head_branch],stderr=devnull) + subprocess.call([GIT,'branch','-q','-D',base_branch],stderr=devnull) + subprocess.call([GIT,'branch','-q','-D',merge_branch],stderr=devnull) + subprocess.call([GIT,'branch','-q','-D',local_merge_branch],stderr=devnull) + + # Push the result. + reply = ask_prompt("Type 'push' to push the result to %s, branch %s." % (host_repo,branch)) + if reply.lower() == 'push': + subprocess.check_call([GIT,'push',host_repo,'refs/heads/'+branch]) + +if __name__ == '__main__': + main() + diff --git a/contrib/verify-commits/README.md b/contrib/verify-commits/README.md new file mode 100644 index 0000000..e9e3f65 --- /dev/null +++ b/contrib/verify-commits/README.md @@ -0,0 +1,26 @@ +Tooling for verification of PGP signed commits +---------------------------------------------- + +This is an incomplete work in progress, but currently includes a pre-push hook +script (`pre-push-hook.sh`) for maintainers to ensure that their own commits +are PGP signed (nearly always merge commits), as well as a script to verify +commits against a trusted keys list. + + +Using verify-commits.sh safely +------------------------------ + +Remember that you can't use an untrusted script to verify itself. This means +that checking out code, then running `verify-commits.sh` against `HEAD` is +_not_ safe, because the version of `verify-commits.sh` that you just ran could +be backdoored. Instead, you need to use a trusted version of verify-commits +prior to checkout to make sure you're checking out only code signed by trusted +keys: + + git fetch origin && \ + ./contrib/verify-commits/verify-commits.sh origin/master && \ + git checkout origin/master + +Note that the above isn't a good UI/UX yet, and needs significant improvements +to make it more convenient and reduce the chance of errors; pull-reqs +improving this process would be much appreciated. diff --git a/contrib/verify-commits/allow-revsig-commits b/contrib/verify-commits/allow-revsig-commits new file mode 100644 index 0000000..e69de29 diff --git a/contrib/verify-commits/gpg.sh b/contrib/verify-commits/gpg.sh new file mode 100755 index 0000000..27b9e3f --- /dev/null +++ b/contrib/verify-commits/gpg.sh @@ -0,0 +1,37 @@ +#!/bin/sh +# Copyright (c) 2014-2016 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + +INPUT=$(cat /dev/stdin) +VALID=false +REVSIG=false +IFS=' +' +for LINE in $(echo "$INPUT" | gpg --trust-model always "$@" 2>/dev/null); do + case "$LINE" in + "[GNUPG:] VALIDSIG "*) + while read KEY; do + case "$LINE" in "[GNUPG:] VALIDSIG $KEY "*) VALID=true;; esac + done < ./contrib/verify-commits/trusted-keys + ;; + "[GNUPG:] REVKEYSIG "*) + [ "$BITCOIN_VERIFY_COMMITS_ALLOW_REVSIG" != 1 ] && exit 1 + while read KEY; do + case "$LINE" in "[GNUPG:] REVKEYSIG ${KEY#????????????????????????} "*) + REVSIG=true + GOODREVSIG="[GNUPG:] GOODSIG ${KEY#????????????????????????} " + esac + done < ./contrib/verify-commits/trusted-keys + ;; + esac +done +if ! $VALID; then + exit 1 +fi +if $VALID && $REVSIG; then + echo "$INPUT" | gpg --trust-model always "$@" | grep "^\[GNUPG:\] \(NEWSIG\|SIG_ID\|VALIDSIG\)" 2>/dev/null + echo "$GOODREVSIG" +else + echo "$INPUT" | gpg --trust-model always "$@" 2>/dev/null +fi diff --git a/contrib/verify-commits/pre-push-hook.sh b/contrib/verify-commits/pre-push-hook.sh new file mode 100755 index 0000000..5cd449d --- /dev/null +++ b/contrib/verify-commits/pre-push-hook.sh @@ -0,0 +1,20 @@ +#!/bin/bash +# Copyright (c) 2014-2015 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + +if ! [[ "$2" =~ ^(git@)?(www.)?github.com(:|/)devrandom/gitian-builder(.git)?$ ]]; then + exit 0 +fi + +while read LINE; do + set -- A $LINE + if [ "$4" != "refs/heads/master" ]; then + continue + fi + if ! ./contrib/verify-commits/verify-commits.sh $3 > /dev/null 2>&1; then + echo "ERROR: A commit is not signed, can't push" + ./contrib/verify-commits/verify-commits.sh + exit 1 + fi +done < /dev/stdin diff --git a/contrib/verify-commits/trusted-git-root b/contrib/verify-commits/trusted-git-root new file mode 100644 index 0000000..8048b8f --- /dev/null +++ b/contrib/verify-commits/trusted-git-root @@ -0,0 +1 @@ +bb4f92f6cbde6ee78e39ae35b0934da3b55e154d diff --git a/contrib/verify-commits/trusted-keys b/contrib/verify-commits/trusted-keys new file mode 100644 index 0000000..d3e500e --- /dev/null +++ b/contrib/verify-commits/trusted-keys @@ -0,0 +1 @@ +498FA3769A88C4AD1B187A7428EB4B0FB7AAF6B0 diff --git a/contrib/verify-commits/verify-commits.sh b/contrib/verify-commits/verify-commits.sh new file mode 100755 index 0000000..cfe4f11 --- /dev/null +++ b/contrib/verify-commits/verify-commits.sh @@ -0,0 +1,62 @@ +#!/bin/sh +# Copyright (c) 2014-2016 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + +# Not technically POSIX-compliant due to use of "local", but almost every +# shell anyone uses today supports it, so its probably fine + +DIR=$(dirname "$0") +[ "/${DIR#/}" != "$DIR" ] && DIR=$(dirname "$(pwd)/$0") + +VERIFIED_ROOT=$(cat "${DIR}/trusted-git-root") +REVSIG_ALLOWED=$(cat "${DIR}/allow-revsig-commits") + +HAVE_FAILED=false +IS_SIGNED () { + if [ $1 = $VERIFIED_ROOT ]; then + return 0; + fi + if [ "${REVSIG_ALLOWED#*$1}" != "$REVSIG_ALLOWED" ]; then + export BITCOIN_VERIFY_COMMITS_ALLOW_REVSIG=1 + else + export BITCOIN_VERIFY_COMMITS_ALLOW_REVSIG=0 + fi + if ! git -c "gpg.program=${DIR}/gpg.sh" verify-commit $1 > /dev/null 2>&1; then + return 1; + fi + local PARENTS + PARENTS=$(git show -s --format=format:%P $1) + for PARENT in $PARENTS; do + if IS_SIGNED $PARENT > /dev/null; then + return 0; + fi + done + if ! "$HAVE_FAILED"; then + echo "No parent of $1 was signed with a trusted key!" > /dev/stderr + echo "Parents are:" > /dev/stderr + for PARENT in $PARENTS; do + git show -s $PARENT > /dev/stderr + done + HAVE_FAILED=true + fi + return 1; +} + +if [ x"$1" = "x" ]; then + TEST_COMMIT="HEAD" +else + TEST_COMMIT="$1" +fi + +IS_SIGNED "$TEST_COMMIT" +RES=$? +if [ "$RES" = 1 ]; then + if ! "$HAVE_FAILED"; then + echo "$TEST_COMMIT was not signed with a trusted key!" + fi +else + echo "There is a valid path from $TEST_COMMIT to $VERIFIED_ROOT where all commits are signed!" +fi + +exit $RES diff --git a/doc/CACHE b/doc/CACHE index ea320fa..9df1b59 100644 --- a/doc/CACHE +++ b/doc/CACHE @@ -5,14 +5,17 @@ is enabled in the build descriptor. Common cache: All descriptors share this cache. It can be useful for storing fetched sources, sharing build assets between descriptors, etc. -To add or update files, copy them to ~/cache/common from the build script. Per-descriptor cache Files installed to this cache can only be seen by this descriptor. Use this to store assets created as part of the build process, to avoid having to rebuild them in future builds. -To add or update files, copy them to ~/cache/$NAME from the build script, where -$NAME is the value of the descriptor's "name" key. + +If caching is enabled, GBUILD_CACHE_ENABLED will be set to 1 in the build +script. In addition, GBUILD_PACKAGE_CACHE and GBUILD_COMMON_CACHE will be set +to their respective paths. GBUILD_PACKAGE_CACHE contains the descriptor's +"name" key as a means of separating caches. To add or update the cached files, +copy them to these paths. Before each build, all files and folders in the cache directories will be transferred to the VM. After each successful build, the caches will be diff --git a/doc/GCC_ISSUES b/doc/GCC_ISSUES deleted file mode 100644 index 103a8c9..0000000 --- a/doc/GCC_ISSUES +++ /dev/null @@ -1,39 +0,0 @@ -# The Problem - -gcc sometimes generates slightly different code with the same semantics. - -## Optimizations - -The following optimizer flags reduce non-determinism when compiling wxWidgets: - - -fno-tree-loop-optimize - -fno-trapping-math - -fno-tree-reassoc - -## Left Over - -Even with the above flags, the compiler still generates this difference in one out of 100 builds of wxWidgets: - - monolib_property.o: elf64-elf_x86_64 - - aef7: 00 - aef8: 45 31 f6 xor %r14d,%r14d - aefb: 48 8d 84 24 50 06 00 00 lea 0x650(%rsp),%rax - - af03: c6 44 24 67 00 movb $0x0,0x67(%rsp) - - af08: 48 83 c2 10 add $0x10,%rdx - - af0c: 48 83 c1 10 add $0x10,%rcx - - af10: 48 81 7c 24 08 ff ff ff cmpq $0xffffff,0x8(%rsp) - - af18: 00 - - af19: c6 44 24 47 00 movb $0x0,0x47(%rsp) - - af1e: c7 44 24 24 00 00 00 00 movl $0x0,0x24(%rsp) - + af03: c7 44 24 24 00 00 00 00 movl $0x0,0x24(%rsp) - + af0b: 48 83 c2 10 add $0x10,%rdx - + af0f: 48 83 c1 10 add $0x10,%rcx - + af13: 48 81 7c 24 08 ff ff ff cmpq $0xffffff,0x8(%rsp) - + af1b: 00 - + af1c: c6 44 24 47 00 movb $0x0,0x47(%rsp) - + af21: c6 44 24 67 00 movb $0x0,0x67(%rsp) - af26: 4c 8d bc 24 28 03 00 00 lea 0x328(%rsp),%r15 - af2e: 48 89 44 24 10 mov %rax,0x10(%rsp) - af33: 48 89 54 24 78 mov %rdx,0x78(%rsp) - diff --git a/etc/lxc.3.config.in b/etc/lxc.3.config.in new file mode 100644 index 0000000..18a8f53 --- /dev/null +++ b/etc/lxc.3.config.in @@ -0,0 +1,38 @@ +lxc.tty.max = 4 +lxc.pty.max = 1024 +lxc.rootfs.path = ROOTFS +lxc.arch = ARCH +lxc.cgroup.devices.deny = a +# /dev/null and zero +lxc.cgroup.devices.allow = c 1:3 rwm +lxc.cgroup.devices.allow = c 1:5 rwm +# consoles +lxc.cgroup.devices.allow = c 5:1 rwm +lxc.cgroup.devices.allow = c 5:0 rwm +lxc.cgroup.devices.allow = c 4:0 rwm +lxc.cgroup.devices.allow = c 4:1 rwm +# /dev/{,u}random +lxc.cgroup.devices.allow = c 1:9 rwm +lxc.cgroup.devices.allow = c 1:8 rwm +lxc.cgroup.devices.allow = c 136:* rwm +lxc.cgroup.devices.allow = c 5:2 rwm +# rtc +lxc.cgroup.devices.allow = c 254:0 rwm + +# mounts points +lxc.mount.entry=proc ROOTFS/proc proc nodev,noexec,nosuid 0 0 +lxc.mount.entry=sysfs ROOTFS/sys sysfs defaults 0 0 + +# Container with network virtualized using a pre-configured bridge named br0 and +# veth pair virtual network devices +# On the host, run: ifconfig br0 up 10.0.2.2 +# Alternatively, you can use another IP range for the bridge interface, in this case set +# the environment variables GITIAN_HOST_IP and LXC_GUEST_IP appropriately. +lxc.net.0.type = veth +lxc.net.0.flags = up +lxc.net.0.link = GUESTLINK +lxc.net.0.ipv4.address = GUESTIP/24 +lxc.net.0.ipv4.gateway = auto + +lxc.uts.name = gitian + diff --git a/etc/lxc.config.in b/etc/lxc.config.in index 74a9c82..8879f43 100644 --- a/etc/lxc.config.in +++ b/etc/lxc.config.in @@ -32,3 +32,6 @@ lxc.network.type = veth lxc.network.flags = up lxc.network.link = GUESTLINK lxc.network.ipv4 = GUESTIP/24 +lxc.network.ipv4.gateway = auto + +lxc.utsname = gitian diff --git a/libexec/config-lxc b/libexec/config-lxc index ddeabcf..e858fde 100755 --- a/libexec/config-lxc +++ b/libexec/config-lxc @@ -10,4 +10,15 @@ if [ -z "$LXC_BRIDGE" ]; then LXC_BRIDGE=br0 fi -sed "s;ROOTFS;$wd/target-$LXC_SUITE-$LXC_ARCH;;s;ARCH;$LXC_ARCH;g;;s;GUESTIP;$LXC_GUEST_IP;g;s;GUESTLINK;$LXC_BRIDGE;g" < etc/lxc.config.in > var/lxc.config +OLD_IFS=$IFS +IFS=. +VERSION=($(lxc-start --version)) +IFS=$OLD_IFS + +if [ $VERSION -ge 3 ]; then + LXC_CONFIG=etc/lxc.3.config.in +else + LXC_CONFIG=etc/lxc.config.in +fi + +sed "s;ROOTFS;$wd/target-$LXC_SUITE-$LXC_ARCH;;s;ARCH;$LXC_ARCH;g;;s;GUESTIP;$LXC_GUEST_IP;g;s;GUESTLINK;$LXC_BRIDGE;g" < $LXC_CONFIG > var/lxc.config diff --git a/libexec/copy-from-target b/libexec/copy-from-target index f82da66..02806c3 100755 --- a/libexec/copy-from-target +++ b/libexec/copy-from-target @@ -2,8 +2,8 @@ . gconfig -TUSER=ubuntu -QUIET_FLAG= +TUSER=${DISTRO:-ubuntu} +QUIET_FLAG="-vP" usage() { echo "Usage: ${0##*/} [OPTION]... " @@ -46,9 +46,13 @@ if [ $# = 0 ] ; then exit 1 fi -if [ -z "$USE_LXC" ]; then - scp $QUIET_FLAG -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_dsa -P $VM_SSH_PORT -r $TUSER@localhost:$1 $2 -else +if [ "$USE_DOCKER" = "1" ]; then + # Use tar, so that files are created with the correct owner on the host + docker exec -u $TUSER gitian-target tar -C `dirname "$1"` -cf - `basename "$1"` | tar -C "$2" -xf - +elif [ "$USE_LXC" = "1" ]; then config-lxc - sudo $LXC_EXECUTE -n gitian -f var/lxc.config -- sudo -i -u $TUSER tar -cf - "$1" | tar -C "$2" -xkf - + sudo $LXC_EXECUTE -n gitian -f var/lxc.config -- sudo -i -u $TUSER tar -C `dirname "$1"` -cf - `basename "$1"` | tar -C "$2" -xf - +else + src="${1%/}" # remove trailing / which triggers special rsync behaviour + rsync --checksum -a $QUIET_FLAG -e "ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT" "$TUSER@localhost:${src}" "$2" fi diff --git a/libexec/copy-to-target b/libexec/copy-to-target index 5ea8705..37d045f 100755 --- a/libexec/copy-to-target +++ b/libexec/copy-to-target @@ -2,8 +2,8 @@ . gconfig -TUSER=ubuntu -QUIET_FLAG= +TUSER=${DISTRO:-ubuntu} +QUIET_FLAG="-vP" usage() { echo "Usage: ${0##*/} [OPTION]... " @@ -46,9 +46,14 @@ if [ $# = 0 ] ; then exit 1 fi -if [ -z "$USE_LXC" ]; then - scp $QUIET_FLAG -r -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_dsa -P $VM_SSH_PORT $1 $TUSER@localhost:$2 -else +if [ "$USE_DOCKER" = "1" ]; then + docker exec -u $TUSER gitian-target mkdir -p "/home/$TUSER/$2" + docker cp "$1" gitian-target:"/home/$TUSER/$2" + docker exec -u root gitian-target chown -R $TUSER:$TUSER "/home/$TUSER/$2" +elif [ "$USE_LXC" = "1" ]; then config-lxc tar -C `dirname "$1"` -cf - `basename "$1"` | sudo $LXC_EXECUTE -n gitian -f var/lxc.config -- sudo -i -u $TUSER tar -C "$2" -xf - +else + src="${1%/}" # remove trailing / which triggers special rsync behaviour + rsync --checksum -a $QUIET_FLAG -e "ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT" "${src}" "$TUSER@localhost:$2" fi diff --git a/libexec/gconfig b/libexec/gconfig index fa69055..a046bf2 100644 --- a/libexec/gconfig +++ b/libexec/gconfig @@ -1,4 +1,11 @@ VM_SSH_PORT=2223 -if [ -z "$LXC_EXECUTE" ]; then - LXC_EXECUTE=lxc-start +if [ "$USE_LXC" = "1" ]; then + if [ -z "$LXC_EXECUTE" ]; then + ver=`lxc-start --version` + if dpkg --compare-versions $ver ge 1.0.0 ; then + LXC_EXECUTE=lxc-execute + else + LXC_EXECUTE=lxc-start + fi + fi fi diff --git a/libexec/make-clean-vm b/libexec/make-clean-vm index 68b7b8a..2de8377 100755 --- a/libexec/make-clean-vm +++ b/libexec/make-clean-vm @@ -1,14 +1,16 @@ #!/bin/sh set -e -SUITE=lucid +SUITE=xenial ARCH=amd64 VMSW=KVM -if [ -n "$USE_LXC" ]; then +if [ "$USE_LXC" = "1" ]; then VMSW=LXC -elif [ -n "$USE_VBOX" ]; then +elif [ "$USE_VBOX" = "1" ]; then VMSW=VBOX +elif [ "$USE_DOCKER" = "1" ]; then + VMSW=DOCKER fi usage() { @@ -17,7 +19,7 @@ usage() { echo cat << EOF --help display this help and exit - --suite U build suite U instead of lucid + --suite U build suite U instead of xenial --arch A build architecture A (e.g. i386) instead of amd64 EOF } @@ -56,14 +58,17 @@ OUT=target-$SUITE-$ARCH case $VMSW in KVM) - qemu-img create -f qcow2 -o backing_file="$BASE.qcow2" "$OUT.qcow2" + qemu-img create -f qcow2 -o backing_fmt=qcow2,backing_file="$BASE.qcow2" "$OUT.qcow2" ;; LXC) - cp -a $BASE $OUT + cp -a --sparse=always $BASE $OUT libexec/config-bootstrap-fixup on-target -u root bash < target-bin/bootstrap-fixup ;; VBOX) VBoxManage snapshot "Gitian-${SUITE}-${ARCH}" restore "Gitian-Clean" ;; + DOCKER) + true #Docker doesn't need to do anything + ;; esac diff --git a/libexec/on-target b/libexec/on-target index cf05d0a..26c066d 100755 --- a/libexec/on-target +++ b/libexec/on-target @@ -2,7 +2,7 @@ . gconfig -TUSER=ubuntu +TUSER=${DISTRO:-ubuntu} usage() { echo "Usage: ${0##*/} [OPTION]... " @@ -46,9 +46,11 @@ fi # exit 1 #fi -if [ -z "$USE_LXC" ]; then - ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_dsa -p $VM_SSH_PORT $TUSER@localhost $* -else +if [ "$USE_DOCKER" = "1" ]; then + docker exec -u $TUSER -i gitian-target $* +elif [ "$USE_LXC" = "1" ]; then config-lxc sudo $LXC_EXECUTE -n gitian -f var/lxc.config -- sudo -u $TUSER $ENV -i -- $* +else + ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT $TUSER@localhost $* fi diff --git a/libexec/start-target b/libexec/start-target index 95376cc..5e6a502 100755 --- a/libexec/start-target +++ b/libexec/start-target @@ -6,10 +6,12 @@ ARCH=qemu$1 SUFFIX=$2 VMSW=KVM -if [ -n "$USE_LXC" ]; then +if [ "$USE_LXC" = "1" ]; then VMSW=LXC -elif [ -n "$USE_VBOX" ]; then +elif [ "$USE_VBOX" = "1" ]; then VMSW=VBOX +elif [ "$USE_DOCKER" = "1" ]; then + VMSW=DOCKER fi case $VMSW in @@ -22,7 +24,7 @@ case $VMSW in [ -n "$KVM" ] || KVM=qemu-system-x86_64 ;; esac - $KVM -enable-kvm -m ${VMEM:-2000} -smp ${NPROCS:-2} -drive file=target-$SUFFIX.qcow2,cache=writeback -net nic,model=virtio -net user,hostfwd=tcp:127.0.0.1:$VM_SSH_PORT-:22 -vnc 127.0.0.1:16 > var/target.log 2>&1 & + $KVM -enable-kvm -m ${VMEM:-2000} -smp ${NPROCS:-2} -drive file=target-$SUFFIX.qcow2,cache=writeback,if=virtio -net nic,model=virtio -net user,hostfwd=tcp:127.0.0.1:$VM_SSH_PORT-:22 -vnc 127.0.0.1:16 > var/target.log 2>&1 & echo $! > var/target.pid wait rm var/target.pid @@ -31,7 +33,14 @@ case $VMSW in true #sudo lxc-start -n gitian -c var/target.log -f lxc.config ;; VBOX) - VBoxManage startvm "Gitian-${2}" # --type headless + VBoxManage startvm "Gitian-${2}" --type headless echo "Gitian-${2}" > var/target.vmname ;; + DOCKER) + EXTRA_ARGS="" + if [ -n "$GITIAN_ALLOW_PRIVILEGED" ]; then + EXTRA_ARGS="--privileged" + fi + docker run -d --name gitian-target $EXTRA_ARGS base-$SUFFIX:latest > /dev/null + ;; esac diff --git a/libexec/stop-target b/libexec/stop-target index 6db547d..169c63a 100755 --- a/libexec/stop-target +++ b/libexec/stop-target @@ -1,17 +1,19 @@ #!/bin/sh VMSW=KVM -if [ -n "$USE_LXC" ]; then +if [ "$USE_LXC" = "1" ]; then VMSW=LXC -elif [ -n "$USE_VBOX" ]; then +elif [ "$USE_VBOX" = "1" ]; then VMSW=VBOX +elif [ "$USE_DOCKER" = "1" ]; then + VMSW=DOCKER fi case $VMSW in KVM) if [ ! -e var/target.pid ]; then exit; fi - on-target -u root halt + on-target -u root poweroff sleep 5 if [ ! -e var/target.pid ]; then exit; fi @@ -30,4 +32,8 @@ case $VMSW in VBoxManage controlvm `cat var/target.vmname` savestate rm var/target.vmname ;; + DOCKER) + docker container stop gitian-target > /dev/null + docker container rm gitian-target > /dev/null + ;; esac diff --git a/target-bin/bootstrap-fixup.in b/target-bin/bootstrap-fixup.in index 1908417..3cc4da4 100755 --- a/target-bin/bootstrap-fixup.in +++ b/target-bin/bootstrap-fixup.in @@ -2,7 +2,48 @@ set -e -. /etc/lsb-release +DISTRIB_NAME=`lsb_release -is` +DISTRIB_CODENAME=`lsb_release -cs` +DISTRIB_NUMBER=`lsb_release -rs` -echo "deb http://HOSTIP:3142/archive.ubuntu.com/ubuntu $DISTRIB_CODENAME main universe" > $1/etc/apt/sources.list -echo "deb http://HOSTIP:3142/archive.ubuntu.com/ubuntu $DISTRIB_CODENAME-updates main universe" >> $1/etc/apt/sources.list +if [ $DISTRIB_NAME = "Ubuntu" ]; then + echo "deb http://HOSTIP:3142/archive.ubuntu.com/ubuntu $DISTRIB_CODENAME main universe" > $1/etc/apt/sources.list + echo "deb http://HOSTIP:3142/security.ubuntu.com/ubuntu $DISTRIB_CODENAME-security main universe" >> $1/etc/apt/sources.list + echo "deb http://HOSTIP:3142/archive.ubuntu.com/ubuntu $DISTRIB_CODENAME-updates main universe" >> $1/etc/apt/sources.list +elif [ $DISTRIB_NAME = "Debian" ]; then + echo "deb http://HOSTIP:3142/ftp.debian.org/debian $DISTRIB_CODENAME main" > $1/etc/apt/sources.list + + # This line format changed with the release of Debian 11 Buster + # https://wiki.debian.org/NewInBullseye + if [ $DISTRIB_NUMBER -ge 11 ]; then + echo "deb http://HOSTIP:3142/security.debian.org/debian-security $DISTRIB_CODENAME-security main" >> $1/etc/apt/sources.list + else + echo "deb http://HOSTIP:3142/security.debian.org/ $DISTRIB_CODENAME/updates main" >> $1/etc/apt/sources.list + fi + + echo "deb http://HOSTIP:3142/ftp.debian.org/debian $DISTRIB_CODENAME-updates main" >> $1/etc/apt/sources.list + # grub-legacy conflicts grub-pc dependencies + # No grub-legacy on Ubuntu, just on Debian + # Work around bcron-run conflict due to cron being removed + # Needed for KVM, but apparently errors out for LXC, so the true ignores the error + apt-get purge -y grub-legacy bcron-run &> /dev/null || true +fi +echo '127.0.1.1 gitian' >> /etc/hosts + +# If LXC +if grep /lxc/gitian /proc/1/cgroup > /dev/null || grep container=lxc /proc/1/environ > /dev/null; then + adduser --disabled-password --gecos ${DISTRIB_NAME,,} --quiet ${DISTRIB_NAME,,} || true + apt-get purge -y rsyslog || true + dpkg-divert --local --rename --add /sbin/initctl + ln -sf /bin/true /sbin/initctl + dpkg-divert --local --rename --add /usr/bin/ischroot + ln -sf /bin/true /usr/bin/ischroot + # Prevent interaction with init during upgrades + dpkg-divert --local --rename --add /usr/sbin/policy-rc.d + echo 'exit 101' > /usr/sbin/policy-rc.d + chmod +x /usr/sbin/policy-rc.d +## These are superceded by the policy-rc.d fix above +# for pkg in lxc cgmanager udev plymouth dmsetup upstart; do +# echo $pkg hold | dpkg --set-selections || true +# done +fi diff --git a/target-bin/grab-packages.sh b/target-bin/grab-packages.sh index adf8e34..c06e3e5 100644 --- a/target-bin/grab-packages.sh +++ b/target-bin/grab-packages.sh @@ -6,7 +6,8 @@ set -e cd /var/cache/apt/archives -#apt-get clean - -dpkg-query -W -f '${Package}\n' | xargs -n 50 apt-get install --reinstall -y -d > /dev/null +# make sure all packages with installed versions are downloaded +# (except for held packages, which may not be available for download) +dpkg-query -W -f '${Status}\t${Package}=${Version}\n' | grep -v ^hold | cut -f2- | xargs -n 50 apt-get install -q --reinstall -y -d > /tmp/download.log +grep "cannot be downloaded" /tmp/download.log && { echo Could not download some packages, please run gbuild --upgrade 1>&2 ; exit 1 ; } sha256sum *.deb | sort --key 2 diff --git a/target-bin/upgrade-system.sh b/target-bin/upgrade-system.sh new file mode 100644 index 0000000..9384229 --- /dev/null +++ b/target-bin/upgrade-system.sh @@ -0,0 +1,15 @@ +#!/bin/sh + +# Upgrade system + +set -e + +mkdir -p /var/cache/gitian + +# remove obsolete grub, it causes package dependency issues +apt-get -q -y purge grub > /dev/null 2>&1 || true + +# upgrade packages +DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade > /dev/null > /var/cache/gitian/upgrade.log 2>&1 + +touch /var/cache/gitian/initial-upgrade