diff --git a/.gitignore b/.gitignore index 043c12a..c0d0ba5 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,8 @@ base* sigs target-bin/bootstrap-fixup .vagrant +docker +*.Dockerfile +cache +target-* +!target-bin/ diff --git a/CODEOWNERS b/CODEOWNERS new file mode 100644 index 0000000..68438f5 --- /dev/null +++ b/CODEOWNERS @@ -0,0 +1 @@ +* @devrandom diff --git a/README.md b/README.md index 1558367..ee0f84d 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,7 @@ +# MAINTENANCE MODE + +Due to the move of Bitcoin Core to [Guix](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#building), this repository is switching to maintenance mode. Only serious bugs (including security issues) will be considered going forward. + # Gitian Read about the project goals at the [project home page](https://gitian.org/). @@ -18,16 +22,19 @@ This performs a build inside a VM, with deterministic inputs and outputs. If th From AUR: * [apt-cacher-ng](https://aur.archlinux.org/packages/apt-cacher-ng/) (you may have to play with permissions (chown to apt-cacher-ng) on files to get apt-cacher-ng to start) -* [debian-archive-keyring](https://aur.archlinux.org/packages/debian-archive-keyring/) (for making Debian guests) -* [debootstrap](https://aur.archlinux.org/packages/debootstrap/) +* [debootstrap](https://aur.archlinux.org/packages/debootstrap-git/) * [dpkg](https://aur.archlinux.org/packages/dpkg/) * [gnupg1](https://aur.archlinux.org/packages/gnupg1/) * [multipath-tools](https://aur.archlinux.org/packages/multipath-tools/) (for kpartx) -* [ubuntu-keyring](https://aur.archlinux.org/packages/ubuntu-keyring/) (for making Ubuntu guests) -From Launchpad: +Non-AUR packages: + +* [debian-archive-keyring](https://packages.debian.org/jessie/debian-archive-keyring) (for making Debian guests) +* [ubuntu-keyring](https://packages.ubuntu.com/search?keywords=ubuntu-keyring) (for making Ubuntu guests) -* [vmbuilder](https://launchpad.net/vmbuilder) +From newroco on GitHub: + +* [vmbuilder](https://github.com/newroco/vmbuilder) Also, I had to modify the default /etc/sudoers file to uncomment the `secure_path` line, because vmbuilder isn't found otherwise when the `env -i ... sudo vmbuilder ...` line is executed (because the i flag resets the environment variables including the PATH). @@ -48,6 +55,10 @@ If you'd like to use LXC mode instead, install it as follows: sudo apt-get install lxc +If you'd like to use docker mode instead, install it as follows: + + sudo apt-get install docker-ce + ### Debian: See Ubuntu, and also run the following on Debian Jessie or newer: @@ -72,9 +83,7 @@ Install virtualbox from http://www.virtualbox.org, and make sure `VBoxManage` is ## Debian Guests -Gitian now supports Debian guests in addition to Ubuntu guests. Note that this doesn't mean you can allow the builders to choose to use either Debian or Ubuntu guests. The person creating the Gitian descriptor will need to choose a particular distro and suite for the guest and all builders must use that particular distro and suite, otherwise the software won't reproduce for everyone. - -The official vmbuilder only includes support for Ubuntu guests, so you need to install [Joseph Bisch's fork of vmbuilder](https://github.com/josephbisch/vmbuilder), which adds a Debian plugin. +Gitian supports Debian guests in addition to Ubuntu guests. Note that this doesn't mean you can allow the builders to choose to use either Debian or Ubuntu guests. The person creating the Gitian descriptor will need to choose a particular distro and suite for the guest and all builders must use that particular distro and suite, otherwise the software won't reproduce for everyone. To create a Debian guest: @@ -82,7 +91,7 @@ To create a Debian guest: There is currently no support for LXC Debian guests. There is just KVM support. LXC support for Debian guests is planned to be added soon. -Only Debian Jessie guests have been tested with Gitian. Debian Jessie is the current stable release of Debian at this time. If you have success (or trouble) with other versions of Debian, please let us know. +Only Debian Jessie guests have been tested with Gitian. If you have success (or trouble) with other versions of Debian, please let us know. If you are creating a Gitian descriptor, you can now specify a distro. If no distro is provided, the default is to assume Ubuntu. Since Ubuntu is assumed, older Gitian descriptors that don't specify a distro will still work as they always have. @@ -103,6 +112,15 @@ Set the `USE_LXC` environment variable to use `LXC` instead of `KVM`: export USE_LXC=1 +### Docker + + bin/make-base-vm --docker + bin/make-base-vm --docker --arch i386 + +Set the `USE_DOCKER` environment variable to use `DOCKER` instead of `KVM`: + + export USE_DOCKER=1 + ### VirtualBox Command-line `VBoxManage` must be in your `$PATH`. @@ -111,12 +129,12 @@ Command-line `VBoxManage` must be in your `$PATH`. `make-base-vm` cannot yet make VirtualBox virtual machines ( _patches welcome_, it should be possible to use `VBoxManage`, boot-from-network Linux images and PXE booting to do it). So you must either get or manually create VirtualBox machines that: -1. Are named `Gitian--` -- e.g. Gitian-lucid-i386 for a 32-bit, Ubuntu 10 machine. +1. Are named `Gitian--` -- e.g. Gitian-xenial-i386 for a 32-bit, Ubuntu 16 machine. 2. Have a booted-up snapshot named `Gitian-Clean` . The build script resets the VM to that snapshot to get reproducible builds. 3. Has the VM's NAT networking setup to forward port `localhost:2223` on the host machine to port `22` of the VM; e.g.: ``` - VBoxManage modifyvm Gitian-lucid-i386 --natpf1 "guestssh,tcp,,2223,,22" + VBoxManage modifyvm Gitian-xenial-i386 --natpf1 "guestssh,tcp,,2223,,22" ``` The final setup needed is to create an `ssh` key that will be used to login to the virtual machine: @@ -140,17 +158,17 @@ Set the `USE_VBOX` environment variable to use `VBOX` instead of `KVM`: If you have everything set-up properly, you should be able to: PATH=$PATH:$(pwd)/libexec - make-clean-vm --suite lucid --arch i386 + make-clean-vm --suite xenial --arch i386 # on-target needs $DISTRO to be set to debian if using a Debian guest # (when running gbuild, $DISTRO is set based on the descriptor, so this line isn't needed) - DiSTRO=debian + DISTRO=debian # For LXC: - LXC_ARCH=i386 LXC_SUITE=lucid on-target ls -la + LXC_ARCH=i386 LXC_SUITE=xenial on-target ls -la # For KVM: - start-target 32 lucid-i386 & + start-target 32 xenial-i386 & # wait a few seconds for VM to start on-target ls -la stop-target @@ -185,7 +203,7 @@ After you've merged everybody's signatures, verify them: * Log files are captured to the _var_ directory * You can run the utilities in libexec by running `PATH="libexec:$PATH"` -* To start the target VM run `start-target 32 lucid-i386` or `start-target 64 lucid-amd64` +* To start the target VM run `start-target 32 xenial-i386` or `start-target 64 xenial-amd64` * To ssh into the target run `on-target` (after setting $DISTRO to debian if using a Debian guest) or `on-target -u root` * On the target, the _build_ directory contains the code as it is compiled and _install_ contains intermediate libraries * By convention, the script in `.yml` starts with any environment setup you would need to manually compile things on the target @@ -205,7 +223,7 @@ Right now `lxc-start` is the default, but you can force `lxc-execute` (useful fo export LXC_EXECUTE=lxc-execute -Recent distributions allow lxc-execute / lxc-start to be run by non-priviledged users, so you might be able to rip-out the `sudo` calls in `libexec/*`. +Recent distributions allow lxc-execute / lxc-start to be run by non-privileged users, so you might be able to rip-out the `sudo` calls in `libexec/*`. If you have a runaway `lxc-start` command, just use `kill -9` on it. diff --git a/RELEASE_NOTES b/RELEASE_NOTES index ed40dec..10762ed 100644 --- a/RELEASE_NOTES +++ b/RELEASE_NOTES @@ -1,3 +1,22 @@ +2019-05-23 +---------- + +- Add `--skip-fetch` argument that skips fetching the latest remote source + +2019-05-13 +---------- + +- No longer fetch repository from remote when the directory already exists + +2017-02-14 +---------- + +- VirtualBox launches are now headless. You can use the VirtualBox Manager to open the console UI if needed. +- Debian on VirtualBox is supported via Vagrant Cloud images +- Note that Debian on kvm is currently not supported because vmbuilder fails in the grub install stage +- git submodule support - any submodules are cloned and checked out +- Note that lxc-execute in Ubuntu 17.10 has a showstopper bug in stdin handling + 2015-12-12 ---------- diff --git a/Vagrantfile b/Vagrantfile index 47f6dc7..817d401 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -5,34 +5,52 @@ set -eu sudo apt-get update -y sudo apt-get upgrade -y -sudo apt-get install -y autoconf2.13 automake build-essential bsdmainutils faketime g++ g++-mingw-w64 git-core libqt4-dev libtool libz-dev mingw-w64 nsis pciutils pkg-config psmisc subversion unzip zip +sudo apt-get install -y autoconf2.13 automake build-essential bsdmainutils faketime g++ g++-mingw-w64 git libqt4-dev libtool libz-dev mingw-w64 nsis pciutils pkg-config psmisc subversion unzip zip echo "ok" SCRIPT archs = ["amd64", "i386"] -suites = ["precise", "quantal", "raring", "saucy", "trusty"] +ubuntu_suites = ["precise", "quantal", "raring", "saucy", "trusty", "xenial", "bionic"] +debian_suites = ["jessie", "stretch"] if ARGV[0] == "up" and ARGV.length == 1 puts "Specify a name of the form 'suite-architecture'" - puts " suites: " + suites.join(', ') + puts " ubuntu suites: " + ubuntu_suites.join(', ') + puts " debian suites (x86_64 only): " + debian_suites.join(', ') puts " architectures: " + archs.join(', ') Process.exit 1 end -Vagrant.configure("2") do |config| +# vagrant 1.9.1 (Ubuntu 17.10) compat +if Vagrant::DEFAULT_SERVER_URL =~ /hashicorp/ + Vagrant::DEFAULT_SERVER_URL.replace('https://vagrantcloud.com') +end +Vagrant.configure("2") do |config| config.vm.provision "shell", inline: $script config.vm.network :forwarded_port, id: "ssh", guest: 22, host: 2223 - suites.each do |suite| + debian_suites.each do |suite| + name = "#{suite}-amd64" + box = "debian/#{suite}64" + + config.vm.define name do |config| + config.vm.box = box + config.vm.provider :virtualbox do |vb| + vb.name = "Gitian-#{name}" + end + end + end + + ubuntu_suites.each do |suite| archs.each do |arch| name = "#{suite}-#{arch}" config.vm.define name do |config| config.vm.box = name - config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/#{suite}/current/#{suite}-server-cloudimg-#{arch}-vagrant-disk1.box" + config.vm.box_url = "https://cloud-images.ubuntu.com/#{suite}/current/#{suite}-server-cloudimg-#{arch}-vagrant.box" config.vm.provider :virtualbox do |vb| vb.name = "Gitian-#{name}" end diff --git a/bin/gbuild b/bin/gbuild index 483d73a..4bfc8e2 100755 --- a/bin/gbuild +++ b/bin/gbuild @@ -1,4 +1,4 @@ -#!/usr/bin/ruby +#!/usr/bin/env ruby require 'optparse' require 'yaml' @@ -10,11 +10,13 @@ require 'pathname' @bitness = { 'i386' => 32, 'amd64' => 64, + 'linux64' => 64, } @arches = { 'i386' => 'i386', 'amd64' => 'x86_64', + 'linux64' => 'linux64', } def system!(cmd) @@ -35,12 +37,13 @@ def info(str) puts str unless @options[:quiet] end -def build_one_configuration(suite, arch, build_desc, reference_datetime) +def build_one_configuration(suite, arch, build_desc) + FileUtils.rm_f("var/install.log") FileUtils.rm_f("var/build.log") bits = @bitness[arch] or raise "unknown architecture ${arch}" - if ENV["USE_LXC"] + if ENV["USE_LXC"] == "1" ENV["LXC_ARCH"] = arch ENV["LXC_SUITE"] = suite end @@ -72,6 +75,10 @@ def build_one_configuration(suite, arch, build_desc, reference_datetime) system! "on-target true" + system! "on-target -u root tee -a /etc/sudoers.d/#{ENV['DISTRO'] || 'ubuntu'} > /dev/null << EOF +%#{ENV['DISTRO'] || 'ubuntu'} ALL=(ALL) NOPASSWD: ALL +EOF" if build_desc["sudo"] and @options[:allow_sudo] + info "Preparing build environment" system! "on-target setarch #{@arches[arch]} bash < target-bin/init-build.sh" @@ -90,15 +97,45 @@ def build_one_configuration(suite, arch, build_desc, reference_datetime) end end + if build_desc["multiarch"] + info "Adding multiarch support (log in var/install.log)" + for a in build_desc["multiarch"] + system! "on-target -u root dpkg --add-architecture #{a} >> var/install.log 2>&1" + end + end + + if build_desc["repositories"] + info "Adding repositories to the sources list (log in var/install.log)" + for r in build_desc["repositories"] + system! "on-target -u root tee -a /etc/apt/sources.list >> var/install.log 2>&1 << EOF +#{r["source"]} +EOF" + end + end + info "Updating apt-get repository (log in var/install.log)" - system! "on-target -u root apt-get update > var/install.log 2>&1" + system! "on-target -u root apt-get update >> var/install.log 2>&1" info "Installing additional packages (log in var/install.log)" - system! "on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install #{build_desc["packages"].join(" ")} > var/install.log 2>&1" + system! "on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -y install #{build_desc["packages"].join(" ")} >> var/install.log 2>&1" + + if build_desc["repositories"] + for r in build_desc["repositories"] + info "Installing additional packages from repository #{r["distribution"]} (log in var/install.log)" + system! "on-target -u root -e DEBIAN_FRONTEND=noninteractive apt-get -t #{r["distribution"]} --no-install-recommends -y install #{r["packages"].join(" ")} >> var/install.log 2>&1" + end + end + + if build_desc["alternatives"] + info "Set alternatives (log in var/install.log)" + for a in build_desc["alternatives"] + system! "on-target -u root update-alternatives --set #{a["package"]} #{a["path"]} >> var/install.log 2>&1" + end + end if @options[:upgrade] || system("on-target -u root '[ ! -e /var/cache/gitian/initial-upgrade ]'") - info "Upgrading system, may take a while" - system! "on-target -u root bash < target-bin/upgrade-system.sh > var/install.log 2>&1" + info "Upgrading system, may take a while (log in var/install.log)" + system! "on-target -u root bash < target-bin/upgrade-system.sh >> var/install.log 2>&1" end info "Creating package manifest" system! "on-target -u root bash < target-bin/grab-packages.sh > var/base-#{suitearch}.manifest" @@ -119,22 +156,38 @@ def build_one_configuration(suite, arch, build_desc, reference_datetime) script.puts "GBUILD_COMMON_CACHE=$HOME/cache/common" end script.puts "MAKEOPTS=(-j#{@options[:num_procs]})" - (ref_date, ref_time) = reference_datetime.split - script.puts "REFERENCE_DATETIME='#{reference_datetime}'" - script.puts "REFERENCE_DATE='#{ref_date}'" - script.puts "REFERENCE_TIME='#{ref_time}'" + script.puts "NUM_PROCS=#{@options[:num_procs]}" + script.puts "NUM_MEM=#{@options[:memory]}" script.puts + author_date = nil build_desc["remotes"].each do |remote| dir = sanitize(remote["dir"], remote["dir"]) + + Dir.chdir("inputs/#{dir}") do + author_date = `TZ=UTC git log --date='format-local:%F %T' --format="%ad" -1`.strip + raise "error looking up author date in #{dir}" unless $?.exitstatus == 0 + end + system! "copy-to-target #{@quiet_flag} inputs/#{dir} build/" script.puts "(cd build/#{dir} && git reset -q --hard && git clean -q -f -d)" end + script.puts + ref_datetime = build_desc["reference_datetime"] || author_date + (ref_date, ref_time) = ref_datetime.split + script.puts "REFERENCE_DATETIME='#{ref_datetime}'" + script.puts "REFERENCE_DATE='#{ref_date}'" + script.puts "REFERENCE_TIME='#{ref_time}'" + script.puts script.puts "cd build" script.puts build_desc["script"] end info "Running build script (log in var/build.log)" - system! "on-target setarch #{@arches[arch]} bash -x < var/build-script > var/build.log 2>&1" + if ENV["CI"] + system! "on-target setarch #{@arches[arch]} bash -x < var/build-script 2>&1 | tee var/build.log" + else + system! "on-target setarch #{@arches[arch]} bash -x < var/build-script > var/build.log 2>&1" + end end ################################ @@ -142,6 +195,9 @@ end OptionParser.new do |opts| opts.banner = "Usage: build [options] .yml" + opts.on("--allow-sudo", "override SECURITY on the target VM and allow the use of sudo with no password for the default user") do |v| + @options[:allow_sudo] = v + end opts.on("-i", "--skip-image", "reuse current target image") do |v| @options[:skip_image] = v end @@ -166,9 +222,21 @@ OptionParser.new do |opts| opts.on("-o", "--cache-read-only", "only use existing cache files, do not update them") do |v| @options[:cache_ro] = v end + opts.on("--skip-fetch", "skip fetching the latest git objects and refs from the remote source") do |v| + @options[:skip_fetch] = v + end + opts.on("--fetch-branches", "fetch branches from the remote source") do |v| + @options[:fetch_branches] = v + end + opts.on("--fetch-tags", "fetch tags from the remote source") do |v| + @options[:fetch_tags] = v + end + opts.on("--skip-cleanup", "skip cleaning up the target VM. this may be useful for copying additional files from the target after the build") do |v| + @options[:skip_cleanup] = v + end end.parse! -if !ENV["USE_LXC"] and !File.exist?("/dev/kvm") +if ENV["USE_LXC"] != "1" and ENV["USE_DOCKER"] != "1" and ENV["USE_VBOX"] != "1" and !File.exist?("/dev/kvm") $stderr.puts "\n************* WARNING: kvm not loaded, this will probably not work out\n\n" end @@ -207,7 +275,15 @@ end distro = build_desc["distro"] || "ubuntu" suites = build_desc["suites"] or raise "must supply suites" archs = build_desc["architectures"] or raise "must supply architectures" -reference_datetime = build_desc["reference_datetime"] or raise "must supply reference_datetime" +build_desc["reference_datetime"] or build_desc["remotes"].size > 0 or raise "must supply `reference_datetime` or `remotes`" +docker_image_digests = build_desc["docker_image_digests"] || [] + +# if docker_image_digests are supplied, it must be the same length as suites +if docker_image_digests.size > 0 and suites.size != docker_image_digests.size + raise "`suites` and `docker_image_digests` must both be the same size if both are supplied" +elsif ENV["USE_DOCKER"] == "1" and docker_image_digests.size > 0 and suites.size == docker_image_digests.size + suites = docker_image_digests +end ENV['DISTRO'] = distro @@ -217,7 +293,9 @@ in_sums << desc_sum build_desc["files"].each do |filename| filename = sanitize(filename, "files section") - in_sums << `cd inputs && sha256sum #{filename}` + Dir.chdir("inputs") do + in_sums << `sha256sum #{filename}` + end end commits = {} @@ -247,14 +325,34 @@ build_desc["remotes"].each do |remote| remote["url"] = urls[remote["dir"]] end dir = sanitize(remote["dir"], remote["dir"]) + commit = sanitize(remote["commit"], remote["commit"]) unless File.exist?("inputs/#{dir}") - system!("git init inputs/#{dir}") + system!("git init inputs/#{dir}") + end + if !@options[:skip_fetch] + if @options[:fetch_branches] + system!("cd inputs/#{dir} && git fetch -f --update-head-ok #{sanitize_path(remote["url"], remote["url"])} +refs/heads/*:refs/heads/*") + end + commit_fetch = commit + if @options[:fetch_tags] + system!("cd inputs/#{dir} && git fetch -f --update-head-ok #{sanitize_path(remote["url"], remote["url"])} +refs/tags/*:refs/tags/*") + else + Dir.chdir("inputs/#{dir}") do + refinfo = `git ls-remote #{sanitize_path(remote["url"], remote["url"])} #{commit}` + if refinfo.include? "\trefs/tags/" + commit_fetch = "tag " + commit + end + end + end + system!("cd inputs/#{dir} && git fetch -f --no-tags --update-head-ok #{sanitize_path(remote["url"], remote["url"])} #{commit_fetch}") + system!("cd inputs/#{dir} && git checkout -q FETCH_HEAD") + else + system!("cd inputs/#{dir} && git checkout -q #{commit}") + end + system!("cd inputs/#{dir} && git submodule update --init --recursive --force") + Dir.chdir("inputs/#{dir}") do + commit = `git log --format=%H -1`.strip end - system!("cd inputs/#{dir} && git fetch --update-head-ok #{sanitize_path(remote["url"], remote["url"])} +refs/tags/*:refs/tags/* +refs/heads/*:refs/heads/*") - commit = sanitize(remote["commit"], remote["commit"]) - commit = `cd inputs/#{dir} && git log --format=%H -1 #{commit}`.strip - raise "error looking up commit for tag #{remote["commit"]}" unless $?.exitstatus == 0 - system!("cd inputs/#{dir} && git checkout -q #{commit}") in_sums << "git:#{commit} #{dir}" end @@ -267,7 +365,7 @@ suites.each do |suite| arch = sanitize(arch, "architecture") # Build! - build_one_configuration(suite, arch, build_desc, reference_datetime) + build_one_configuration(suite, arch, build_desc) info "Grabbing results from target" system! "copy-from-target #{@quiet_flag} out #{build_dir}" @@ -282,6 +380,11 @@ suites.each do |suite| end end +unless @options[:skip_cleanup] + info "Cleaning up target" + system "stop-target" +end + out_dir = File.join(build_dir, "out") out_sums = {} cache_common_dir = File.join(cache_dir, "common") @@ -294,7 +397,9 @@ Dir.glob(File.join(out_dir, '**', '*'), File::FNM_DOTMATCH).sort.each do |file_i next if File.directory?(file_in_out) file = file_in_out.sub(out_dir + File::SEPARATOR, '') file = sanitize_path(file, file_in_out) - out_sums[file] = `cd #{out_dir} && sha256sum #{file}` + Dir.chdir(out_dir) do + out_sums[file] = `sha256sum #{file}` + end raise "failed to sum #{file}" unless $? == 0 puts out_sums[file] unless @options[:quiet] end @@ -304,7 +409,9 @@ if enable_cache next if File.directory?(file_in_out) file = file_in_out.sub(cache_common_dir + File::SEPARATOR, '') file = sanitize_path(file, file_in_out) - cache_common_sums[file] = `cd #{cache_common_dir} && sha256sum #{file}` + Dir.chdir(cache_common_dir) do + cache_common_sums[file] = `sha256sum #{file}` + end raise "failed to sum #{file}" unless $? == 0 end @@ -312,7 +419,9 @@ if enable_cache next if File.directory?(file_in_out) file = file_in_out.sub(cache_package_dir + File::SEPARATOR, '') file = sanitize_path(file, file_in_out) - cache_package_sums[file] = `cd #{cache_package_dir} && sha256sum #{file}` + Dir.chdir(cache_package_dir) do + cache_package_sums[file] = `sha256sum #{file}` + end raise "failed to sum #{file}" unless $? == 0 end end diff --git a/bin/gsign b/bin/gsign index a03c810..a560c23 100755 --- a/bin/gsign +++ b/bin/gsign @@ -1,4 +1,4 @@ -#!/usr/bin/ruby +#!/usr/bin/env ruby require 'optparse' require 'yaml' @@ -67,7 +67,7 @@ package_name = sanitize(package_name, "package name") result_file = "#{package_name}-res.yml" result_path = File.join(result_dir, result_file) -File.exists?(result_path) or raise "#{result_path} does not exist" +File.exist?(result_path) or raise "#{result_path} does not exist" result = YAML.load_file(result_path) destination = @options[:destination] || File.join(base_dir, "sigs", package_name) diff --git a/bin/gverify b/bin/gverify index 5eab37b..63bbded 100755 --- a/bin/gverify +++ b/bin/gverify @@ -1,10 +1,12 @@ -#!/usr/bin/ruby +#!/usr/bin/env ruby require 'optparse' require 'yaml' require 'fileutils' require 'pathname' +bold = ["\033[0m", "\033[1m"] + @options = {} def system!(cmd) @@ -17,7 +19,7 @@ def sanitize(str, where) end def sanitize_path(str, where) - raise "unsanitary string in #{where}" if (str =~ /[^@\w\/.:+-]/) + raise "unsanitary string in #{where}" if (str =~ /[^@\w\\ '\/.:+-]/) str end @@ -33,6 +35,11 @@ OptionParser.new do |opts| opts.on("-v", "--verbose", "be more verbose") do |v| @options[:verbose] = v end + @options[:markup] = true + opts.on("-m", "--[no-]markup", "markup the output using ANSI escape codes") do |m| + @options[:markup] = m + end + opts.on("-r REL", "--release REL", "release name") do |v| @options[:release] = v end @@ -43,6 +50,10 @@ OptionParser.new do |opts| opts.on("-c SIGNER", "--compare-to SIGNER", "compare other manifests to SIGNER's, if not given pick first") do |v| @options[:compareto] = v end + + opts.on("-p PROG", "--verify-program PROG", "specify verification program to use (default is gpg)") do |v| + @options[:program] = v + end end.parse! base_dir = Pathname.new(__FILE__).expand_path.dirname.parent @@ -62,10 +73,13 @@ destination = @options[:destination] || File.join(base_dir, "sigs", package_name release = @options[:release] || "current" release = sanitize(release, "release") verbose = @options[:verbose] +bold = ['', ''] unless @options[:markup] + +program = @options[:program] || "gpg" release_path = File.join(destination, release) -File.exists?(release_path) or raise "#{release_path} does not exist" +File.exist?(release_path) or raise "#{release_path} does not exist" result_file = "#{package_name}-build.assert" sig_file = "#{result_file}.sig" @@ -99,8 +113,8 @@ Dir.foreach(release_path) do |signer_dir| end result = YAML.load_file(result_path) - system("gpg --keyserver pgp.mit.edu --recv-keys `gpg --quiet --batch --verify \"#{File.join(signer_path, 'signature.pgp')}\" \"#{result_path}\" 2>&1 | head -n1 | grep \"key ID\" | awk '{ print $15 }'` > /dev/null 2>&1") - out = `gpg --quiet --batch --verify \"#{sig_path}\" \"#{result_path}\" 2>&1` + system("#{program} --keyserver pgp.mit.edu --recv-keys `#{program} --quiet --batch --verify \"#{File.join(signer_path, 'signature.pgp')}\" \"#{result_path}\" 2>&1 | head -n1 | grep \"key ID\" | awk '{ print $15 }'` > /dev/null 2>&1") + out = `#{program} --quiet --batch --verify \"#{sig_path}\" \"#{result_path}\" 2>&1` if $? != 0 out.each_line do |line| if line =~ /^gpg: Signature made/ @@ -109,7 +123,8 @@ Dir.foreach(release_path) do |signer_dir| puts line end end - puts "#{signer_dir}: BAD SIGNATURE" + puts "#{bold[1]}#{signer_dir}: BAD SIGNATURE#{bold[0]}" + puts did_fail = true elsif current_manifest and (result['out_manifest'] != current_manifest or result['release'] != release or result['name'] != package_name) out.each_line do |line| @@ -123,7 +138,8 @@ Dir.foreach(release_path) do |signer_dir| puts line end end - puts "#{signer_dir}: MISMATCH" + puts "#{bold[1]}#{signer_dir}: MISMATCH#{bold[0]}" + puts if verbose lines1 = current_manifest.each_line lines2 = result['out_manifest'].each_line @@ -147,7 +163,8 @@ Dir.foreach(release_path) do |signer_dir| puts line end end - puts "#{signer_dir}: OK" + puts "#{bold[1]}#{signer_dir}: OK#{bold[0]}" + puts end if !current_manifest # take first manifest as 'current' to compare against diff --git a/bin/make-base-vm b/bin/make-base-vm index 5376baa..30e4fbf 100755 --- a/bin/make-base-vm +++ b/bin/make-base-vm @@ -2,29 +2,54 @@ set -e DISTRO=ubuntu -SUITE=lucid +SUITE=xenial ARCH=amd64 -MIRROR_BASE=http://${MIRROR_HOST:-127.0.0.1}:3142 +DISKSIZE=12287 LXC=0 VBOX=0 +DOCKER=0 +DOCKER_IMAGE_HASH="" usage() { echo "Usage: ${0##*/} [OPTION]..." echo "Make a base client." echo cat << EOF - --help display this help and exit - --distro D build distro D (e.g. debian) instead of ubuntu - --suite U build suite U instead of lucid - --arch A build architecture A (e.g. i386) instead of amd64 - --lxc use lxc instead of kvm - --vbox use VirtualBox instead of kvm + --help display this help and exit + --distro D build distro D (e.g. debian) instead of ubuntu + --suite U build suite U instead of xenial + --arch A build architecture A (e.g. i386) instead of amd64 + --disksize S disk/image size S in MB (default 12287) + --lxc use lxc instead of kvm + --vbox use VirtualBox instead of kvm + --docker use docker instead of kvm + --docker-image-hash D digest of the docker image to build from The MIRROR_HOST environment variable can be used to change the apt-cacher host. It should be something that both the host and the target VM can reach. It may be set to 127.0.0.1, in which case it will be changed to 10.0.2.2 on the guest (or GITIAN_HOST_IP if it is defined) 10.0.2.2 is the host IP as visible from the guest under qemu networking. + + The DEBOOTSTRAP_DIR (but also GITIAN_SUDO_USE_DEBOOTSTRAP_DIR, see below!) + environment variable can be set to select a directory + that will contain data like in "/usr/share/debootstrap/". This allows user to + make a copy of this files to some local dir and modify them locally: + e.g. set env variable "DEBOOTSTRAP_DIR=./mydeboot/", then copy or link + system's version of files there, and modify them there + (e.g. copy your debootstrap-script file "xenial" to "./mydeboot/scripts/"). + + Set env GITIAN_SUDO_USE_DEBOOTSTRAP_DIR="yes" to allow sudo for debootstrap + to use flags like --preserve-env that are required for DEBOOTSTRAP_DIR to work. + It must be equal string "yes". + This is done as separate variable to make it clear that we modify sudo + behaviour here regarding security (though anyway env is cleared with + whitelist so should be perfectly safe). + + The --docker-image-hash option can be used to specify the hash of a particular + base image to use. These hashes can be found under the "RepoDigests" field of + "docker image inspect ". They will be reported in the form "sha256:"; + only need the part is needed EOF } @@ -47,6 +72,10 @@ if [ $# != 0 ] ; then ARCH="$2" shift 2 ;; + --disksize) + DISKSIZE="$2" + shift 2 + ;; --lxc) LXC=1 shift 1 @@ -55,6 +84,14 @@ if [ $# != 0 ] ; then VBOX=1 shift 1 ;; + --docker) + DOCKER=1 + shift 1 + ;; + --docker-image-digest) + DOCKER_IMAGE_HASH="$2" + shift 2 + ;; --*) echo "unrecognized option $1" exit 1 @@ -66,17 +103,21 @@ if [ $# != 0 ] ; then done fi -if [ $DISTRO = "debian" -a $LXC = "1" ]; then - echo "There is no support for Debian guests using LXC currently. Please use KVM or another distro for now." - exit 1 +if [ $DOCKER = "1" ]; then + MIRROR_DEFAULT=172.17.0.1 +else + MIRROR_DEFAULT=127.0.0.1 fi +MIRROR_BASE=http://${MIRROR_HOST:-$MIRROR_DEFAULT}:3142 if [ $DISTRO = "ubuntu" ]; then MIRROR=$MIRROR_BASE/archive.ubuntu.com/ubuntu SECURITY_MIRROR=$MIRROR_BASE/security.ubuntu.com/ubuntu + components=main,universe elif [ $DISTRO = "debian" ]; then MIRROR=$MIRROR_BASE/ftp.debian.org/debian SECURITY_MIRROR=$MIRROR_BASE/security.debian.org/ + components=main,contrib fi mkdir -p var @@ -100,12 +141,13 @@ elif [ $DISTRO = "debian" ]; then FLAVOUR=686-pae fi + LOCALE_PKG=language-pack-en if [ $DISTRO = "debian" ]; then LOCALE_PKG=locales fi -addpkg=pciutils,build-essential,git-core,subversion,$LOCALE_PKG,wget,lsb-release +addpkg=pciutils,build-essential,git,subversion,$LOCALE_PKG,wget,lsb-release,sudo if [ $DISTRO = "ubuntu" ]; then # Need comma at end to work around an issue with apt for Debian <= Wheezy regarding empty strings @@ -137,6 +179,40 @@ fi # Remove cron to work around vmbuilder issue when umounting /dev on target removepkg=cron +if [ $DOCKER = "1" ]; then + + addpkg=`echo $addpkg | tr ',' ' '` + + mkdir -p docker + cd docker + + if [ -n "$DOCKER_IMAGE_HASH" ]; then + base_image="$DISTRO@sha256:$DOCKER_IMAGE_HASH" + OUT=base-$DOCKER_IMAGE_HASH-$ARCH + else + base_image="$DISTRO:$SUITE" + fi + + # Generate the dockerfile + cat << EOF > $OUT.Dockerfile +FROM $base_image + +ENV DEBIAN_FRONTEND=noninteractive +RUN echo 'Acquire::http { Proxy "$MIRROR_BASE"; };' > /etc/apt/apt.conf.d/50cacher +RUN apt-get update && apt-get --no-install-recommends -y install $addpkg + +RUN useradd -ms /bin/bash -U $DISTRO +USER $DISTRO:$DISTRO +WORKDIR /home/$DISTRO + +CMD ["sleep", "infinity"] +EOF + + docker build --pull -f $OUT.Dockerfile -t $OUT . + + exit 0 +fi + if [ $VBOX = "1" ]; then NAME="$SUITE-$ARCH" if ! vagrant status | grep "$NAME" | grep "not created" > /dev/null; then @@ -144,7 +220,16 @@ if [ $VBOX = "1" ]; then exit 1 fi + DISTRO_USER_CREATE=0 + if [ $DISTRO = "debian" ]; then + # we use a vagrant provider + DISTRO_USER_CREATE=1 + fi + vagrant up "$NAME" + if [ $DISTRO_USER_CREATE = "1" ]; then + vagrant ssh "$NAME" -c "sudo useradd -m -s /bin/bash $DISTRO" + fi vagrant ssh "$NAME" -c "sudo mkdir -p /root/.ssh && sudo chmod 700 /root/.ssh" vagrant ssh "$NAME" -c "sudo sh -c 'cat >> /root/.ssh/authorized_keys'" < var/id_rsa.pub @@ -164,8 +249,32 @@ if [ $LXC = "1" ]; then fi sudo rm -rf $OUT-bootstrap # Need universe for lxc in lucid - env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 sudo debootstrap --arch=$ARCH --include=$addpkg --exclude=$removepkg --components=main,universe $SUITE $OUT-bootstrap $MIRROR - dd if=/dev/zero of=$OUT-lxc bs=1M count=1 seek=10240 + + unset preserve_env + if [ "$GITIAN_SUDO_USE_DEBOOTSTRAP_DIR" = "yes" ]; then + echo "sudo will preserve (some) env flags" + preserve_env=yes # if you would want to set false then unset this variable + fi + env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 DEBOOTSTRAP_DIR="$DEBOOTSTRAP_DIR" sudo ${preserve_env+--preserve-env} debootstrap --arch=$ARCH --include=$addpkg --exclude=$removepkg --components=$components $SUITE $OUT-bootstrap $MIRROR + # Fix lxc issue + if [ -f $OUT-bootstrap/usr/lib/lxc/lxc-init ] + then + sudo cp $OUT-bootstrap/usr/lib/lxc/lxc-init $OUT-bootstrap/usr/sbin/init.lxc + else + if [ $ARCH = "amd64" ] + then + if [ -f $OUT-bootstrap/usr/lib/x86_64-linux-gnu/lxc/lxc-init ] + then + sudo cp $OUT-bootstrap/usr/lib/x86_64-linux-gnu/lxc/lxc-init $OUT-bootstrap/usr/sbin/init.lxc + fi + else + if [ -f $OUT-bootstrap/usr/lib/i386-linux-gnu/lxc/lxc-init ] + then + sudo cp $OUT-bootstrap/usr/lib/i386-linux-gnu/lxc/lxc-init $OUT-bootstrap/usr/sbin/init.lxc + fi + fi + fi + dd if=/dev/zero of=$OUT-lxc bs=1M count=1 seek=$DISKSIZE /sbin/mkfs.ext4 -F $OUT-lxc t=`mktemp -d gitian.XXXXXXXX` sudo mount $OUT-lxc $t @@ -184,7 +293,7 @@ else libexec/config-bootstrap-fixup rm -rf $OUT - env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 sudo vmbuilder kvm $DISTRO --rootsize 10240 --arch=$ARCH --suite=$SUITE --addpkg=$addpkg --removepkg=$removepkg --ssh-key=var/id_rsa.pub --ssh-user-key=var/id_rsa.pub --mirror=$MIRROR --security-mirror=$SECURITY_MIRROR --dest=$OUT --flavour=$FLAVOUR --firstboot=`pwd`/target-bin/bootstrap-fixup + env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 sudo vmbuilder kvm $DISTRO --rootsize $DISKSIZE --arch=$ARCH --suite=$SUITE --addpkg=$addpkg --removepkg=$removepkg --ssh-key=var/id_rsa.pub --ssh-user-key=var/id_rsa.pub --mirror=$MIRROR --security-mirror=$SECURITY_MIRROR --dest=$OUT --flavour=$FLAVOUR --firstboot=`pwd`/target-bin/bootstrap-fixup mv $OUT/*.qcow2 $OUT.qcow2 rm -rf $OUT # bootstrap-fixup is done on first boot diff --git a/contrib/README.md b/contrib/README.md new file mode 100644 index 0000000..b313986 --- /dev/null +++ b/contrib/README.md @@ -0,0 +1,9 @@ +Repository Tools +--------------------- + +### [Developer tools](/contrib/devtools) ### +Specific tools for developers working on this repository. +Contains the script `github-merge.py` for merging github pull requests securely and signing them using GPG. + +### [Verify-Commits](/contrib/verify-commits) ### +Tool to verify that every merge commit was signed by a developer using the above `github-merge.py` script. diff --git a/contrib/devtools/README.md b/contrib/devtools/README.md new file mode 100644 index 0000000..1af476c --- /dev/null +++ b/contrib/devtools/README.md @@ -0,0 +1,37 @@ +Contents +======== +This directory contains tools for developers working on this repository. + +github-merge.py +=============== + +A small script to automate merging pull-requests securely and sign them with GPG. + +For example: + + ./github-merge.py 3077 + +(in any git repository) will help you merge pull request #3077 for the +devrandom/gitian-builder repository. + +What it does: +* Fetch master and the pull request. +* Locally construct a merge commit. +* Show the diff that merge results in. +* Ask you to verify the resulting source tree (so you can do a make +check or whatever). +* Ask you whether to GPG sign the merge commit. +* Ask you whether to push the result upstream. + +This means that there are no potential race conditions (where a +pullreq gets updated while you're reviewing it, but before you click +merge), and when using GPG signatures, that even a compromised github +couldn't mess with the sources. + +Setup +--------- +Configuring the github-merge tool for this repository is done in the following way: + + git config githubmerge.repository devrandom/gitian-builder + git config githubmerge.testcmd "make -j4 check" (adapt to whatever you want to use for testing) + git config --global user.signingkey mykeyid (if you want to GPG sign) diff --git a/contrib/devtools/github-merge.py b/contrib/devtools/github-merge.py new file mode 100755 index 0000000..f82362f --- /dev/null +++ b/contrib/devtools/github-merge.py @@ -0,0 +1,251 @@ +#!/usr/bin/env python3 +# Copyright (c) 2016 Bitcoin Core Developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + +# This script will locally construct a merge commit for a pull request on a +# github repository, inspect it, sign it and optionally push it. + +# The following temporary branches are created/overwritten and deleted: +# * pull/$PULL/base (the current master we're merging onto) +# * pull/$PULL/head (the current state of the remote pull request) +# * pull/$PULL/merge (github's merge) +# * pull/$PULL/local-merge (our merge) + +# In case of a clean merge that is accepted by the user, the local branch with +# name $BRANCH is overwritten with the merged result, and optionally pushed. +from __future__ import division,print_function,unicode_literals +import os,sys +from sys import stdin,stdout,stderr +import argparse +import subprocess +import json,codecs +try: + from urllib.request import Request,urlopen +except: + from urllib2 import Request,urlopen + +# External tools (can be overridden using environment) +GIT = os.getenv('GIT','git') +BASH = os.getenv('BASH','bash') + +# OS specific configuration for terminal attributes +ATTR_RESET = '' +ATTR_PR = '' +COMMIT_FORMAT = '%h %s (%an)%d' +if os.name == 'posix': # if posix, assume we can use basic terminal escapes + ATTR_RESET = '\033[0m' + ATTR_PR = '\033[1;36m' + COMMIT_FORMAT = '%C(bold blue)%h%Creset %s %C(cyan)(%an)%Creset%C(green)%d%Creset' + +def git_config_get(option, default=None): + ''' + Get named configuration option from git repository. + ''' + try: + return subprocess.check_output([GIT,'config','--get',option]).rstrip().decode('utf-8') + except subprocess.CalledProcessError as e: + return default + +def retrieve_pr_info(repo,pull): + ''' + Retrieve pull request information from github. + Return None if no title can be found, or an error happens. + ''' + try: + req = Request("https://api.github.com/repos/"+repo+"/pulls/"+pull) + result = urlopen(req) + reader = codecs.getreader('utf-8') + obj = json.load(reader(result)) + return obj + except Exception as e: + print('Warning: unable to retrieve pull information from github: %s' % e) + return None + +def ask_prompt(text): + print(text,end=" ",file=stderr) + stderr.flush() + reply = stdin.readline().rstrip() + print("",file=stderr) + return reply + +def parse_arguments(): + epilog = ''' + In addition, you can set the following git configuration variables: + githubmerge.repository (mandatory), + user.signingkey (mandatory), + githubmerge.host (default: git@github.com), + githubmerge.branch (no default), + githubmerge.testcmd (default: none). + ''' + parser = argparse.ArgumentParser(description='Utility to merge, sign and push github pull requests', + epilog=epilog) + parser.add_argument('pull', metavar='PULL', type=int, nargs=1, + help='Pull request ID to merge') + parser.add_argument('branch', metavar='BRANCH', type=str, nargs='?', + default=None, help='Branch to merge against (default: githubmerge.branch setting, or base branch for pull, or \'master\')') + return parser.parse_args() + +def main(): + # Extract settings from git repo + repo = git_config_get('githubmerge.repository') + host = git_config_get('githubmerge.host','git@github.com') + opt_branch = git_config_get('githubmerge.branch',None) + testcmd = git_config_get('githubmerge.testcmd') + signingkey = git_config_get('user.signingkey') + if repo is None: + print("ERROR: No repository configured. Use this command to set:", file=stderr) + print("git config githubmerge.repository /", file=stderr) + exit(1) + if signingkey is None: + print("ERROR: No GPG signing key set. Set one using:",file=stderr) + print("git config --global user.signingkey ",file=stderr) + exit(1) + + host_repo = host+":"+repo # shortcut for push/pull target + + # Extract settings from command line + args = parse_arguments() + pull = str(args.pull[0]) + + # Receive pull information from github + info = retrieve_pr_info(repo,pull) + if info is None: + exit(1) + title = info['title'] + # precedence order for destination branch argument: + # - command line argument + # - githubmerge.branch setting + # - base branch for pull (as retrieved from github) + # - 'master' + branch = args.branch or opt_branch or info['base']['ref'] or 'master' + + # Initialize source branches + head_branch = 'pull/'+pull+'/head' + base_branch = 'pull/'+pull+'/base' + merge_branch = 'pull/'+pull+'/merge' + local_merge_branch = 'pull/'+pull+'/local-merge' + + devnull = open(os.devnull,'w') + try: + subprocess.check_call([GIT,'checkout','-q',branch]) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot check out branch %s." % (branch), file=stderr) + exit(3) + try: + subprocess.check_call([GIT,'fetch','-q',host_repo,'+refs/pull/'+pull+'/*:refs/heads/pull/'+pull+'/*']) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot find pull request #%s on %s." % (pull,host_repo), file=stderr) + exit(3) + try: + subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+head_branch], stdout=devnull, stderr=stdout) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot find head of pull request #%s on %s." % (pull,host_repo), file=stderr) + exit(3) + try: + subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+merge_branch], stdout=devnull, stderr=stdout) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot find merge of pull request #%s on %s." % (pull,host_repo), file=stderr) + exit(3) + try: + subprocess.check_call([GIT,'fetch','-q',host_repo,'+refs/heads/'+branch+':refs/heads/'+base_branch]) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot find branch %s on %s." % (branch,host_repo), file=stderr) + exit(3) + subprocess.check_call([GIT,'checkout','-q',base_branch]) + subprocess.call([GIT,'branch','-q','-D',local_merge_branch], stderr=devnull) + subprocess.check_call([GIT,'checkout','-q','-b',local_merge_branch]) + + try: + # Create unsigned merge commit. + if title: + firstline = 'Merge #%s: %s' % (pull,title) + else: + firstline = 'Merge #%s' % (pull,) + message = firstline + '\n\n' + message += subprocess.check_output([GIT,'log','--no-merges','--topo-order','--pretty=format:%h %s (%an)',base_branch+'..'+head_branch]).decode('utf-8') + try: + subprocess.check_call([GIT,'merge','-q','--commit','--no-edit','--no-ff','-m',message.encode('utf-8'),head_branch]) + except subprocess.CalledProcessError as e: + print("ERROR: Cannot be merged cleanly.",file=stderr) + subprocess.check_call([GIT,'merge','--abort']) + exit(4) + logmsg = subprocess.check_output([GIT,'log','--pretty=format:%s','-n','1']).decode('utf-8') + if logmsg.rstrip() != firstline.rstrip(): + print("ERROR: Creating merge failed (already merged?).",file=stderr) + exit(4) + + print('%s#%s%s %s %sinto %s%s' % (ATTR_RESET+ATTR_PR,pull,ATTR_RESET,title,ATTR_RESET+ATTR_PR,branch,ATTR_RESET)) + subprocess.check_call([GIT,'log','--graph','--topo-order','--pretty=format:'+COMMIT_FORMAT,base_branch+'..'+head_branch]) + print() + # Run test command if configured. + if testcmd: + # Go up to the repository's root. + toplevel = subprocess.check_output([GIT,'rev-parse','--show-toplevel']).strip() + os.chdir(toplevel) + if subprocess.call(testcmd,shell=True): + print("ERROR: Running %s failed." % testcmd,file=stderr) + exit(5) + + # Show the created merge. + diff = subprocess.check_output([GIT,'diff',merge_branch+'..'+local_merge_branch]) + subprocess.check_call([GIT,'diff',base_branch+'..'+local_merge_branch]) + if diff: + print("WARNING: merge differs from github!",file=stderr) + reply = ask_prompt("Type 'ignore' to continue.") + if reply.lower() == 'ignore': + print("Difference with github ignored.",file=stderr) + else: + exit(6) + reply = ask_prompt("Press 'd' to accept the diff.") + if reply.lower() == 'd': + print("Diff accepted.",file=stderr) + else: + print("ERROR: Diff rejected.",file=stderr) + exit(6) + else: + # Verify the result manually. + print("Dropping you on a shell so you can try building/testing the merged source.",file=stderr) + print("Run 'git diff HEAD~' to show the changes being merged.",file=stderr) + print("Type 'exit' when done.",file=stderr) + if os.path.isfile('/etc/debian_version'): # Show pull number on Debian default prompt + os.putenv('debian_chroot',pull) + subprocess.call([BASH,'-i']) + reply = ask_prompt("Type 'm' to accept the merge.") + if reply.lower() == 'm': + print("Merge accepted.",file=stderr) + else: + print("ERROR: Merge rejected.",file=stderr) + exit(7) + + # Sign the merge commit. + reply = ask_prompt("Type 's' to sign off on the merge.") + if reply == 's': + try: + subprocess.check_call([GIT,'commit','-q','--gpg-sign','--amend','--no-edit']) + except subprocess.CalledProcessError as e: + print("Error signing, exiting.",file=stderr) + exit(1) + else: + print("Not signing off on merge, exiting.",file=stderr) + exit(1) + + # Put the result in branch. + subprocess.check_call([GIT,'checkout','-q',branch]) + subprocess.check_call([GIT,'reset','-q','--hard',local_merge_branch]) + finally: + # Clean up temporary branches. + subprocess.call([GIT,'checkout','-q',branch]) + subprocess.call([GIT,'branch','-q','-D',head_branch],stderr=devnull) + subprocess.call([GIT,'branch','-q','-D',base_branch],stderr=devnull) + subprocess.call([GIT,'branch','-q','-D',merge_branch],stderr=devnull) + subprocess.call([GIT,'branch','-q','-D',local_merge_branch],stderr=devnull) + + # Push the result. + reply = ask_prompt("Type 'push' to push the result to %s, branch %s." % (host_repo,branch)) + if reply.lower() == 'push': + subprocess.check_call([GIT,'push',host_repo,'refs/heads/'+branch]) + +if __name__ == '__main__': + main() + diff --git a/contrib/verify-commits/README.md b/contrib/verify-commits/README.md new file mode 100644 index 0000000..e9e3f65 --- /dev/null +++ b/contrib/verify-commits/README.md @@ -0,0 +1,26 @@ +Tooling for verification of PGP signed commits +---------------------------------------------- + +This is an incomplete work in progress, but currently includes a pre-push hook +script (`pre-push-hook.sh`) for maintainers to ensure that their own commits +are PGP signed (nearly always merge commits), as well as a script to verify +commits against a trusted keys list. + + +Using verify-commits.sh safely +------------------------------ + +Remember that you can't use an untrusted script to verify itself. This means +that checking out code, then running `verify-commits.sh` against `HEAD` is +_not_ safe, because the version of `verify-commits.sh` that you just ran could +be backdoored. Instead, you need to use a trusted version of verify-commits +prior to checkout to make sure you're checking out only code signed by trusted +keys: + + git fetch origin && \ + ./contrib/verify-commits/verify-commits.sh origin/master && \ + git checkout origin/master + +Note that the above isn't a good UI/UX yet, and needs significant improvements +to make it more convenient and reduce the chance of errors; pull-reqs +improving this process would be much appreciated. diff --git a/contrib/verify-commits/allow-revsig-commits b/contrib/verify-commits/allow-revsig-commits new file mode 100644 index 0000000..e69de29 diff --git a/contrib/verify-commits/gpg.sh b/contrib/verify-commits/gpg.sh new file mode 100755 index 0000000..27b9e3f --- /dev/null +++ b/contrib/verify-commits/gpg.sh @@ -0,0 +1,37 @@ +#!/bin/sh +# Copyright (c) 2014-2016 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + +INPUT=$(cat /dev/stdin) +VALID=false +REVSIG=false +IFS=' +' +for LINE in $(echo "$INPUT" | gpg --trust-model always "$@" 2>/dev/null); do + case "$LINE" in + "[GNUPG:] VALIDSIG "*) + while read KEY; do + case "$LINE" in "[GNUPG:] VALIDSIG $KEY "*) VALID=true;; esac + done < ./contrib/verify-commits/trusted-keys + ;; + "[GNUPG:] REVKEYSIG "*) + [ "$BITCOIN_VERIFY_COMMITS_ALLOW_REVSIG" != 1 ] && exit 1 + while read KEY; do + case "$LINE" in "[GNUPG:] REVKEYSIG ${KEY#????????????????????????} "*) + REVSIG=true + GOODREVSIG="[GNUPG:] GOODSIG ${KEY#????????????????????????} " + esac + done < ./contrib/verify-commits/trusted-keys + ;; + esac +done +if ! $VALID; then + exit 1 +fi +if $VALID && $REVSIG; then + echo "$INPUT" | gpg --trust-model always "$@" | grep "^\[GNUPG:\] \(NEWSIG\|SIG_ID\|VALIDSIG\)" 2>/dev/null + echo "$GOODREVSIG" +else + echo "$INPUT" | gpg --trust-model always "$@" 2>/dev/null +fi diff --git a/contrib/verify-commits/pre-push-hook.sh b/contrib/verify-commits/pre-push-hook.sh new file mode 100755 index 0000000..5cd449d --- /dev/null +++ b/contrib/verify-commits/pre-push-hook.sh @@ -0,0 +1,20 @@ +#!/bin/bash +# Copyright (c) 2014-2015 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + +if ! [[ "$2" =~ ^(git@)?(www.)?github.com(:|/)devrandom/gitian-builder(.git)?$ ]]; then + exit 0 +fi + +while read LINE; do + set -- A $LINE + if [ "$4" != "refs/heads/master" ]; then + continue + fi + if ! ./contrib/verify-commits/verify-commits.sh $3 > /dev/null 2>&1; then + echo "ERROR: A commit is not signed, can't push" + ./contrib/verify-commits/verify-commits.sh + exit 1 + fi +done < /dev/stdin diff --git a/contrib/verify-commits/trusted-git-root b/contrib/verify-commits/trusted-git-root new file mode 100644 index 0000000..8048b8f --- /dev/null +++ b/contrib/verify-commits/trusted-git-root @@ -0,0 +1 @@ +bb4f92f6cbde6ee78e39ae35b0934da3b55e154d diff --git a/contrib/verify-commits/trusted-keys b/contrib/verify-commits/trusted-keys new file mode 100644 index 0000000..d3e500e --- /dev/null +++ b/contrib/verify-commits/trusted-keys @@ -0,0 +1 @@ +498FA3769A88C4AD1B187A7428EB4B0FB7AAF6B0 diff --git a/contrib/verify-commits/verify-commits.sh b/contrib/verify-commits/verify-commits.sh new file mode 100755 index 0000000..cfe4f11 --- /dev/null +++ b/contrib/verify-commits/verify-commits.sh @@ -0,0 +1,62 @@ +#!/bin/sh +# Copyright (c) 2014-2016 The Bitcoin Core developers +# Distributed under the MIT software license, see the accompanying +# file COPYING or http://www.opensource.org/licenses/mit-license.php. + +# Not technically POSIX-compliant due to use of "local", but almost every +# shell anyone uses today supports it, so its probably fine + +DIR=$(dirname "$0") +[ "/${DIR#/}" != "$DIR" ] && DIR=$(dirname "$(pwd)/$0") + +VERIFIED_ROOT=$(cat "${DIR}/trusted-git-root") +REVSIG_ALLOWED=$(cat "${DIR}/allow-revsig-commits") + +HAVE_FAILED=false +IS_SIGNED () { + if [ $1 = $VERIFIED_ROOT ]; then + return 0; + fi + if [ "${REVSIG_ALLOWED#*$1}" != "$REVSIG_ALLOWED" ]; then + export BITCOIN_VERIFY_COMMITS_ALLOW_REVSIG=1 + else + export BITCOIN_VERIFY_COMMITS_ALLOW_REVSIG=0 + fi + if ! git -c "gpg.program=${DIR}/gpg.sh" verify-commit $1 > /dev/null 2>&1; then + return 1; + fi + local PARENTS + PARENTS=$(git show -s --format=format:%P $1) + for PARENT in $PARENTS; do + if IS_SIGNED $PARENT > /dev/null; then + return 0; + fi + done + if ! "$HAVE_FAILED"; then + echo "No parent of $1 was signed with a trusted key!" > /dev/stderr + echo "Parents are:" > /dev/stderr + for PARENT in $PARENTS; do + git show -s $PARENT > /dev/stderr + done + HAVE_FAILED=true + fi + return 1; +} + +if [ x"$1" = "x" ]; then + TEST_COMMIT="HEAD" +else + TEST_COMMIT="$1" +fi + +IS_SIGNED "$TEST_COMMIT" +RES=$? +if [ "$RES" = 1 ]; then + if ! "$HAVE_FAILED"; then + echo "$TEST_COMMIT was not signed with a trusted key!" + fi +else + echo "There is a valid path from $TEST_COMMIT to $VERIFIED_ROOT where all commits are signed!" +fi + +exit $RES diff --git a/doc/GCC_ISSUES b/doc/GCC_ISSUES deleted file mode 100644 index 103a8c9..0000000 --- a/doc/GCC_ISSUES +++ /dev/null @@ -1,39 +0,0 @@ -# The Problem - -gcc sometimes generates slightly different code with the same semantics. - -## Optimizations - -The following optimizer flags reduce non-determinism when compiling wxWidgets: - - -fno-tree-loop-optimize - -fno-trapping-math - -fno-tree-reassoc - -## Left Over - -Even with the above flags, the compiler still generates this difference in one out of 100 builds of wxWidgets: - - monolib_property.o: elf64-elf_x86_64 - - aef7: 00 - aef8: 45 31 f6 xor %r14d,%r14d - aefb: 48 8d 84 24 50 06 00 00 lea 0x650(%rsp),%rax - - af03: c6 44 24 67 00 movb $0x0,0x67(%rsp) - - af08: 48 83 c2 10 add $0x10,%rdx - - af0c: 48 83 c1 10 add $0x10,%rcx - - af10: 48 81 7c 24 08 ff ff ff cmpq $0xffffff,0x8(%rsp) - - af18: 00 - - af19: c6 44 24 47 00 movb $0x0,0x47(%rsp) - - af1e: c7 44 24 24 00 00 00 00 movl $0x0,0x24(%rsp) - + af03: c7 44 24 24 00 00 00 00 movl $0x0,0x24(%rsp) - + af0b: 48 83 c2 10 add $0x10,%rdx - + af0f: 48 83 c1 10 add $0x10,%rcx - + af13: 48 81 7c 24 08 ff ff ff cmpq $0xffffff,0x8(%rsp) - + af1b: 00 - + af1c: c6 44 24 47 00 movb $0x0,0x47(%rsp) - + af21: c6 44 24 67 00 movb $0x0,0x67(%rsp) - af26: 4c 8d bc 24 28 03 00 00 lea 0x328(%rsp),%r15 - af2e: 48 89 44 24 10 mov %rax,0x10(%rsp) - af33: 48 89 54 24 78 mov %rdx,0x78(%rsp) - diff --git a/etc/lxc.3.config.in b/etc/lxc.3.config.in new file mode 100644 index 0000000..18a8f53 --- /dev/null +++ b/etc/lxc.3.config.in @@ -0,0 +1,38 @@ +lxc.tty.max = 4 +lxc.pty.max = 1024 +lxc.rootfs.path = ROOTFS +lxc.arch = ARCH +lxc.cgroup.devices.deny = a +# /dev/null and zero +lxc.cgroup.devices.allow = c 1:3 rwm +lxc.cgroup.devices.allow = c 1:5 rwm +# consoles +lxc.cgroup.devices.allow = c 5:1 rwm +lxc.cgroup.devices.allow = c 5:0 rwm +lxc.cgroup.devices.allow = c 4:0 rwm +lxc.cgroup.devices.allow = c 4:1 rwm +# /dev/{,u}random +lxc.cgroup.devices.allow = c 1:9 rwm +lxc.cgroup.devices.allow = c 1:8 rwm +lxc.cgroup.devices.allow = c 136:* rwm +lxc.cgroup.devices.allow = c 5:2 rwm +# rtc +lxc.cgroup.devices.allow = c 254:0 rwm + +# mounts points +lxc.mount.entry=proc ROOTFS/proc proc nodev,noexec,nosuid 0 0 +lxc.mount.entry=sysfs ROOTFS/sys sysfs defaults 0 0 + +# Container with network virtualized using a pre-configured bridge named br0 and +# veth pair virtual network devices +# On the host, run: ifconfig br0 up 10.0.2.2 +# Alternatively, you can use another IP range for the bridge interface, in this case set +# the environment variables GITIAN_HOST_IP and LXC_GUEST_IP appropriately. +lxc.net.0.type = veth +lxc.net.0.flags = up +lxc.net.0.link = GUESTLINK +lxc.net.0.ipv4.address = GUESTIP/24 +lxc.net.0.ipv4.gateway = auto + +lxc.uts.name = gitian + diff --git a/libexec/config-lxc b/libexec/config-lxc index ddeabcf..e858fde 100755 --- a/libexec/config-lxc +++ b/libexec/config-lxc @@ -10,4 +10,15 @@ if [ -z "$LXC_BRIDGE" ]; then LXC_BRIDGE=br0 fi -sed "s;ROOTFS;$wd/target-$LXC_SUITE-$LXC_ARCH;;s;ARCH;$LXC_ARCH;g;;s;GUESTIP;$LXC_GUEST_IP;g;s;GUESTLINK;$LXC_BRIDGE;g" < etc/lxc.config.in > var/lxc.config +OLD_IFS=$IFS +IFS=. +VERSION=($(lxc-start --version)) +IFS=$OLD_IFS + +if [ $VERSION -ge 3 ]; then + LXC_CONFIG=etc/lxc.3.config.in +else + LXC_CONFIG=etc/lxc.config.in +fi + +sed "s;ROOTFS;$wd/target-$LXC_SUITE-$LXC_ARCH;;s;ARCH;$LXC_ARCH;g;;s;GUESTIP;$LXC_GUEST_IP;g;s;GUESTLINK;$LXC_BRIDGE;g" < $LXC_CONFIG > var/lxc.config diff --git a/libexec/copy-from-target b/libexec/copy-from-target index 08c6ecf..02806c3 100755 --- a/libexec/copy-from-target +++ b/libexec/copy-from-target @@ -46,10 +46,13 @@ if [ $# = 0 ] ; then exit 1 fi -if [ -z "$USE_LXC" ]; then - src="${1%/}" # remove trailing / which triggers special rsync behaviour - rsync --checksum -a $QUIET_FLAG -e "ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT" "$TUSER@localhost:${src}" "$2" -else +if [ "$USE_DOCKER" = "1" ]; then + # Use tar, so that files are created with the correct owner on the host + docker exec -u $TUSER gitian-target tar -C `dirname "$1"` -cf - `basename "$1"` | tar -C "$2" -xf - +elif [ "$USE_LXC" = "1" ]; then config-lxc sudo $LXC_EXECUTE -n gitian -f var/lxc.config -- sudo -i -u $TUSER tar -C `dirname "$1"` -cf - `basename "$1"` | tar -C "$2" -xf - +else + src="${1%/}" # remove trailing / which triggers special rsync behaviour + rsync --checksum -a $QUIET_FLAG -e "ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT" "$TUSER@localhost:${src}" "$2" fi diff --git a/libexec/copy-to-target b/libexec/copy-to-target index 6670862..37d045f 100755 --- a/libexec/copy-to-target +++ b/libexec/copy-to-target @@ -46,10 +46,14 @@ if [ $# = 0 ] ; then exit 1 fi -if [ -z "$USE_LXC" ]; then - src="${1%/}" # remove trailing / which triggers special rsync behaviour - rsync --checksum -a $QUIET_FLAG -e "ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT" "${src}" "$TUSER@localhost:$2" -else +if [ "$USE_DOCKER" = "1" ]; then + docker exec -u $TUSER gitian-target mkdir -p "/home/$TUSER/$2" + docker cp "$1" gitian-target:"/home/$TUSER/$2" + docker exec -u root gitian-target chown -R $TUSER:$TUSER "/home/$TUSER/$2" +elif [ "$USE_LXC" = "1" ]; then config-lxc tar -C `dirname "$1"` -cf - `basename "$1"` | sudo $LXC_EXECUTE -n gitian -f var/lxc.config -- sudo -i -u $TUSER tar -C "$2" -xf - +else + src="${1%/}" # remove trailing / which triggers special rsync behaviour + rsync --checksum -a $QUIET_FLAG -e "ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT" "${src}" "$TUSER@localhost:$2" fi diff --git a/libexec/gconfig b/libexec/gconfig index 204be94..a046bf2 100644 --- a/libexec/gconfig +++ b/libexec/gconfig @@ -1,5 +1,5 @@ VM_SSH_PORT=2223 -if [ -n "$USE_LXC" ]; then +if [ "$USE_LXC" = "1" ]; then if [ -z "$LXC_EXECUTE" ]; then ver=`lxc-start --version` if dpkg --compare-versions $ver ge 1.0.0 ; then diff --git a/libexec/make-clean-vm b/libexec/make-clean-vm index d99bf8e..2de8377 100755 --- a/libexec/make-clean-vm +++ b/libexec/make-clean-vm @@ -1,14 +1,16 @@ #!/bin/sh set -e -SUITE=lucid +SUITE=xenial ARCH=amd64 VMSW=KVM -if [ -n "$USE_LXC" ]; then +if [ "$USE_LXC" = "1" ]; then VMSW=LXC -elif [ -n "$USE_VBOX" ]; then +elif [ "$USE_VBOX" = "1" ]; then VMSW=VBOX +elif [ "$USE_DOCKER" = "1" ]; then + VMSW=DOCKER fi usage() { @@ -17,7 +19,7 @@ usage() { echo cat << EOF --help display this help and exit - --suite U build suite U instead of lucid + --suite U build suite U instead of xenial --arch A build architecture A (e.g. i386) instead of amd64 EOF } @@ -56,7 +58,7 @@ OUT=target-$SUITE-$ARCH case $VMSW in KVM) - qemu-img create -f qcow2 -o backing_file="$BASE.qcow2" "$OUT.qcow2" + qemu-img create -f qcow2 -o backing_fmt=qcow2,backing_file="$BASE.qcow2" "$OUT.qcow2" ;; LXC) cp -a --sparse=always $BASE $OUT @@ -66,4 +68,7 @@ case $VMSW in VBOX) VBoxManage snapshot "Gitian-${SUITE}-${ARCH}" restore "Gitian-Clean" ;; + DOCKER) + true #Docker doesn't need to do anything + ;; esac diff --git a/libexec/on-target b/libexec/on-target index 78eab62..26c066d 100755 --- a/libexec/on-target +++ b/libexec/on-target @@ -46,9 +46,11 @@ fi # exit 1 #fi -if [ -z "$USE_LXC" ]; then - ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT $TUSER@localhost $* -else +if [ "$USE_DOCKER" = "1" ]; then + docker exec -u $TUSER -i gitian-target $* +elif [ "$USE_LXC" = "1" ]; then config-lxc sudo $LXC_EXECUTE -n gitian -f var/lxc.config -- sudo -u $TUSER $ENV -i -- $* +else + ssh -oConnectTimeout=30 -oNoHostAuthenticationForLocalhost=yes -i ${GITIAN_BASE:-.}/var/id_rsa -p $VM_SSH_PORT $TUSER@localhost $* fi diff --git a/libexec/start-target b/libexec/start-target index 467ed63..5e6a502 100755 --- a/libexec/start-target +++ b/libexec/start-target @@ -6,10 +6,12 @@ ARCH=qemu$1 SUFFIX=$2 VMSW=KVM -if [ -n "$USE_LXC" ]; then +if [ "$USE_LXC" = "1" ]; then VMSW=LXC -elif [ -n "$USE_VBOX" ]; then +elif [ "$USE_VBOX" = "1" ]; then VMSW=VBOX +elif [ "$USE_DOCKER" = "1" ]; then + VMSW=DOCKER fi case $VMSW in @@ -31,7 +33,14 @@ case $VMSW in true #sudo lxc-start -n gitian -c var/target.log -f lxc.config ;; VBOX) - VBoxManage startvm "Gitian-${2}" # --type headless + VBoxManage startvm "Gitian-${2}" --type headless echo "Gitian-${2}" > var/target.vmname ;; + DOCKER) + EXTRA_ARGS="" + if [ -n "$GITIAN_ALLOW_PRIVILEGED" ]; then + EXTRA_ARGS="--privileged" + fi + docker run -d --name gitian-target $EXTRA_ARGS base-$SUFFIX:latest > /dev/null + ;; esac diff --git a/libexec/stop-target b/libexec/stop-target index 6db547d..169c63a 100755 --- a/libexec/stop-target +++ b/libexec/stop-target @@ -1,17 +1,19 @@ #!/bin/sh VMSW=KVM -if [ -n "$USE_LXC" ]; then +if [ "$USE_LXC" = "1" ]; then VMSW=LXC -elif [ -n "$USE_VBOX" ]; then +elif [ "$USE_VBOX" = "1" ]; then VMSW=VBOX +elif [ "$USE_DOCKER" = "1" ]; then + VMSW=DOCKER fi case $VMSW in KVM) if [ ! -e var/target.pid ]; then exit; fi - on-target -u root halt + on-target -u root poweroff sleep 5 if [ ! -e var/target.pid ]; then exit; fi @@ -30,4 +32,8 @@ case $VMSW in VBoxManage controlvm `cat var/target.vmname` savestate rm var/target.vmname ;; + DOCKER) + docker container stop gitian-target > /dev/null + docker container rm gitian-target > /dev/null + ;; esac diff --git a/target-bin/bootstrap-fixup.in b/target-bin/bootstrap-fixup.in index 6ffc0d2..3596039 100755 --- a/target-bin/bootstrap-fixup.in +++ b/target-bin/bootstrap-fixup.in @@ -4,6 +4,7 @@ set -e DISTRIB_NAME=`lsb_release -is` DISTRIB_CODENAME=`lsb_release -cs` +DISTRIB_NUMBER=`lsb_release -rs` if [ $DISTRIB_NAME = "Ubuntu" ]; then echo "deb http://HOSTIP:3142/archive.ubuntu.com/ubuntu $DISTRIB_CODENAME main universe" > $1/etc/apt/sources.list @@ -11,19 +12,28 @@ if [ $DISTRIB_NAME = "Ubuntu" ]; then echo "deb http://HOSTIP:3142/archive.ubuntu.com/ubuntu $DISTRIB_CODENAME-updates main universe" >> $1/etc/apt/sources.list elif [ $DISTRIB_NAME = "Debian" ]; then echo "deb http://HOSTIP:3142/ftp.debian.org/debian $DISTRIB_CODENAME main" > $1/etc/apt/sources.list - echo "deb http://HOSTIP:3142/security.debian.org/ $DISTRIB_CODENAME/updates main" >> $1/etc/apt/sources.list + + # This line format changed with the release of Debian 11 Buster + # https://wiki.debian.org/NewInBullseye + if [ $DISTRIB_NUMBER -ge 11 ]; then + echo "deb http://HOSTIP:3142/security.debian.org/debian-security $DISTRIB_CODENAME-security main" >> $1/etc/apt/sources.list + else + echo "deb http://HOSTIP:3142/security.debian.org/ $DISTRIB_CODENAME/updates main" >> $1/etc/apt/sources.list + fi + echo "deb http://HOSTIP:3142/ftp.debian.org/debian $DISTRIB_CODENAME-updates main" >> $1/etc/apt/sources.list # grub-legacy conflicts grub-pc dependencies # No grub-legacy on Ubuntu, just on Debian # Work around bcron-run conflict due to cron being removed - apt-get purge -y grub-legacy bcron-run &> /dev/null + # Needed for KVM, but apparently errors out for LXC, so the true ignores the error + apt-get purge -y grub-legacy bcron-run &> /dev/null || true fi echo '127.0.1.1 gitian' >> /etc/hosts # If LXC -if grep /lxc/gitian /proc/1/cgroup > /dev/null; then - adduser --disabled-password --gecos ubuntu --quiet ubuntu || true - apt-get remove -y rsyslog || true +if grep /lxc/gitian /proc/1/cgroup > /dev/null || grep container=lxc /proc/1/environ > /dev/null || grep lxc- /proc/1/comm > /dev/null; then + adduser --disabled-password --gecos ${DISTRIB_NAME,,} --quiet ${DISTRIB_NAME,,} || true + apt-get purge -y rsyslog || true dpkg-divert --local --rename --add /sbin/initctl ln -sf /bin/true /sbin/initctl dpkg-divert --local --rename --add /usr/bin/ischroot