Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release pipeline #2620

Merged
merged 11 commits into from
Jul 2, 2023
Merged

Release pipeline #2620

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
493ca3e
Release pipeline
owenthereal Jun 17, 2023
fc50569
Locate bison for Windows build
owenthereal Jun 18, 2023
b47d016
Also install flex for Windows build
owenthereal Jun 18, 2023
d295348
Add matrix to test all available GH Actions images
owenthereal Jun 25, 2023
4c15a17
Enable all tests for Windows
owenthereal Jul 1, 2023
25a0608
Run `brew update-reset` when `brew update` fails
owenthereal Jul 1, 2023
6cce62d
Select gcc binaries to release
owenthereal Jul 1, 2023
69e27dd
Only enable CI when push to master & tag
owenthereal Jul 1, 2023
2480dd2
Try out statick build
owenthereal Jul 1, 2023
4a4ba88
No need to enforce the same `AM_INIT_AUTOMAKE` version for MacOS
owenthereal Jul 1, 2023
6520242
Disable static build for ubuntu-20.04
owenthereal Jul 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
224 changes: 224 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,224 @@
name: CI
on:
push:
branches:
- master
tags:
- "v*"
pull_request:
owenthereal marked this conversation as resolved.
Show resolved Hide resolved
jobs:
linux:
strategy:
fail-fast: false
matrix:
compiler: [gcc, clang]
image: [ubuntu-20.04, ubuntu-22.04]
include:
- compiler: gcc
image: ubuntu-20.04
configure_flag: ''
- compiler: gcc
image: ubuntu-22.04
configure_flag: --enable-static --enable-all-static
- compiler: clang
image: ubuntu-20.04
configure_flag: ''
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh so only ubuntu-20.04 and clang fails using static? other ubuntus are fine? 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah no also gcc, misread above

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can reproduce with ubuntu docker image:

$ docker run -ti -v $PWD:$PWD -w $PWD ubuntu:20.04
... <configure, build etc>...
root@83a347734b03:/Users/wader/src/jq# gdb --args ./jq
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
....
(gdb) r
Starting program: /Users/wader/src/jq/jq
warning: Error disabling address space randomization: Operation not permitted

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00000000004ec78c in __register_frame_info_bases.part.0 ()
#2  0x0000000000401ec1 in frame_dummy ()
#3  0x0000000000000001 in ?? ()
#4  0x00000000004edd3c in __libc_csu_init ()
#5  0x00000000004ed48e in __libc_start_main ()
#6  0x0000000000401dde in _start ()
(gdb)

Dugg a bit, i suspect it might be something like https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95989 some version of glibc pthreads don't like static linking. Usually you can make it work by providing a lots of link flags to tell it to resolve weak symbols and whatnot, but it's a mess.

Also got me thinking: do the jq cli need threads? looking at the code it seems to be about making libjq possible to use concurrently? but it's probably no easy task to make it optional

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also see that jq do call some libc functions like getpwuid that i suspect might trigger NSS usage that i think might trigger dynamic loading and might crash hmm

In my experience if one really want static binaries linking with musl is the "safest"... but that might have some other issues.... but again in my experience with modern musl versions there are few issues

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw alpine, that uses musl, already package jq and i haven't had any issues with it and looking at their bug tracker for jq issues https://gitlab.alpinelinux.org/alpine/aports/-/issues/?search=jq&sort=created_date&state=closed&first_page_size=20 i don't see anything that looks like reported random strange crashes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a test using CC=musl-gcc (needs package musl musl-dev musl-tools) and it seems to work fine and make check passes

root@83a347734b03:/Users/wader/src/jq# file jq
jq: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
root@83a347734b03:/Users/wader/src/jq# ls -lh jq
-rwxr-xr-x 1 root root 1.8M Jul  2 14:04 jq
root@83a347734b03:/Users/wader/src/jq# strip jq
root@83a347734b03:/Users/wader/src/jq# file jq
jq: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
root@83a347734b03:/Users/wader/src/jq# ls -lh jq
-rwxr-xr-x 1 root root 1.1M Jul  2 14:04 jq
root@83a347734b03:/Users/wader/src/jq# ./jq --version
jq-1.6-215-gf88c4e5-dirty
root@83a347734b03:/Users/wader/src/jq# ./jq -n 1+2
3

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find! I'm going to merge this PR so that you have something to base off.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 sounds good

Yeap i think alpine for static binaries could be a good alternative, then i think we could truely have one binary that works on any linux kernel. @itchyny seemed a bit concerned about it but maybe can be convinced :) the alternative think is find a good distro and glibc version that produce known "good" static binaries, i guess that is was the previous releases did?

Out of curiosity had a look at the current jq 1.6 linux64 binary:

$ ldd jq-linux64
	not a dynamic executable
$ file jq-linux64
jq-linux64: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=c159e8a134ee222019dfbac0a1af4bc94b302eb0, with debug_info, not stripped

but couldn't find what libc variant/version is it, glibc something?

also notice that jq 1.6 had 32bit builds

- compiler: clang
image: ubuntu-22.04
configure_flag: --enable-static --enable-all-static
runs-on: ${{ matrix.image }}
env:
CC: ${{ matrix.compiler }}
SUFFIX: linux-${{ matrix.image}}-${{ matrix.compiler }}
steps:
- name: Clone repository
uses: actions/checkout@v3
with:
submodules: true
- name: Install packages
run: |
sudo apt-get update -qq
sudo apt-get install -y \
automake \
autoconf \
bison \
flex \
gdb \
python3
- name: Build
run: |
autoreconf -fi
./configure --disable-dependency-tracking \
--disable-silent-rules \
--disable-maintainer-mode \
--disable-valgrind \
--with-oniguruma=builtin \
${{ matrix.configure_flag }} \
YACC="$(which bison) -y"
make
- name: Test
run: |
make check
- name: Upload Test Logs
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: test-logs-${{ env.SUFFIX }}
retention-days: 7
path: |
test-suite.log
tests/*.log
- name: Upload Artifacts
uses: actions/upload-artifact@v3
with:
name: jq-${{ env.SUFFIX }}
if-no-files-found: error
retention-days: 7
path: |
jq
macos:
strategy:
fail-fast: false
matrix:
compiler: [gcc, clang]
image: [macos-11, macos-12, macos-13]
runs-on: ${{ matrix.image }}
env:
CC: ${{ matrix.compiler }}
SUFFIX: macos-${{ matrix.image}}-${{ matrix.compiler }}
steps:
- name: Clone repository
uses: actions/checkout@v3
with:
submodules: true
- name: Install packages
run: |
# brew update sometimes fails with "Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask failed!"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed this on some CI builds but for some reason (randomly?) only on macOS 12

Copy link
Member Author

@owenthereal owenthereal Jul 1, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, that's what I saw. Perhaps the GH Actions macos-12 image has some issue? The update-reset fallback helps with that, as you see.

brew update || brew update-reset
brew install \
autoconf \
automake \
libtool \
flex \
bison
- name: Build
run: |
autoreconf -fi
./configure --disable-dependency-tracking \
--disable-silent-rules \
--disable-maintainer-mode \
--disable-valgrind \
--with-oniguruma=builtin \
--enable-static \
--enable-all-static \
YACC="$(brew --prefix)/opt/bison/bin/bison -y"
make
- name: Test
run: |
make check
- name: Upload Test Logs
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: test-logs-${{ env.SUFFIX }}
retention-days: 7
path: |
test-suite.log
tests/*.log
- name: Upload Artifacts
uses: actions/upload-artifact@v3
with:
name: jq-${{ env.SUFFIX }}
if-no-files-found: error
retention-days: 7
path: |
jq
windows:
strategy:
fail-fast: false
matrix:
compiler: [gcc]
image: [windows-2019, windows-2022]
runs-on: ${{ matrix.image }}
env:
CC: ${{ matrix.compiler }}
SUFFIX: windows-${{ matrix.image}}-${{ matrix.compiler }}
steps:
- name: Clone repository
uses: actions/checkout@v3
with:
submodules: true
- uses: msys2/setup-msys2@v2
with:
update: true
install: >-
base-devel
git
clang
autoconf
automake
libtool
bison
flex
- name: Build
shell: msys2 {0}
run: |
autoreconf -fi
./configure --disable-dependency-tracking \
--disable-silent-rules \
--disable-maintainer-mode \
--disable-valgrind \
--with-oniguruma=builtin \
--disable-shared \
--enable-static \
--enable-all-static \
YACC="$(which bison) -y"
make
- name: Test
shell: msys2 {0}
run: |
make check
- name: Upload Test Logs
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: test-logs-${{ env.SUFFIX }}
retention-days: 7
path: |
test-suite.log
tests/*.log
- name: Upload Artifacts
uses: actions/upload-artifact@v3
with:
name: jq-${{ env.SUFFIX }}
if-no-files-found: error
retention-days: 7
path: |
jq.exe
release:
runs-on: ubuntu-latest
permissions:
contents: write
needs: [linux, macos, windows]
if: startsWith(github.event.ref, 'refs/tags/v')
steps:
- name: Clone repository
uses: actions/checkout@v3
with:
submodules: true
- name: Merge built artifacts
uses: actions/download-artifact@v3
with:
path: artifacts
- name: Upload release
env:
TAG_NAME: ${{ github.ref_name }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
mkdir release
cp artifacts/jq-linux-ubuntu-22.04-gcc/jq release/jq-linux-amd64
cp artifacts/jq-macos-macos-13-gcc/jq release/jq-macos-amd64
cp artifacts/jq-windows-windows-2022-gcc/jq.exe release/jq-windows-amd64.exe

gh release create $TAG_NAME --draft --title "jq ${TAG_NAME#v}" --generate-notes
gh release upload $TAG_NAME --clobber release/jq-*
70 changes: 0 additions & 70 deletions .github/workflows/linux.yml
View file
Open in desktop

This file was deleted.

62 changes: 0 additions & 62 deletions .github/workflows/macos.yml
View file
Open in desktop

This file was deleted.

Loading