Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use QEMU and "arch-test" to avoid bad binaries in the future #121

Merged
merged 2 commits into from
Dec 20, 2022
Merged

Use QEMU and "arch-test" to avoid bad binaries in the future #121

merged 2 commits into from
Dec 20, 2022

Conversation

tianon
Copy link
Owner

@tianon tianon commented Dec 19, 2022

As now noted on https://github.com/tianon/gosu/releases/tag/1.15, our new releases are affected by golang/go#56426, which is broken in Go 1.18+ and not fixed until Go 1.20. 😞

This is an attempt to pre-test for that via QEMU user-mode emulation and actually testing our end binaries (because this reproduces the failure).

Another failure that I think this is also going to run into is that from Go 1.18+ it seems our binaries now often hang in QEMU user-mode emulation (most reliably on ARM), so that's fun.

@tianon tianon force-pushed the qemu branch 2 times, most recently from 3cc30fd to 1960dca Compare December 19, 2022 23:57
@tianon
Copy link
Owner Author

tianon commented Dec 19, 2022

There's the QEMU bug 😭

https://github.com/tianon/gosu/actions/runs/3736167285/jobs/6340182931#step:4:457

(it hangs indefinitely in whatever this failure mode is, hence all the timeout invocations)

@tianon
Copy link
Owner Author

tianon commented Dec 20, 2022

There we go, there's our MIPS failure! https://github.com/tianon/gosu/actions/runs/3736241126/jobs/6340334082#step:4:10830

Long Silly Backtrace: 
+ timeout 1s /go/bin/gosu-mips64el nobody id
trap:5114, a123456=[0,0,0,0,0,0]
trap:5114, a123456=[0,0,0,0,0,0]
results: got {r1=0,r2=5114,errno=0}, want {r1=0,r2=0,errno=0
fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted

goroutine 0 [idle]:
results: got {r1=0,r2=5114,errno=0}, want {r1=0,r2=0,errno=0
fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc00003fc90, fp:0xc00003fcd8} stack=[0xc000038000,0xc000040000)
0x000000c00003fb90:  0x0000000000000000  0x0000000000000000 
0x000000c00003fba0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fbb0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fbc0:  0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac>  0x0000000000000000 
0x000000c00003fbd0:  0x0000000000000000  0x000000c00003fc28 
0x000000c00003fbe0:  0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028>  0x0000000000000021 
0x000000c00003fbf0:  0x000000c000002340  0x0000000000000000 
0x000000c00003fc00:  0x0000000000000000  0x0000000000000000 
0x000000c00003fc10:  0x000000c00003fcf0  0x000000c00003fd70 
0x000000c00003fc20:  0x000000000008cf74 <runtime.sigtramp+0x0000000000000054>  0x000000c000000021 
0x000000c00003fc30:  0x000000c00003fcf0  0x000000c00003fd70 
0x000000c00003fc40:  0x000000c0000024e0  0x0000000000000000 
0x000000c00003fc50:  0x0000000000000000  0x0000000000000000 
0x000000c00003fc60:  0x0000000000000000  0x0000000000000000 
0x000000c00003fc70:  0x0000000000000000  0x000000c0000024e0 
0x000000c00003fc80:  0x000000c00003fcf0  0x000000c00003fd70 
0x000000c00003fc90: <0x0000004001001008  0x0000000000000021 
0x000000c00003fca0:  0x000000c00003fcf0  0x000000c00003fd70 
0x000000c00003fcb0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fcc0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fcd0:  0x0000000000000000 >0x0000000000000000 
0x000000c00003fce0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fcf0:  0xfffffffa00000021  0x0000000000000000 
0x000000c00003fd00:  0x00000000000002a0  0x0000000000000000 
0x000000c00003fd10:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd20:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd30:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd40:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd50:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd60:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd70:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd80:  0x000000c000038000  0x0000000000008000 
0x000000c00003fd90:  0x0000000000000000  0x0000000000000000 
0x000000c00003fda0:  0x000000c000045ee8  0x0000000000000004 
0x000000c00003fdb0:  0x000000003b9aca00  0x000000000026ec38 
0x000000c00003fdc0:  0x0000000000000080  0x0000000000000000 
0x000000c00003fdd0:  0x0000000000000001 
runtime.throw({0x183f3e, 0x44})
	/usr/local/go/src/runtime/panic.go:992 +0x6c
runtime.runPerThreadSyscall()
	/usr/local/go/src/runtime/os_linux.go:874 +0x410
runtime.sighandler(0x21, 0xc00003fcf0, 0xc00003fd70, 0xc0000024e0)
	/usr/local/go/src/runtime/signal_unix.go:631 +0x770
runtime.sigtrampgo(0x21, 0xc00003fcf0, 0xc00003fd70)
	/usr/local/go/src/runtime/signal_unix.go:477 +0x228
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc00003fc90, fp:0xc00003fcd8} stack=[0xc000038000,0xc000040000)
0x000000c00003fb90:  0x0000000000000000  0x0000000000000000 
0x000000c00003fba0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fbb0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fbc0:  0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac>  0x0000000000000000 
0x000000c00003fbd0:  0x0000000000000000  0x000000c00003fc28 
0x000000c00003fbe0:  0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028>  0x0000000000000021 
0x000000c00003fbf0:  0x000000c000002340  0x0000000000000000 
0x000000c00003fc00:  0x0000000000000000  0x0000000000000000 
0x000000c00003fc10:  0x000000c00003fcf0  0x000000c00003fd70 
0x000000c00003fc20:  0x000000000008cf74 <runtime.sigtramp+0x0000000000000054>  0x000000c000000021 
0x000000c00003fc30:  0x000000c00003fcf0  0x000000c00003fd70 
0x000000c00003fc40:  0x000000c0000024e0  0x0000000000000000 
0x000000c00003fc50:  0x0000000000000000  0x0000000000000000 
0x000000c00003fc60:  0x0000000000000000  0x0000000000000000 
0x000000c00003fc70:  0x0000000000000000  0x000000c0000024e0 
0x000000c00003fc80:  0x000000c00003fcf0  0x000000c00003fd70 
0x000000c00003fc90: <0x0000004001001008  0x0000000000000021 
0x000000c00003fca0:  0x000000c00003fcf0  0x000000c00003fd70 
0x000000c00003fcb0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fcc0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fcd0:  0x0000000000000000 >0x0000000000000000 
0x000000c00003fce0:  0x0000000000000000  0x0000000000000000 
0x000000c00003fcf0:  0xfffffffa00000021  0x0000000000000000 
0x000000c00003fd00:  0x00000000000002a0  0x0000000000000000 
0x000000c00003fd10:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd20:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd30:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd40:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd50:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd60:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd70:  0x0000000000000000  0x0000000000000000 
0x000000c00003fd80:  0x000000c000038000  0x0000000000008000 
0x000000c00003fd90:  0x0000000000000000  0x0000000000000000 
0x000000c00003fda0:  0x000000c000045ee8  0x0000000000000004 
0x000000c00003fdb0:  0x000000003b9aca00  0x000000000026ec38 
0x000000c00003fdc0:  0x0000000000000080  0x0000000000000000 
0x000000c00003fdd0:  0x0000000000000001 
runtime.sigtramp()
	/usr/local/go/src/runtime/sys_linux_mips64x.s:435 +0x54
trap:5114, a123456=[0,0,0,0,0,0]
results: got {r1=0,r2=5114,errno=0}, want {r1=0,r2=0,errno=0
fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted

goroutine 1 [running, locked to thread]:
	goroutine running on other thread; stack unavailable

goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc000057c90, fp:0xc000057cd8} stack=[0xc000050000,0xc000058000)
0x000000c000057b90:  0x0000000000000000  0x0000000000000000 
0x000000c000057ba0:  0x0000000000000000  0x0000000000000000 
0x000000c000057bb0:  0x0000000000000000  0x0000000000000000 
0x000000c000057bc0:  0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac>  0x0000000000000000 
0x000000c000057bd0:  0x0000000000000000  0x000000c000057c28 
0x000000c000057be0:  0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028>  0x0000000000000021 
0x000000c000057bf0:  0x000000c000002b60  0x0000000000000000 
0x000000c000057c00:  0x0000000000000000  0x0000000000000000 
0x000000c000057c10:  0x000000c000057cf0  0x000000c000057d70 
0x000000c000057c20:  0x000000000008cf74 <runtime.sigtramp+0x0000000000000054>  0x000000c000000021 
0x000000c000057c30:  0x000000c000057cf0  0x000000c000057d70 
0x000000c000057c40:  0x000000c000002d00  0x0000000000000000 
0x000000c000057c50:  0x0000000000000000  0x0000000000000000 
0x000000c000057c60:  0x0000000000000000  0x0000000000000000 
0x000000c000057c70:  0x0000000000000000  0x000000c000002d00 
0x000000c000057c80:  0x000000c000057cf0  0x000000c000057d70 
0x000000c000057c90: <0x0000004001001008  0x0000000000000021 
0x000000c000057ca0:  0x000000c000057cf0  0x000000c000057d70 
0x000000c000057cb0:  0x0000000000000000  0x0000000000000000 
0x000000c000057cc0:  0x0000000000000000  0x0000000000000000 
0x000000c000057cd0:  0x0000000000000000 >0x0000000000000000 
0x000000c000057ce0:  0x0000000000000000  0x0000000000000000 
0x000000c000057cf0:  0xfffffffa00000021  0x0000000000000000 
0x000000c000057d00:  0x00000000000002a0  0x0000000000000000 
0x000000c000057d10:  0x0000000000000000  0x0000000000000000 
0x000000c000057d20:  0x0000000000000000  0x0000000000000000 
0x000000c000057d30:  0x0000000000000000  0x0000000000000000 
0x000000c000057d40:  0x0000000000000000  0x0000000000000000 
0x000000c000057d50:  0x0000000000000000  0x0000000000000000 
0x000000c000057d60:  0x0000000000000000  0x0000000000000000 
0x000000c000057d70:  0x0000000000000000  0x0000000000000000 
0x000000c000057d80:  0x000000c000050000  0x0000000000008000 
0x000000c000057d90:  0x0000000000000000  0x0000000000000000 
0x000000c000057da0:  0x0000000000000000  0x000000000000144a 
0x000000c000057db0:  0x000000c000002d00  0x000000c000036950 
0x000000c000057dc0:  0x0000000000000080  0x0000000000000000 
0x000000c000057dd0:  0x0000000000000000 
runtime.throw({0x183f3e, 0x44})
	/usr/local/go/src/runtime/panic.go:992 +0x6c
runtime.runPerThreadSyscall()
	/usr/local/go/src/runtime/os_linux.go:874 +0x410
runtime.sighandler(0x21, 0xc000057cf0, 0xc000057d70, 0xc000002d00)
	/usr/local/go/src/runtime/signal_unix.go:631 +0x770
runtime.sigtrampgo(0x21, 0xc000057cf0, 0xc000057d70)
	/usr/local/go/src/runtime/signal_unix.go:477 +0x228
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc000057c90, fp:0xc000057cd8} stack=[0xc000050000,0xc000058000)
0x000000c000057b90:  0x0000000000000000  0x0000000000000000 
0x000000c000057ba0:  0x0000000000000000  0x0000000000000000 
0x000000c000057bb0:  0x0000000000000000  0x0000000000000000 
0x000000c000057bc0:  0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac>  0x0000000000000000 
0x000000c000057bd0:  0x0000000000000000  0x000000c000057c28 
0x000000c000057be0:  0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028>  0x0000000000000021 
0x000000c000057bf0:  0x000000c000002b60  0x0000000000000000 
0x000000c000057c00:  0x0000000000000000  0x0000000000000000 
0x000000c000057c10:  0x000000c000057cf0  0x000000c000057d70 
0x000000c000057c20:  0x000000000008cf74 <runtime.sigtramp+0x0000000000000054>  0x000000c000000021 
0x000000c000057c30:  0x000000c000057cf0  0x000000c000057d70 
0x000000c000057c40:  0x000000c000002d00  0x0000000000000000 
0x000000c000057c50:  0x0000000000000000  0x0000000000000000 
0x000000c000057c60:  0x0000000000000000  0x0000000000000000 
0x000000c000057c70:  0x0000000000000000  0x000000c000002d00 
0x000000c000057c80:  0x000000c000057cf0  0x000000c000057d70 
0x000000c000057c90: <0x0000004001001008  0x0000000000000021 
0x000000c000057ca0:  0x000000c000057cf0  0x000000c000057d70 
0x000000c000057cb0:  0x0000000000000000  0x0000000000000000 
0x000000c000057cc0:  0x0000000000000000  0x0000000000000000 
0x000000c000057cd0:  0x0000000000000000 >0x0000000000000000 
0x000000c000057ce0:  0x0000000000000000  0x0000000000000000 
0x000000c000057cf0:  0xfffffffa00000021  0x0000000000000000 
0x000000c000057d00:  0x00000000000002a0  0x0000000000000000 
0x000000c000057d10:  0x0000000000000000  0x0000000000000000 
0x000000c000057d20:  0x0000000000000000  0x0000000000000000 
0x000000c000057d30:  0x0000000000000000  0x0000000000000000 
0x000000c000057d40:  0x0000000000000000  0x0000000000000000 
0x000000c000057d50:  0x0000000000000000  0x0000000000000000 
0x000000c000057d60:  0x0000000000000000  0x0000000000000000 
0x000000c000057d70:  0x0000000000000000  0x0000000000000000 
0x000000c000057d80:  0x000000c000050000  0x0000000000008000 
0x000000c000057d90:  0x0000000000000000  0x0000000000000000 
0x000000c000057da0:  0x0000000000000000  0x000000000000144a 
0x000000c000057db0:  0x000000c000002d00  0x000000c000036950 
0x000000c000057dc0:  0x0000000000000080  0x0000000000000000 
0x000000c000057dd0:  0x0000000000000000 
runtime.sigtramp()
	/usr/local/go/src/runtime/sys_linux_mips64x.s:435 +0x54

goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc000061c90, fp:0xc000061cd8} stack=[0xc00005a000,0xc000062000)
0x000000c000061b90:  0x0000000000000000  0x0000000000000000 
0x000000c000061ba0:  0x0000000000000000  0x0000000000000000 
0x000000c000061bb0:  0x0000000000000000  0x0000000000000000 
0x000000c000061bc0:  0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac>  0x0000000000000000 
0x000000c000061bd0:  0x0000000000000000  0x000000c000061c28 
0x000000c000061be0:  0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028>  0x0000000000000021 
0x000000c000061bf0:  0x000000c000002ea0  0x0000000000000000 
0x000000c000061c00:  0x0000000000000000  0x0000000000000000 
0x000000c000061c10:  0x000000c000061cf0  0x000000c000061d70 
0x000000c000061c20:  0x000000000008cf74 <runtime.sigtramp+0x0000000000000054>  0x000000c000000021 
0x000000c000061c30:  0x000000c000061cf0  0x000000c000061d70 
0x000000c000061c40:  0x000000c000003040  0x0000000000000000 
0x000000c000061c50:  0x0000000000000000  0x0000000000000000 
0x000000c000061c60:  0x0000000000000000  0x0000000000000000 
0x000000c000061c70:  0x0000000000000000  0x000000c000003040 
0x000000c000061c80:  0x000000c000061cf0  0x000000c000061d70 
0x000000c000061c90: <0x0000004001001008  0x0000000000000021 
0x000000c000061ca0:  0x000000c000061cf0  0x000000c000061d70 
0x000000c000061cb0:  0x0000000000000000  0x0000000000000000 
0x000000c000061cc0:  0x0000000000000000  0x0000000000000000 
0x000000c000061cd0:  0x0000000000000000 >0x0000000000000000 
0x000000c000061ce0:  0x0000000000000000  0x0000000000000000 
0x000000c000061cf0:  0xfffffffa00000021  0x0000000000000000 
0x000000c000061d00:  0x00000000000002a0  0x0000000000000000 
0x000000c000061d10:  0x0000000000000000  0x0000000000000000 
0x000000c000061d20:  0x0000000000000000  0x0000000000000000 
0x000000c000061d30:  0x0000000000000000  0x0000000000000000 
0x000000c000061d40:  0x0000000000000000  0x0000000000000000 
0x000000c000061d50:  0x0000000000000000  0x0000000000000000 
0x000000c000061d60:  0x0000000000000000  0x0000000000000000 
0x000000c000061d70:  0x0000000000000000  0x0000000000000000 
0x000000c000061d80:  0x000000c00005a000  0x0000000000008000 
0x000000c000061d90:  0x0000000000000000  0x0000000000000000 
0x000000c000061da0:  0x0000000000000000  0x000000000000144a 
0x000000c000061db0:  0x000000c000003040  0x000000000029b098 
0x000000c000061dc0:  0x0000000000000080  0x0000000000000000 
0x000000c000061dd0:  0x0000000000000000 
runtime.throw({0x183f3e, 0x44})
	/usr/local/go/src/runtime/panic.go:992 +0x6c
runtime.runPerThreadSyscall()
	/usr/local/go/src/runtime/os_linux.go:874 +0x410
runtime.sighandler(0x21, 0xc000061cf0, 0xc000061d70, 0xc000003040)
	/usr/local/go/src/runtime/signal_unix.go:631 +0x770
runtime.sigtrampgo(0x21, 0xc000061cf0, 0xc000061d70)
	/usr/local/go/src/runtime/signal_unix.go:477 +0x228
runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008
stack: frame={sp:0xc000061c90, fp:0xc000061cd8} stack=[0xc00005a000,0xc000062000)
0x000000c000061b90:  0x0000000000000000  0x0000000000000000 
0x000000c000061ba0:  0x0000000000000000  0x0000000000000000 
0x000000c000061bb0:  0x0000000000000000  0x0000000000000000 
0x000000c000061bc0:  0x000000000006c3dc <runtime.sigtrampgo+0x00000000000001ac>  0x0000000000000000 
0x000000c000061bd0:  0x0000000000000000  0x000000c000061c28 
0x000000c000061be0:  0x000000000006c258 <runtime.sigtrampgo+0x0000000000000028>  0x0000000000000021 
0x000000c000061bf0:  0x000000c000002ea0  0x0000000000000000 
0x000000c000061c00:  0x0000000000000000  0x0000000000000000 
0x000000c000061c10:  0x000000c000061cf0  0x000000c000061d70 
0x000000c000061c20:  0x000000000008cf74 <runtime.sigtramp+0x0000000000000054>  0x000000c000000021 
0x000000c000061c30:  0x000000c000061cf0  0x000000c000061d70 
0x000000c000061c40:  0x000000c000003040  0x0000000000000000 
0x000000c000061c50:  0x0000000000000000  0x0000000000000000 
0x000000c000061c60:  0x0000000000000000  0x0000000000000000 
0x000000c000061c70:  0x0000000000000000  0x000000c000003040 
0x000000c000061c80:  0x000000c000061cf0  0x000000c000061d70 
0x000000c000061c90: <0x0000004001001008  0x0000000000000021 
0x000000c000061ca0:  0x000000c000061cf0  0x000000c000061d70 
0x000000c000061cb0:  0x0000000000000000  0x0000000000000000 
0x000000c000061cc0:  0x0000000000000000  0x0000000000000000 
0x000000c000061cd0:  0x0000000000000000 >0x0000000000000000 
0x000000c000061ce0:  0x0000000000000000  0x0000000000000000 
0x000000c000061cf0:  0xfffffffa00000021  0x0000000000000000 
0x000000c000061d00:  0x00000000000002a0  0x0000000000000000 
0x000000c000061d10:  0x0000000000000000  0x0000000000000000 
0x000000c000061d20:  0x0000000000000000  0x0000000000000000 
0x000000c000061d30:  0x0000000000000000  0x0000000000000000 
0x000000c000061d40:  0x0000000000000000  0x0000000000000000 
0x000000c000061d50:  0x0000000000000000  0x0000000000000000 
0x000000c000061d60:  0x0000000000000000  0x0000000000000000 
0x000000c000061d70:  0x0000000000000000  0x0000000000000000 
0x000000c000061d80:  0x000000c00005a000  0x0000000000008000 
0x000000c000061d90:  0x0000000000000000  0x0000000000000000 
0x000000c000061da0:  0x0000000000000000  0x000000000000144a 
0x000000c000061db0:  0x000000c000003040  0x000000000029b098 
0x000000c000061dc0:  0x0000000000000080  0x0000000000000000 
0x000000c000061dd0:  0x0000000000000000 
runtime.sigtramp()
	/usr/local/go/src/runtime/sys_linux_mips64x.s:435 +0x54

@tianon
Backport golang/go@2c7c98c in our builds
8eb1914 
This fixes our `mips64le` builds.
@tianon
Copy link
Owner Author

tianon commented Dec 20, 2022

A nice benefit of container-based builds: we can just apply that upstream patch to our Go sources during build and the bug is now fixed. 😂

@tianon tianon merged commit bb69d2a into tianon:master Dec 20, 2022
@tianon tianon deleted the qemu branch December 20, 2022 00:39
@yosifkit
Copy link
Collaborator

yosifkit commented Jan 5, 2024

The mips64le binary is also affected in the Bookworm package since it used Go 1.19 for building. 😢 https://packages.debian.org/bookworm/gosu https://packages.debian.org/bookworm/golang.

(Just to post this somewhere since fixing it in Debian Bookworm would be unlikely since it would need a patch applied to the golang package to fix it but also no one has really noticed 🙈)

@yosifkit yosifkit mentioned this pull request Jan 5, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tianon @yosifkit