Repository navigation
Description
I don't know if these are getting more frequent but they feel like they are.
09:30:35 make[2]: Entering directory '/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_openssl110_x64/test/addons/zlib-binding/build'
09:30:35 CXX(target) Release/obj.target/binding/binding.o
09:30:35 SOLINK_MODULE(target) Release/obj.target/binding.node
09:30:35 COPY Release/binding.node
09:30:35 make[2]: Leaving directory '/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_openssl110_x64/test/addons/zlib-binding/build'
09:30:35 gyp info ok
09:30:35 touch test/addons/.buildstamp
09:30:35 make[1]: write error: stdout
09:30:35 Makefile:475: recipe for target 'run-ci' failed
make[2]: Entering directory '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1604-64/test/addons/zlib-binding/build'
CXX(target) Release/obj.target/binding/binding.o
SOLINK_MODULE(target) Release/obj.target/binding.node
COPY Release/binding.node
make[2]: Leaving directory '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1604-64/test/addons/zlib-binding/build'
gyp info ok
touch test/addons/.buildstamp
Makefile:475: recipe for target 'run-ci' failed
make: *** [run-ci] Error 2
They may not be the same error but they look similar. Always seem to be near the addon tests, always seem to be just after touch test/addons/.buildstamp and always on Linux. Mostly they seem to be on the more modern machines but I can't say this for sure. I'm noticing it the most on debian8 and perhaps it's a performance thing too. I've been assuming it's some kind of Makefile problem with the way we set up dependencies.
Since make[1]: write error: stdout seems to be something that shows up fairly frequently with these I'm giving something else a go. There's reports of problems with Make's use of stdout when working on parallel jobs, and we do a lot of that. So Make v4 introduced an --output-sync option. I've enabled that on node-test-commit-linux as follows:
if [ $(make -v | grep 'GNU Make 4' -c) -ne 0 ]; then
MAKE_ARGS="$MAKE_ARGS --output-sync=target"
fi
if [ "$RUN_TESTS" = "RUN" ]; then
NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE make run-ci $MAKE_ARGS
else
Ubuntu 12 and 14 and I think both CentOS 5 and 6 fail this test and only have Make 3.8.
I'd appreciate help keeping an eye on Linux runs to see if it makes a difference. Otherwise I'd love others to take a look at trying to reproduce and fix this!
Activity
rvagg commentedon Feb 12, 2018
This one just now on plinux, looks like the same category but there are no helpful clues that I can see
https://ci.nodejs.org/job/node-test-commit-plinux/15329/nodes=ppcle-ubuntu1404/
rvagg commentedon Feb 12, 2018
Oh, actually there are other errors further up in the output:
Unfortunately I don't believe Make 4 is available for this one so even if my trick works for the others it won't work here.
This is always in addon builds. Perhaps theres a way we can make those builds serial-only even when you do a
-j?rvagg commentedon Feb 13, 2018
OK, this is interesting, similar failure on FreeBSD now @ https://ci.nodejs.org/job/node-test-commit-freebsd/15465/nodes=freebsd10-64
... much lower ...
rvagg commentedon Feb 13, 2018
FreeBSD 10 has gmake v4 so I tried adding
--output-sync=targetto the Jenkins config but now I get a whole lot offcntl(): Bad file descriptorduring build and I don't think we're getting everything we should and at this stage I'm not even sure it's working (will keep watching https://ci.nodejs.org/job/node-test-commit-freebsd/15470/ to see if it does). So I don't think that's an acceptable solution unfortunately.So how about this hypothesis—gyp is invoked to make the addons, but it's only recently that we had a new addons test directory added. Previously we only had test/addons, but now we have test/addons-napi. When you're doing
-j2(or greater) breaks it up by directories and now that there are two addons directories it'll build them both in parallel, whereas before it'd just do the test/addons serial. Now we have gyp output competing for stdout and for some reason it trips over itself while doing so.Does anyone here have the expertise and time to dig further into this? If I'm right then it might involve digging in to gyp and the way Python does stdout during this process and how it gets tangled up with make while doing so.
@bnoordhuis does this sound like it's up your alley?
rvagg commentedon Feb 14, 2018
https://ci.nodejs.org/job/node-test-commit-linux/16402/ has two of these failures but on ubuntu12 and ubuntu14 which don't support --output-sync. I haven't seen failures on more modern linuxes so I'm getting more confident this is a stdout race problem that we're going to have to solve.
maclover7 commentedon May 24, 2018
I forget what issue/PR, but I believe this is resolved now. Please reopen if I'm mistaken.
manzee commentedon Apr 28, 2020
What is the final solution for this??
rvagg commentedon Apr 28, 2020
@manzee what's the exact problem you're facing that brings you to this ancient artifact?
manzee commentedon Apr 28, 2020
@rvagg I am facing this error make: write error: stdout
nodejs version 8.17.0
rvagg commentedon Apr 28, 2020
@manzee try giving
--output-sync=targettomake, iirc that helped at the time on Linux but I don't think we've seen this since this issue has closed (I also don't think we're using that in our CI either anymore).rvagg commentedon Apr 28, 2020
^^ fwiw the problem seems to be related to parallel make usage of stdout, so when you have
JOBS=Xor-jX(X>1) you have multiple processes competing for stdout and they don't coordinate very well.manzee commentedon Apr 28, 2020
@rvagg Thanks a lot. It helped.
nitin-gera commentedon Sep 1, 2021
@manzee did this solve your problem?? I am still getting the error
write error: stdout on Jenkins even after adding --output-sync=target to make
I am on CentOS7 and node version 8.0.0