Skip to content

Persistent linux make failures #1114

Closed
Closed

Description

@rvagg

I don't know if these are getting more frequent but they feel like they are.

https://ci.nodejs.org/job/node-test-commit-linux-containered/2180/nodes=ubuntu1604_sharedlibs_openssl110_x64/

09:30:35 make[2]: Entering directory '/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_openssl110_x64/test/addons/zlib-binding/build'
09:30:35   CXX(target) Release/obj.target/binding/binding.o
09:30:35   SOLINK_MODULE(target) Release/obj.target/binding.node
09:30:35   COPY Release/binding.node
09:30:35 make[2]: Leaving directory '/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_openssl110_x64/test/addons/zlib-binding/build'
09:30:35 gyp info ok 
09:30:35 touch test/addons/.buildstamp
09:30:35 make[1]: write error: stdout
09:30:35 Makefile:475: recipe for target 'run-ci' failed

https://ci.nodejs.org/job/node-test-commit-linux/16352/nodes=ubuntu1604-64/

make[2]: Entering directory '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1604-64/test/addons/zlib-binding/build'
  CXX(target) Release/obj.target/binding/binding.o
  SOLINK_MODULE(target) Release/obj.target/binding.node
  COPY Release/binding.node
make[2]: Leaving directory '/home/iojs/build/workspace/node-test-commit-linux/nodes/ubuntu1604-64/test/addons/zlib-binding/build'
gyp info ok 
touch test/addons/.buildstamp
Makefile:475: recipe for target 'run-ci' failed
make: *** [run-ci] Error 2

They may not be the same error but they look similar. Always seem to be near the addon tests, always seem to be just after touch test/addons/.buildstamp and always on Linux. Mostly they seem to be on the more modern machines but I can't say this for sure. I'm noticing it the most on debian8 and perhaps it's a performance thing too. I've been assuming it's some kind of Makefile problem with the way we set up dependencies.

Since make[1]: write error: stdout seems to be something that shows up fairly frequently with these I'm giving something else a go. There's reports of problems with Make's use of stdout when working on parallel jobs, and we do a lot of that. So Make v4 introduced an --output-sync option. I've enabled that on node-test-commit-linux as follows:

if [ $(make -v | grep 'GNU Make 4' -c) -ne 0 ]; then
  MAKE_ARGS="$MAKE_ARGS --output-sync=target"
fi

if [ "$RUN_TESTS" = "RUN" ]; then
  NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python FLAKY_TESTS=$FLAKY_TESTS_MODE make run-ci $MAKE_ARGS
else

Ubuntu 12 and 14 and I think both CentOS 5 and 6 fail this test and only have Make 3.8.

I'd appreciate help keeping an eye on Linux runs to see if it makes a difference. Otherwise I'd love others to take a look at trying to reproduce and fix this!

Activity

  1. rvagg commented on Feb 12, 2018

    @rvagg
    MemberAuthor

    This one just now on plinux, looks like the same category but there are no helpful clues that I can see

    https://ci.nodejs.org/job/node-test-commit-plinux/15329/nodes=ppcle-ubuntu1404/

    gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build', '--jobs', 4 ]
      CC(target) Release/obj.target/test_warning2/test_warning2.o
      SOLINK_MODULE(target) Release/obj.target/test_warning.node
      SOLINK_MODULE(target) Release/obj.target/test_warning2.node
      COPY Release/test_warning.node
      COPY Release/test_warning2.node
    make[2]: Leaving directory `/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons-napi/test_warning/build'
    gyp info ok 
    touch test/addons-napi/.buildstamp
    make[1]: Leaving directory `/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404'
    make: *** [run-ci] Error 2
    
  2. rvagg commented on Feb 12, 2018

    @rvagg
    MemberAuthor

    Oh, actually there are other errors further up in the output:

    make[2]: Entering directory `/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons-napi/test_make_callback_recurse/build'
      CXX(target) Release/obj.target/binding/binding.o
      SOLINK_MODULE(target) Release/obj.target/binding.node
      COPY Release/binding.node
    make[2]: Leaving directory `/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons-napi/test_make_callback_recurse/build'
    gyp info ok 
    gypsh: printf: I/O error
    
    make[2]: Leaving directory `/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/addons/hello-world-esm/build'
    make[2]: write error
    make[1]: *** [test/addons/.buildstamp] Error 1
    make[1]: *** Waiting for unfinished jobs....
    

    Unfortunately I don't believe Make 4 is available for this one so even if my trick works for the others it won't work here.

    This is always in addon builds. Perhaps theres a way we can make those builds serial-only even when you do a -j?

  3. rvagg commented on Feb 13, 2018

    @rvagg
    MemberAuthor

    OK, this is interesting, similar failure on FreeBSD now @ https://ci.nodejs.org/job/node-test-commit-freebsd/15465/nodes=freebsd10-64

    gyp info spawn gmake
    gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build', '--jobs', 2 ]
    gmake[2]: Entering directory '/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/addons/06_factory_of_wrapped_objects/build'
      CXX(target) Release/obj.target/addon/addon.o
      CXX(target) Release/obj.target/addon/myobject.o
      SOLINK_MODULE(target) Release/obj.target/addon.node
    gyp ERR! build error 
    gyp info it worked if it ends with ok
    

    ... much lower ...

    gyp info spawn gmake
    gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build', '--jobs', 2 ]
    gmake[2]: Entering directory '/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/addons-napi/test_warning/build'
      CC(target) Release/obj.target/test_warning/test_warning.o
      CC(target) Release/obj.target/test_warning2/test_warning2.o
      SOLINK_MODULE(target) Release/obj.target/test_warning.node
      SOLINK_MODULE(target) Release/obj.target/test_warning2.node
      COPY Release/test_warning2.node
      COPY Release/test_warning.node
    gmake[2]: Leaving directory '/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/addons-napi/test_warning/build'
    gyp info ok 
    touch test/addons-napi/.buildstamp
    gmake: *** [Makefile:475: run-ci] Error 2
    Build step 'Execute shell' marked build as failure
    
  4. rvagg commented on Feb 13, 2018

    @rvagg
    MemberAuthor

    FreeBSD 10 has gmake v4 so I tried adding --output-sync=target to the Jenkins config but now I get a whole lot of fcntl(): Bad file descriptor during build and I don't think we're getting everything we should and at this stage I'm not even sure it's working (will keep watching https://ci.nodejs.org/job/node-test-commit-freebsd/15470/ to see if it does). So I don't think that's an acceptable solution unfortunately.

    So how about this hypothesis—gyp is invoked to make the addons, but it's only recently that we had a new addons test directory added. Previously we only had test/addons, but now we have test/addons-napi. When you're doing -j2 (or greater) breaks it up by directories and now that there are two addons directories it'll build them both in parallel, whereas before it'd just do the test/addons serial. Now we have gyp output competing for stdout and for some reason it trips over itself while doing so.

    Does anyone here have the expertise and time to dig further into this? If I'm right then it might involve digging in to gyp and the way Python does stdout during this process and how it gets tangled up with make while doing so.

    @bnoordhuis does this sound like it's up your alley?

  5. rvagg commented on Feb 14, 2018

    @rvagg
    MemberAuthor

    https://ci.nodejs.org/job/node-test-commit-linux/16402/ has two of these failures but on ubuntu12 and ubuntu14 which don't support --output-sync. I haven't seen failures on more modern linuxes so I'm getting more confident this is a stdout race problem that we're going to have to solve.

  6. maclover7 commented on May 24, 2018

    @maclover7
    Contributor

    I forget what issue/PR, but I believe this is resolved now. Please reopen if I'm mistaken.

  7. manzee commented on Apr 28, 2020

    @manzee

    What is the final solution for this??

  8. rvagg commented on Apr 28, 2020

    @rvagg
    MemberAuthor

    @manzee what's the exact problem you're facing that brings you to this ancient artifact?

  9. manzee commented on Apr 28, 2020

    @manzee

    @rvagg I am facing this error make: write error: stdout

    nodejs version 8.17.0

  10. rvagg commented on Apr 28, 2020

    @rvagg
    MemberAuthor

    @manzee try giving --output-sync=target to make, iirc that helped at the time on Linux but I don't think we've seen this since this issue has closed (I also don't think we're using that in our CI either anymore).

  11. rvagg commented on Apr 28, 2020

    @rvagg
    MemberAuthor

    ^^ fwiw the problem seems to be related to parallel make usage of stdout, so when you have JOBS=X or -jX (X>1) you have multiple processes competing for stdout and they don't coordinate very well.

  12. manzee commented on Apr 28, 2020

    @manzee

    @rvagg Thanks a lot. It helped.

  13. nitin-gera commented on Sep 1, 2021

    @nitin-gera

    @manzee did this solve your problem?? I am still getting the error
    write error: stdout on Jenkins even after adding --output-sync=target to make
    I am on CentOS7 and node version 8.0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions