Parallel Steps

[HariSekhon]

Hi,

Can you please raise a request to engineering for parallel steps support?

Eg. for logging in to multiple docker registries in parallel, or other actions which cannot be done in separate jobs because you require the local cache / side effects.

Uploading/downloading artifacts from adjacent jobs is not suitable for this use case.

Thanks

Hari

[lukasz-mitka]

I'd like that too. There are scenarios where it doesn't make sense juggling artifacts between jobs.
This would be greatly helpful for self-hosted and bigger runners too, where you can have many cores and handle parallelism well.

An idea how it could be used:

  steps:
    - uses: actions/checkout@v3

    - run: echo "Sequential step"

    - parallel: 
        branch_id_1:
          steps:
            - run: echo "branch 1 sequential step 1"
            - run: echo "branch 1 sequential step 2"

        # Executes steps in parallel to branch1
        branch_id_2:
          steps:
            - run: echo "branch 2 sequential step 1"
            - run: echo "branch 2 sequential step 2"

    - run: echo "More sequential steps after the whole parallel part is done."

[HariSekhon]

This is a good approach as it mirrors Jenkins well-known solution for parallelizing stages.

https://www.jenkins.io/blog/2017/09/25/declarative-1/

https://www.jenkins.io/doc/book/pipeline/syntax/#parallel

[server-craftsman]

Response #1

[floralatin]

Alternative solution:
We can convert steps into actions, and then use reusable workflows to solve the problems you said

[floralatin]

Hi,

Can you please raise a request to engineering for parallel steps support?

Eg. for logging in to multiple docker registries in parallel, or other actions which cannot be done in separate jobs because you require the local cache / side effects.

Uploading/downloading artifacts from adjacent jobs are not suitable for this use case.

Thanks

Hari

Alternative solution:
We can convert steps into actions, and then use reusable workflows to solve the problems you said

[HariSekhon]

Sorry I'm missed how putting steps in a composite action would allow you to parallelize steps such as multiple logins to different registries?

Could you please elaborate further with regards to how this solves step parallelization?

[paulz]

Parallel steps or sub jobs would save us 50% of time waiting for the CI.

[jsoref]

Logging into docker in parallel is probably going to result in things exploding.

Each docker login command tries to mutate ~/.docker/config.json.

Today, you could probably do something like:

(To make these work, you'd want to have them rely on env vars and set the env vars using your workflow based on secrets)

log-into-docker.sh:

HOME=$HOME/.docker-docker
mkdir -p $HOME
docker login ...

log-into-gcr.sh:

HOME=$HOME/.docker-gcr
mkdir -p $HOME
docker login ...

log-into-ghcr.sh:

HOME=$HOME/.docker-ghcr
mkdir -p $HOME
docker login ...

docker-log-in.sh

for task in ./log-into-*.sh; do
  $task &
done
wait
jq '. * input' ~/.docker*/config.js > my.config.js && mv my.config.js ~/.docker/config.js
jq '. * input' ~/.docker*/plaintext-credentials.config.json > my.plaintext.js && mv my.plaintext.js ~/.docker/plaintext-credentials.config.json

[HariSekhon]

If the docker CLI is not parallel safe in rewriting its config.json then that would be better raised as an issue to the docker CLI developers and the issue should be reproduceable locally using GNU parallel.

[jsoref]

Talking to the docker folks is an exercise for the brave. I wouldn't recommend it to my worst enemy.

[piotrekkr]

@jsoref I think instead of for loop with wait pid you can just use wait

If the wait utility is invoked with no operands, it shall wait
until all process IDs known to the invoking shell have terminated
and exit with a zero exit status.

About docker I totally agree. Talking to docker devs is PITA and nothing that improve dev work is actually done e.g. multiple dockerignore files or mounting volumes as non root to container and so on.

[jsoref]

@piotrekkr: thanks on both points.

[sfonseka-provenir]

can we have the parallel steps please

[MichaelVoelkel]

YES, this is the most useful option and it's obvious. Downvote must come from some GitHub dev who does not like to have some work or what? (nah, downvote was about something else, sorry, it's valid)

[lukasz-mitka]

Downvote must come from some GitHub dev who does not like to have some work or what?

Nope, it's coming from me, because this answer doesn't bring any value. Basically a spam.

[MichaelVoelkel]

Because here no-one can request for anything because GitHub action devs don't see it? Okay, I'm not aware where requests can be made. It seems to be anyways useless because there are some many open requests at various pages and I also saw GitHub action devs saying, they will probably not do it

[lukasz-mitka]

The main post is the by HariSekhon is the request. You agree - there's an arrow up button on his post.

---

It seems to be anyways useless

Unfortunately yes :(

---

also saw GitHub action devs saying, they will probably not do it

Link please?

[MichaelVoelkel]

It's from memory, so cannot tell, sorry. I think it was either about better caching or about their unwillingness to make "include yml files for steps" or some other of the obvious shortcomings. But then again, not sure if Github even claims to be a CI/CD solution or just code repo with benefits, so maybe we should not be so hard

[andrew-clear]

The BitBucket has parallel step options. Why not add this option to GitHub actions?

[RichiCoder1]

This is the biggest pain point we've run into coming from Jenkins. You can approximate this feature with cache, artifacts, and matrices but it's considerably slower, more costly, and error prone.

[lukasz-mitka]

How come it's error prone?

[RichiCoder1]

Every project has different needs about what artifacts they need available, how they're shaped, fs permissions, size costs, etc...
It's a one time cost, but it's not fun to figure out.

[Vungle-GordonTian]

Need parallel steps support for our scenario: deploy multi clusters concurrently within the same VPN environment. So it is convinient to achive in steps rather than actions or jobs

[raphaelbadia]

Using a yarn monorepo I'd like to:

1. yarn install
2. then build every project in parallel
3. then run test on every project in parallel

yarn install is slow, and I currently have to repeat it for every workflow like lint, test.... It's a waste of time

[jsoref]

actions/cache will help. As does switching to yarn berry.

We've been iterating over an action that does some of the work to make this better:

https://github.com/GarnerCorp/build-actions/blob/main/yarn/action.yml
(I just spotted two things in this file that I think we'll clean up today.)

You can use actions/upload-artifact and actions/download-artifact to share a build product (e.g. the output tarball from the above action) between jobs (especially a matrix of jobs to run tests).

[thepratt]

Are you using yarn workspaces? If you are, you should be able to run yarn workspaces foreach --parallel [...] https://yarnpkg.com/cli/workspaces/foreach

[michielvanderros-tomtom]

I was looking for a solution for exactly the same problem @raphaelbadia.

I did get caching to work but I'm using PNPM, which I would highly recommend as an easy time saver compared to Yarn. It uses symlinks to a system level package cache, instead of installing dependencies in all your monorepo projects.

[michielvanderros-tomtom]

Check out this approach: Making pnpm run jobs in parallel instead of finding a way to run GitHub steps in parallel: https://stackoverflow.com/questions/68427503/how-can-i-run-pnpm-workspace-projects-as-parallel-jobs-on-github-actions

[koistya]

@michielvanderros-tomtom speaking of PNPM :) Yarn supports pnpm linker out of the box, in addition to pnp and regular node_modules; yet it's developed by a team vs. single person behind pnpm, and has great design with plugin system.

[erikologic]

If its easier, GitHub could also allow sharing jobs workdirs

[jsoref]

You can use actions/upload-artifact / actions/download-artifact today...

[erikologic]

I'm aware of that.
It is not the same as 2 jobs sharing a workdir.

[erikologic]

Have a look here: https://circleci.com/docs/caching/

[lukasz-mitka]

You can use actions/upload-artifact / actions/download-artifact today...

It's super slow. Impractical for anything larger.
actions/cache is faster but has limited capacity.

[hfhbd]

I just want to start a resource in a background, which needs some time to start, eg a docker container or a simulator. This resource should start during the job start. My test step should wait until the test resource is available, or fail after an (optional) timeout.

- jobs:
   - build
     - resources:
        - test-db
           run: docker start someSlowImage
     - steps: 
       - run: assemble
       - run: test
          resources:
            - test-db
               timeout: 5min

[sjhewitt]

I think you can do this using service containers: https://docs.github.com/en/actions/using-containerized-services/about-service-containers

[koistya]

Just add waitFor attribute similarly to Cloud Build, no need to reinvent the wheel

Note how it was implemented in Cloud Build many years ago yet it's super convenient syntax:

steps:
- name: foo
  id: A
- name: bar
  id: B
  waitFor: ["A"]
- name: baz
  id: C
  waitFor: ["A"]

... where step B and step C are intended to run in parallel, but only after step A is completed.

steps:
- name: foo
  id: A
- name: bar
  id: B
  waitFor: ["-"]
- name: baz

... where waitFor: ["-"] indicates that this step doesn't have to wait for the previous step(s) to complete.

https://cloud.google.com/build/docs/configuring-builds/configure-build-step-order

[jsoref]

They might not, but each Google Cloudbuild step runs a separate disposable container and you can't persist state from one to another. Thus anyone making a comparison to them is making a very uniformed comparison.

I'm well aware of both. We're in the process of migrating some builds from Cloudbuild to GitHub Workflows and I've been using GitHub Workflows/Actions for years.

[koistya]

@jsoref you can pass state between steps in Cloud Build as easily as in GitHub Actions steps (docs). For example, the following list of steps — 1 npm install, 2 build A, 3 build B, 4 deploy A, 5 deploy B, works perfectly well in Cloud Build (sharing the /workspace files alongside the installed NPM dependencies and build output). I am currently checking what developers are saying about GitHub Actions on Reddit, and it appears to be a consensus opinion that it is still very immature because some of the core features, such as parallel step execution and others, are missing.

steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-node@v3
  # Install dependencies
  - run: yarn install
  # Build the API and web front-end (app)
  - id: build-api
    run: yarn workspace api build
  - id: build-app
    run: yarn workspace app build
  # Deploy
  - id: deploy-api
    run: yarn workspace api deploy
    waitFor: ["build-api", "build-app"]   # <- This "waitFor" field demonstrates
  - id: deploy-app                        # the elegance of parallel steps feature
    run: yarn workspace app deploy        # in Google Cloud Build. If implemented
    waitFor: ["build-api", "build-app"]   # in GitHub Actions it would reduce the pain
                                          # configuring parallel steps 10-fold 💰 🙌 👍  🎉 👏 💪

^^ Deployment of packages api and app starts in parallel after both api and app packages were successfully built.

[jsoref]

steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-node@v3
  # Install dependencies
  - run: yarn install
  # Build the API and web front-end (app)
  - id: build-api-app
    run: |
      yarn workspace api build &
      yarn workspace app build
      fg %1 || true
  # Deploy
  - id: deploy-api-app
    run: |
      yarn workspace api deploy &
      yarn workspace app deploy
      fg %1 || true

[koistya]

@jsoref In your example above there are no separate steps for building individual packages/projects, with separate success/failure statuses and separate logs, that's the point of this issue. I can't stress enough how painful it is (at least for me) to have this feature (parallel steps) missing in GitHub Actions. 😫 😖 😣 😡 💔 😭 😩 💢 🤬 💥 💢 😤

[jsoref]

You can make that if you want, just create logs for each thing in the main step and then pass the various job outputs to individual steps for reporting.

It really isn't terribly hard

[malindajudo]

As a workaround I was able to use this
https://github.com/marketplace/actions/distributed-task-runner

` name: Distributed Tasks
steps:
- uses: actions/checkout@v3

  - name: Run bash commands in parallel
    uses: sambacha/parallelish@v1
      id: tasks
    with:
      cmd1: echo $BASH_VERSION; date;
      cmd2: echo $BASH_VERSION; date;`

This way I was able to run our tests parallel

[koistya]

Are you able to see the logs separately for different parallel commands in GitHub Actions?

[malindajudo]

@koistya : no, can not see logs separately , but if one of the cmd fails the can see logs

[gtchaboussie]

Hi. Are there any updates on this topic from a Github team member ?
Reading the thread it seems like a strong painpoint.

[andrei-dascalu]

Would be great to have some updates here. Parallel steps would be great, plenty of instances where for example I'd like to know whether multiple steps fail (eg: a lint check as well as unit tests). While jobs can run in parallel, each needs to be setup again. Would be great for a pipeline to be able to run a setup step that for example pulls dependencies and then splits off in parallel steps each starting from the state left by the setup.
Bitbucket is able to do this, for example.

[crispossa]

Would be awesome if we can, for example, compile our js/css files at the same time we are running composer (for PHP projects).

[stefanos-kalantzis]

Just wanted to add, that parallel jobs means waiting for multiple available workers, which can add a lot to the total build time, and in some cases you don't need the extra hardware.

Not sure what's the problem with implementing this. It just adds an option that aparently a lot of people find useful.

[patrykacc]

Hi,

Can you please raise a request to engineering for parallel steps support?

Eg. for logging in to multiple docker registries in parallel, or other actions which cannot be done in separate jobs because you require the local cache / side effects.

Uploading/downloading artifacts from adjacent jobs is not suitable for this use case.

Thanks

Hari

Bitbucket can do it, just saying...

[eightnoteight]

the current way of running steps in parallel (i.e using bash & or a variant of it) has a very bad ux when you think about logging

[ccll]

Don't know much about Actions as I'm new to this (trying to find alternatives to Drone CI), but a simple feature like this took so long time to implement, maybe the design of Actions has some fundamental flaws on create DAG of steps.
In Drone CI DAG was designed and supported in a long time ago, and there is even a view of the DAG graph.

Drone steps are by default parallel, unless you add a 'depends_on' list to steps, which would transform them to a DAG.
I think this might be a better and more flexible design choice than by default sequential.

[artemisart]

Hello, I was considering Drone CI to use parallel steps, any reasons why you're trying to replace it? Thanks!

[Skyppid]

Would love this feature as well. We already have quite long running pipelines and saving some time would be great. We build our .NET backend then pack it as docker image. While creating the image takes significant time we also need to upload our symbols to Sentry for better error analysis. While docker images are being created we could parallelize the upload to Sentry. A new job is no option as we lose the symbol files we require. Right now we run a separate job which fetches git, compiles, etc. all just to upload already built symbols.

Please implement this! :)

[AurimasNav]

I have 5 small steps (10-30sec), each runs a docker container with different arguments. Ideally I would have first step to build docker image and then run those five steps in parallel. Doing this with workflows seems cumbersome and would possibly prolong the total run time with artifact upload and artifact downloads added to share the image between jobs.

[jsoref]

if you can fit more than one docker container in a single github runner, then it definitely makes more sense to use a single step to create the image and then run the steps in parallel. We use docker compose up with a container definition to get parallelism like that.

[AurimasNav]

if you can fit more than one docker container in a single github runner, then it definitely makes more sense to use a single step to create the image and then run the steps in parallel. We use docker compose up with a container definition to get parallelism like that.

correct me if I'm wrong, but in a case with docker compose that would be a single step and you have to dig into the logs to figure out which of the containers failed?

[jsoref]

Yes. We did it. It isn't difficult.

You could probably use docker inspect -f '{{ .State.ExitCode }}' (we didn't do that, but...)