Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHub Actions failes with: "no space left on device" #2875

Closed
1 of 7 tasks
Imipenem opened this issue Mar 8, 2021 · 6 comments
Closed
1 of 7 tasks

GitHub Actions failes with: "no space left on device" #2875

Imipenem opened this issue Mar 8, 2021 · 6 comments

Comments

@Imipenem
Copy link

Imipenem commented Mar 8, 2021

Description
A clear and concise description of what the bug is, and why you consider it to be a bug.
I recently noticed that one of my GitHub workflows failed with the above mentioned error.
Link to one of those runs: https://github.com/Imipenem/mypytest/runs/2060441074.

i followed #709 and added:
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

But this lead to the error that I could not execute some python packages etc which is crucial for my workflow.

So I decided to remove some space consuming packages, but the error still persists.

Area for Triage:

Question, Bug, or Feature?:

Bug

Virtual environments affected

  • Ubuntu 16.04
  • Ubuntu 18.04
  • Ubuntu 20.04
  • macOS 10.15
  • macOS 11.0
  • Windows Server 2016 R2
  • Windows Server 2019

Image version

Image version where you are experiencing the issue.
Version: 20210302.0

Expected behavior
A description of what you expected to happen.
The workflow should pass.

Actual behavior
A description of what is actually happening.
Workflow keeps failing with running out of space

Repro steps
A description with steps to reproduce the issue. If your have a public example or repo to share,
please provide the link.

  1. https://github.com/Imipenem/mypytest/runs/2060441074 link for example WF run (accidently ran WF another time, so link is not the initial run)

Exact error:
Error processing tar file(exit status 1): write /home/user/miniconda/envs/mypytest/lib/python3.9/site-packages/torch/lib/libtorch_cuda_cu.so: no space left on device Error: exit status 1

Any ideas @miketimofeev or others?

Thanks ;)

@al-cheb
Copy link
Contributor

al-cheb commented Mar 9, 2021

Hello, @Imipenem
I think you can delete donet&ghc locations - #2840 (comment)

@Imipenem
Copy link
Author

Imipenem commented Mar 9, 2021

Thanks @al-cheb will try this

@Imipenem
Copy link
Author

Imipenem commented Mar 9, 2021

This worked, thanks @al-cheb

@Imipenem Imipenem closed this as completed Mar 9, 2021
@jlumbroso
Copy link

@Imipenem Having faced this siutation so many times, I decided to stop rewriting the wheel 😁 and wrote a GitHub Actions to do this properly:

https://github.com/marketplace/actions/free-disk-space-ubuntu

Open to suggestions. I noticed these lines:

sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

but I am not sure what they remove?

@miketimofeev
Copy link
Contributor

@jlumbroso the first one doesn't exist anymore on the image. The second one removes all the pre-cached tools (Node, Go, Python, Ruby) that are used by the correspondent actions like https://github.com/actions/setup-python

@jlumbroso
Copy link

Thanks so much @miketimofeev! Thanks to you, I have added an option tool-cache that removes the pre-cached tools, it saves an extra 6GB. You also taught me how to look at the internals of GitHub Actions runners 😁 🙏🏻

https://github.com/jlumbroso/free-disk-space/releases/tag/v1.1.0

lorisleiva added a commit to metaplex-foundation/mpl-token-auth-rules that referenced this issue May 18, 2023
Without it we're getting a "no space left on device" error.

See actions/runner-images#2875
lorisleiva added a commit to metaplex-foundation/mpl-token-auth-rules that referenced this issue May 18, 2023
* Move program folder to programs/token-auth-rules

* Rename packages/sdk to clients/js-solita

* Move cli into clients

* Add configs folder

* Remove old JS configs

* Remove old scripts

* Update package.json and replace yarn with pnpm

* Update GitHub workflows

* Format config folder

* Add Amman client back to JS Solita client

* Add Umi JS client

* Regenerate client with correct program ID

* Add .env file from GitHub actions

* Fix CLI dependencies

* Enable Umi library publishing on NPM

* Use buildjet runner when testing programs

Without it we're getting a "no space left on device" error.

See actions/runner-images#2875
akoshelev added a commit to akoshelev/raw-ipa that referenced this issue Nov 8, 2023
14Gb default space is probably not enough for us anymore

```
[ec2-user@ip-172-31-49-57 ipa]$ du -sh target/
36G     target/
```

see the discussion here
actions/runner-images#2875
akoshelev added a commit to akoshelev/raw-ipa that referenced this issue Nov 8, 2023
14Gb default space is probably not enough for us anymore

```
[ec2-user@ip-172-31-49-57 ipa]$ du -sh target/
36G     target/
```

see the discussion here
actions/runner-images#2875
akoshelev added a commit to akoshelev/raw-ipa that referenced this issue Nov 8, 2023
14Gb default space is probably not enough for us anymore

```
[ec2-user@ip-172-31-49-57 ipa]$ du -sh target/
36G     target/
```

see the discussion here
actions/runner-images#2875
akoshelev added a commit to private-attribution/ipa that referenced this issue Nov 9, 2023
14Gb default space is probably not enough for us anymore

```
[ec2-user@ip-172-31-49-57 ipa]$ du -sh target/
36G     target/
```

see the discussion here
actions/runner-images#2875
sjain-stanford added a commit to cruise-automation/mlir-tcp that referenced this issue Feb 13, 2024
…-hosted runners (#38)

We're hitting `no space left on device` errors during llvm and stablehlo
builds with the free GitHub hosted runners, which come with limited disk
space allocations. Cleaning up the runners as recommended
[here](https://github.com/orgs/community/discussions/25678),
[here](https://github.com/orgs/community/discussions/26351) and
actions/runner-images#2875 have helped
partially (increase root mount from 20G to 47G) but it still isn't
enough for some builds. Temporarily disable the 3p builds on push to
main (they will still run as non-merge gating on PRs), and re-enable
once self-hosted runners are setup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants