Github prebuild fails during exporting layers

[steph409]

Select Topic Area

Bug

Body

I am currently trying to switch a codespace devcontainer to dockercompose. the devcontainer works when I run it locally, but it does not work yet with github codespaces. The issue is that the prebuild does not succeed. I tried playing around with the docker buildkit options as I found this issue, but so far no luck.
During the prebuild, I get the following error:

#32 [devcontainer] preparing layers for inline cache#32 DONE 136.8s

#33 [devcontainer] exporting to image
#33 exporting layers
failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF
Stop: Run: docker compose --project-name testimage -f /var/lib/docker/codespacemount/workspace/testworkspace/.devcontainer/testimage/docker-compose.yml -f /var/lib/docker/codespacemount/.persistedshare/docker-compose.codespaces.yml -f /var/lib/docker/codespacemount/.persistedshare/docker-compose/docker-compose.devcontainer.build-1737965614628.yml build
Error: Command failed: docker compose --project-name testimage -f /var/lib/docker/codespacemount/workspace/testworkspace/.devcontainer/testimage/docker-compose.yml -f /var/lib/docker/codespacemount/.persistedshare/docker-compose.codespaces.yml -f /var/lib/docker/codespacemount/.persistedshare/docker-compose/docker-compose.devcontainer.build-1737965614628.yml build

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[thevindu-w]

This seems the same issue addressed here, docker/buildx#2064. Can you please check that and see if it fixes your problem?

[steph409]

thanks for the quick reply. I saw this issue as well. I am running on the github agents, so I can't run any commands there. I can see in the logs from the prebuild that the docker-buildx-plugin gets installed, but I can't really see which version. Is there a way to specify which version to use in the prebuild?

Github agent: Current runner version: '2.321.0'

from the logs:

#3******** 4.539 The following NEW packages will be installed:
#3******** 4.54********   docker-buildx-plugin docker-ce-cli docker-compose-plugin
#3******** 4.618 ******** upgraded, 3 newly installed, ******** to remove and ******** not upgraded.
#3******** 4.618 Need to get 61.3 MB of archives.
#3******** 4.618 After this operation, 2********1 MB of additional disk space will be used.
#3******** 4.618 Get:1 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-buildx-plugin amd64 ********.2********.********-1~ubuntu.22.********4~jammy [33.2 MB]
#3******** 4.895 Get:2 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-ce-cli amd64 5:27.5.1-1~ubuntu.22.********4~jammy [15.2 MB]
#3******** 5.********27 Get:3 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-compose-plugin amd64 2.32.4-1~ubuntu.22.********4~jammy [12.8 MB]
#3******** 7.173 Fetched 61.3 MB in 1s (1********7 MB/s)
#3******** 7.255 Selecting previously unselected package docker-buildx-plugin.

[steph409]

Hey @thevindu-w ,

I looked deeper into it, and there are 2 things not working:

1. the cache_from feature is not working, and the image is build not matter if the cache is up to date or not. there is an error message in the prebuild logs: Error fetching image details: No manifest found for docker.io/MYREGISTRY.azurecr.io/MYIMAGE:latest. My image is hosted on azure container registry. It has access there - pulling my custom base image is successful from that registry, but in the manifest fetching, it seems to not build the link correctly (it writes docker.io before my registry). the cache_from fails, and it then tries to build the image, which leads to error 2.
2. building the image fails on the github agent with the error message failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF. If this is related to buildx, do you have some advice to find out if the github agent is using an old version of that package - and how to update it?

I would greatly appreciate your advice.
Stephi

[thevindu-w]

Then you need to fix the 1st issue before the 2nd. Where did you specify the MYREGISTRY.azurecr.io/MYIMAGE:latest? is it in the Dockerfile like FROM MYREGISTRY.azurecr.io/MYIMAGE:latest? or any other config file?

[steph409]

my dockercompose file looks like this:

services:
  devcontainer:
    container_name: devcontainer
    build:
      context: .
      dockerfile: Dockerfile
      cache_from:
        - MYREGISTRY.azurecr.io/MYIMAGE:latest
    volumes:
      - ../../:/workspaces/MYREPO
      - data:/data
      - profile:/profile
    command: sleep infinity
    shm_size: 2gb
    privileged: true

if I remove the cache_from and do the following instead, then the prebuild works, as the image is not build but just pulled from the registry. Also in this case, I get the first error message mentioned above that it was not able to pull the manifest, but it is still able to pull the image. However, this then fixes the image to whatever version is in the registry.

services:
  devcontainer:
    container_name: devcontainer
    image: MYREGISTRY.azurecr.io/MYIMAGE:latest
    volumes:
      - ../../:/workspaces/MYREPO
      - data:/data
      - profile:/profile
    command: sleep infinity
    shm_size: 2gb
    privileged: true

[sherazumar]

I ran into a similar issue before, and here’s what worked for me:
Check the Image: Make sure the image MYREGISTRY.azurecr.io/MYIMAGE:latest is accessible. Try pulling it manually with:

docker pull MYREGISTRY.azurecr.io/MYIMAGE:latest

If it’s private, make sure you’re logged into the Azure registry:

az acr login --name

Resource Limits: The shm_size: 2gb is good for processes that need more shared memory, but double-check if your Codespace has enough resources (RAM and CPU). If you suspect resource limits, try reducing the shared memory allocation temporarily:

shm_size: 512mb

Volumes Setup: Verify that the paths you’ve specified in the volumes section exist and are accessible:

volumes:`

• ../../:/workspaces/MYREPO
• data:/data
• profile:/profile

For instance, ensure ../../ resolves correctly in the context of your setup.

Privileged Mode: You’ve set privileged: true. While this is sometimes necessary, it can also cause issues in certain environments. If it’s not essential, try removing it and see if that helps.

Logs for More Details: If the error persists, checking the container logs can provide more context. Use:

docker logs

Network Check: Since this could also be network-related, ensure your Docker environment has proper access to external networks. For example, check your proxy settings if you’re behind one.

Sometimes, these errors are temporary, so retrying the build after verifying the above can help. Let me know how it goes, and I’ll be happy to help troubleshoot further if needed!