Build failure: grafana on aarch64-darwin

[bouk]

Steps To Reproduce

Steps to reproduce the behavior:

1. nix run nixpkgs#grafana on aarch64-darwin

Build log

$ nix run -L nixpkgs#grafana
grafana> unpacking sources
grafana> unpacking source archive /nix/store/0ncdnawnzfzllb291ci8khkkhrwh4vdv-source
grafana> source root is source
grafana> patching sources
grafana> updateAutotoolsGnuConfigScriptsPhase
grafana> configuring
grafana> building
grafana> wire: github.com/grafana/grafana/pkg/server: wrote /private/tmp/nix-build-grafana-10.2.2.drv-0/source/pkg/server/wire_gen.go
grafana> Building subPackage .
grafana> Building subPackage ./.github/workflows/scripts/kinds
grafana> Building subPackage ./kinds
grafana> package github.com/grafana/grafana/kinds: build constraints exclude all Go files in /private/tmp/nix-build-grafana-10.2.2.drv-0/source/kinds
grafana> Building subPackage ./pkg
grafana> package github.com/grafana/grafana/pkg: build constraints exclude all Go files in /private/tmp/nix-build-grafana-10.2.2.drv-0/source/pkg
grafana> Building subPackage ./pkg/api
grafana> pkg/tsdb/mssql/mssql.go:20:2: cannot find module providing package github.com/microsoft/go-mssqldb: import lookup disabled by -mod=vendor
grafana> pkg/tsdb/mssql/mssql.go:21:2: cannot find module providing package github.com/microsoft/go-mssqldb/azuread: import lookup disabled by -mod=vendor
error: builder for '/nix/store/acvann56zagva3jni5mg8yfwxrfn1vrf-grafana-10.2.2.drv' failed with exit code 1;
       last 17 log lines:
       > unpacking sources
       > unpacking source archive /nix/store/0ncdnawnzfzllb291ci8khkkhrwh4vdv-source
       > source root is source
       > patching sources
       > updateAutotoolsGnuConfigScriptsPhase
       > configuring
       > building
       > wire: github.com/grafana/grafana/pkg/server: wrote /private/tmp/nix-build-grafana-10.2.2.drv-0/source/pkg/server/wire_gen.go
       > Building subPackage .
       > Building subPackage ./.github/workflows/scripts/kinds
       > Building subPackage ./kinds
       > package github.com/grafana/grafana/kinds: build constraints exclude all Go files in /private/tmp/nix-build-grafana-10.2.2.drv-0/source/kinds
       > Building subPackage ./pkg
       > package github.com/grafana/grafana/pkg: build constraints exclude all Go files in /private/tmp/nix-build-grafana-10.2.2.drv-0/source/pkg
       > Building subPackage ./pkg/api
       > pkg/tsdb/mssql/mssql.go:20:2: cannot find module providing package github.com/microsoft/go-mssqldb: import lookup disabled by -mod=vendor
       > pkg/tsdb/mssql/mssql.go:21:2: cannot find module providing package github.com/microsoft/go-mssqldb/azuread: import lookup disabled by -mod=vendor
       For full logs, run 'nix log /nix/store/acvann56zagva3jni5mg8yfwxrfn1vrf-grafana-10.2.2.drv'.

Additional context

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"aarch64-darwin"`
 - host os: `Darwin 23.1.0, macOS 14.1.2`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.18.1`
 - channels(bouke): `""`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

[tel]

I'm seeing the same error, nix-shell -p nix-info --run "nix-info -m" gives:

 - system: `"aarch64-darwin"`
 - host os: `Darwin 23.1.0, macOS 14.1.2`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

I'm also working in a flake environment rev 95d0067cc02182d7c98bb207ae84f571508382ba.

I attempted to build grafana.overrideAttrs { deleteVendor = true; } as well and it fails for the same reason.

[tel]

I also confirmed in source that the vendor directories github.com/microsoft/go-mssqldb and github.com/microsoft/go-mssqldb/azuread don't exist. Perhaps these need to be vendored prior to build?

[tel]

Flipping back through commits where Grafana was changed, I can get it to compile at Nixpkgs cd723bf928f0b6c33c3d8e968b255074be3b4ce8. That's the parent of this commit which made changes to the grafana derivation

bff4a35

[tel]

@Ma27 I'd like to see if you have any thoughts as to what might have changed with your commit bff4a35

[Ma27]

Perhaps these need to be vendored prior to build?

The vendor directory will be created in a different derivation, the contents will be copied at the appropriate place before the Go compiler gets invoked. So AFAIU everything should(tm) be there at the time where the build fails according to your logs.

I don't really know about the state of aarch64-darwin support on Grafana upstream. Did you check the issues (and CI builds) there? I may be wrong, but to me this currently seems like an issue in the upstream codebase or one of its dependencies.

[tel]

Thanks! That’s helpful to know.

I didn’t check upstream, I’m not super familiar with Grafana or Go. Was just doing a quick prototype and was able to get it working with nixpkgs just before your commit but not after.

We’re not moving forward with the prototype so I’m probably not going to keep looking on this issue.

[Ma27]

@tel so I got around to taking a first look (sorry that it took me so long!)

One question: when you do nix-build -A grafana.goModules, do you also have a directory called github.com/microsoft~nix~case~hack~1 in your result directory? That's most likely the explanation for that.

Please confirm that I'm not off chasing a different issue. But if that's the case, then we probably found a Nix bug that needs to be reported upstream.

[tel]

I'm on a flake-based system, so I ran nix build nixpkgs#grafana.goModules. Nevertheless, the path result/github.com/microsoft~nix~case~hack~1/go-mssqldb did get generated. Hopefully that's helpful!

[Ma27]

I'm on a flake-based system, so I ran nix build nixpkgs#grafana.goModules

FWIW it doesn't matter whether or not you're using flakes (I'm too in fact), you can still use nix-build on local git checkouts.

Anyways, as I said, I'm pretty sure that this is a Nix bug. I'll open a bug over there.

[bouk]

I think this is probably triggered by the fact that some of the go modules grafana depends on use Microsoft and others 'microsoft' for the organization name: https://github.com/grafana/grafana/blob/b5d14d03d7388894bbee47a93f25707539ff1519/go.mod#L437

so probably an interaction of these things. I've submitted a PR to go-winio to rename the module microsoft/go-winio#314

[shivaraj-bh]

@Ma27 I got it building for both aarch64 and x86_64 darwin, see the override. To read more about how I got it to build, you can refer to comments, starting from here.

[shivaraj-bh]

I am open to upstreaming it to nixpkgs, if that’s desired.

[Ma27]

@shivaraj-bh would you mind checking if #292997 solves the core issue for you (you'll probably need to adjust yarn's supported systems locally, but other than that there shouldn't be much to do).

Because if that's the case, that'd be my preferred solution (and another argument in favor of that patch).

[shivaraj-bh]

@Ma27 If I just add the darwin cache for yarn, it fails at:

Edit: I tried this on #292997

 > Building subPackage ./pkg/cmd/grafana
       > pkg/tsdb/mssql/mssql.go:22:2: cannot find module providing package github.com/microsoft/go-mssqldb: import lookup disabled by -mod=vendor
       > pkg/tsdb/mssql/mssql.go:23:2: cannot find module providing package github.com/microsoft/go-mssqldb/azuread: import lookup disabled by -mod=vendor

Which is the same issue as reported by OP. To solve this, I had used proxyVendor = true;. When I use this, the go-modules no longer has the github.com/microsoft\~nix\~case\~hack\~1, but there is github.com/microsoft and github.com/!microsoft. I am guessing proxyVendor somehow works around the case-insensitivity problem.

[shivaraj-bh]

This patch on your fork helps me build grafana on darwin:

diff --git a/pkgs/servers/monitoring/grafana/default.nix b/pkgs/servers/monitoring/grafana/default.nix
index 00688dc0aec8..f18e32296c20 100644
--- a/pkgs/servers/monitoring/grafana/default.nix
+++ b/pkgs/servers/monitoring/grafana/default.nix
@@ -36,6 +36,8 @@ buildGoModule rec {

   subPackages = [ "pkg/cmd/grafana" "pkg/cmd/grafana-server" "pkg/cmd/grafana-cli" ];

+  excludedPackages = [ "xorm" ];
+
   src = fetchFromGitHub {
     owner = "grafana";
     repo = "grafana";
@@ -58,7 +60,7 @@ buildGoModule rec {
       export HOME="$(mktemp -d)"
       yarn config set enableTelemetry 0
       yarn config set cacheFolder $out
-      yarn config set --json supportedArchitectures.os '[ "linux" ]'
+      yarn config set --json supportedArchitectures.os '[ "linux", "darwin" ]'
       yarn config set --json supportedArchitectures.cpu '["arm", "arm64", "ia32", "x64"]'
       yarn
       runHook postBuild
@@ -67,12 +69,14 @@ buildGoModule rec {
     dontInstall = true;
     dontFixup = true;
     outputHashMode = "recursive";
-    outputHash = "sha256-70eMa8E483f/Bz7iy+4Seap1EfIdjD5krnt6W9CUows=";
+    outputHash = "sha256-pqInPfZEg2tcp8BXg1nnMddRZ1yyZ6KQa2flWd4IZSU=";
   };

   disallowedRequisites = [ offlineCache ];

-  vendorHash = "sha256-nGv/DBNnQ4AOJtrsYIGLCrV1xNmBN0dDf6u46R3TAHo=";
+  proxyVendor = true;
+
+  vendorHash = "sha256-Ig7Vj3HzMVOjT6Tn8aS4W000uKGoPOCFh2yIfyEWn78=";

   nativeBuildInputs = [ wire yarn jq moreutils removeReferencesTo python3 ];

@@ -91,10 +95,6 @@ buildGoModule rec {
     GOARCH= CGO_ENABLED=0 go generate ./public/app/plugins/gen.go
     GOARCH= CGO_ENABLED=0 go generate ./pkg/kindsys/report.go

-    # Work around `main module (github.com/grafana/grafana) does not contain package github.com/grafana/grafana/pkg/util/xorm`.
-    # Apparently these files confuse the dependency resolution for the go builder implemented here.
-    rm pkg/util/xorm/go.{mod,sum}
-
     # The testcase makes an API call against grafana.com:
     #
     # [...]

[shivaraj-bh]

rant: grafana check takes longer time than the build itself

[Ma27]

Yeah, known issue.
#292997 is a potential fix (at the cost of skipping all tests).
See also #285362 & #284568

[Ma27]

@shivaraj-bh would you mind rebasing your changes onto master and PR them? Grafana is now a little more pleasant to work with given that it builds way faster and I this should probably be fixed (with or without a proper fix in Nix).

Also, broken Darwin appears to have unintended side-effects for Hydra: #296193 (comment)

[shivaraj-bh]

would you mind rebasing your changes onto master and PR them?

Will do

[shivaraj-bh]

Well, grafana 10.4.0 received a lot of changes, so I had to fix a few more issues, but the PR is finally ready ^