Shellcheck fails on M2 (without Rosetta installed)

[sureshjoshi]

After running pants peek :: in the Pants main repo, I get the following error:

Error launching process: Os { code: 86, kind: Uncategorized, message: "Bad CPU type in executable" }

From the ShellCheck subsystem, I see that the Mac ARM version points to the x86 binaries (there are no Mac ARM binaries in the GH releases):

class Shellcheck(TemplatedExternalTool):
    options_scope = "shellcheck"
    name = "Shellcheck"
    help = "A linter for shell scripts."

    default_version = "v0.8.0"
    default_known_versions = [
        "v0.8.0|macos_arm64 |e065d4afb2620cc8c1d420a9b3e6243c84ff1a693c1ff0e38f279c8f31e86634|4049756",
        "v0.8.0|macos_x86_64|e065d4afb2620cc8c1d420a9b3e6243c84ff1a693c1ff0e38f279c8f31e86634|4049756",
        "v0.8.0|linux_arm64 |9f47bbff5624babfa712eb9d64ece14c6c46327122d0c54983f627ae3a30a4ac|2996468",
        "v0.8.0|linux_x86_64|ab6ee1b178f014d1b86d1e24da20d1139656c8b0ed34d2867fbb834dad02bf0a|1403852",
    ]

    default_url_template = (
        "https://github.com/koalaman/shellcheck/releases/download/{version}/shellcheck-"
        "{version}.{platform}.tar.xz"
    )
    default_url_platform_mapping = {
        "macos_arm64": "darwin.x86_64",
        "macos_x86_64": "darwin.x86_64",
        "linux_arm64": "linux.aarch64",
        "linux_x86_64": "linux.x86_64",
    }

Mitigation

I think that installing Rosetta might fix this...

Context

• Error launching process: Os { code: 86, kind: Uncategorized, message: "Bad CPU type in executable" } #14416
• Fails on apple M2 chip  koalaman/shellcheck#2680
• Release binaries for darwin-aarch64 koalaman/shellcheck#2714

Detailed Error

native_engine.IntrinsicError: Failed to execute: Process {
    argv: [
        "./shellcheck-v0.8.0/shellcheck",
        "--format=json",
        "cargo",
    ],
    env: {},
    working_directory: None,
    input_digests: InputDigests {
        complete: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<74712eda6b804fa6f81dea6d4b7008be9e369f40d2c7c2fce92bb59ca25a9ba3>,
                size_bytes: 174,
            },
            tree: "Some(..)",
        },
        nailgun: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>,
                size_bytes: 0,
            },
            tree: "Some(..)",
        },
        inputs: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<74712eda6b804fa6f81dea6d4b7008be9e369f40d2c7c2fce92bb59ca25a9ba3>,
                size_bytes: 174,
            },
            tree: "Some(..)",
        },
        immutable_inputs: {},
        use_nailgun: {},
    },
    output_files: {},
    output_directories: {},
    timeout: None,
    execution_slot_variable: None,
    concurrency_available: 0,
    description: "Detect Shell imports for cargo",
    level: Debug,
    append_only_caches: {},
    jdk_home: None,
    cache_scope: Always,
    execution_environment: ProcessExecutionEnvironment {
        name: None,
        platform: Macos_arm64,
        strategy: Local,
    },
    remote_cache_speculation_delay: 0ns,
}

Error launching process: Os { code: 86, kind: Uncategorized, message: "Bad CPU type in executable" }

[sureshjoshi]

I still think one big question is why this was run in the first place... I was peeking, so not sure why a lint rule suddenly ran.

[gauthamnair]

Ok, I see it is because we use ShellCheck for dependency inference. That is why the failure to run shellcheck is getting in the way even when we are not linting.

pants/src/python/pants/backend/shell/dependency_inference.py

Lines 96 to 111 in 9382fe2

	async def parse_shell_imports(
	request: ParseShellImportsRequest, shellcheck: Shellcheck, platform: Platform
	) -> ParsedShellImports:
	# We use Shellcheck to parse for us by running it against each file in isolation, which means
	# that all `source` statements will error. Then, we can extract the problematic paths from the
	# JSON output.
	downloaded_shellcheck = await Get(
	DownloadedExternalTool, ExternalToolRequest, shellcheck.get_request(platform)
	)
	input_digest = await Get(Digest, MergeDigests([request.digest, downloaded_shellcheck.digest]))
	process_result = await Get(
	FallibleProcessResult,
	Process(
	# NB: We do not load up `[shellcheck].{args,config}` because it would risk breaking
	# determinism of dependency inference in an unexpected way.
	[downloaded_shellcheck.exe, "--format=json", request.fp],

[gauthamnair]

Shellcheck is available from VSCode: https://github.com/vscode-shellcheck/shellcheck-binaries/releases

and this is the recommendation of shellcheck itself:

https://github.com/koalaman/shellcheck/blob/90d3172dfec30a7569f95b32479ae97af73b8b2e/README.md?plain=1#L236-L237

There are currently no official binaries for Apple Silicon, but third party builds are available via ShellCheck for Visual Studio Code.

I'll make a PR to add references to this source of binaries.