Provenance causes `Error: no matching manifest for linux/amd64 in the manifest list entries` error in `balena deploy`

[shaunco]

When I build a Docker container using docker/build-push-action with provenance: mode=max and sbom: true it creates a multi-arch manifest that really is just single-arch + provenance, targeting arm64 (aarch64):

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:ff90c4b1a4c0d4299919d8c0bb9260ffedd3506836cb12c87935215eb305d78c",
      "size": 6169,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:23e95dc47ff87c2068c3ebcf5d1d0510a1781b4d6b58126a4cdcdd105864ce23",
      "size": 842,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:ff90c4b1a4c0d4299919d8c0bb9260ffedd3506836cb12c87935215eb305d78c",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

If I then have a Dockerfile that has a FROM that references an ARG pointing to the manifest:

ARG DTAG
FROM ${DTAG}

and a docker-compose.yml with:

version: '2.4'

services:
  myimage:
    build:
      context: .
      dockerfile: ./Dockerfile
      args:
        DTAG: '{{ .DTAG }}'

This works fine for docker-compose build --build-arg DTAG=myregistry.com/myimage:sha-2e991d20-arm64, but if I try to use balena deploy myaarch64fleet --debug, with a .balena/balena.yml file that has:

build-variables:
  global:
    - DTAG=myregistry.com/myimage:sha-2e991d20-arm64

I get:

[Debug]   Parsing input...
[Debug]   Loading project...
[Debug]   Resolving project...
[Debug]   docker-compose.yml file found at "/home/runner/work/test/balena"
[Debug]   Creating project...
[Build]   Building services...
[Build]   myimage          Preparing...
[Info]    Building for aarch64/generic-aarch64
[Debug]   Found build tasks:
[Debug]       myimage: build [.]
[Debug]   Resolving services with [generic-aarch64|aarch64]
[Debug]   Found project types:
[Debug]       myimage: Standard Dockerfile
[Debug]   Prepared tasks; building...
[Debug]   myimage: Image manifest data unavailable for ${DTAG}
[Build]   Built 1 services in 0:00
Error:    Deploy failed
no matching manifest for linux/amd64 in the manifest list entries

Error: no matching manifest for linux/amd64 in the manifest list entries
    at Stream.<anonymous> (/snapshot/balena-cli/node_modules/@balena/compose/dist/build/builder.js:107:23)
    at Stream.write (/snapshot/balena-cli/node_modules/through/index.js:26:11)
    at Stream.ondata (node:internal/streams/legacy:20:31)
    at Stream.emit (node:events:537:28)
    at drain (/snapshot/balena-cli/node_modules/through/index.js:36:16)
    at Stream.<anonymous> (/snapshot/balena-cli/node_modules/through/index.js:45:5)
    at Parser.onToken (/snapshot/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.write (/snapshot/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/snapshot/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.write (/snapshot/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (node:internal/streams/readable:766:22)
    at IncomingMessage.emit (node:events:537:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at HTTPParser.parserOnBody (node:_http_common:130:24)

So, first off, this all works fine if Provenance/SBOM is disabled. Next, I'm sure you're wondering why use an ARG for the FROM. We do this because we generate a multi-arch docker image for use in a bunch of places, and when we use it on Balena, we point to just one architecture to keep Balena happy.

With no provenance/SBOM, that is a "mediaType": "application/vnd.docker.distribution.manifest.v2+json", but with provenance/SBOM it is a "mediaType": "application/vnd.oci.image.index.v1+json". #22 seems like it fixed this, but something is still off.

In the above error, I am targeting a linux/arm64 platform but running the build on a linux/amd64 platform... and it is failing to find the amd64 manifest. If I target a linux/amd64 on the linux/amd64 builder it works fine.

Additionally, the [Debug] myimage: Image manifest data unavailable for ${DTAG} appears that ARG parsing needs to happen before walking the docker-compose services.

I'm still digging, but I figured 6 hours in I would get a ticket started...

[shaunco]

If I point DTAG to the top-level multi-arch, which looks like:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:6007b352b1369746d5744533617d3a4f54d37444a072ccce0546c711be203a7a",
      "size": 6356,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:392df90fbf789e9d61d347efdd75ad18500895817feb67c730b25434f44f935f",
      "size": 842,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:6007b352b1369746d5744533617d3a4f54d37444a072ccce0546c711be203a7a",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:ff90c4b1a4c0d4299919d8c0bb9260ffedd3506836cb12c87935215eb305d78c",
      "size": 6169,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:23e95dc47ff87c2068c3ebcf5d1d0510a1781b4d6b58126a4cdcdd105864ce23",
      "size": 842,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:ff90c4b1a4c0d4299919d8c0bb9260ffedd3506836cb12c87935215eb305d78c",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

balena deploy succeeds, but it gets linux/amd64 images in it even when targeting a fleet that is arm64/aarch64.

Is there any way to specify the target platform architecture on the balena cmdline? I don't see any such option for balena deploy, and this issue of creating the wrong images for the target fleet's architecture is how we got to the wackiness in the first post where our matrix build fills in the DTAG with a specific target architecture to match the fleet because balena deploy seems to always just use the architecture of the machine where balena deploy is called.

[shaunco]

I tried adding platform: linux/____ to the service in docker-compose.yml, but it still tried to access the wrong platform.

I ended up just using sed to swap the ${DTAG} value in Dockerfile with what I was setting DTAG to in .balena/balena.yml for each branch of my matrix build and all seems good.

So, I suppose this just comes down to balena-compose not supporting ARGs in the FROM of a Dockerfile that is built whereas docker-compose supports this. It would appear that issue only happens though when the ARG points to a application/vnd.oci.image.index.v1+json instead of a application/vnd.docker.distribution.manifest.v2+json, where the platform selection logic picks the platform of the host where balena cli is running instead of the fleet platform or docker-compose platform value... and that was just exposed because when Provenance is enabled it is a manifest of manifests.

Even with it "working" now, I still get this in the log:

[Debug]   myimage: Using platform option for build: linux/arm64/v8
.
.
.
[Build]   myimage          Step 1/4 : ARG DTAG=DTAG_REQUIRED
[Build]   myimage          Step 2/4 : FROM myregistry.com/myimage:sha-2e991d20-arm64
[Build]   myimage           ---> f92f4a0b5067
[Build]   myimager          Step 3/4 : LABEL maintainer "xyz<info@xyz.com>"
Warning:  myimage           ---> [Warning] The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested
[Build]   myimage           ---> Running in 9ea771d73ada
[Build]   myimage           ---> Removed intermediate container 9ea771d73ada
[Build]   myimage           ---> 32572e43783b

... which looks like platform selection is still broken, but it was forced.