Cannot Look Up Images in Self Hosted Gitlab Container Registry

[jDmacD]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us what version of Renovate you run.

36.60.0

If you're self-hosting Renovate, select which platform you are using.

GitLab self-hosted

Was this something which used to work for you, and then stopped?

I am trying to get this working for the first time

Describe the problem

I cannot get renovate to lookup images in our private registry. All the configuration is done though CI/CD env vars.

## Repository problems

Renovate tried to run on this repository, but found these problems.

* WARN: Package lookup failures

---

### :warning: Dependency Lookup Warnings :warning:

* Renovate failed to look up the following dependencies: `Failed to look up docker package git-registry.comany.com/docker/image-builder/debian-buildx`.

Files affected: `.ci/default.yml`

RENOVATE_HOST_RULES
[{"matchHost": "${CI_REGISTRY}","username": "renovate-bot","password": "${RENOVATE_TOKEN}", "hostType": "docker"}]

RENOVATE_EXTRA_FLAGS
--autodiscover=true --onboarding=true --autodiscover-filter=architecture/*,/company/vault-.*/,multimedia/*,/multimedia/*,Multimedia/*,/Multimedia/*

Token permissions
api, read_api, read_user, read_repository, write_repository, read_registry

renovate.json

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:base"
  ],
  "regexManagers": [
    {
      "fileMatch":[
        ".+\\Dockerfile$"
      ],
      "matchStrings": [
        ".*ARG.*VERSION=(?<currentValue>.*)\\s#\\srenovate:\\sdatasource=(?<datasource>.*?)\\sdepName=(?<depName>.*?)\\s"
      ]
    }   
  ]
}

Relevant debug logs

Logs

       "warnings": [
         "Failed to look up docker package git-registry.company.com/docker/image-builder/debian-buildx"
       ],
       "files": [".ci/default.yml"]

[
    {
        "name": "renovate",
        "hostname": "runner-kx4nhkle-project-877-concurrent-0",
        "pid": 16,
        "level": 20,
        "logContext": "WkDgedbvtK0ghAo1bfFtx",
        "repository": "docker/image-builder",
        "msg": "GET https://git.company.com/jwt/auth?scope=repository%3Adocker%2Fimage-builder%2Fdebian-buildx%3Apull&service=container_registry = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=403 retryCount=0, duration=61)",
        "time": "2023-08-25T16:17:10.148Z",
        "v": 0
    },
    {
        "name": "renovate",
        "hostname": "runner-kx4nhkle-project-877-concurrent-0",
        "pid": 16,
        "level": 20,
        "logContext": "WkDgedbvtK0ghAo1bfFtx",
        "repository": "docker/image-builder",
        "registryHost": "https://git-registry.company.com",
        "dockerRepository": "docker/image-builder/debian-buildx",
        "msg": "Not allowed to access docker registry",
        "time": "2023-08-25T16:17:10.149Z",
        "v": 0
    },
    {
        "name": "renovate",
        "hostname": "runner-kx4nhkle-project-877-concurrent-0",
        "pid": 16,
        "level": 20,
        "logContext": "WkDgedbvtK0ghAo1bfFtx",
        "repository": "docker/image-builder",
        "err": {
            "name": "HTTPError",
            "code": "ERR_NON_2XX_3XX_RESPONSE",
            "timings": {
                "start": 1692980230086,
                "socket": 1692980230086,
                "lookup": 1692980230087,
                "connect": 1692980230101,
                "secureConnect": 1692980230104,
                "upload": 1692980230104,
                "response": 1692980230146,
                "end": 1692980230147,
                "phases": {
                    "wait": 0,
                    "dns": 1,
                    "tcp": 14,
                    "tls": 3,
                    "request": 0,
                    "firstByte": 42,
                    "download": 1,
                    "total": 61
                }
            },
            "message": "Response code 403 (Forbidden)",
            "stack": "HTTPError: Response code 403 (Forbidden)\n    at Request.<anonymous> (/opt/containerbase/tools/renovate/36.60.0/node_modules/got/dist/source/as-promise/index.js:118:42)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)",
            "options": {
                "headers": {
                    "user-agent": "RenovateBot/36.60.0 (https://github.com/renovatebot/renovate)",
                    "accept": "application/json",
                    "accept-encoding": "gzip, deflate, br"
                },
                "url": "https://git.company.com/jwt/auth?scope=repository%3Adocker%2Fimage-builder%2Fdebian-buildx%3Apull&service=container_registry",
                "hostType": "docker",
                "username": "",
                "password": "",
                "method": "GET",
                "http2": false
            },
            "response": {
                "statusCode": 403,
                "statusMessage": "Forbidden",
                "body": {
                    "errors": [
                        {
                            "code": "DENIED",
                            "message": "access forbidden"
                        }
                    ],
                    "http_status": 403
                },
                "headers": {
                    "cache-control": "no-cache, no-cache=\"set-cookie\"",
                    "content-security-policy": "",
                    "content-type": "application/json; charset=utf-8",
                    "date": "Fri, 25 Aug 2023 16:17:10 GMT",
                    "page-title": "GitLab",
                    "permissions-policy": "interest-cohort=()",
                    "server": "nginx",
                    "set-cookie": [
                        "AWSELB=1501DDD908F3FC572F89A3ABEBAEB28B4CE45618A39E833AFCD7654709C6E395A6058F1347D86E650CBF1F4F7FDBA7634DA261814205FEA0F2D9536822A5AAA2A4EF70B2CC;PATH=/;MAX-AGE=300",
                        "AWSELBCORS=1501DDD908F3FC572F89A3ABEBAEB28B4CE45618A39E833AFCD7654709C6E395A6058F1347D86E650CBF1F4F7FDBA7634DA261814205FEA0F2D9536822A5AAA2A4EF70B2CC;PATH=/;MAX-AGE=300;SECURE;SAMESITE=None"
                    ],
                    "vary": "Accept",
                    "x-content-type-options": "nosniff",
                    "x-download-options": "noopen",
                    "x-frame-options": "SAMEORIGIN",
                    "x-gitlab-custom-error": "1",
                    "x-permitted-cross-domain-policies": "none",
                    "x-request-id": "01H8PQRDYTSF3N9X9B2TDXSJTF",
                    "x-runtime": "0.022665",
                    "x-ua-compatible": "IE=edge",
                    "x-xss-protection": "1; mode=block",
                    "content-length": "77",
                    "connection": "Close"
                },
                "httpVersion": "1.1",
                "retryCount": 0
            }
        },
        "msg": "Response code 403 (Forbidden)",
        "time": "2023-08-25T16:17:10.152Z",
        "v": 0
    }
]

Have you created a minimal reproduction repository?

I have explained in the description why a minimal reproduction is impossible

[rarkins]

Does it appear that Renovate is querying the correct endpoints to begin with? If so then it's likely an auth issue and not a path/url issue

[jDmacD]

I did a test running renovate outside the pipeline, against a single repo with the same credentials and it seems to authenticate fine

docker run \
	--env RENOVATE_TOKEN=glpat-1234567 \
	--env RENOVATE_PLATFORM=gitlab \
	--env RENOVATE_ENDPOINT=https://git.company.com/api/v4/ \
	--env RENOVATE_ONBOARDING=false \
	--env RENOVATE_LOG_FILE_LEVEL=DEBUG \
	--env GITHUB_COM_TOKEN=ghp_1234567 \
	-v $(pwd)/renovate/:/tmp/renovate/ \
	-ti \
	--rm  \
	renovate/renovate:36.60.0-slim@sha256:661ad6cb0ee3274b02c524dad1ee916982b81cd16c6487f400bcc6497d4db1c4 \
	--host-rules='[{"hostType": "docker", "username": "renovate-bot", "password": "glpat-1234567","matchHost": "git-registry.company.com"}]' \
	--log-file=/tmp/renovate/renovate.log \
	docker/image-builder

{
    "gitlabci": [
        {
            "packageFile": ".ci/default.yml",
            "deps": [
                {
                    "depName": "git-registry.company.com/docker/image-builder/debian-buildx",
                    "currentValue": "0.0.1",
                    "replaceString": "git-registry.company.com/docker/image-builder/debian-buildx:0.0.1",
                    "autoReplaceStringTemplate": "{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}",
                    "datasource": "docker",
                    "depType": "image",
                    "updates": [],
                    "packageName": "git-registry.company.com/docker/image-builder/debian-buildx",
                    "versioning": "docker",
                    "warnings": [],
                    "sourceUrl": "https://git.company.com/docker/image-builder",
                    "registryUrl": "https://git-registry.company.com",
                    "currentVersion": "0.0.1",
                    "fixedVersion": "0.0.1"
                }
            ]
        }
    ]
}

[jDmacD]

This is strange. When I enable RENOVATE_LOG_FILE_LEVEL=TRACE in the pipeline, renovate authenticates correctly and everything works fine.

[
    {
        "repository": "docker/image-builder",
        "registryHost": "https://git-registry.company.com",
        "dockerRepository": "docker/image-builder/debian-buildx",
        "authUrl": "https://git.company.com/jwt/auth?scope=repository%3Adocker%2Fimage-builder%2Fdebian-buildx%3Apull&service=container_registry",
        "msg": "Obtaining docker registry token",
        "time": "2023-08-28T12:45:33.516Z",
        "v": 0
    },
    {
        "repository": "docker/image-builder",
        "url": "https://git.company.com/jwt/auth?scope=repository%3Adocker%2Fimage-builder%2Fdebian-buildx%3Apull&service=container_registry",
        "msg": "Authorization disabled",
        "time": "2023-08-28T12:45:33.517Z",
        "v": 0
    },
    {
        "repository": "docker/image-builder",
        "url": "https://git.company.com/jwt/auth?scope=repository%3Adocker%2Fimage-builder%2Fdebian-buildx%3Apull&service=container_registry",
        "options": {
            "method": "get",
            "context": {
                "hostType": "docker"
            },
            "hostType": "docker",
            "timeout": 60000,
            "headers": {
                "accept": "application/json",
                "authorization": "***********",
                "user-agent": "RenovateBot/36.60.0 (https://github.com/renovatebot/renovate)"
            },
            "noAuth": true,
            "responseType": "json",
            "hooks": {
                "beforeRedirect": [
                    "[function]"
                ]
            }
        },
        "msg": "got request",
        "time": "2023-08-28T12:45:33.517Z",
        "v": 0
    }
]

[aamarques]

Hope someone is looking at this too old topic :)
I have the same error message, and I found it's related to the way Renovate deals with TAGs. In my case, we don't use tags, but the SHA256 value. So, renovate can't deal with it in a normal situation because the expected is registry/image:tag:sha and not registry/image:sha.

How can we solve it ?

[jamietanna]

@aamarques sorry I'd missed you commenting here from mend/renovate-ce-ee#759

Can you please raise this as a separate Discussion? Especially due to the age of this Discussion, it's unlikely to be the same root cause