how can we pull images from a private registry from colima kubernetes?

[apple-corps]

My current .docker/config.json looks like below, but alternatively I could configure a credentials helper. However I am unable to pull images from the colima configured kubernetes from the private registry. How can we pull images from a private registry from colima kubernetes?

{
        "auths": {
                "my-reg-1": {
                        "auth": "some-value"
                },
                "my-reg-2": {
                        "auth": "a-different-value"
                }
        },
        "currentContext": "colima"
}

[mgs255]

Hi - I'm trying to achieve the same sort of thing and have spent quite some time fiddling trying to get this to work. I've made some progress, but there are some issues I'd like to overcome.

First off I'm using an MacOS 13.5, trying to use:

• colima and lima installed via brew at HEAD
• kubernetes 1.24.x
• containerd
• a private registry which in our case is a Harbor registry proxy (https://goharbor.io/) pulling through to ECR.
• traefik ingress disabled (was eventually intending on using nginx)

According to the K3S docs, the private registry configuration should be automatically picked up from /etc/rancher/k3s/registries.yaml. However this doesn't seem to work for me. However, if I stop the k3s service: sudo rc-service k3s stop, then manually run the k3s service using:

sudo /usr/local/bin/k3s server --debug \
    --private-registry /etc/rancher/k3s/registries.yaml \
    --disable traefik

That does allow me to pull down images from ECR using the registry config:

mirrors:
  <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com:
    endpoint:
     - "https://harbor.mycompany.com"
    rewrite:
      "^(.*)": "myproject/$1"
configs:
  "harbor.mycompany.com":
    auth:
      username: <username>
      password: <harbor_cli_token>

I'd be interested to know if anyone is able to progress this further. It would be really useful to be able to get this working for local development. For completeness I'm using the following colima configuration

# Number of CPUs to be allocated to the virtual machine.
# Default: 2
cpu: 2

# Size of the disk in GiB to be allocated to the virtual machine.
# NOTE: changing this has no effect after the virtual machine has been created.
# Default: 60
disk: 60

# Size of the memory in GiB to be allocated to the virtual machine.
# Default: 2
memory: 2

# Architecture of the virtual machine (x86_64, aarch64, host).
# Default: host
arch: aarch64

# Container runtime to be used (docker, containerd).
# Default: docker
runtime: containerd

# Kubernetes configuration for the virtual machine.
kubernetes:
  # Enable kubernetes.
  # Default: false
  enabled: true
  
  # Kubernetes version to use.
  # This needs to exactly match a k3s version https://github.com/k3s-io/k3s/releases
  # Default: latest stable release
  version: v1.24.15+k3s1
  
  # Additional args to pass to k3s https://docs.k3s.io/cli/server
  # Default: traefik is disabled
  k3sArgs:
    - --disable traefik
    - --private-registry /etc/rancher/k3s/registries.yaml

# Auto-activate on the Host for client access.
# Setting to true does the following on startup
#  - sets as active Docker context (for Docker runtime).
#  - sets as active Kubernetes context (if Kubernetes is enabled).
# Default: true
autoActivate: true

# Network configurations for the virtual machine.
network:
  # Assign reachable IP address to the virtual machine.
  # NOTE: this is currently macOS only and ignored on Linux.
  # Default: false
  address: true
  
  # Custom DNS resolvers for the virtual machine.
  #
  # EXAMPLE
  # dns: [8.8.8.8, 1.1.1.1]
  #
  # Default: []
  dns: []
  
  # DNS hostnames to resolve to custom targets using the internal resolver.
  # This setting has no effect if a custom DNS resolver list is supplied above.
  # It does not configure the /etc/hosts files of any machine or container.
  # The value can be an IP address or another host.
  #
  # EXAMPLE
  # dnsHosts:
  #   example.com: 1.2.3.4
  dnsHosts:
    host.docker.internal: host.lima.internal
  
  # Network driver to use (slirp, gvproxy), (requires vmType `qemu`)
  #   - slirp is the default user mode networking provided by Qemu
  #   - gvproxy is an alternative to VPNKit based on gVisor https://github.com/containers/gvisor-tap-vsock
  # Default: gvproxy
  driver: gvproxy

# ===================================================================== #
# ADVANCED CONFIGURATION
# ===================================================================== #

# Forward the host's SSH agent to the virtual machine.
# Default: false
forwardAgent: false

# Docker daemon configuration that maps directly to daemon.json.
# https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file.
# NOTE: some settings may affect Colima's ability to start docker. e.g. `hosts`.
#
# EXAMPLE - disable buildkit
# docker:
#   features:
#     buildkit: false
#
# EXAMPLE - add insecure registries
# docker:
#   insecure-registries:
#     - myregistry.com:5000
#     - host.docker.internal:5000
#
# Colima default behaviour: buildkit enabled
# Default: {}
docker: {}

# Virtual Machine type (qemu, vz)
# NOTE: this is macOS 13 only. For Linux and macOS <13.0, qemu is always used.
#
# vz is macOS virtualization framework and requires macOS 13
#
# Default: qemu
vmType: vz

# Utilise rosetta for amd64 emulation (requires m1 mac and vmType `vz`)
# Default: false
rosetta: false

# Volume mount driver for the virtual machine (virtiofs, 9p, sshfs).
#
# virtiofs is limited to macOS and vmType `vz`. It is the fastest of the options.
#
# 9p is the recommended and the most stable option for vmType `qemu`.
#
# sshfs is faster than 9p but the least reliable of the options (when there are lots
# of concurrent reads or writes).
#
# Default: virtiofs (for vz), sshfs (for qemu)
mountType: virtiofs

# Propagate inotify file events to the VM.
# NOTE: this is experimental.
mountInotify: false

# The CPU type for the virtual machine (requires vmType `qemu`).
# Options available for host emulation can be checked with: `qemu-system-$(arch) -cpu help`.
# Instructions are also supported by appending to the cpu type e.g. "qemu64,+ssse3".
# Default: host
cpuType: host

# For a more general purpose virtual machine, Ubuntu container is optionally provided
# as a layer on the virtual machine.
# The underlying virtual machine is still accessible via `colima ssh --layer=false` or running `colima` in
# the Ubuntu session.
#
# Default: false
layer: false

# Custom provision scripts for the virtual machine.
# Provisioning scripts are executed on startup and therefore needs to be idempotent.
#
# EXAMPLE - script exected as root
# provision:
#   - mode: system
#     script: apk add htop vim
#
# EXAMPLE - script exected as user
# provision:
#   - mode: user
#     script: |
#       [ -f ~/.provision ] && exit 0;
#       echo provisioning as $USER...
#       touch ~/.provision
#
# Default: []
provision: []

# Modify ~/.ssh/config automatically to include a SSH config for the virtual machine.
# SSH config will still be generated in ~/.colima/ssh_config regardless.
# Default: true
sshConfig: true

# Configure volume mounts for the virtual machine.
# Colima mounts user's home directory by default to provide a familiar
# user experience.
#
# EXAMPLE
# mounts:
#   - location: ~/secrets
#     writable: false
#   - location: ~/projects
#     writable: true
#
# Colima default behaviour: $HOME and /tmp/colima are mounted as writable.
# Default: []
mounts: []

# Environment variables for the virtual machine.
#
# EXAMPLE
# env:
#   KEY: value
#   ANOTHER_KEY: another value
#
# Default: {}
env: {}

[blast-hardcheese]

Some ad-hoc documentation for how to get this set up for Google Artifact Registry (GAR) in k3s managed by colima backed by containerd:

1. Create a service account
2. Run the following gcloud --project <PROJECT> artifacts repositories add-iam-policy-binding infrastructure --member="serviceAccount:<SA-NAME>@<PROJECT>.iam.gserviceaccount.com" --role='roles/artifactregistry.reader' --location <LOCATION> to grant artifactregistry.reader access to your SA
3. Create and download a Key for the new service account. This will come as a JSON file with the following keys: auth_provider_x509_cert_url, auth_uri, client_email, client_id, client_x509_cert_url, private_key, private_key_id, project_id, token_uri, type, universe_domain
4. Use base64 -w0 on linux (or base64 without -w0 if you're on a mac) to encode the contents of the JSON file into one long string: base64 -w0 < your-sa-53123abcd.json
5. Finally, create a docker-registry secret specifying the GAR server, your SA's IAM email, the hard-coded username of _json_key_base64 (see documentation for references to this magic username), and the password is the single long base64-encoded string from the previous step:

   kubectl create secret docker-registry <SECRET-NAME> \
     --docker-server=<LOCATION>-docker.pkg.dev \
     --docker-email=<SA-NAME>@<PROJECT>.iam.gserviceaccount.com \
     --docker-username=_json_key_base64 \
     --docker-password="<BASE64-KEY>"
   

6. Finally, in your K8s manifests that use this registry, you must reference the secret we created in step 5

   spec:
     imageName: <LOCATION>-docker.pkg.dev/<PROJECT>/<REGISTRY>/...
     imagePullSecrets:
     - name: <SECRET-NAME>

Hopefully this helps somebody else, I wasted so much time trying to muck about with /etc/rancher/k3s/registries.yaml and even configuring containerd's hosts.toml.

A brief retrospective is that what may have been happening here is k3s + colima's containerd prevents k3s from writing hosts.toml (which is what seems to be happening today, from reading documentation which I no longer have at hand as well as the code for k3s directly).

Because that hosts.toml doesn't get generated and connected, colima's containerd never picks up the registries config from k3s, necessitating managing that file independently (does k3s' registries.yaml even matter at that point?)

What's likely causing yet more problems is that containerd has a concept of image namespace that's distinct from kubernetes' idea of namespace. My guess is that containerd is probably not associating requests from the k8s.io to need the configured credentials for the supplied GAR image namespace.

I ran out of patience and found the alternate above solution, but perhaps this is enough context that somebody else can finish knitting these things together. Presumably it all "just worked" with the docker backend, but we've got growing pains with the containerd backend.