redis: connection pool: failed to dial after 5 attempts: dial tcp: lookup argocd-redis: i/o timeout

[bianbbc87]

Describe the bug
When deploying the argocd-agent in a multi-cluster KIND environment, the agent repeatedly failed to connect to the Redis instance in the principal cluster. Logs from both argocd-agent-agent (workload cluster) and argocd-agent-principal (control-plane cluster) showed DNS resolution failures for the hostname argocd-redis. The error persisted even though the Redis service was running and reachable within the principal cluster.

Steps to reproduce the behaviour

1. Create two KIND clusters: one for the principal (kind-argocd-hub) and one for the agent (agent cluster).
2. Deploy Argo CD components (including argocd-redis) in the argocd namespace on the principal cluster.
3. Deploy argocd-agent-agent in the agent cluster using mTLS configuration.
4. Observe agent logs after startup.

Expected behavior
The agent should establish a successful mTLS connection to the principal and communicate through the Redis proxy without DNS lookup failures.

Additional context
Add any other context about the problem here.
Agent logs:

redis: connection pool: failed to dial after 5 attempts: dial tcp: lookup argocd-redis: i/o timeout
time="..." level=error msg="Failed to get cluster info from cache" error="dial tcp: lookup argocd-redis: i/o timeout"

Principal logs:

failed to get connection info from cluster: dial tcp: lookup argocd-redis: i/o timeout

This occurred because the agent attempted to resolve the Redis hostname from a separate cluster where the DNS record did not exist.

[bianbbc87]

Previously, the tcp: lookup argocd-redis error occurred because the agent couldn’t resolve the Redis service DNS.
To fix this, we modified the following NetworkPolicy files to allow DNS lookup queries:

• argocd-agent/install/kubernetes/agent/agent-networkpolicy-redis.yaml
• argocd-agent/install/kubernetes/principal/principal-networkpolicy-redis.yaml

ports:
- port: 6379
  protocol: TCP # existing rule
# added DNS query rules below
- to:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: kube-system
  ports:
  - port: 53
    protocol: UDP
  - port: 53
    protocol: TCP

[bianbbc87]

After resolving the DNS issue, a new problem appeared both argocd-agent-agent and argocd-agent-principal pods started failing to connect to Redis with the following error:

dial tcp <ClusterIP>:6379: i/o timeout

This was later fixed by adding ingress permissions for both agent pods in the Redis NetworkPolicy via Kustomize.

ex: argocd-agent/install/kubernetes/argo-cd/agent-managed/kustomization.yaml

namespace: argocd
# ...
- target:
    group: networking.k8s.io
    version: v1
    kind: NetworkPolicy
    name: argocd-redis-network-policy
  patch: |-
    - op: add
      path: /spec/ingress/0/from/-
      value:
        podSelector:
          matchLabels:
            app.kubernetes.io/name: argocd-agent-agent

[jannfis]

Hey @bianbbc87, both the agent and the principal (and the Argo CDs on each cluster) should use their own, dedicated Redis available to them in their own cluster.

I know that https://argocd-agent.readthedocs.io/latest/concepts/argocd-integration/ talks about an integration pattern where the Redis is shared. However, this is a pattern from when we did not have the Redis proxy yet. We probably should note in the docs that this pattern is deprecated, or get rid of that pattern altogether.

[bianbbc87]

Hey @bianbbc87, both the agent and the principal (and the Argo CDs on each cluster) should use their own, dedicated Redis available to them in their own cluster.

I know that https://argocd-agent.readthedocs.io/latest/concepts/argocd-integration/ talks about an integration pattern where the Redis is shared. However, this is a pattern from when we did not have the Redis proxy yet. We probably should note in the docs that this pattern is deprecated, or get rid of that pattern altogether.

thank you for your comment !
but I followed the getting started guide(it was managed mode) step by step, and even after applying the latest NetworkPolicy patch from issue #566, the same connection problem persists.
Since I’m running this setup with kind, could that be the reason?

also i wonder how do I explicitly control or configure Pattern 2 in my environment.

[bianbbc87]

Additionally, both my principal and agent clusters already have their own separate Redis instances, so I'm not sure why they’re still trying to send requests to each other’s Redis.

[bianbbc87]

i tried set this way
#614

[jannfis]

also i wonder how do I explicitly control or configure Pattern 2 in my environment.

I think pattern 2 is the default when you don't perform special configuration (i.e. a standard Argo CD installation on each cluster, no special configuration for either agent or principal required).

I'm not sure why they’re still trying to send requests to each other’s Redis.

By default, both principal and agent will try to connect to the argocd-redis service in the same namespace as they are running, using the service DNS name:

argocd-agent/install/kubernetes/agent/agent-params-cm.yaml

Line 88 in d314838

agent.redis.address: "argocd-redis:6379"

and

argocd-agent/install/kubernetes/principal/principal-params-cm.yaml

Line 142 in d314838

principal.redis.server.address: "argocd-redis:6379"

[bianbbc87]

@jannfis

Okay, I also believe my environment is following Pattern 2.
i found the problem. It turned out that the ConfigMap was misconfigured, so the Redis server IP that the agent was trying to connect to was incorrect.

However, in my environment, both the principal and the agent are still experiencing DNS lookup issues when trying to access argocd-redis via their own DNS.
I’m currently investigating whether this issue is related to kind (although I think that’s unlikely). I struggled a bit with overlapping CIDR settings while creating the clusters with kind, so it’s possible that some of those changes caused this conflict.

I’ll continue analyzing the root cause and share an update once I resolve it.
Thank you.

[bianbbc87]

Hi @jannfis,
I think the dns lookup failures we discussed earlier might be related to coredns egress restrictions.

In my setup, both the principal and agent Pods were unable to resolve argocd-redis because udp/tcp 53 traffic to kube-system (coredns) was blocked by networkpolicy.
After allowing that egress, the lookups succeeded.

Does that sound consistent with your understanding of how dns resolution works in Pattern 2?

[akaroth]

I had similar issues when setting this up with kind. Hard time with connectivity on Mac but able to get it to work on linux. Network policies for redis is an issue and i fixed manually post which the issue resolved.

[bianbbc87]

I had similar issues when setting this up with kind. Hard time with connectivity on Mac but able to get it to work on linux. Network policies for redis is an issue and i fixed manually post which the issue resolved.

I also encountered a redis connection error while setting it up with KinD on mac os.
Would you mind reviewing my pr when you get a chance?
Thanks a lot!