db_service Failed to connect to Proxy Server through tunnel, but trying to connect to 127.0.0.1:3022, not the configured proxy

[vgrig-us]

Expected behavior:
db_service is expected to connect to proxy service on auth or proxy server (running on another node) configured

Current behavior:
After successfully registering, error show teleport with db_service configured fails to connect to 127.0.0.1:3022
I am new to teleport, so - be gentle. :-)

Bug details:

• Teleport version - 18.2.10
• Recreation steps - happens on initial start and every restart
• Debug logs
  2025-10-29T16:50:03.844052-04:00 ca01-is-app-05 systemd[1]: Started teleport.service - Teleport Service.
  2025-10-29T16:50:03.980224-04:00 ca01-is-app-05 /usr/local/bin/teleport[2212]: 2025-10-29T16:50:03.979-04:00 INFO Starting Teleport with a config file version:18.2.10 config_file:/etc/teleport.yaml common/teleport.go:879
  2025-10-29T16:50:04.483163-04:00 ca01-is-app-05 /usr/local/bin/teleport[2212]: 2025-10-29T16:50:04.482-04:00 INFO [PROC:1] Service is creating new listener. pid:2212.1 type:debug address:/var/lib/teleport/debug.sock service/signals.go:242
  2025-10-29T16:50:04.486986-04:00 ca01-is-app-05 /usr/local/bin/teleport[2212]: 2025-10-29T16:50:04.484-04:00 INFO [PROC:1] Joining the cluster with a secure token. pid:2212.1 service/connect.go:532
  2025-10-29T16:50:04.487071-04:00 ca01-is-app-05 /usr/local/bin/teleport[2212]: 2025-10-29T16:50:04.484-04:00 INFO Attempting registration. method:via proxy server join/join.go:397
  2025-10-29T16:50:04.532876-04:00 ca01-is-app-05 /usr/local/bin/teleport[2212]: 2025-10-29T16:50:04.532-04:00 INFO Successfully registered. method:via proxy server join/join.go:404
  2025-10-29T16:50:04.535841-04:00 ca01-is-app-05 /usr/local/bin/teleport[2212]: 2025-10-29T16:50:04.535-04:00 INFO [PROC:1] Successfully obtained credentials to connect to the cluster. pid:2212.1 identity:Instance service/connect.go:383
  2025-10-29T16:50:04.541274-04:00 ca01-is-app-05 /usr/local/bin/teleport[2212]: 2025-10-29T16:50:04.541-04:00 ERRO [PROC:1] Failed to establish connection to cluster. pid:2212.1 identity:Instance error:[#012ERROR REPORT:#012Original Error: *errors.errorString Failed to connect to Proxy Server through tunnel: connection error: desc = "transport: Error while dialing: failed to dial: dial tcp 127.0.0.1:3022: connect: connection refused"#012Stack Trace:teleport connect and remote authorities #12#011github.com/gravitational/teleport/lib/service/connect.go:1276 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient#012#011github.com/gravitational/teleport/lib/service/connect.go:1170 github.com/gravitational/teleport/lib/service.(*TeleportProcess).getConnector#012#011github.com/gravitational/teleport/lib/service/connect.go:393 github.com/gravitational/teleport/lib/service.(*TeleportProcess).firstTimeConnect#012#011github.com/gravitational/teleport/lib/service/connect.go:220 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect#012#011github.com/gravitational/teleport/lib/service/connect.go:193 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService#012#011github.com/gravitational/teleport/lib/service/connect.go:87 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService#012#011github.com/gravitational/teleport/lib/service/service.go:3600 github.com/gravitational/teleport/lib/service.(*TeleportProcess).RegisterWithAuthServer.func1#012#011github.com/gravitational/teleport/lib/service/supervisor.go:606 github.com/gravitational/teleport/lib/service.(*LocalService).Serve#012#011github.com/gravitational/teleport/lib/service/supervisor.go:329 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1#012#011runtime/asm_amd64.s:1700 runtime.goexit#012User Message: Failed to connect to Proxy Server through tunnel: connection error: desc = "transport: Error while dialing: failed to dial: dial tcp 127.0.0.1:3022: connect: connection refused"] service/connect.go:102

[zmb3]

Hi @vgrig-us, in order to help you we'll need you to provide steps for us to recreate the issue. You mentioned you're new to Teleport, so we're likely dealing with a configuration issue here.

Let's start with the configuration file for your DB service and see if anything looks off.

[vgrig-us]

here you go. and i did check DNS resolution.

---

version: v3
teleport:
  nodename: ca01-is-app-05.example.com
  proxy_server: https://ca01-is-auth-02.example.com:3080
  data_dir: /var/lib/teleport
  join_params:
    token_name: "token"
    method: token
  log:
    output: syslog
    severity: INFO
    format:
      output: text
  ca_pin: "sha256:56696c66338ffa891c8dacaeba97707c881a0a4233f29b6xxxx"
  diag_addr: ""
db_service:
  enabled: "yes"
  databases:
    - name: "mariadb-ca01-is-app-05"  # Label for the DB in Teleport
      protocol: "mysql"  # MariaDB uses MySQL protocol
      uri: "127.0.0.1:3306"  # Adjust if MariaDB is on a different host/port
      static_labels:
        env: dev-test-ca
        type: standalone

[zmb3]

That looks good. Let's look at the proxy configuration next.

[vgrig-us]

See below. Also - proxy service running on both auth and public proxy - same error when trying both. And all teleport ssh services work properly (including when i try it on db_service host.

version: v3
teleport:
  nodename: ca01-is-auth-02.example.com
  data_dir: /var/lib/teleport
  join_params:
    token_name: "token"
    method: token
  connection_limits:
    max_connections: 100
  log:
    output: syslog
    severity: INFO
    format:
      output: text
auth_service:
  enabled: "yes"
  cluster_name: teleport.example.com
  listen_addr: 0.0.0.0:3025
ssh_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:22222
  public_addr: 172.21.21.22:22222
proxy_service:
  enabled: "yes"
  tunnel_listen_addr: 0.0.0.0:3022
  proxy_protocol: off
  https_keypairs:
    - key_file: /etc/letsencrypt/live/example.com/privkey.pem
      cert_file: /etc/letsencrypt/live/example.com/fullchain.pem
  https_keypairs_reload_interval: 0s

[zmb3]

proxy_service:
  enabled: "yes"
  tunnel_listen_addr: 0.0.0.0:3022

The tunnel_listen_addr looks off. Any particular reason to override the default there?

[vgrig-us]

I added tunnel_listen_addr after i started to have this problem. As far as i understand, this is legacy - listening for tunnel on 3022, not used anymore on regular basis. If i remove that settings, error from db_service server changes to "Error while dialing: failed to dial: dial tcp 127.0.0.1:3080" (with "Failed to connect to Proxy Server through tunnel" still in logs).

[zmb3]

You shouldn't need to explicitly set tunnel_listen_addr.

Your DB service will dial whatever proxy_server you configure, which looks to be:

proxy_server: https://ca01-is-auth-02.example.com:3080

I would remove the https:// from there so you just have host:port.

If it's still trying to dial 127.0.0.1 then that would likely be a DNS issue.

[vgrig-us]

Ok, i'll go over settings again over next couple of days - see if i can find solution.
Thank you for your help! :-)

[vgrig-us]

Ok, so my auth server runs proxy as well - same hostname. if i configure ssh server on app server and give it auth_server: - everything works. and if i give it proxy_server: - it tries to connect to 127.0.0.1. how's that even possible? is there a requirement for reverse DNS resolution if proxy_server: used in the config?
One possible other reason - is there any issue of using letsencrypt wildcard certificates for proxy server? since both servers are on the same domain, does ssh/db_service looks at the wilcard cert of the proxy server and assumes it's its own and connect to 127.0.0.1?

[vgrig-us]

OK, i had to add:
public_addr: ca01-is-auth-02.example.com:3080
tunnel_listen_addr: 0.0.0.0:3022
to proxy config and it works now. weird, but usable.
thank you again for your help! :-)