emq集群的节点会出现emqx_authn_http相关的告警

[tywin2024]

环境

• Kubernetes: v1.26.3
• EMQX Operator Version: quay.io/tigera/operator:v1.29.0
• EMQX Version: 5.5.0 (Open Source)
• EMQX API Version: apps.emqx.io/v2beta1

创建emqx集群的yaml文件

apiVersion: [apps.emqx.io/v2beta1](http://apps.emqx.io/v2beta1)
kind: EMQX
metadata:
name: emqx
namespace: real
spec:
image: emqx:5.5.0
coreTemplate:
spec:
replicas: 3
volumeClaimTemplates:
storageClassName: standard
resources:
requests:
storage: 10Gi
accessModes:
- ReadWriteOnce
extraVolumes:
- name: ssl-self-sign
secret:
secretName: ssl-self-sign
extraVolumeMounts:
- name: ssl-self-sign
mountPath: /mounted/cert
replicantTemplate:
spec:
replicas: 3
extraVolumes:
- name: ssl-self-sign
secret:
secretName: ssl-self-sign
extraVolumeMounts:
- name: ssl-self-sign
mountPath: /mounted/cert
dashboardServiceTemplate:
spec:
type: NodePort
ports:
- name: dashboard
nodePort: 30811
port: 18083
protocol: TCP
targetPort: 18083
listenersServiceTemplate:
spec:
type: NodePort
ports:
- name: ssl-default
nodePort: 32349
port: 8883
protocol: TCP
targetPort: 8883
- name: tcp-default
nodePort: 32350
port: 1883
protocol: TCP
targetPort: 1883
- name: ws-default
nodePort: 32347
port: 8083
protocol: TCP
targetPort: 8083
- name: wss-default
nodePort: 32348
port: 8084
protocol: TCP
targetPort: 8084
- name: tcp-program
nodePort: 32351
port: 1993
protocol: TCP
targetPort: 1993

问题信息

我在EMQ的监控告警中看到了多条类似下面的告警信息

emqx_authn_http:6269953  resource down: closed [[email protected]](mailto:[email protected]) 系统 2025-04-17 01:28:22

reason: resource_down
resource_id: emqx_authn_http:11126819

如果是连接不稳定，这种情况可能的原因是什么，这是云上同一个内网中的服务器，应该不存在延迟之类的问题，服务器的资源消耗不算大

在emqx容器日志

2025-04-17T22:34:50.261151+08:00 [error] msg: http_connector_get_status_failed, mfa: emqx_bridge_http_connector:do_get_status/2(535), reason: closed, worker: <0.91569875.0>
2025-04-17T22:34:50.264632+08:00 [warning] msg: alarm_is_activated, mfa: emqx_alarm:do_actions/3(418), message: <<"resource down: closed">>, name: <<"emqx_authn_http:6287081">>
2025-04-17T22:34:50.264876+08:00 [warning] msg: health_check_failed, mfa: emqx_resource_manager:handle_connected_health_check/1(893), id: <<"emqx_authn_http:6287081">>, status: disconnected
···
#在15秒后，有开始连接的日志
2025-04-17T22:35:05.265923+08:00 [info] msg: starting_http_connector, mfa: emqx_bridge_http_connector:on_start/2(203), config: #{ssl => #{depth => 10,verify => verify_peer,hibernate_after => 5000,enable => false,ciphers => [],versions => ['tlsv1.3','tlsv1.2'],log_level => notice,secure_renegotiate => true,reuse_sessions => true,user_lookup_fun => {fun emqx_tls_psk:lookup/3,undefined}},connect_timeout => 2000,mechanism => password_based,pool_size => 20,enable => true,body => #{password => <<"******">>,userName => <<"${username}">>},headers => #{},url => <<"http://iot-device:41***/iotDevice/deviceAuth/mqttAuth">>,method => get,backend => http,request_timeout => 5000,pool_type => random,enable_pipelining => 100,base_url => #{port => 41***,scheme => http,path => "/",host => "iot-device"}}, connector: <<"emqx_authn_http:6287081">>
2025-04-17T22:35:05.266193+08:00 [warning] msg: emqx_connector_on_start_already_started, mfa: emqx_bridge_http_connector:start_pool/2(256), pool_name: <<"emqx_authn_http:6287081">>
2025-04-17T22:35:05.270613+08:00 [warning] msg: alarm_is_deactivated, mfa: emqx_alarm:do_actions/3(424), name: <<"emqx_authn_http:6287081">>

[tywin2024]

以下是补充的日志，另外似乎集群还有吞日志的情况，部分日志出现两天后自动消失了

{"time":1744981097122591,"level":"error","msg":"http_connector_get_status_failed","mfa":"emqx_bridge_http_connector:do_get_status/2(535)","worker":"<0.113003721.0>","reason":"closed","pid":"<0.42732945.2>"}
{"time":1744981097126333,"level":"warning","msg":"alarm_is_activated","mfa":"emqx_alarm:do_actions/3(418)","pid":"<0.2692.0>","name":"emqx_authn_http:11126878","message":"resource down: closed"}
{"time":1744981097126622,"level":"warning","msg":"health_check_failed","mfa":"emqx_resource_manager:handle_connected_health_check/1(893)","status":"disconnected","pid":"<0.113006056.0>","id":"emqx_authn_http:11126878"}
{"time":1744981112127173,"level":"info","msg":"starting_http_connector","mfa":"emqx_bridge_http_connector:on_start/2(203)","connector":"emqx_authn_http:11126878","config":{"url":"http://iot-device:41***/iotDeviceManager/deviceAuth/mqttAuth","ssl":{"versions":["tlsv1.3","tlsv1.2"],"verify":"verify_peer","user_lookup_fun":"{fun emqx_tls_psk:lookup/3,undefined}","secure_renegotiate":"true","reuse_sessions":"true","log_level":"notice","hibernate_after":5000,"enable":"false","depth":10,"ciphers":[]},"request_timeout":5000,"pool_type":"random","pool_size":20,"method":"get","mechanism":"password_based","headers":{},"enable_pipelining":100,"enable":"true","connect_timeout":2000,"body":{"userName":"${username}","password":"******"},"base_url":{"scheme":"http","port":41***,"path":"/","host":"iot-device"},"backend":"http"},"pid":"<0.113006056.0>"}
{"time":1744981112127568,"level":"warning","msg":"emqx_connector_on_start_already_started","mfa":"emqx_bridge_http_connector:start_pool/2(256)","pool_name":"emqx_authn_http:11126878","pid":"<0.113006056.0>"}
{"time":1744981112132115,"level":"warning","msg":"alarm_is_deactivated","mfa":"emqx_alarm:do_actions/3(424)","pid":"<0.2692.0>","name":"emqx_authn_http:11126878"}

[zmstone]

HTTP 客户端健康检查时报 closed 错误的原因是服务器关闭了 http 连接，或者网络问题。

[tywin2024]

http服务和emqx集群都在相同的K8s集群的3台工作节点服务器上，内网的不应该有网络问题，http服务也没有关闭过，另外下面的日志中第一条记录中的三个节点并不存在，且我的emqx集群也已经运行了4个月了

{"time":1744956905766836,"level":"error","msg":"cluster_rpc_peers_lagging","mfa":"emqx_cluster_rpc:multicall/5(162)","tnx_id":52,"nodes":["[email protected]","[email protected]","[email protected]"],"status":"stopped_nodes","pid":"<0.40974330.2>"}
{"time":1744958026480040,"level":"error","msg":"http_connector_get_status_failed","mfa":"emqx_bridge_http_connector:do_get_status/2(535)","worker":"<0.112758912.0>","reason":"closed","pid":"<0.41042704.2>"}
{"time":1744958026483443,"level":"warning","msg":"alarm_is_activated","mfa":"emqx_alarm:do_actions/3(418)","pid":"<0.2683.0>","name":"emqx_authn_http:11126819","message":"resource down: closed"}
{"time":1744958026483811,"level":"warning","msg":"health_check_failed","mfa":"emqx_resource_manager:handle_connected_health_check/1(893)","status":"disconnected","pid":"<0.112761807.0>","id":"emqx_authn_http:11126819"}
{"time":1744958041484693,"level":"info","msg":"starting_http_connector","mfa":"emqx_bridge_http_connector:on_start/2(203)","connector":"emqx_authn_http:11126819","config":{"url":"http://iot-device:41***/iotDevice/deviceAuth/mqttAuth","ssl":{"versions":["tlsv1.3","tlsv1.2"],"verify":"verify_peer","user_lookup_fun":"{fun emqx_tls_psk:lookup/3,undefined}","secure_renegotiate":"true","reuse_sessions":"true","log_level":"notice","hibernate_after":5000,"enable":"false","depth":10,"ciphers":[]},"request_timeout":5000,"pool_type":"random","pool_size":20,"method":"get","mechanism":"password_based","headers":{},"enable_pipelining":100,"enable":"true","connect_timeout":2000,"body":{"userName":"${username}","password":"******"},"base_url":{"scheme":"http","port":41***,"path":"/","host":"iot-device"},"backend":"http"},"pid":"<0.112761807.0>"}
{"time":1744958041485165,"level":"warning","msg":"emqx_connector_on_start_already_started","mfa":"emqx_bridge_http_connector:start_pool/2(256)","pool_name":"emqx_authn_http:11126819","pid":"<0.112761807.0>"}
{"time":1744958041489921,"level":"warning","msg":"alarm_is_deactivated","mfa":"emqx_alarm:do_actions/3(424)","pid":"<0.2683.0>","name":"emqx_authn_http:11126819"}

[zmstone]

cluster_rpc_peers_lagging 报错的节点可以用命令 emqx eval "emqx_cluster_rpc:force_leave_clean('[email protected]')" 来移除。

这个错误跟 认证器的 HTTP 连接告警没关系。