challenge backend failing health checks with GCE ingress

[ryane]

Background:

I have two existing GCE ingresses running without kube-lego. This is what the health checks look like:

$ gcloud compute http-health-checks list
NAME                            HOST  PORT   REQUEST_PATH
k8s-be-30804--eeba4b4d12737265        30804  /login
k8s-be-32516--eeba4b4d12737265        32516  /
k8s-be-32742--eeba4b4d12737265        32742  /healthz

The /login path is for a legacy service that does not expose a dedicated, unauthenticated health(z) endpoint. /login is used b/c it doesn't require authentication and returns a 200. The deployment behind this ingress exposes /login as a readinessProbe.

Issue:

Now I want to test kube-lego (0.1.2) with the echoserver sample. After deploying the echoserver ingress (and updating dns), I see a lot of errors in the kube-lego logs indicating that the reachability test is failing with wrong status code '502'. Sure enough, looking at the GCE load balancer, I can see that none of my nodes for the /.well-known/acme-challenge/* backend are healthy.

gcloud compute backend-services get-health k8s-be-32044--eeba4b4d12737265  | grep healthState
  - healthState: UNHEALTHY
  - healthState: UNHEALTHY
  - healthState: UNHEALTHY

When I look a the health checks again, I see that there are two /login paths. That doesn't seem right. The health check for the /.well-known/acme-challenge/* should be /healthz, right?

$ gcloud compute http-health-checks list
NAME                            HOST  PORT   REQUEST_PATH
k8s-be-30804--eeba4b4d12737265        30804  /login
k8s-be-32044--eeba4b4d12737265        32044  /login
k8s-be-32243--eeba4b4d12737265        32243  /
k8s-be-32516--eeba4b4d12737265        32516  /
k8s-be-32742--eeba4b4d12737265        32742  /healthz

$ gcloud compute http-health-checks describe k8s-be-32044--eeba4b4d12737265
checkIntervalSec: 1
creationTimestamp: '2016-09-07T07:12:18.014-07:00'
description: kubernetes L7 health check from readiness probe.
healthyThreshold: 1
host: ''
id: '304508149891440301'
kind: compute#httpHealthCheck
name: k8s-be-32044--eeba4b4d12737265
port: 32044
requestPath: /login
selfLink: https://www.googleapis.com/compute/v1/projects/imm-gce/global/httpHealthChecks/k8s-be-32044--eeba4b4d12737265
timeoutSec: 1
unhealthyThreshold: 10

If I manually change the path for the k8s-be-32044--eeba4b4d12737265 health check to /healthz and then wait a few minutes, the backend for /.well-known/acme-challenge/* becomes healthy and kube-lego is able to get a certificate.

I can't figure out how the health check for the challenge endpoint is getting created with /login instead of /healthz. Is it somehow picking it up from the pre-existing ingress? Or am I doing some wrong here? Any ideas? Thanks!

[simonswine]

Ok I see. The normal thing would be that GCE-L7-LBC uses the readinessProbe from the kube-lego pod. You are sure that exists in the kube-lego pod?

Otherwise it could be a bug in https://github.com/kubernetes/contrib/tree/master/ingress/controllers/gce. Do you use GKE or self spun up cluster? The version of the gce ingress controller would be interesting...

[ianwremmel]

I ran into the same issue - ended up with a check for /login instead of /healthz even though the kube-lego deployment specified

        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 1

[ianwremmel]

manually pointing the health check (using the google cloud console) at /healthz:8080 fixed the problem, but very weird that it was required (and i've love to get this reconfigured to work via code instead of gui).

[ianwremmel]

to answer your questions to @ryane, I'm using GKE and the ingress controller is gcr.io/google_containers/defaultbackend:1.0

[ianwremmel]

ok, so this is weird, but it seems like the health checks fail until I run

kubectl port-forward <kube-lego-pod> 8080:8080 --namespace kube-lego

[ryane]

I am also using GKE (v1.3.7) and gcr.io/google_containers/defaultbackend:1.0

[teelahti]

I faced the same problem today and spent several hours trying to figure out what is going on. Kube-lego 0.1.2, Kubernetes 1.4.3 on GKE. After deployment of kube-lego and a service (pretty much identical with the samples) I get a setup where the /.well-known URL map is assigned with a health check to root "/" like this:

$ gcloud compute http-health-checks describe k8s-be-30810--f991dead2d5f4e42
checkIntervalSec: 1
creationTimestamp: '2016-10-19T03:55:29.380-07:00'
description: Default kubernetes L7 Loadbalancing health check.
healthyThreshold: 1
host: ''
id: '652431145411735502'
kind: compute#httpHealthCheck
name: k8s-be-30810--f991dead2d5f4e42
port: 30810
requestPath: /
selfLink: https://www.googleapis.com/compute/v1/projects/foo/global/httpHealthChecks/k8s-be-30810--f991dead2d5f4e42
timeoutSec: 1
unhealthyThreshold: 10

This health check is of course failing. No certificates are issued because the /.well-known check ends in HTTP 502 due to GCP load balancer not finding a working backend. When I change the requestPath to /healthz with

$ gcloud compute http-health-checks update k8s-be-30810--f991dead2d5f4e42 --request-path=/healthz

...the whole system starts to work. I'd say there definitely is a bug, but I cannot pinpoint exactly where in the codebase.

[munnerz]

I'm also running into the issue on GKE, and as @teelahti states, updating the health check on gcloud fixes it.

Throwing in my 2c, how does the GLBC lookup the readinessProbe? My assumption is that it's by looking at the selector on the service?

If so, I'd imagine it's failing as the created service does not have a selector - kube-lego manually creates the Endpoints resource in order to handle the case where the ingress resource created by the user exists in a namespace other than kube-lego (https://github.com/jetstack/kube-lego/blob/master/pkg/provider/gce/gce.go#L94)

For reference, this is the service created by kube-lego:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubernetes.io/kube-lego-managed: "true"
  name: kube-lego-gce
  namespace: sock-shop
spec:
  clusterIP: 10.175.247.41
  ports:
  - nodePort: 32228
    port: 8080
    protocol: TCP
    targetPort: 8080
  sessionAffinity: None
  type: NodePort

I'm unsure of the best way to workaround this issue. The service itself must appear in the namespace of the ingress resource the user has created else the ingress will be unable to find the service. What happens with the nginx ingress controller? Does it simply not perform health checks?

One simple option would be to serve a 200 response on the / endpoint - I'm unsure if this will have any negative consequence, but can't immediately think of any?

[simonswine]

@munnerz thanks for your investigation on this. It looks like this commit in the GLBC controller breaks the way how we handle kube-lego across namespaces:

kubernetes-retired/contrib@857a815

For now I added a 200 return for / in 8e89f4d

[eriweb]

@simonswine any chance you could do patch release with this fix?

[simonswine]

@terbolous This workaround has been released with 0.1.3

Feel free to reopen, if this is not solving it...

[paolomainardi]

I had the same problem, health check created with login, then manually changed to /healthz and it worked.

Using GCE + kube-lego:0.1.3