Varnish does not start

[Tarasovych]

Describe the bug
Varnish does not start. No errors in pod's logs

To Reproduce
Steps to reproduce the behavior:

1. Install Helm chart 0.7.2
2. Use values:

        # Default values for kube-httpcache.
        # This is a YAML-formatted file.
        # Declare variables to be passed into your templates.
        
        replicaCount: 1
        
        image:
          repository: quay.io/mittwald/kube-httpcache
          pullPolicy: IfNotPresent
          tag: "stable"
        
        imagePullSecrets: []
        nameOverride: ""
        fullnameOverride: ""
        
        # Enable StatefulSet (Deployment is default)
        useStatefulset:
          enabled: false
        
        # Enable configMap for Varnish Template File (see below vclTemplate)
        # OR use extravolume with name "template" if the file is too big
        configmap:
          enabled: true
        
        # kube-httpcache specific configuration
        cache:
          # name of frontend service
          # frontendService: kube-httpcache-headless
          # name of backend service
          backendService: backend-service
          # name of backend service namespace
          # backendServiceNamespace: backend-service-namespace
          # watching for frontend changes is true by default
          frontendWatch: true
          # watching for backend changes is true by default
          backendWatch: true
          # Varnish storage backend type (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html)
          varnishStorage: malloc # default,malloc,umem,file...
          # Varnish storage backend size
          storageSize: 128M # K(ibibytes), M(ebibytes), G(ibibytes), T(ebibytes) ... unlimited
          # Varnish transient storage backend type (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html)
          #varnishTransientStorage: malloc
          # Varnish transient storage backend size
          #transientStorageSize: 128M # K(ibibytes), M(ebibytes), G(ibibytes), T(ebibytes) ... unlimited
          # Secret for Varnish admin credentials
          secret: "HIDDEN"
          # Read admin credentials from user provided secret
          #existingSecret: kubecache-secret
        
        cacheExtraArgs: {}
          # cacheExtraArgs: |
          # - -v=8
          # - -varnish-additional-parameters=vcc_allow_inline_c=on
        
        serviceAccount:
          # Specifies whether a service account should be created
          enabled: true
          # Annotations to add to the service account
          annotations: {}
          # The name of the service account to use.
          # If not set and create is true, a name is generated using the fullname template
          name: ""
        
        rbac:
          enabled: true
        
        # create a prometheus operator ServiceMonitor
        serviceMonitor:
          enabled: false
        
          additionalLabels: {}
          ## Scrape interval. If not set, the Prometheus default scrape interval is used.
          interval: 10s
          ## Scrape Timeout. If not set, the Prometheus default scrape timeout is used.
          scrapeTimeout: ""
        
        
        podSecurityPolicy:
          enabled: false
          # name: unrestricted-psp
        
        annotations: {}
        podAnnotations: {}
        podLabels: {}
        
        podSecurityContext: {}
        # fsGroup: 2000
        
        securityContext: {}
          # capabilities:
          #   drop:
          #   - ALL
          # readOnlyRootFilesystem: true
          # runAsNonRoot: true
        # runAsUser: 1000
        
        lifecycle: {}
          # preStop:
          #   exec:
          #     command:
          #     - /bin/sh
          #     - -c
          #     - touch /etc/varnish/fail_probes; sleep 25
        
        topologySpreadConstraints: {}
          # - topologyKey: topology.kubernetes.io/zone
          #   maxSkew: 1
          #   whenUnsatisfiable: ScheduleAnyway
          #   labelSelector:
          #     matchLabels:
          #       app.kubernetes.io/name: kube-httpcache
          # - topologyKey: kubernetes.io/hostname
          #   maxSkew: 1
          #   whenUnsatisfiable: ScheduleAnyway
          #   labelSelector:
          #     matchLabels:
          #       app.kubernetes.io/name: kube-httpcache
        
        initContainers: {}
        # initContainers: |
        #   - args:
        #     - -c
        #     - |
        #       echo "Copying external varnish template from..."
        #     command:
        #     - sh
        #     image: busybox:latest
        #     imagePullPolicy: IfNotPresent
        #     name: varnishtemplate
        #     resources: {}
        #     terminationMessagePath: /dev/termination-log
        #     terminationMessagePolicy: File
        #     volumeMounts:
        #     - name: template
        #       mountPath: /etc/varnish/tmpl
        
        extraContainers: []
        # - name: my-sidecar
        #   image: myapp/my-sidecar
        #   command:
        #   - my-sidecar-command
        
        extraVolumes: {}
        # extraVolumes:
        #   - emptyDir: {}
        #     name: template
        
        extraMounts: {}
        # extraMounts:
        #   - name: geoip
        #     mountPath: /var/lib/geoip
        
        extraEnvVars: {}
        #extraEnvVars:
        #  - name: foo
        #    value: bar
        
        
        extraEnvFromConfig: {}
        #extraEnvFromConfig:
        #  - configMapRef:
        #      name: my-configmap-name
        #  - secretRef:
        #      name: my-secret-name
        
        
        exporter:
          enabled: false
          securityContext: {}
            # capabilities:
            #   drop:
            #   - ALL
            # readOnlyRootFilesystem: true
            # runAsNonRoot: true
          # runAsUser: 1000
          resources: {}
          livenessProbe: {}
          # livenessProbe:
          #   httpGet:
          #    path: /
          #    port: 6083
          readinessProbe: {}
        
        service:
          type: ClusterIP
          port: 80
          target: 8080
        
        ingress:
          enabled: false
          annotations: {}
          # kubernetes.io/tls-acme: "true"
          className: nginx
          hosts: []
          # hosts:
          #   - host: www.example.com
          #     paths:
          #       - path: /
          #         pathType: Prefix
          #         backend:
          #           service:
          #             name: kube-httpcache
          #             port:
          #               number: 80
          #       - path: /backend
          #         backend:
          #           name: backend-service
          #           port:
          #             number: 8080
          #   - host: www2.example.com
          #     paths:
          #       - path: /
          #         pathType: Prefix
          #         backend:
          #           name: kube-httpcache
          #           port:
          #             number: 80
          tls: []
          #  - secretName: chart-example-tls
          #    hosts:
          #      - chart-example.local
        
        resources: {}
          # We usually recommend not to specify default resources and to leave this as a conscious
          # choice for the user. This also increases chances charts run on environments with little
          # resources, such as Minikube. If you do want to specify resources, uncomment the following
          # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
          # limits:
          #   cpu: 100m
          #   memory: 128Mi
          # requests:
          #   cpu: 100m
          #   memory: 128Mi
        
        autoscaling:
          enabled: false
          minReplicas: 1
          maxReplicas: 100
          targetCPUUtilizationPercentage: 80
          # targetMemoryUtilizationPercentage: 80
        
        nodeSelector: {}
        
        tolerations: []
        
        #terminationGracePeriodSeconds: 60
        
        affinity: {}
        
        livenessProbe: {}
        # livenessProbe:
        #   httpGet:
        #    path: /
        #    port: 6083
        readinessProbe: {}
        
        vclTemplate: |
          vcl 4.0;
        
          import std;
          import directors;
        
          // ".Frontends" is a slice that contains all known Varnish instances
          // (as selected by the service specified by -frontend-service).
          // The backend name needs to be the Pod name, since this value is compared
          // to the server identity ("server.identity" [1]) later.
          //
          //   [1]: https://varnish-cache.org/docs/6.4/reference/vcl.html#local-server-remote-and-client
          {{ range .Frontends }}
          backend {{ .Name }} {
              .host = "{{ .Host }}";
              .port = "{{ .Port }}";
          }
          {{- end }}
        
          {{ range .Backends }}
          backend be-{{ .Name }} {
              .host = "{{ .Host }}";
              .port = "{{ .Port }}";
          }
          {{- end }}
        
          sub vcl_init {
              new cluster = directors.hash();
        
              {{ range .Frontends -}}
              cluster.add_backend({{ .Name }}, 1);
              {{ end }}
        
              new lb = directors.round_robin();
        
              {{ range .Backends -}}
              lb.add_backend(be-{{ .Name }});
              {{ end }}
          }
        
          sub vcl_recv
          {
              # Set backend hint for non cachable objects.
              set req.backend_hint = lb.backend();
        
              # ...
        
              # Routing logic. Pass a request to an appropriate Varnish node.
              # See https://info.varnish-software.com/blog/creating-self-routing-varnish-cluster for more info.
              unset req.http.x-cache;
              set req.backend_hint = cluster.backend(req.url);
              set req.http.x-shard = req.backend_hint;
              if (req.http.x-shard != server.identity) {
                  return(pass);
              }
              set req.backend_hint = lb.backend();
        
              # ...
        
              return(hash);
          }

3. Pod's logs are

I0601 13:07:34.640328       1 main.go:31] running kube-httpcache with following options: {Kubernetes:{Config: RetryBackoffString:30s RetryBackoff:30s} Frontend:{Address:0.0.0.0 Port:8080 Watch:true Namespace:kube-httpcache Service:kube-httpcache PortName:http} Backend:{Watch:true Namespace:kube-httpcache Service:backend-service Port: PortName:http} Signaller:{Enable:true Address:0.0.0.0 Port:8090 WorkersCount:1 MaxRetries:5 RetryBackoffString:30s RetryBackoff:30s QueueLength:0 MaxConnsPerHost:-1 MaxIdleConns:-1 MaxIdleConnsPerHost:-1 UpstreamRequestTimeoutString: UpstreamRequestTimeout:0s} Admin:{Address:0.0.0.0 Port:6083} Varnish:{SecretFile:/etc/varnish/k8s-secret/secret Storage:malloc,128M TransientStorage:malloc,128m AdditionalParameters: VCLTemplate:/etc/varnish/tmpl/default.vcl.tmpl VCLTemplatePoll:false WorkingDir:} Readiness:{Enable:true Address:0.0.0.0:9102}}
I0601 13:07:34.640398       1 main.go:38] using in-cluster configuration
I0601 13:07:34.641165       1 run.go:15] waiting for initial configuration before starting Varnish

It hangs on waiting for initial configuration before starting Varnish

Expected behavior
Varnish starts

Environment:

• Kubernetes version: 1.24, EKS
• kube-httpcache version: 0.7.2

[martin-helmich]

Thanks for the report -- This is odd; after the "waiting for initial configuration before starting Varnish" message, the controller should watch the backend endpoints, and render its configuration based on the observed endpoints.

It's been a few year since I've last touched that particular piece of code, but I'm wondering if there's an initial GET missing when setting up the endpoint watch:

kube-httpcache/pkg/watcher/endpoints_watch.go

Lines 23 to 27 in 60f555d

	func (v *EndpointWatcher) watch(ctx context.Context, updates chan *EndpointConfig, errors chan error) {
	for {
	w, err := v.client.CoreV1().Endpoints(v.namespace).Watch(ctx, metav1.ListOptions{
	FieldSelector: fields.OneTermEqualSelector("metadata.name", v.serviceName).String(),
	})

Otherwise, the entire startup of the controller would depend on the endpoints object being updated (by chance) to actually start. I'll investigate some more.

[martin-helmich]

I was able to confirm this issue, in the case (and only then) when the watched backend service did either not exist or have any endpoints. Can you confirm that your backend service both exists and has endpoints when the issue occurs?

[guilhermehbueno]

I'm facing the same problem. I could noticed the /tmp/vcl file is properly set with the backends and the kubectl get endpoints is properly listing the backend services' ips.

Note: I'm deploying all the files in a single file. I'll try to do it individually after.

[guilhermehbueno]

I'm facing the same problem. I could noticed the /tmp/vcl file is properly set with the backends and the kubectl get endpoints is properly listing the backend services' ips.

Note: I'm deploying all the files in a single file. I'll try to do it individually after.

Nevermind. I was with syntax problems in my Varnish settings.