/
Kafka pods failing

Kafka pods failing

If kafka pods are failing with

Session 0x0 for sever localhost/0:0:0:0:0:0:0:1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused

and zookeeper pods are silently (pods still green) failing with PKIX cert verification issue, steps to fix are:

  1. Delete Secrets:

    1. gv-kafka-cluster-cluster-operator-certs

    2. gv-kafka-cluster-entity-topic-operator-certs

    3. gv-kafka-cluster-entity-user-operator-certs

    4. gv-kafka-cluster-kafka-brokers

  2. Wait for the Operator to recreate these Secrets

  3. Delete Pods:

    1. gv-kafka-cluster-zookeeper-0

    2. gv-kafka-cluster-kafka-0

  4. Wait for them to be recreated

  5. Redeploy all services which are using kafka - forcing them to recreate missing topics. (For example classification pipeline won’t full start if regex topics are missing )

Data stored in Kafka might be lost, but these clusters are dysfunctional anyway

 

Related content

K3s Installation
K3s Installation
Read with this
Keycloak pods failing
Keycloak pods failing
More like this
Initial k3s install not fully functioning on Centos 7.9
Initial k3s install not fully functioning on Centos 7.9
More like this

Classified as Getvisibility - Partner/Customer Confidential