Liveness Readiness Probs in Kubernetes

Liveness and readiness probes are needed for the resilience of a service. They have two basic requirements: fast and show the actual state of the service. Outdated status is not helpful to keep the system ready when it is needed. The guideline from the Kubernetes team is to validate the connectivity to the service. It is enough for a simple application. However, when a system has multiple dependencies, say service1 calls service2 and service3, it is necessary to check service2 and service3 along with service1 to determine the health of servcie1.

I was looking for some suggestions in that area but had no luck. I found some implementation, where health check is a chain of synchronous calls to all dependent services. However, this implementation may introduce an unintentional delay to the probs.

An asynchronous pull-based model will be more suitable to handle this scenario. I am using golang to demonstrate the pattern. It creates a background routine to update the state of the service on a predefined interval.

How to use the module in your service:

Initialized the monitoring object like the following:

monitor := getStatusMonitor() 
defer monitor.stop()  
//chain all health checks 
monitor.addCheck(healthCheck1) 
monitor.addCheck(healthCheck2) 
montor.start()

The handler for prob will use the monitor to return the state, like the following:

if getStatusMonitor().isReady() {  
   w.WriteHeader(200)  
   w.Write([]byte("ok")) 
} else {  
   w.WriteHeader(500)  
   w.Write([]byte("healthz failed")) 
}

Here is the implementation of the pattern

type statusMonitor struct {
	ready          bool
	chStop         chan int
	monitorRateSec int
	healthChecks   []func() bool
}

var monitorOnce sync.Once
var monitor *statusMonitor

func (monitor *statusMonitor) addCheck(f func() bool) {
	monitor.healthChecks = append(monitor.healthChecks, f)
}

func getStatusMonitor() *statusMonitor {
  monitorOnce.Do(func() {
  monitor = &statusMonitor{ready: true, chStop: make(chan int)}
    rate, v := os.LookupEnv("MONITOR_RATE_SEC")
    if !v {
      monitor.monitorRateSec = 10
    } else {
      i, err := strconv.Atoi(rate)
      if err == nil {
	monitor.monitorRateSec = i
      } else {
	monitor.monitorRateSec = 10
      }
    }
  })
 return monitor
}

func (monitor *statusMonitor) isReady() bool {
  return monitor.ready
}

func (monitor *statusMonitor) start() {
  go func() {
     for {
	  select {
	  case <-monitor.chStop:
	     return
	  default:
	     monitor.ready = monitor.getStatus()
             time.Sleep(time.Duration(monitor.monitorRateSec) *     time.Second)
	}
     }
   }()
}

func (monitor *statusMonitor) getStatus() bool {
  for _, f := range monitor.healthChecks {
    if !f() {
      return false
    }
 }
 return true
}

func (monitor *statusMonitor) stop() {
  close(monitor.chStop)
}

The complete source code is available in the following github repo

https://github.com/manideepbora/livenessprobe

Let me know your thought!