Part 7 – Health Probes

Why a Running Spring Boot Application Doesn’t Always Mean It’s Ready

Sarah watches the deployment dashboard.

Everything appears normal.

NAME                                READY   STATUS    RESTARTS   AGE
payment-service-84bd67f6df-j9dkt    1/1     Running   0          18s

The Pod is Running.

The Deployment is progressing.

There are no warning events.

The release manager smiles.

But within seconds, the monitoring dashboard begins turning red.

Customer payment requests are failing.

The API Gateway is returning HTTP 503 – Service Unavailable.

Operations immediately ask the development team:

“The Pod is running. Why isn’t the application working?”

If you’ve worked with Kubernetes long enough, you’ve probably encountered this exact situation.

The answer lies in one of the most misunderstood concepts in Kubernetes.

A Running Pod is not necessarily a Ready application.

Understanding this distinction is critical for building reliable enterprise microservices.


Understanding the Pod Lifecycle

Most developers think a Pod moves directly from Pending to Running.

In reality, several important phases occur before a customer request ever reaches your application.

Pending
    │
    ▼
Scheduled
    │
    ▼
Image Pulled
    │
    ▼
Container Started
    │
    ▼
JVM Started
    │
    ▼
Spring Boot Started
    │
    ▼
Database Connected
    │
    ▼
Caches Loaded
    │
    ▼
Messaging Connected
    │
    ▼
READY

The important observation is this:

The Pod enters the Running phase very early.

Your application, however, may still be performing significant initialization work.

For a simple “Hello World” application, startup might complete in a second or two.

For an enterprise banking service, startup can involve dozens of dependencies.


What Happens During Spring Boot Startup?

Let’s examine what our Payment Service actually does before it is capable of processing a payment.

Start JVM

↓

Load Spring Context

↓

Component Scan

↓

Create Beans

↓

Initialize Security

↓

Create Database Connection Pool

↓

Run Flyway/Liquibase Migration

↓

Connect Redis

↓

Load Master Data

↓

Initialize Solace Publisher

↓

Subscribe to Event Queues

↓

Warm Frequently Used Cache

↓

Register Health Indicators

↓

Embedded Tomcat Starts

↓

Application Ready

Depending on the application, this process may take anywhere from a few seconds to several minutes.

Imagine customer traffic arriving halfway through this initialization.

Suppose Oracle connections are still being established.

The API receives:

POST /payments

The application attempts to retrieve customer information.

The database connection pool isn’t ready yet.

Result:

500 Internal Server Error

Nothing is technically wrong with the application.

It simply wasn’t ready to accept requests.

This is precisely the problem Kubernetes health probes solve.


Why Kubernetes Needs Health Probes

Imagine managing a hospital.

A doctor walks into the building.

Can you immediately assign patients?

Not necessarily.

The doctor may still be:

  • Changing into scrubs
  • Reviewing patient records
  • Preparing equipment
  • Completing shift handover

Simply being inside the hospital doesn’t mean the doctor is ready to treat patients.

Applications behave the same way.

Running does not equal ready.

Kubernetes therefore asks applications three different questions.

Have you started?

↓

Are you still alive?

↓

Are you ready to receive traffic?

Each question serves a completely different purpose.


The Three Types of Health Probes

Kubernetes provides three independent health checks.

                 Spring Boot Application

                        │

        ┌───────────────┼────────────────┐

        ▼               ▼                ▼

 Startup Probe    Liveness Probe   Readiness Probe

 "Did you      "Are you still    "Can customers
 start?"        healthy?"         use you?"

Understanding when each probe executes is more important than memorizing their YAML syntax.

Let’s examine them one by one.


Startup Probe

The Startup Probe is used only during application initialization.

Its job is simple.

Has the application finished starting?

Until the Startup Probe succeeds:

  • Liveness Probe is ignored.
  • Readiness Probe is ignored.

This prevents Kubernetes from killing applications that legitimately require longer startup times.


Enterprise Banking Example

Suppose our Payment Service performs the following startup sequence.

JVM

↓

Spring Boot

↓

Oracle Connection

↓

Redis Connection

↓

Load Holiday Calendar

↓

Load Currency Exchange Rates

↓

Connect Solace

↓

Warm Payment Rules Cache

↓

READY

This process takes:

95 seconds

Meanwhile, the default Liveness Probe starts checking after only 30 seconds.

Without a Startup Probe, Kubernetes observes:

Application not responding.

The kubelet assumes something has gone wrong.

It restarts the container.

The new container starts.

Again, it needs 95 seconds.

Again, Kubernetes restarts it.

The result is an endless restart cycle.

Start

↓

Killed

↓

Restart

↓

Killed

↓

Restart

↓

Killed

Eventually, the Pod enters the dreaded state:

CrashLoopBackOff

Ironically, nothing is wrong with the application.

Kubernetes simply became impatient.

A properly configured Startup Probe prevents this scenario.


Liveness Probe

Once startup completes, the Startup Probe is disabled permanently.

The Liveness Probe now becomes active.

Its question is different.

Is the application still alive?

Notice the wording carefully.

Not:

“Is it fast?”

Not:

“Is Oracle available?”

Not:

“Is Redis connected?”

Only:

“Is the application itself healthy?”

Suppose a deadlock occurs inside the JVM.

All request-processing threads become blocked.

CPU usage drops.

Memory remains stable.

The process continues running.

From Linux’s perspective:

Everything appears normal.

From the customer’s perspective:

The application is completely unusable.

This is exactly the kind of failure a Liveness Probe detects.

If the probe fails repeatedly, the kubelet concludes that recovery is unlikely.

It terminates the container and starts a fresh one.

In many cases, this automatic restart restores service without any manual intervention.

This is one of the simplest yet most powerful examples of Kubernetes’ self-healing capabilities.


Readiness Probe

The Readiness Probe answers perhaps the most important question.

Should this Pod receive customer traffic?

Notice that the application may still be alive.

It may even be functioning correctly.

But that doesn’t necessarily mean it should process requests.

Suppose Oracle undergoes scheduled maintenance.

Our Payment Service detects that database connectivity has been lost.

Instead of crashing, it marks itself as:

Not Ready

Immediately, Kubernetes removes the Pod from the Service endpoints.

Before

Service

↓

Pod A

Pod B

Pod C

-----------------

After

Service

↓

Pod A

Pod C

Customer traffic continues flowing to the remaining healthy Pods.

No restart occurs.

No deployment rollback occurs.

The unhealthy Pod simply stops receiving requests until it becomes ready again.

This distinction between Liveness and Readiness is fundamental.

Liveness answers:

“Should Kubernetes restart me?”

Readiness answers:

“Should Kubernetes send me traffic?”

Those are very different decisions.


Spring Boot Actuator Integration

Spring Boot makes implementing health probes remarkably straightforward through Spring Boot Actuator.

A common configuration exposes dedicated endpoints.

/actuator/health

/actuator/health/liveness

/actuator/health/readiness

These endpoints report the application’s internal state in a format Kubernetes can understand.

For example:

{
  "status": "UP"
}

or

{
  "status": "DOWN"
}

Spring Boot 3.x integrates seamlessly with Kubernetes, allowing liveness and readiness states to be managed independently.

For example:

  • Database unavailable? Report Readiness = DOWN.
  • JVM deadlocked? Report Liveness = DOWN.
  • Still loading reference data? Report Startup = DOWN.

This gives Kubernetes precise information about how to respond.


A Production Scenario

Let’s revisit our banking platform.

The Payment Service depends on:

  • Oracle Database
  • Redis Cache
  • Solace Messaging
  • Customer Service
  • Fraud Service

Suppose Redis becomes temporarily unavailable.

Should Kubernetes restart the Pod?

Probably not.

Redis may recover within a few seconds.

Restarting every Payment Service Pod would only make matters worse.

Instead:

Application Alive

↓

Readiness = DOWN

↓

Traffic Stops

↓

Redis Recovers

↓

Readiness = UP

↓

Traffic Resumes

Customers experience minimal disruption, and the application avoids unnecessary restarts.

This is precisely why designing meaningful health endpoints is so important.


Common Mistakes

Many teams implement probes incorrectly.

Some common examples include:

Returning 200 OK Unconditionally

A health endpoint that always returns success defeats the purpose of health monitoring.

If the application is stuck or unable to serve requests, Kubernetes should know.


Including Every Dependency in Liveness

Developers sometimes fail the Liveness Probe because a downstream service is unavailable.

For example:

Fraud Service Down

↓

Liveness = DOWN

↓

Restart Payment Service

Restarting won’t fix the Fraud Service.

The Payment Service simply enters a restart loop.

External dependency failures usually belong in Readiness, not Liveness.


Aggressive Timeouts

Enterprise applications often require additional startup time after deployment.

Probe intervals that are too aggressive can cause unnecessary restarts.

Always measure startup times under realistic production conditions before configuring probe thresholds.


The Self-Healing Feedback Loop

Let’s step back and appreciate what Kubernetes has achieved.

Application Failure

↓

Health Probe Detects Problem

↓

kubelet Reports Failure

↓

Container Restarted

↓

Startup Probe Executes

↓

Readiness Probe Passes

↓

Traffic Restored

No engineer logged into a server.

No manual restart.

No service desk ticket.

No operational intervention.

This continuous monitoring and recovery loop is one of the defining characteristics of Kubernetes.

It transforms failure from an operational emergency into an expected, automated event.


What Happens Next?

Our Payment Service is finally ready.

The Startup Probe has completed successfully.

The Liveness Probe confirms the JVM is healthy.

The Readiness Probe reports that the application is prepared to process requests.

For the first time since Sarah executed:

kubectl apply -f payment-service.yaml

the Service controller adds the Pod to its list of available endpoints.

The API Gateway can now begin routing customer requests to the new Payment Service instance.

But another important question remains.

How does Kubernetes distribute traffic across multiple Pods?

How does it know where each healthy Pod is?

How are failed Pods automatically removed from load balancing?

Those answers lie within one of Kubernetes’ most fundamental networking abstractions.

In the next article, we’ll explore Services, Endpoints, and EndpointSlices, following the very first customer payment request as it travels from the API Gateway to one of Sarah’s newly deployed Payment Service Pods.

Next Article

Part 8 – Kubernetes Services: How Customer Requests Reach the Right Pod

We’ll cover:

  • Why Pods should never be accessed directly
  • ClusterIP, NodePort, LoadBalancer, and ExternalName
  • Endpoint and EndpointSlice objects
  • kube-proxy and packet forwarding
  • iptables vs IPVS
  • Service discovery and DNS
  • Load balancing across replicas
  • Rolling updates without downtime
  • Enterprise networking best practices
  • Common production troubleshooting scenarios

Leave a Reply

Your email address will not be published. Required fields are marked *