Production applications should not be treated as black boxes. When an API slows down, fails, or consumes too much memory, teams need answers quickly:

• Is the application healthy?
• Which API is slow?
• Is the JVM running out of memory?
• Which request failed?
• Which downstream service caused the delay?

This is where observability helps.

Observability is the ability to understand the internal state of a running application by examining the signals it produces. The three key signals are:

1. Metrics
2. Logs
3. Traces

In this hands-on workshop series, we will build a simple Employee Management API using Java 17, Spring Boot 4, Spring Data JPA, and H2 Database. We will then progressively add metrics, centralized logging, and distributed tracing.

---

What We Will Build

Our sample application will support basic employee CRUD operations:

• Create an employee
• Get all employees
• Get employee by ID
• Update employee
• Delete employee

We will then add observability in three stages.

Part 1: Spring Boot Actuator + Micrometer + Prometheus + Grafana
Part 2: Spring Boot Logging + Filebeat + Logstash + Elasticsearch + Kibana
Part 3: Spring Boot Tracing + OpenTelemetry + Jaeger

---

Understanding Observability

1. Metrics

Metrics are numerical measurements collected over time.

Examples include:

• Number of HTTP requests
• Number of failed requests
• API response time
• JVM heap memory consumption
• CPU utilization
• Database connection pool usage
• Number of employees created per hour

Metrics are highly efficient because monitoring systems aggregate them over time. They are ideal for dashboards, charts, alerting, trend analysis, and capacity planning.

For example, if the API response time suddenly increases, a metric can reveal that the p95 response time went from 200 ms to 2 seconds.

2. Logs

Logs are detailed, timestamped records of events occurring inside an application.

2026-10-16 10:15:20 INFO  EmployeeService - Employee created with id=101
2026-10-16 10:15:35 ERROR EmployeeService - Employee not found with id=999

Logs help answer questions such as:

• Why did this request fail?
• Which exception occurred?
• Which employee ID was involved?
• What was the request payload or validation failure?

Metrics tell us that an error rate increased. Logs explain the cause.

3. Traces

Traces follow one request as it travels through one or more services.

For example, a client request to create an employee may:

Client
  |
  | POST /api/employees
  v
Employee Service
  |
  +--> Validate employee request
  |
  +--> Save employee to database
  |
  +--> Call Department Service
  |
  v
Return HTTP 201 Created

A trace contains multiple spans. Each span represents a unit of work, such as a database call, HTTP call, or method execution.

Traces are particularly useful in microservices, where one request can travel through several services before a response is returned.

---

Overall Observability Architecture

                         +--------------------+
                         |       Client       |
                         +--------------------+
                                   |
                                   v
                    +--------------------------------+
                    |     Employee Spring Boot API    |
                    |                                |
                    |  Actuator + Micrometer         |
                    |  Logging Framework             |
                    |  Micrometer Tracing            |
                    +--------------------------------+
                       |              |             |
             Metrics   |              | Logs        | Traces
                       v              v             v
              +---------------+  +----------+  +------------------+
              |  Prometheus   |  | Filebeat |  | OTEL Collector   |
              +---------------+  +----------+  +------------------+
                       |              |             |
                       v              v             v
              +---------------+  +-----------+  +------------------+
              |    Grafana    |  | Logstash  |  |      Jaeger      |
              +---------------+  +-----------+  +------------------+
                                      |
                                      v
                                +-------------+
                                |Elasticsearch|
                                +-------------+
                                      |
                                      v
                                +-------------+
                                |   Kibana    |
                                +-------------+

Important: Prometheus pulls metrics from the application. It does not pull logs. Logs need a log shipper such as Filebeat, or a direct logging appender.

---

Part 1: Metrics with Spring Boot Actuator, Micrometer, Prometheus, and Grafana

Step 1: Create the Spring Boot Project

Create a Maven-based Spring Boot 4 application with Java 17.

Add these dependencies to pom.xml.

<dependencies>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-webmvc</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

</dependencies>

---

Step 2: Create the Employee Entity

@Entity
@Table(name = "employees")
public class Employee {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    private String name;

    @Column(unique = true)
    private String email;

    private String department;

    private BigDecimal salary;

    // Getters and setters
}

---

Step 3: Create the Repository

public interface EmployeeRepository extends JpaRepository<Employee, Long> {
}

---

Step 4: Create the Employee Service

We will create a custom Micrometer counter named employees.created.

@Service
public class EmployeeService {

    private final EmployeeRepository employeeRepository;
    private final Counter employeesCreatedCounter;

    public EmployeeService(EmployeeRepository employeeRepository,
                           MeterRegistry meterRegistry) {

        this.employeeRepository = employeeRepository;

        this.employeesCreatedCounter = Counter.builder("employees.created")
                .description("Total number of employees created")
                .register(meterRegistry);
    }

    public Employee createEmployee(Employee employee) {
        Employee savedEmployee = employeeRepository.save(employee);
        employeesCreatedCounter.increment();
        return savedEmployee;
    }

    public List<Employee> getAllEmployees() {
        return employeeRepository.findAll();
    }

    public Employee getEmployeeById(Long id) {
        return employeeRepository.findById(id)
                .orElseThrow(() ->
                        new NoSuchElementException("Employee not found: " + id));
    }

    public Employee updateEmployee(Long id, Employee employeeRequest) {
        Employee employee = getEmployeeById(id);

        employee.setName(employeeRequest.getName());
        employee.setEmail(employeeRequest.getEmail());
        employee.setDepartment(employeeRequest.getDepartment());
        employee.setSalary(employeeRequest.getSalary());

        return employeeRepository.save(employee);
    }

    public void deleteEmployee(Long id) {
        employeeRepository.delete(getEmployeeById(id));
    }
}

The custom counter will appear in Prometheus as:

employees_created_total

This lets us build Grafana charts such as “How many employees were created in the past hour?”

---

Step 5: Create the REST Controller

@RestController
@RequestMapping("/api/employees")
public class EmployeeController {

    private final EmployeeService employeeService;

    public EmployeeController(EmployeeService employeeService) {
        this.employeeService = employeeService;
    }

    @PostMapping
    @ResponseStatus(HttpStatus.CREATED)
    public Employee createEmployee(@RequestBody Employee employee) {
        return employeeService.createEmployee(employee);
    }

    @GetMapping
    public List<Employee> getAllEmployees() {
        return employeeService.getAllEmployees();
    }

    @GetMapping("/{id}")
    public Employee getEmployeeById(@PathVariable Long id) {
        return employeeService.getEmployeeById(id);
    }

    @PutMapping("/{id}")
    public Employee updateEmployee(@PathVariable Long id,
                                   @RequestBody Employee employee) {
        return employeeService.updateEmployee(id, employee);
    }

    @DeleteMapping("/{id}")
    @ResponseStatus(HttpStatus.NO_CONTENT)
    public void deleteEmployee(@PathVariable Long id) {
        employeeService.deleteEmployee(id);
    }
}

---

Step 6: Configure H2, Actuator, and Metrics

Add the following configuration to application.yml.

spring:
  application:
    name: employee-service

  datasource:
    url: jdbc:h2:mem:employees
    driver-class-name: org.h2.Driver
    username: sa
    password:

  jpa:
    hibernate:
      ddl-auto: update

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus

  endpoint:
    health:
      show-details: when_authorized

  metrics:
    tags:
      application: ${spring.application.name}
      environment: local

    distribution:
      percentiles-histogram:
        http.server.requests: true

      percentiles:
        http.server.requests: 0.5,0.95,0.99

      slo:
        http.server.requests: 50ms,100ms,250ms,500ms,1s

This configuration enables:

• HTTP response-time histograms
• p50, p95, and p99 latency calculations
• Service-level response-time buckets
• Prometheus scrape endpoint
• Standard application tags

---

What Does Spring Boot Actuator Provide?

Spring Boot Actuator automatically exposes valuable metrics.

Area	Example metrics
HTTP APIs	Request count, status code, URI, method, response time
JVM	Heap memory, non-heap memory, garbage collection, threads
CPU	Process CPU usage, system CPU usage
Uptime	Process uptime and application start time
Database	Hikari connection pool usage
Disk	Available and total disk space
Logging	Number of INFO, WARN, and ERROR log events
Startup	Application started and ready time

Useful Actuator endpoints include:

http://localhost:8080/actuator/health
http://localhost:8080/actuator/info
http://localhost:8080/actuator/metrics
http://localhost:8080/actuator/metrics/jvm.memory.used
http://localhost:8080/actuator/metrics/http.server.requests
http://localhost:8080/actuator/prometheus

The /actuator/metrics endpoint is useful for inspecting metric names during development. Prometheus should scrape /actuator/prometheus in production.

---

Step 7: Connect Prometheus

Create a file named prometheus.yml.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: employee-service
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ["host.docker.internal:8080"]

Run Prometheus using Docker.

docker run --rm \
  -p 9090:9090 \
  -v ./prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Open Prometheus:

http://localhost:9090

---

How Metrics Move Through the System

+-----------------------+
| Employee Spring Boot  |
| Application           |
+-----------------------+
           |
           | Records metrics using Micrometer
           v
+-----------------------+
| Spring Boot Actuator  |
| /actuator/prometheus |
+-----------------------+
           ^
           |
           | Prometheus scrapes every 15 seconds
           |
+-----------------------+
| Prometheus            |
| Stores time series    |
+-----------------------+
           ^
           |
           | Grafana sends PromQL queries
           |
+-----------------------+
| Grafana Dashboard     |
| Charts and alerts     |
+-----------------------+

---

Step 8: Build Grafana Dashboards

Add Prometheus as a Grafana data source.

URL: http://host.docker.internal:9090

Use the following PromQL queries.

Request Rate

sum(rate(http_server_requests_seconds_count[5m]))

HTTP 5xx Error Rate

sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))

p95 API Response Time

histogram_quantile(
  0.95,
  sum by (le, uri) (
    rate(http_server_requests_seconds_bucket[5m])
  )
)

JVM Heap Memory Used

sum(jvm_memory_used_bytes{area="heap"})

JVM Heap Usage Percentage

sum(jvm_memory_used_bytes{area="heap"})
/
sum(jvm_memory_max_bytes{area="heap"})
* 100

Process CPU Usage

process_cpu_usage * 100

Application Uptime

process_uptime_seconds

Employees Created in the Last Hour

sum(increase(employees_created_total[1h]))

Active Database Connections

hikaricp_connections_active

---

Recommended Grafana Dashboard Panels

Create the following panels:

1. API request rate
2. HTTP status-code distribution
3. HTTP 5xx error percentage
4. p50, p95, and p99 API response time
5. JVM heap memory utilization
6. CPU utilization
7. JVM thread count
8. Garbage-collection pause time
9. Hikari active database connections
10. Application uptime
11. Employees created per hour

Note: Spring Boot gives application-level HTTP metrics, but full machine or container network metrics usually come from Node Exporter, cAdvisor, or cloud-provider monitoring.

---

Grafana Alerting

Grafana can evaluate metrics and send notifications when a condition becomes abnormal.

Useful alert examples:

Alert	Example condition
Service down	Application health endpoint cannot be scraped
High error rate	5xx requests exceed 2% for five minutes
Slow API	p95 latency exceeds 500 ms for ten minutes
Memory pressure	JVM heap usage exceeds 85%
Database pressure	Active DB connections exceed 80% of pool size
Missing business activity	No employees created during a defined business period

Grafana alerting supports:

• Alert rules
• Alert labels and annotations
• Contact points
• Notification policies
• Email notifications
• Slack notifications
• Microsoft Teams notifications
• PagerDuty notifications
• Webhook notifications
• Alert grouping
• Alert silences
• Mute timings
• Alert history
• Alert notification templates

Prometheus metric
      |
      v
Grafana alert rule
      |
      v
Alert condition matches
      |
      v
Notification policy
      |
      v
Contact point
      |
      +--> Slack
      +--> Email
      +--> Microsoft Teams
      +--> PagerDuty
      +--> Webhook

---

Part 2: Centralized Logging with ELK

Metrics tell us that an API is slow or failing. Logs help us understand the reason.

In the next blog, we will configure structured JSON logging for the same Employee Service.

+----------------------+
| Employee Service     |
| JSON logs to console |
+----------------------+
           |
           v
+----------------------+
| Docker / Host logs   |
+----------------------+
           |
           v
+----------------------+
| Filebeat             |
+----------------------+
           |
           v
+----------------------+
| Logstash             |
+----------------------+
           |
           v
+----------------------+
| Elasticsearch        |
+----------------------+
           |
           v
+----------------------+
| Kibana               |
+----------------------+

We will capture fields such as:

{
  "@timestamp": "2026-10-16T10:15:20.245Z",
  "level": "INFO",
  "service": "employee-service",
  "traceId": "c18ce8d2e5f4d199",
  "spanId": "f6d4a3c8e9b601a2",
  "event": "employee_created",
  "employeeId": 101
}

In Kibana, we will learn how to:

• Search application logs
• Filter logs by severity
• Find exceptions
• Search by employee ID
• Search by trace ID
• Build error dashboards
• Create log-based alert rules
• Detect error spikes and repeated failures

---

Part 3: Distributed Tracing with OpenTelemetry and Jaeger

In the final blog, we will create two services:

Employee Service
Department Service

The Employee Service will call the Department Service. We will use OpenTelemetry and Jaeger to trace the entire request.

Client
  |
  | POST /api/employees
  v
Employee Service
  |
  | GET /api/departments/{id}
  v
Department Service
  |
  v
Database / Response

Both services export trace spans through OpenTelemetry.

+--------------------+
| Employee Service   |
+--------------------+
          |
          | OTLP spans
          v
+--------------------+
| OpenTelemetry      |
| Collector          |
+--------------------+
          |
          v
+--------------------+
| Jaeger             |
+--------------------+

In Jaeger, we will learn how to:

• Search traces by service
• Search traces by operation
• Identify slow spans
• View service dependencies
• Compare latency between services
• Locate failed downstream calls
• Use trace IDs to correlate traces with Kibana logs

A trace ID provides technical request correlation across services. A business correlation ID can also be propagated when a business workflow needs a recognizable identifier.

---

Final Outcome

At the end of this workshop series, we will have a complete observability demonstration.

METRICS
Spring Boot Actuator
        |
        v
Prometheus
        |
        v
Grafana dashboards and alerts


LOGS
Spring Boot JSON logs
        |
        v
Filebeat
        |
        v
Logstash
        |
        v
Elasticsearch
        |
        v
Kibana search, dashboards, and alerts


TRACES
Spring Boot Micrometer Tracing
        |
        v
OpenTelemetry Collector
        |
        v
Jaeger trace visualization

Observability works best when all three signals are used together:

• Metrics detect the problem.
• Traces identify where the problem happened.
• Logs explain why the problem happened.

In the next post, we will configure structured logging and build the ELK pipeline for the Employee Management API.