In Part 1, we added metrics with Spring Boot Actuator, Prometheus, and Grafana.

In Part 2, we centralized structured application logs using Filebeat, Logstash, Elasticsearch, and Kibana.

In this final part, we will add distributed tracing.

Distributed tracing helps answer an important question:

When one user request travels through multiple services, where did time get spent and where did it fail?

We will split the Employee Management API into two microservices:

Employee Service
Department Service

The Employee Service will call the Department Service while retrieving an employee’s department details. We will use:

Spring Boot 4
Micrometer Tracing
OpenTelemetry
Jaeger
Java 17

At the end, we will be able to follow a single request across both services.

What Is Distributed Tracing?

A trace represents the full lifecycle of one request.

For example, when a user requests employee details:

Client
  |
  | GET /api/employees/101/department
  v
Employee Service
  |
  | GET /api/departments/10
  v
Department Service
  |
  v
Return department information

The trace contains multiple spans.

Trace: GET employee department

Span 1: HTTP GET /api/employees/101/department
    |
    +-- Span 2: Employee Service department lookup
            |
            +-- Span 3: HTTP GET /api/departments/10
                    |
                    +-- Span 4: Department Service processing

Each span captures information such as:

Service name
Operation name
Start time
Duration
Error details
Parent span
Trace ID
Span ID

Tracing Architecture

+-------------+
|   Client    |
+-------------+
       |
       | GET /api/employees/{id}/department
       v
+------------------------+
| Employee Service       |
|                        |
| traceId: abc123        |
| spanId:  employee-span |
+------------------------+
       |
       | traceparent header
       | GET /api/departments/{id}
       v
+------------------------+
| Department Service     |
|                        |
| traceId: abc123        |
| spanId: department-span|
+------------------------+
       |
       | OTLP spans
       v
+------------------------+
| Jaeger Collector       |
+------------------------+
       |
       v
+------------------------+
| Jaeger UI              |
| Search and trace view  |
+------------------------+

The same trace ID is propagated from Employee Service to Department Service. That allows Jaeger to group both service calls into one complete trace.

Step 1: Create the Department Service

Create a second Spring Boot project named department-service.

Add these dependencies.

<dependencies>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-webmvc</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-opentelemetry</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

</dependencies>

For Spring Boot 4, spring-boot-starter-opentelemetry provides Spring Boot’s supported OpenTelemetry tracing integration. Spring Boot tracing documentation

Step 2: Create the Department Model

public record DepartmentResponse(
        Long id,
        String name,
        String manager
) {
}

Step 3: Create the Department Controller

@RestController
@RequestMapping("/api/departments")
public class DepartmentController {

    private static final Logger log =
            LoggerFactory.getLogger(DepartmentController.class);

    private static final Map<Long, DepartmentResponse> DEPARTMENTS =
            Map.of(
                    10L, new DepartmentResponse(
                            10L,
                            "Engineering",
                            "Priya Sharma"
                    ),
                    20L, new DepartmentResponse(
                            20L,
                            "Human Resources",
                            "Rahul Mehta"
                    ),
                    30L, new DepartmentResponse(
                            30L,
                            "Finance",
                            "Anita Kapoor"
                    )
            );

    @GetMapping("/{id}")
    public DepartmentResponse getDepartment(@PathVariable Long id) {

        log.atInfo()
                .addKeyValue("event", "department_lookup")
                .addKeyValue("departmentId", id)
                .log("Fetching department details");

        DepartmentResponse department = DEPARTMENTS.get(id);

        if (department == null) {
            log.atWarn()
                    .addKeyValue("event", "department_not_found")
                    .addKeyValue("departmentId", id)
                    .log("Department not found");

            throw new NoSuchElementException(
                    "Department not found: " + id);
        }

        return department;
    }
}

Step 4: Configure the Department Service

Create application.yml.

server:
  port: 8081

spring:
  application:
    name: department-service

logging:
  structured:
    format:
      console: logstash

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus

  tracing:
    sampling:
      probability: 1.0

  opentelemetry:
    resource-attributes:
      service.name: ${spring.application.name}
      deployment.environment: local

    tracing:
      export:
        otlp:
          endpoint: http://localhost:4318/v1/traces

For local development, we use a sampling probability of 1.0, meaning every request is traced.

In production, do not automatically trace every request in a high-volume application. Spring Boot defaults to sampling only a percentage of requests to prevent overwhelming the tracing backend.

Step 5: Add OpenTelemetry to Employee Service

Add the OpenTelemetry starter to the existing Employee Service.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-opentelemetry</artifactId>
</dependency>

Update the Employee Service application.yml.

server:
  port: 8080

spring:
  application:
    name: employee-service

department:
  service:
    base-url: http://localhost:8081

logging:
  structured:
    format:
      console: logstash
      file: logstash

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus

  tracing:
    sampling:
      probability: 1.0

  opentelemetry:
    resource-attributes:
      service.name: ${spring.application.name}
      deployment.environment: local

    tracing:
      export:
        otlp:
          endpoint: http://localhost:4318/v1/traces

Step 6: Create a Department Client in Employee Service

Spring Boot automatically configures RestClient.Builder.

Use that auto-configured builder. It is important because Spring Boot adds the tracing instrumentation needed to propagate trace context to downstream services.

Create DepartmentClient.java.

@Service
public class DepartmentClient {

    private final RestClient restClient;

    public DepartmentClient(
            RestClient.Builder restClientBuilder,
            @Value("${department.service.base-url}") String baseUrl) {

        this.restClient = restClientBuilder
                .baseUrl(baseUrl)
                .build();
    }

    public DepartmentResponse getDepartment(Long departmentId) {
        return restClient.get()
                .uri("/api/departments/{id}", departmentId)
                .retrieve()
                .body(DepartmentResponse.class);
    }
}

Do not create the client with RestClient.create() if you want automatic tracing propagation. Use the auto-configured RestClient.Builder.

Spring Boot automatically propagates trace context when you create HTTP clients with its auto-configured RestClient.Builder or WebClient.Builder. Spring Boot REST client documentation

Step 7: Add a Custom Trace Span

Automatic spans are created for incoming and outgoing HTTP requests. We can also create a custom span around important business activity.

Update EmployeeService.

@Service
public class EmployeeService {

    private static final Logger log =
            LoggerFactory.getLogger(EmployeeService.class);

    private final EmployeeRepository employeeRepository;
    private final DepartmentClient departmentClient;
    private final ObservationRegistry observationRegistry;

    public EmployeeService(
            EmployeeRepository employeeRepository,
            DepartmentClient departmentClient,
            ObservationRegistry observationRegistry) {

        this.employeeRepository = employeeRepository;
        this.departmentClient = departmentClient;
        this.observationRegistry = observationRegistry;
    }

    public EmployeeDepartmentResponse getEmployeeDepartment(Long employeeId) {

        Employee employee = employeeRepository.findById(employeeId)
                .orElseThrow(() -> new NoSuchElementException(
                        "Employee not found: " + employeeId));

        Long departmentId = resolveDepartmentId(employee.getDepartment());

        return Observation
                .createNotStarted(
                        "employee.department.lookup",
                        observationRegistry
                )
                .observe(() -> {

                    log.atInfo()
                            .addKeyValue("event", "employee_department_lookup")
                            .addKeyValue("employeeId", employeeId)
                            .addKeyValue("departmentId", departmentId)
                            .log("Looking up employee department");

                    DepartmentResponse department =
                            departmentClient.getDepartment(departmentId);

                    return new EmployeeDepartmentResponse(
                            employee.getId(),
                            employee.getName(),
                            employee.getEmail(),
                            department
                    );
                });
    }

    private Long resolveDepartmentId(String department) {
        return switch (department) {
            case "Engineering" -> 10L;
            case "Human Resources" -> 20L;
            case "Finance" -> 30L;
            default -> throw new NoSuchElementException(
                    "Department mapping not found: " + department);
        };
    }
}

Create the response record.

public record EmployeeDepartmentResponse(
        Long employeeId,
        String employeeName,
        String email,
        DepartmentResponse department
) {
}

Add an endpoint to EmployeeController.

@GetMapping("/{id}/department")
public EmployeeDepartmentResponse getEmployeeDepartment(
        @PathVariable Long id) {

    return employeeService.getEmployeeDepartment(id);
}

The custom observation appears in Jaeger as a business-level span:

employee.department.lookup

This is more meaningful than viewing only low-level HTTP calls.

Step 8: Run Jaeger Locally

Run Jaeger All-in-One with Docker.

docker run --rm \
  --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:1.76.0

Open Jaeger in the browser.

http://localhost:16686

Jaeger’s all-in-one image exposes:

Port	Purpose
`16686`	Jaeger user interface
`4317`	OTLP gRPC collector
`4318`	OTLP HTTP collector

Jaeger supports OTLP HTTP trace ingestion at /v1/traces on port 4318. Jaeger getting started documentation

Step 9: Generate a Trace

Start the services.

Department Service: http://localhost:8081
Employee Service:   http://localhost:8080
Jaeger UI:          http://localhost:16686

First, create an employee.

curl -X POST http://localhost:8080/api/employees \
  -H "Content-Type: application/json" \
  -d "{\"name\":\"Asha\",\"email\":\"asha@example.com\",\"department\":\"Engineering\",\"salary\":75000}"

Then request the employee’s department.

curl http://localhost:8080/api/employees/1/department

Expected response:

{
  "employeeId": 1,
  "employeeName": "Asha",
  "email": "asha@example.com",
  "department": {
    "id": 10,
    "name": "Engineering",
    "manager": "Priya Sharma"
  }
}

Step 10: View the Trace in Jaeger

Open the Jaeger UI.

http://localhost:16686

Select:

Service: employee-service
Operation: GET /api/employees/{id}/department

Click Find Traces.

Open a trace to see the complete request flow.

GET /api/employees/{id}/department
        |
        +-- employee.department.lookup
                |
                +-- HTTP GET Department Service
                        |
                        +-- GET /api/departments/{id}

A healthy trace might look like this:

Employee Service                    Department Service

|------------------------------|
GET employee department         |
|--- employee lookup ----------|
      |-------------------------------|
      HTTP GET department             |
      |                                |
      |                                GET department
      |                                |---------|
      |<-------------------------------|
|--------------------------------------|

If the Department Service is slow, its span will be visibly longer. If it fails, Jaeger highlights the error-related span.

Trace ID, Span ID, and Correlation ID

These terms are related but not identical.

Identifier	Purpose
Trace ID	Identifies the complete end-to-end request
Span ID	Identifies one operation within a trace
Correlation ID	An application-defined ID for grouping related business work

Example:

Trace ID:       8b8d33d1774a44c2ae5f43ae0270f188
Employee Span:  127aea045d8aef10
Department Span:f2665f60307303ee
Correlation ID: employee-demo-001

In this workshop:

Jaeger uses the trace ID to show the full request journey.
The structured logs from Part 2 can contain trace and span context.
The optional X-Correlation-Id header remains useful for business-level investigation.

Spring Boot adds tracing correlation IDs to logging when tracing is enabled, making it possible to move from a Jaeger trace to matching Kibana logs. Spring Boot tracing and log correlation

Correlating Jaeger Traces with Kibana Logs

A practical investigation workflow looks like this:

Grafana detects slow API response
        |
        v
Open Jaeger trace for the request
        |
        v
Identify slow or failed service span
        |
        v
Copy the trace ID
        |
        v
Search Kibana logs using traceId
        |
        v
Read detailed exception and business context

Example Kibana search:

traceId: 8b8d33d1774a44c2ae5f43ae0270f188

This is the value of combining all three observability pillars:

Metrics: Detect the problem
Traces: Locate the problem
Logs: Explain the problem

Common Tracing Problems

Traces do not appear in Jaeger

Check the following:

- Jaeger is running.
- Port 4318 is exposed.
- The OTLP endpoint ends with /v1/traces.
- Both services include spring-boot-starter-opentelemetry.
- Sampling probability is 1.0 for the local demonstration.
- Both applications can reach the Jaeger collector.

Employee Service and Department Service appear as separate traces

Check that the Employee Service creates its HTTP client with the auto-configured RestClient.Builder.

Correct: RestClient.Builder injected by Spring Boot
Avoid:   RestClient.create(...)

Trace IDs do not appear in logs

Check that:

- Tracing is enabled.
- Structured logging is enabled.
- The logging framework is using Spring Boot’s default setup.
- The log event happens inside an active HTTP request or observation.

Too many traces in production

Reduce sampling.

management:
  tracing:
    sampling:
      probability: 0.1

This samples approximately 10% of requests.

Use higher sampling temporarily when investigating a production issue, but consider trace volume, storage cost, and sensitive data before doing so.

Final Workshop Architecture

                            +-------------------+
                            |      Client       |
                            +-------------------+
                                      |
          +---------------------------+----------------------------+
          |                           |                            |
          v                           v                            v
+---------------------+     +--------------------+      +----------------------+
| Metrics             |     | Logs               |      | Traces               |
|                     |     |                    |      |                      |
| Actuator            |     | Structured JSON    |      | Micrometer Tracing   |
| Micrometer          |     | Filebeat           |      | OpenTelemetry        |
+---------------------+     | Logstash           |      +----------------------+
          |                 | Elasticsearch      |                 |
          v                 +--------------------+                 v
+---------------------+                |                  +----------------------+
| Prometheus          |                v                  | Jaeger Collector     |
+---------------------+        +---------------+          +----------------------+
          |                   | Kibana        |                     |
          v                   +---------------+                     v
+---------------------+                                      +---------------+
| Grafana             |                                      | Jaeger UI     |
| Dashboards, alerts  |                                      | Trace search  |
+---------------------+                                      +---------------+

Series Conclusion

We built a complete observability foundation for a Spring Boot microservices application.

Part 1: Metrics
Spring Boot Actuator → Prometheus → Grafana

Part 2: Logs
Spring Boot JSON Logs → Filebeat → Logstash → Elasticsearch → Kibana

Part 3: Traces
Spring Boot Micrometer Tracing → OpenTelemetry → Jaeger

The three pillars work together:

Signal	Best used for
Metrics	Dashboards, trends, SLOs, and alerts
Logs	Error details, exceptions, and business context
Traces	End-to-end request flow and service latency

With these foundations in place, teams can detect failures early, investigate them faster, and understand how requests behave across distributed systems.

Part 3: Distributed Tracing in Spring Boot with OpenTelemetry and Jaeger

What Is Distributed Tracing?

Tracing Architecture

Step 1: Create the Department Service

Step 2: Create the Department Model

Step 3: Create the Department Controller

Step 4: Configure the Department Service

Step 5: Add OpenTelemetry to Employee Service

Step 6: Create a Department Client in Employee Service

Step 7: Add a Custom Trace Span

Step 8: Run Jaeger Locally

Step 9: Generate a Trace

Step 10: View the Trace in Jaeger

Trace ID, Span ID, and Correlation ID

Correlating Jaeger Traces with Kibana Logs

Common Tracing Problems

Traces do not appear in Jaeger

Employee Service and Department Service appear as separate traces

Trace IDs do not appear in logs

Too many traces in production

Final Workshop Architecture

Series Conclusion

Leave a Reply Cancel reply

What Is Distributed Tracing?

Tracing Architecture

Step 1: Create the Department Service

Step 2: Create the Department Model

Step 3: Create the Department Controller

Step 4: Configure the Department Service

Step 5: Add OpenTelemetry to Employee Service

Step 6: Create a Department Client in Employee Service

Step 7: Add a Custom Trace Span

Step 8: Run Jaeger Locally

Step 9: Generate a Trace

Step 10: View the Trace in Jaeger

Trace ID, Span ID, and Correlation ID

Correlating Jaeger Traces with Kibana Logs

Common Tracing Problems

Traces do not appear in Jaeger

Employee Service and Department Service appear as separate traces

Trace IDs do not appear in logs

Too many traces in production

Final Workshop Architecture

Series Conclusion

Leave a Reply Cancel reply

Part 7: Understanding Collectors – The Backbone of Stream Aggregation

Part 42: The Java 8 → Java 21 Journey – Building Modern Enterprise Microservices and Looking Ahead to Java 26

Part 31: Java 12 – Switch Expressions – Writing Cleaner, Safer and More Expressive Decision Logic

Spring Batch and Quartz Scheduler Concepts Explained