In the first part of this workshop, we added metrics to our Employee Management API using Spring Boot Actuator, Micrometer, Prometheus, and Grafana.

Metrics answer questions such as:

How many requests does the application receive?
What is the API error rate?
Is JVM heap memory increasing?
Is p95 response time above the expected limit?

But metrics do not explain the detailed reason behind a failure.

For that, we need logs.

In this post, we will add structured JSON logs to the Employee Management API and send them to the Elastic Stack:

Spring Boot Application
        |
        | JSON logs
        v
Filebeat
        |
        v
Logstash
        |
        v
Elasticsearch
        |
        v
Kibana

At the end, we will be able to search, filter, visualize, and alert on application logs.

What We Will Build

We will extend the Employee Management API from Part 1.

The application already supports:

Create employee
Get all employees
Get employee by ID
Update employee
Delete employee

In this post, we will add:

Structured JSON logging
A request correlation ID
Filebeat log shipping
Logstash processing
Elasticsearch storage
Kibana search and dashboards
Kibana log-based alerting

Why Structured Logging?

Traditional logs are easy for humans to read but difficult for machines to reliably parse.

2026-10-16 10:15:20 INFO EmployeeService - Employee created with id=101

A structured JSON log contains named fields.

{
  "@timestamp": "2026-10-16T10:15:20.245Z",
  "level": "INFO",
  "message": "Employee created",
  "event": "employee_created",
  "employeeId": 101,
  "department": "Engineering",
  "correlationId": "3d10ff2a-417b-4d51-befb-fbcc750b7b77"
}

This makes it easy to search Kibana for:

level: ERROR
event: employee_created
employeeId: 101
correlationId: 3d10ff2a-417b-4d51-befb-fbcc750b7b77

Spring Boot 4 supports structured logging out of the box in ECS, GELF, and Logstash JSON formats. We will use the Logstash JSON format. Spring Boot structured logging documentation

ELK Architecture

+--------------------------+
| Employee Spring Boot API |
|                          |
| Logback / SLF4J          |
| JSON structured logs     |
+--------------------------+
             |
             | Writes JSON log events
             v
+--------------------------+
| Application log file     |
| employee-service.json    |
+--------------------------+
             |
             | Filebeat reads and ships events
             v
+--------------------------+
| Filebeat                 |
+--------------------------+
             |
             | Beats protocol, port 5044
             v
+--------------------------+
| Logstash                 |
| Parse / enrich / route   |
+--------------------------+
             |
             | Index documents
             v
+--------------------------+
| Elasticsearch            |
+--------------------------+
             |
             | Search and visualize
             v
+--------------------------+
| Kibana                   |
+--------------------------+

Filebeat is the component that tails the application log file. Logstash receives and processes those events. Prometheus is not part of this pipeline because Prometheus is for metrics, not logs.

Step 1: Add Structured Logging Configuration

Add the following to the Employee Service application.yml.

spring:
  application:
    name: employee-service

logging:
  file:
    name: /var/log/app/employee-service.json

  structured:
    format:
      console: logstash
      file: logstash

    json:
      add:
        application: ${spring.application.name}
        environment: local

This gives us:

JSON logs in the console
JSON logs in /var/log/app/employee-service.json
Application and environment fields on every log event

A typical log record will look similar to this:

{
  "@timestamp": "2026-10-16T10:15:20.245Z",
  "@version": "1",
  "message": "Employee created",
  "logger_name": "com.example.employee.EmployeeService",
  "thread_name": "http-nio-8080-exec-1",
  "level": "INFO",
  "level_value": 20000,
  "application": "employee-service",
  "environment": "local"
}

Step 2: Add a Correlation ID to Each Request

A correlation ID lets us find every log line generated by one request.

For example, a single POST /api/employees request may generate logs from:

The controller
The service layer
The database layer
A downstream HTTP call
Exception handling

Create CorrelationIdFilter.java.

@Component
public class CorrelationIdFilter extends OncePerRequestFilter {

    private static final String CORRELATION_ID = "correlationId";
    private static final String HEADER_NAME = "X-Correlation-Id";

    @Override
    protected void doFilterInternal(
            HttpServletRequest request,
            HttpServletResponse response,
            FilterChain filterChain)
            throws ServletException, IOException {

        String correlationId = request.getHeader(HEADER_NAME);

        if (correlationId == null || correlationId.isBlank()) {
            correlationId = UUID.randomUUID().toString();
        }

        MDC.put(CORRELATION_ID, correlationId);
        response.setHeader(HEADER_NAME, correlationId);

        try {
            filterChain.doFilter(request, response);
        }
        finally {
            MDC.remove(CORRELATION_ID);
        }
    }
}

The MDC, or Mapped Diagnostic Context, stores contextual fields for the current request. Spring Boot’s Logstash JSON format includes MDC values automatically.

After this change, each log entry contains:

{
  "correlationId": "3d10ff2a-417b-4d51-befb-fbcc750b7b77"
}

Send a request with your own correlation ID.

curl -X POST http://localhost:8080/api/employees \
  -H "Content-Type: application/json" \
  -H "X-Correlation-Id: employee-demo-001" \
  -d "{\"name\":\"Asha\",\"email\":\"asha@example.com\",\"department\":\"Engineering\",\"salary\":75000}"

The response includes the same header.

X-Correlation-Id: employee-demo-001

Later, in Kibana, searching for employee-demo-001 will show every related log event.

Step 3: Add Useful Business Logs

Update EmployeeService to add useful, structured business events.

@Service
public class EmployeeService {

    private static final Logger log =
            LoggerFactory.getLogger(EmployeeService.class);

    private final EmployeeRepository employeeRepository;
    private final Counter employeesCreatedCounter;

    public EmployeeService(EmployeeRepository employeeRepository,
                           MeterRegistry meterRegistry) {
        this.employeeRepository = employeeRepository;

        this.employeesCreatedCounter = Counter.builder("employees.created")
                .description("Total number of employees created")
                .register(meterRegistry);
    }

    public Employee createEmployee(Employee employee) {
        log.atInfo()
                .addKeyValue("event", "employee_creation_requested")
                .addKeyValue("email", employee.getEmail())
                .addKeyValue("department", employee.getDepartment())
                .log("Creating employee");

        Employee savedEmployee = employeeRepository.save(employee);
        employeesCreatedCounter.increment();

        log.atInfo()
                .addKeyValue("event", "employee_created")
                .addKeyValue("employeeId", savedEmployee.getId())
                .addKeyValue("department", savedEmployee.getDepartment())
                .log("Employee created successfully");

        return savedEmployee;
    }

    public Employee getEmployeeById(Long id) {
        return employeeRepository.findById(id)
                .orElseThrow(() -> {
                    log.atWarn()
                            .addKeyValue("event", "employee_not_found")
                            .addKeyValue("employeeId", id)
                            .log("Employee not found");

                    return new NoSuchElementException(
                            "Employee not found: " + id);
                });
    }

    public void deleteEmployee(Long id) {
        Employee employee = getEmployeeById(id);

        log.atInfo()
                .addKeyValue("event", "employee_deleted")
                .addKeyValue("employeeId", employee.getId())
                .addKeyValue("department", employee.getDepartment())
                .log("Deleting employee");

        employeeRepository.delete(employee);
    }
}

Avoid logging sensitive data such as passwords, tokens, complete payment information, or unnecessary personally identifiable information.

Step 4: Create the Docker Compose Environment

Create a file named docker-compose.yml.

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.5
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
    ports:
      - "9200:9200"
    networks:
      - observability-network

  kibana:
    image: docker.elastic.co/kibana/kibana:9.5.5
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - observability-network

  logstash:
    image: docker.elastic.co/logstash/logstash:9.5.5
    container_name: logstash
    ports:
      - "5044:5044"
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    depends_on:
      - elasticsearch
    networks:
      - observability-network

  filebeat:
    image: docker.elastic.co/beats/filebeat:9.5.5
    container_name: filebeat
    user: root
    command: ["--strict.perms=false"]
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - employee-logs:/var/log/app:ro
    depends_on:
      - logstash
    networks:
      - observability-network

volumes:
  employee-logs:

networks:
  observability-network:
    driver: bridge

Use matching Elastic Stack versions for Elasticsearch, Kibana, Logstash, and Filebeat. For production, enable Elasticsearch security, TLS, authentication, role-based access, and encrypted Beats-to-Logstash communication.

Step 5: Configure Logstash

Create a directory named logstash/pipeline.

Inside it, create employee-logs.conf.

input {
  beats {
    port => 5044
  }
}

filter {
  mutate {
    add_field => {
      "pipeline" => "employee-logs"
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "employee-logs-%{+YYYY.MM.dd}"
  }

  stdout {
    codec => rubydebug
  }
}

This Logstash pipeline:

Receives Filebeat events on port 5044
Adds the pipeline name
Stores logs in a daily Elasticsearch index
Prints events to the Logstash console for debugging

Filebeat sends data to Logstash over the Beats protocol, and Logstash commonly listens on port 5044. Elastic Filebeat Logstash output documentation

Step 6: Configure Filebeat

Create a directory named filebeat.

Inside it, create filebeat.yml.

filebeat.inputs:
  - type: filestream
    id: employee-service-json-logs
    enabled: true
    paths:
      - /var/log/app/employee-service.json

    parsers:
      - ndjson:
          target: ""
          add_error_key: true

    fields:
      service_name: employee-service
      environment: local

    fields_under_root: true

output.logstash:
  hosts: ["logstash:5044"]

This configuration tells Filebeat to:

Read the JSON application log file
Parse one JSON object per line
Add service metadata
Forward the events to Logstash

Step 7: Mount the Application Log Volume

If the Spring Boot application runs in Docker, add this volume to the Employee Service in its Docker Compose configuration.

services:
  employee-service:
    build: .
    ports:
      - "8080:8080"
    volumes:
      - employee-logs:/var/log/app
    networks:
      - observability-network

volumes:
  employee-logs:

Now both containers access the same log volume:

Employee Service writes logs
        |
        v
employee-logs Docker volume
        |
        v
Filebeat reads logs

Start the ELK stack.

docker compose up -d

Open Kibana:

http://localhost:5601

Step 8: Create a Kibana Data View

After generating a few API requests, open Kibana and create a data view.

Index pattern: employee-logs-*
Timestamp field: @timestamp

You can now open Discover and search application logs.

Useful searches include:

level: ERROR

event: employee_created

correlationId: employee-demo-001

employeeId: 101

application: employee-service AND level: WARN

Suggested Kibana Dashboards

Create a dashboard with these visualizations:

Dashboard panel	Purpose
Log volume by level	Compare INFO, WARN, and ERROR event counts
Error events over time	Identify application failure spikes
Top exceptions	Find recurring failures
Top API paths with errors	Identify unstable endpoints
Employee creation events by department	Visualize business activity
Employee not found events	Identify invalid client behavior
Logs by correlation ID	Investigate a single request

Kibana Alerting

Kibana can evaluate log data on a schedule and send notifications when defined conditions are met.

Create alert rules for scenarios such as:

Alert	Condition
Error spike	More than 20 `ERROR` logs in five minutes
Employee creation failure	`employee_creation_failed` event appears
Repeated missing employee requests	More than 50 `employee_not_found` events in ten minutes
Database failures	`DataIntegrityViolationException` or SQL errors detected
No application activity	No Employee Service logs for ten minutes

Kibana alerting includes:

Rules
Alert conditions
Check schedules
Connectors
Recovery actions
Alert summaries
Alert history
Alert acknowledgements
Muting and snoozing

Kibana connectors can send notifications through services such as email, Slack, Microsoft Teams, webhooks, incident-management tools, and other supported integrations. Kibana alerting documentation

Elasticsearch logs
        |
        v
Kibana rule checks every minute
        |
        v
Condition matched?
        |
        +--> No: continue monitoring
        |
        +--> Yes: create alert
                     |
                     v
               Connector action
                     |
                     +--> Email
                     +--> Slack
                     +--> Microsoft Teams
                     +--> Webhook

Troubleshooting Checklist

Filebeat cannot connect to Logstash

Check that:

- Logstash is running
- Port 5044 is exposed
- Filebeat uses logstash:5044
- The Logstash Beats input is configured
- Elasticsearch output is disabled in Filebeat

No logs appear in Kibana

Check that:

- The Spring Boot application created the log file
- Filebeat can read the mounted volume
- Filebeat is sending events
- Logstash is receiving events
- Elasticsearch has employee-logs-* indexes
- The Kibana data view uses employee-logs-*

JSON parsing errors

Check that:

- Spring Boot structured logging is enabled
- Each line in the log file is one complete JSON object
- The Filebeat ndjson parser is configured

Key Takeaways

In this post, we turned local application logs into searchable, centralized observability data.

Spring Boot structured JSON logs
        |
        v
Filebeat ships the logs
        |
        v
Logstash processes the logs
        |
        v
Elasticsearch stores the logs
        |
        v
Kibana searches, visualizes, and alerts

Metrics tell us that a problem exists.

Logs explain what happened.

In the next post, we will add distributed tracing using OpenTelemetry and Jaeger. We will split the application into Employee Service and Department Service, then follow one request across both services from beginning to end.

Part 2: Centralized Logging in Spring Boot with ELK Stack — Filebeat, Logstash, Elasticsearch, and Kibana

What We Will Build

Why Structured Logging?

ELK Architecture

Step 1: Add Structured Logging Configuration

Step 2: Add a Correlation ID to Each Request

Step 3: Add Useful Business Logs

Step 4: Create the Docker Compose Environment

Step 5: Configure Logstash

Step 6: Configure Filebeat

Step 7: Mount the Application Log Volume

Step 8: Create a Kibana Data View

Suggested Kibana Dashboards

Kibana Alerting

Troubleshooting Checklist

Filebeat cannot connect to Logstash

No logs appear in Kibana

JSON parsing errors

Key Takeaways

Leave a Reply Cancel reply