Building a Generic Data Synchronization Platform Using Spring Batch and Quartz Scheduler

Posted on by Rahul Mittal

Description / Meta Description

Learn how Spring Batch and Quartz Scheduler can be combined to build scalable, fault-tolerant data synchronization applications. Understand Spring Batch metadata tables, Quartz scheduling tables, job execution tracking, restartability, and how to build a reusable framework for syncing data between systems using REST APIs.

Enterprise applications frequently need to synchronize data between systems.

Common examples include:

  • CRM → Data Warehouse
  • ERP → Reporting Database
  • Product Catalog → Search Platform
  • Customer System → Analytics Platform
  • Third-Party APIs → Internal Databases

A typical requirement looks like:

Source System
     │
     ▼

REST API

     │
     ▼

Spring Batch Job

     │
     ▼

Target Database

The synchronization must:

  • Run automatically every 24 hours
  • Handle millions of records
  • Recover from failures
  • Maintain execution history
  • Support future synchronization jobs

This is where Spring Batch and Quartz Scheduler work extremely well together.

Why Spring Batch?

Spring Batch is designed for:

Large Data Processing
Chunk Processing
Scheduling
Restartability
Job Monitoring
Error Recovery

Unlike a simple scheduler, Spring Batch provides:

  • Job metadata
  • Execution history
  • Restart support
  • Parallel processing
  • Retry mechanisms
  • Skip logic

making it ideal for enterprise data synchronization.

Why Quartz?

Quartz is responsible for:

Job Scheduling
Cron Execution
Calendar-Based Scheduling
Clustered Scheduling

Quartz decides:

WHEN

a job runs.

Spring Batch decides:

HOW

the data is processed.

High-Level Architecture

Quartz Scheduler
        │
        ▼

Spring Batch Job
        │
        ▼

REST API Reader
        │
        ▼

Processor
        │
        ▼

Database Writer

Execution flow:

Every 24 Hours
        │
        ▼

Quartz Trigger
        │
        ▼

Spring Batch Job
        │
        ▼

Fetch Data
        │
        ▼

Transform Data
        │
        ▼

Store Data

Spring Batch Metadata Tables

One of the biggest advantages of Spring Batch is automatic job tracking.

When Spring Batch starts, it creates several metadata tables.

BATCH_JOB_INSTANCE

Stores:

Unique Job Definitions

Example:

DATA_SYNC_JOB
CUSTOMER_SYNC_JOB
PRODUCT_SYNC_JOB

Each logical job gets an entry.

BATCH_JOB_EXECUTION

Stores:

Job Execution History

Example:

Job Start Time
Job End Time
Status
Exit Code

Status examples:

STARTING
STARTED
COMPLETED
FAILED
STOPPED

BATCH_JOB_EXECUTION_PARAMS

Stores:

Job Parameters

Example:

startDate
endDate
syncType

Useful for reprocessing.

BATCH_STEP_EXECUTION

Stores:

Step-Level Statistics

Example:

Read Count
Write Count
Skip Count
Commit Count

Very useful for monitoring.

BATCH_STEP_EXECUTION_CONTEXT

Stores:

Step State Information

Used for restartability.

BATCH_JOB_EXECUTION_CONTEXT

Stores:

Job-Level State

Used for job recovery.

Spring Batch Metadata Visualization

BATCH_JOB_INSTANCE
         │
         ▼

BATCH_JOB_EXECUTION
         │
         ▼

BATCH_STEP_EXECUTION

This gives complete execution visibility.

Quartz Tables

Quartz also creates its own tables.

Common ones include:

QRTZ_JOB_DETAILS

Stores:

Registered Jobs

Example:

DATA_SYNC_JOB

QRTZ_TRIGGERS

Stores:

Trigger Definitions

Example:

Every 24 Hours

QRTZ_CRON_TRIGGERS

Stores:

Cron Expressions

Example:

0 0 1 * * ?

Run daily at 1 AM.

QRTZ_FIRED_TRIGGERS

Stores:

Currently Running Jobs

Useful for monitoring.

Example Data Sync Requirement

Suppose we need to synchronize products from:

Source System

using:

GET /products

API.

Target:

PRODUCT_MASTER

table in Oracle.

Reader

The Reader calls the REST API.

ItemReader<ProductDTO>

Flow:

Call REST API
      │
      ▼

Receive JSON
      │
      ▼

Convert To DTO

Example:

List<ProductDTO> products =
        restTemplate.getForObject(
            apiUrl,
            List.class);

Processor

The processor transforms data.

Example:

ProductEntity process(
        ProductDTO dto)

Possible tasks:

Validation
Mapping
Data Enrichment
Default Values

Writer

Writer persists records.

Example:

JpaItemWriter<ProductEntity>

Writes data into:

PRODUCT_MASTER

table.

Chunk Processing

Instead of:

1 Record At A Time

Spring Batch processes chunks.

Example:

.chunk(500)

Flow:

Read 500
Process 500
Write 500
Commit

Benefits:

Better Performance
Reduced Memory Usage
Restartability

Scheduling Every 24 Hours

Quartz Trigger:

CronScheduleBuilder
    .cronSchedule(
        "0 0 1 * * ?");

Meaning:

1:00 AM Every Day

Execution:

Quartz
   │
   ▼
Spring Batch

Restartability

Suppose:

100,000 Records

need synchronization.

Failure occurs after:

75,000 Records

Without Spring Batch:

Restart From Beginning

With Spring Batch:

Resume From Last Checkpoint

using metadata tables.

Huge operational advantage.

Making the Solution Generic

Many organizations eventually need:

Customer Sync
Product Sync
Account Sync
Transaction Sync
Inventory Sync

Instead of creating separate frameworks:

Build one generic platform.

Generic Job Configuration

Create:

SYNC_JOB_CONFIG

table.

Example:

Job NameAPI URLTarget TableSchedule
PRODUCT_SYNC/productsPRODUCT_MASTERDaily
CUSTOMER_SYNC/customersCUSTOMER_MASTERDaily

Generic Reader

RestApiReader<T>

Reads from configurable API endpoints.

Generic Processor

GenericProcessor<S,T>

Maps source to target.

Generic Writer

JpaItemWriter<T>

Persists target entities.

Future Job Addition

To add a new synchronization:

Insert Config Row

instead of:

Write New Framework

Example:

ACCOUNT_SYNC

becomes:

Configuration Change

rather than:

Code Change

where possible.

Monitoring Dashboard Possibilities

Using Batch metadata tables:

Display:

Job Name
Last Run
Status
Duration
Records Processed
Failure Reason

This provides production visibility.

Benefits of Spring Batch + Quartz

Scheduling

Quartz provides enterprise-grade scheduling.

Execution Tracking

Spring Batch stores complete history.

Restartability

Failed jobs resume safely.

Scalability

Supports:

Partitioning
Multi-Threading
Remote Chunking

for large datasets.

Reusability

Generic framework supports future sync jobs.

Auditability

Execution metadata remains available indefinitely.

Best Practices

Use Incremental Synchronization

Avoid:

20 Years Of Data
Every Day

Instead:

Last Updated Timestamp

based synchronization.

Store Last Successful Run

Example:

2026-01-15 01:00:00

Fetch only newer records.

Use Chunk Processing

Recommended:

100
500
1000

depending on payload size.

Add Retry Logic

Temporary API failures should not fail entire jobs.

Add Dead Letter Handling

Failed records should be stored separately.

Enable Monitoring

Leverage:

BATCH_JOB_EXECUTION
BATCH_STEP_EXECUTION

for operational dashboards.

Final Thoughts

Spring Batch and Quartz together provide a powerful foundation for building enterprise-grade data synchronization platforms.

Quartz handles:

When To Run

Spring Batch handles:

How To Process

Using built-in metadata tables such as:

BATCH_JOB_INSTANCE
BATCH_JOB_EXECUTION
BATCH_STEP_EXECUTION

organizations gain complete visibility into job execution, restartability, auditing, and operational monitoring.

For a data synchronization use case involving REST APIs and database persistence every 24 hours, a generic Spring Batch + Quartz framework can evolve into a reusable platform supporting dozens of future synchronization jobs with minimal additional development effort.

This approach is widely used in large enterprises where reliability, observability, scalability, and maintainability are critical requirements.

Leave a Reply

Your email address will not be published. Required fields are marked *