Dead Letter Queues DLQs are an essential part of any enterprise event-driven architecture. They prevent a continuously failing message from blocking the main processing pipeline while preserving the message for investigation and recovery.
However, a DLQ should not become a dumping ground that requires someone to manually process messages one by one.
A robust production architecture should combine:
- Automatic retries for transient failures
- Controlled movement to a DLQ
- Automated DLQ reprocessing where appropriate
- Monitoring and alerting
- Idempotent processing
- Manual replay for exceptional cases
This article looks at a practical DLQ architecture using Spring Cloud Function, with examples for both AWS SQS and Solace PubSub+.
1. What exactly should go to a DLQ?
Consider a typical microservice flow:
Client
|
v
Base Engine
|
v
Mediator
|
v
Choreo
|
v
Atomic
|
v
Downstream System
Messages can fail for very different reasons.
Transient failure
For example:
Atomic
|
+—> Downstream
|
+—> HTTP 503
+—> Timeout
+—> Connection refused
These failures are normally candidates for retry.
Permanent failure
Examples:
Invalid request
Invalid business data
Unsupported message version
Missing mandatory field
Corrupted payload
Unknown transaction type
Repeatedly retrying these messages will not solve the problem.
Application defect
For example:
NullPointerException
SerializationException
Database exception
Unexpected runtime exception
Some of these may be transient, while others require a code fix.
Therefore, the first principle is:
Do not treat every exception as a DLQ-worthy permanent failure. Classify failures before deciding what to do with the message.
2. The recommended DLQ architecture
A production architecture should look approximately like this:
┌──────────────────┐
│ Main Queue │
└────────┬─────────┘
│
▼
Spring Function
│
┌───────┴────────┐
│ │
SUCCESS ERROR
│ │
▼ ▼
ACK Retry mechanism
│
┌───────┴───────┐
│ │
Retry Permanent
│ │
▼ ▼
Main Queue DLQ
│
┌────────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Auto Retry Investigate Alert
│
▼
Main Queue
The important distinction is between retry and DLQ.
DLQ should generally represent:
“The normal processing pipeline could not successfully process this message after the permitted recovery attempts.”
3. Should DLQ messages be processed manually?
No—not by default.
Manual processing should be the exception.
Imagine a system processing:
20,000 messages/minute
and a downstream outage causes:
5,000 messages
to enter the DLQ.
Having an operations team manually process 5,000 messages is neither scalable nor safe.
Instead:
DLQ
|
v
DLQ Processor
|
+—- transient failure —> Retry
|
+—- permanent failure —> Manual investigation
|
+—- business failure —-> Business exception queue
Manual intervention should normally be reserved for:
- Invalid business data
- Security-sensitive transactions
- Corrupted payloads
- Unknown message versions
- Code defects requiring a deployment
- Messages requiring business approval
4. Retry before DLQ
The first line of defence should be retry.
For example:
Attempt 1
|
+– failure
|
Attempt 2
|
+– failure
|
Attempt 3
|
+– failure
|
DLQ
But retries should not normally happen immediately.
A better strategy is exponential backoff:
Retry 1 → 10 seconds
Retry 2 → 30 seconds
Retry 3 → 2 minutes
Retry 4 → 10 minutes
Retry 5 → DLQ
The exact values should be configurable.
This is particularly important when the downstream system itself is experiencing an outage.
Otherwise, your retry mechanism can create a retry storm.
5. Spring Cloud Function model
With Spring Cloud Function, your consumer can be represented as:
@Bean
public Consumer<Message<Request>> processRequest() {
return message -> {
process(message.getPayload());
};
}
The messaging infrastructure is responsible for delivering the message to the function.
The function should focus on:
Receive
↓
Validate
↓
Process
↓
Call downstream
↓
Success / Exception
The messaging binder should be responsible for the mechanics of:
ACK
Retry
REDELIVERY
DLQ/Error Queue
This separation is particularly useful when you want the same application code to work with different messaging providers.
6. AWS SQS approach
With AWS SQS, the typical architecture is:
SNS Topic
|
v
Main SQS Queue
|
v
Spring Cloud Function
|
+—- Success —> Delete/Acknowledge
|
+—- Failure —> Retry
|
v
maxReceiveCount
|
v
DLQ
SQS supports a redrive policy where a message is moved to a DLQ after the configured maximum receive count.
For example:
maxReceiveCount = 5
means the message can be received multiple times before being moved to the DLQ.
7. AWS DLQ processing
Once messages are in the DLQ, there are two possible approaches.
Option A — Redrive
Move messages from DLQ back to the original queue.
DLQ
|
v
Original Queue
|
v
Consumer
This is useful when the problem was transient.
For example:
10:00 AM → downstream unavailable
10:05 AM → messages enter DLQ
10:20 AM → downstream recovered
10:25 AM → redrive DLQ
Option B — Dedicated DLQ processor
A separate Spring Cloud Function consumes the DLQ.
DLQ
|
v
DLQ Consumer
|
+—- transient —> Retry Queue
|
+—- permanent —> Error Store
|
+—- valid —> Main Queue
I generally prefer this approach for enterprise systems because it provides greater control.
8. Why a separate DLQ listener is useful
Do not necessarily modify your business consumer to handle both normal and DLQ messages.
Instead:
processRequest()
handles the normal queue.
And:
processDlqMessage()
handles the DLQ.
For example:
@Bean
public Consumer<Message<Request>> processRequest() {
return message -> {
process(message.getPayload());
};
}
and:
@Bean
public Consumer<Message<Request>> processDlqMessage() {
return message -> {
processDlq(message.getPayload());
};
}
The DLQ function can implement additional logic:
Read DLQ message
|
v
Determine failure reason
|
+—- transient —> retry
|
+—- permanent —> error store
|
+—- unknown —–> manual investigation
9. Solace is slightly different
This is where an important distinction needs to be made.
With Solace PubSub+, there are two concepts that are often incorrectly called a “DLQ”:
Error Queue
An Error Queue is used when the application receives the message but cannot process it successfully.
Dead Message Queue DMQ
A DMQ is used by Solace broker features such as message TTL expiration or exceeding maximum redelivery attempts.
These are not exactly the same thing. Solace’s Spring Cloud Stream binder explicitly distinguishes Error Queues from DMQs. [GitHub](https://github.com/SolaceProducts/solace-spring-cloud/blob/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter/README.adoc?utm_source=chatgpt.com)
This distinction is extremely important when designing your architecture.
10. Solace message flow
A typical Solace architecture looks like:
Producer
|
v
Topic
|
v
Consumer Queue
|
v
Spring Cloud Function
|
+—— SUCCESS
| |
| v
| ACK
|
+—— FAILURE
|
v
Redelivery
|
v
Maximum attempts
|
v
DMQ
Alternatively, using the Spring Cloud Stream Solace binder:
Consumer Queue
|
v
Spring Function
|
X
Processing failure
|
v
Solace Error Queue
The binder supports an autoBindErrorQueue property specifically for this purpose. When enabled, failed messages can be republished to a durable error queue after internal retries are exhausted. [GitHub](https://github.com/SolaceProducts/solace-spring-cloud/blob/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter/README.adoc?utm_source=chatgpt.com)
11. Configuring a Solace Error Queue
With the Solace Spring Cloud Stream binder, the important property is:
spring:
cloud:
stream:
bindings:
processRequest-in-0:
destination: request/topic
group: atomic-consumer
solace:
bindings:
processRequest-in-0:
consumer:
autoBindErrorQueue: true
The binder can automatically provision the error queue.
By default, the generated error queue follows a naming structure similar to:
scst/error/wk/<group>/plain/<destination>
The exact naming can be customized using:
errorQueueNameExpression
and an externally provisioned queue can also be used. [GitHub](https://github.com/SolaceProducts/solace-spring-cloud/blob/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter/README.adoc?utm_source=chatgpt.com)
For enterprise environments, I would generally recommend pre-provisioning important queues administratively rather than allowing every production application deployment to create infrastructure dynamically.
12. Configure an explicit Solace error queue
For example:
spring:
cloud:
stream:
bindings:
processRequest-in-0:
destination: request/topic
group: atomic-consumer
solace:
bindings:
processRequest-in-0:
consumer:
queueNameExpression: “‘ATOMIC.REQUEST.Q'”
autoBindErrorQueue: true
errorQueueNameExpression: “‘ATOMIC.REQUEST.ERROR.Q'”
This gives you a much cleaner enterprise topology:
ATOMIC.REQUEST.Q
|
v
Atomic Function
|
X
failure
|
v
ATOMIC.REQUEST.ERROR.Q
The Solace binder documents autoBindErrorQueue as disabled by default, so it must be explicitly enabled if you want this behaviour. [GitHub](https://github.com/SolaceProducts/solace-spring-cloud/blob/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter/README.adoc?utm_source=chatgpt.com)
13. Error Queue vs DMQ in Solace
This distinction deserves its own section.
Error Queue
Application received message
|
X
Application error
|
v
Error Queue
The application has actually consumed the message from the broker, but processing failed.
DMQ
Application Queue
|
v
Redelivery
|
v
Maximum redelivery exceeded
|
v
DMQ
The broker can move the message to the DMQ because of broker-level delivery policies such as maximum redelivery or message expiry. [GitHub](https://github.com/SolaceProducts/solace-spring-cloud/blob/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter/README.adoc?utm_source=chatgpt.com)
Therefore, don’t simply call every Solace failure queue a “DLQ”.
A better enterprise terminology is:
Application Error Queue
+
Solace DMQ
14. What happens when the downstream system times out?
This is one of the most important scenarios in your architecture.
Consider:
Base Engine
|
v
Atomic
|
v
Downstream
Suppose:
Atomic → Downstream
|
X
timeout
The Atomic service throws:
throw new DownstreamTimeoutException();
The message should not immediately go to the DLQ.
Instead:
Downstream Timeout
|
v
Retry
|
+—- success —> ACK
|
+—- failure —> Retry
|
v
Max attempts
|
v
DLQ
15. Classify exceptions
A very useful pattern is to classify exceptions.
public enum FailureType {
TRANSIENT,
PERMANENT,
BUSINESS,
TECHNICAL
}
For example:
| Exception | Classification | Action |
| Connection timeout | TRANSIENT | Retry |
| HTTP 503 | TRANSIENT | Retry |
| HTTP 429 | TRANSIENT | Backoff |
| Invalid JSON | PERMANENT | DLQ |
| Missing mandatory field | BUSINESS | Error queue |
| Unsupported version | BUSINESS | Error queue |
| NullPointerException | TECHNICAL | DLQ + alert |
| DB temporarily unavailable | TRANSIENT | Retry |
This prevents the DLQ from being filled unnecessarily.
16. The most important design principle: Idempotency
DLQ processing introduces another important issue:
The same message can be processed multiple times.
For example:
Attempt 1
|
Downstream successfully processes
|
Application crashes before ACK
|
Message redelivered
|
Attempt 2
Now the downstream system may receive the same transaction twice.
Therefore, your application should use an idempotency key such as:
messageId
transactionId
correlationId
businessReference
For example:
Transaction ID
|
v
Redis / DB
|
+—- already processed —> skip
|
+—- new —————–> process
This becomes even more important when a DLQ message is manually replayed.
17. Do not blindly replay DLQ messages
A dangerous design is:
DLQ
|
v
Replay everything
|
v
Main Queue
Imagine 10,000 messages are in the DLQ because the downstream system is unavailable.
Replaying all 10,000 immediately can create:
DLQ
|
+–> 10,000 messages
|
v
Main Queue
|
v
Downstream overload
Instead use controlled replay:
DLQ
|
v
DLQ Processor
|
v
Rate Limiter
|
v
Retry Queue
|
v
Main Queue
|
v
Downstream
This is especially relevant to your earlier requirement around rate limiting message consumption and publishing.
18. Recommended DLQ processor
A dedicated DLQ processor can look conceptually like:
@Bean
public Consumer<Message<Request>> processDlq() {
return message -> {
Request request = message.getPayload();
FailureDetails failure =
extractFailureDetails(message);
switch (failure.type()) {
case TRANSIENT:
retry(request);
break;
case BUSINESS:
moveToBusinessErrorStore(request);
break;
case PERMANENT:
moveToPermanentFailureStore(request);
break;
case TECHNICAL:
raiseAlert(request);
break;
}
};
}
The actual implementation should also consider idempotency, retry count, rate limiting and observability.
19. Recommended Solace architecture for your platform
For your Spring Cloud Function architecture, I would recommend:
┌─────────────────────┐
│ Solace Topic │
└──────────┬──────────┘
|
v
┌─────────────────────┐
│ Atomic Request Queue│
└──────────┬──────────┘
|
v
┌─────────────────────┐
│ Spring Cloud │
│ Function │
└──────────┬──────────┘
|
Downstream Call
|
┌─────────────┴─────────────┐
│ │
SUCCESS FAILURE
│ │
v v
ACK Internal Retry
|
Max Attempts
|
v
┌─────────────────────┐
│ Error Queue │
│ ATOMIC.ERROR.Q │
└──────────┬──────────┘
|
v
DLQ Processor
|
┌─────────────────┼────────────────┐
│ │ │
Transient Business Permanent
│ │ │
v v v
Retry Queue Error Store Alert
|
v
Main Queue
20. What about the Solace DMQ?
The DMQ should be treated as a last-resort broker-level safety net.
For example:
ATOMIC.REQUEST.Q
|
| redelivery
|
v
max redelivery exceeded
|
v
Solace DMQ
You can then have a separate operational process for DMQ messages:
DMQ
|
v
DMQ Processor
|
+—- recoverable —> replay
|
+—- invalid ——-> quarantine
|
+—- unknown ——-> manual investigation
Do not mix the application Error Queue and broker DMQ without documenting the distinction.
21. One DLQ listener or multiple listeners?
I recommend a generic DLQ processing framework rather than implementing completely independent logic for every queue.
For example:
DLQ Processor
|
+—- ATOMIC.ERROR.Q
|
+—- CHOREO.ERROR.Q
|
+—- MEDIATOR.ERROR.Q
The processor can inspect metadata:
{
“originalQueue”: “ATOMIC.REQUEST.Q”,
“originalDestination”: “atomic/request”,
“failureType”: “TRANSIENT”,
“retryCount”: 3,
“correlationId”: “12345”,
“transactionId”: “ABC123”
}
and determine the appropriate recovery strategy.
However, business-specific recovery logic should remain with the owning service.
Don’t build one giant “DLQ service” containing every application’s business rules.
22. Observability is mandatory
A production DLQ solution should expose metrics such as:
messages.processed
messages.failed
messages.retried
messages.dlq
messages.replayed
messages.permanently.failed
downstream.timeout
downstream.5xx
For Solace, monitor:
Queue depth
Message spool
Redelivery count
Consumer count
Oldest message age
Ingress rate
Egress rate
DMQ count
Error queue count
This allows you to distinguish:
100 DLQ messages
from:
100 DLQ messages and increasing rapidly
The second scenario is much more serious.
23. Manual replay should still exist
Even with a highly automated DLQ architecture, operations teams need a controlled replay mechanism.
For example:
DLQ Management UI
|
+– View message
|
+– View failure
|
+– View retry history
|
+– Replay
|
+– Discard
|
+– Quarantine
Replay should require appropriate authorization and should preferably support:
Replay one message
Replay selected messages
Replay by transaction
Replay by failure type
Replay by timestamp
Avoid giving operators unrestricted access to replay the entire queue.
24. Recommended end-to-end strategy
For your architecture, I would implement the following policy:
Normal processing
Message
↓
Spring Cloud Function
↓
Business processing
↓
Downstream
↓
SUCCESS
↓
ACK
Transient failure
Message
↓
Downstream timeout
↓
Retry
↓
Retry
↓
Retry
↓
Success
Persistent transient failure
Message
↓
Retry
↓
Retry
↓
Retry exhausted
↓
Error Queue / DLQ
↓
DLQ Processor
↓
Delayed retry
↓
Main Queue
Permanent failure
Message
↓
Validation failure
↓
Error Queue
↓
Error Store
↓
Alert
↓
Manual correction
↓
Controlled replay
25. Final recommendation
For a Spring Cloud Function + Solace/AWS messaging abstraction, I would design DLQ handling as part of your common messaging library.
Something like:
Messaging Library
|
┌────────────┴────────────┐
│ │
Solace AWS SQS
│ │
Error Queue DLQ
│ │
└────────────┬────────────┘
│
v
Common DLQ Model
|
v
DLQ Processor
|
┌───────────────┼────────────────┐
│ │ │
Retry Quarantine Manual
│ │ Replay
v v │
Main Queue Error Store <────────┘
The business service should not need to know whether the underlying messaging provider is Solace or AWS SQS.
Your abstraction can expose something conceptually like:
public interface DeadLetterHandler {
void handle(DeadLetterMessage message);
void retry(DeadLetterMessage message);
void quarantine(DeadLetterMessage message);
void replay(DeadLetterMessage message);
}
Then:
Solace implementation
↓
Solace Error Queue / DMQ
AWS implementation
↓
SQS DLQ
while the application-level recovery model remains consistent.
The key takeaway
DLQ is not the end of the message’s life. It is the beginning of a controlled recovery workflow.
For your architecture, I would therefore avoid “DLQ = manually process messages”.
Instead use:
Retry → Error Queue/DLQ → Automated DLQ Processor → Rate-limited Replay → Manual Intervention for genuine exceptions.
The Solace Spring Cloud Stream binder already provides useful primitives for this model, including internal retries, redelivery, error-queue republishing, configurable error queues and acknowledgement behaviour. [GitHub](https://github.com/SolaceProducts/solace-spring-cloud/blob/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter/README.adoc?utm_source=chatgpt.com)
For Solace specifically, also remember:
Error Queue ≠ DMQ.
An Error Queue is appropriate for application processing failures; a DMQ is a broker-level mechanism for messages that are rerouted because of things such as expiry or maximum redelivery. [GitHub](https://github.com/SolaceProducts/solace-spring-cloud/blob/master/solace-spring-cloud-starters/solace-spring-cloud-stream-starter/README.adoc?utm_source=chatgpt.com)
Suggested production pattern for your platform:
Spring Cloud Function
|
┌───────────┴───────────┐
│ │
Solace AWS
│ │
Error Queue SQS DLQ
│ │
└───────────┬───────────┘
│
Common DLQ Model
│
v
DLQ Processor
|
┌──────────┼──────────┐
│ │ │
Retry Quarantine Alert
│
v
Rate Limiter
│
v
Main Queue
│
v
Downstream
This gives you a provider-independent DLQ strategy while still taking advantage of the native Solace and AWS capabilities.
