Single Node Strategy

Purpose

The Single Node Strategy addresses the need for a fault-tolerant and resumable Kafka consumer offset management mechanism tailored for single-node Apache Camel Kafka integrations. Unlike more complex multi-node or clustered resume strategies, this strategy maintains offset state by publishing and consuming offset records exclusively through a dedicated Kafka topic. This approach enables offset recovery after consumer restarts or failures without relying on external storage systems.

Functionality

At its core, the Single Node Strategy asynchronously loads offset data from a configured Kafka topic into a local cache and updates offsets by publishing new records back to the same topic. This bidirectional flow ensures a consistent, durable, and recoverable offset state.

Key Workflows

• Asynchronous Cache Loading
  Upon initialization, the strategy spins up a background consumer thread that subscribes to the configured resume topic. This thread continuously polls Kafka and deserializes offset records, populating a local cache used for seeking consumer partitions on startup or rebalance.

• Offset Updates via Kafka Producer
  Offset changes are serialized and sent asynchronously as Kafka producer records to the resume topic. A write lock ensures thread-safe producer access, while callbacks handle error logging and update notifications.

• Partition Seek on Subscription
  When subscribing the internal consumer to the resume topic, the strategy optionally rewinds consumption by a number of messages corresponding to the cache capacity. This allows rehydration of the cache with recent offset states, ensuring accuracy.

• Lifecycle Management
  The strategy manages lifecycle events such as start (creating the producer), stop (closing consumer and producer), and resource cleanup, integrating smoothly with the Apache Camel lifecycle and concurrency utilities.

Illustrative Snippet

// Publishing offset update asynchronously with thread safety
writeLock.lock();
try {
    produce(keyBuffer.array(), valueBuffer.array(), updateCallBack);
} finally {
    writeLock.unlock();
}

This snippet highlights critical synchronization around offset publishing to prevent concurrent write conflicts.

Relationship

This strategy is a specialized implementation under the broader **Resume Strategies** topic, which collectively manage how Kafka consumer offsets are persisted and restored for fault tolerance. It complements other strategies by focusing on single-node use cases, where offset state is held solely within Kafka topics rather than distributed external systems.

• Integrates with the Kafka Consumer by providing offset data for partition seeking during rebalance events.

• Works alongside Commit Managers and Manual Offset Commit Support to ensure offsets are committed consistently.

• Utilizes the Resume Adapter and Resume Cache abstractions for serialization, deserialization, and in-memory offset caching.

• Extends the general Resume Strategy interface, inheriting lifecycle and configuration responsibilities.

Uniquely, it introduces asynchronous cache loading and offset refreshing from Kafka topics directly, a mechanism not covered by other resume strategies.

Diagram

sequenceDiagram
    participant Strategy as SingleNodeKafkaResumeStrategy
    participant Kafka as Kafka Topic (Resume Topic)
    participant Cache as Local Offset Cache
    participant Producer as Kafka Producer
    participant Consumer as Kafka Consumer

    Note over Strategy: Initialization
    Strategy->>Consumer: createConsumer() & subscribe()
    Consumer->>Kafka: poll() loop
    Kafka-->>Consumer: offset records
    Consumer->>Strategy: deliver records
    Strategy->>Cache: deserialize and cache offsets

    Note over Strategy: Offset Update
    Strategy->>Producer: produce(offsetKey, offsetValue)
    Producer->>Kafka: send asynchronously
    Kafka-->>Producer: ack or error
    Producer->>Strategy: callback updates status

    Note over Strategy: Shutdown
    Strategy->>Producer: close()
    Strategy->>Consumer: wakeup() & close()

This sequence diagram captures the core asynchronous caching and offset update flows central to the Single Node Strategy. It emphasizes the continuous polling for offset data to populate the cache and the asynchronous offset publishing to Kafka topics.