Kafka Is Not A Job Queue

Project Overview

A core campaign management system was utilizing AWS MSK (Managed Streaming for Apache Kafka) at a cost of ~$1,000/month. The system was suffering from queue blocking and disproportionate infrastructure overhead. The root cause was an architectural mismatch: we were using an event streaming platform to do a job queue’s work.

The Architectural Mismatch

The backend consisted of an API that accepted campaign launch requests and a worker unit that executed outbound tasks (email, SMS, voicemail) with specific step delays, retries and status checks.

Kafka sat between them. While Kafka provides a strong boundary between request paths and background processors, it is fundamentally designed for ordered event streams, log retention and partition-based consumption.

Our campaign workers, however, required job queue semantics: independent tasks with distinct retry counts, delays, priorities and concurrency limits. Because the setup used a single topic and consumer group, unrelated campaign tasks were forced into a strictly ordered line. A slow job (like a rate-limited voicemail batch) would block faster jobs (like email sends) competing for the same consumption path, severely degrading the system.

The Migration: BullMQ and Redis

To align the architecture with business needs, Kafka was replaced with BullMQ running on Redis. Instead of a single shared stream, the system uses dedicated queues per task type, allowing independent concurrency limits based on the cost, external rate limits and speed of the work.

export const WORKER_CONCURRENCY = {
  [JobType.CAMPAIGN]: 100,
  [JobType.CAMPAIGN_STEP]: 40,
  [JobType.EMAIL]: 30,
  [JobType.SMS]: 50,
  [JobType.VOICEMAIL]: 1,
  [JobType.WATCH]: 100
};

Simplified Handoff

The API no longer publishes to a Kafka topic; it adds a job directly to the Redis queue with an idempotency key to protect against duplicate processing.

const queue = QueueManager.getInstance().getQueue<CampaignJob>('CAMPAIGN');
await queue.add('launch', { campaignId, userId }, { jobId: idempotencyKey });

Results & Business Impact

Moving from a managed streaming platform to an in-memory job queue successfully eliminated the queue-blocking issues. Moreover, it drastically reduced our infrastructure footprint.

Item	Monthly Cost
Before: AWS MSK	~$1,000
After: Redis	~$180
Saved	~$820/month (82% Reduction)

Takeaway

Treating a job queue as an event stream introduces unnecessary complexity. Kafka’s partition and broker model is designed for high-throughput, ordered event streams shared across multiple independent consumers. For independent, delayed and rate-limited background tasks, a dedicated job queue provides better control primitives and significantly lower operational overhead.