Message Queues आणि Async Processing

Message Queue म्हणजे producers (messages पाठवणारे) आणि consumers (messages process करणारे) यांच्यामधला buffer. Direct synchronous calls ऐवजी async message passing — यामुळे systems loose-coupled, scalable आणि resilient होतात.

Message Queue कशासाठी?

• ▸Email/SMS notifications — request async मध्ये process करा
• ▸Image/Video processing — upload झाल्यावर background मध्ये process
• ▸Order processing — e-commerce orders queue मध्ये, inventory/payment async
• ▸Log processing — millions of logs Kafka मध्ये, batch process
• ▸Microservices communication — services directly communicate नाही करत

Without Queue (Synchronous):
User signup → API → Send Email → (wait 3 sec) → Response
Problem: User 3 seconds wait करतो. Email service down = signup fail!

With Queue (Asynchronous):
User signup → API → Queue मध्ये email task push → Instant Response ✅
                              ↓ (background)
                    Email Worker → Email send करतो

Popular Message Queues

• ▸Apache Kafka — high throughput, log streaming, event sourcing. LinkedIn, Uber वापरतात. millions of messages/sec handle करतो.
• ▸RabbitMQ — traditional message broker. Complex routing, task queues साठी.
• ▸AWS SQS — managed queue service, serverless friendly.
• ▸Redis Pub/Sub — simple, low-latency. Real-time notifications साठी.

const Queue = require('bull');
const emailQueue = new Queue('email', { redis: redisConfig });

// Producer — API मध्ये
app.post('/signup', async (req, res) => {
  const user = await createUser(req.body);

  // Queue मध्ये email job add करा (non-blocking)
  await emailQueue.add({ userId: user.id, email: user.email });

  res.json({ success: true }); // Instant response!
});

// Consumer — background worker मध्ये
emailQueue.process(async (job) => {
  const { userId, email } = job.data;
  await sendWelcomeEmail(email);
  console.log(`Email sent to ${email}`);
});