November 15, 2024
Project Nebula: Serverless Event Bus
Architecting a global nervous system for distributed applications.
Project Nebula
Role: Lead Architect
Stack: AWS (Lambda, EventBridge, DynamoDB), Go, Terraform
Status: Production
The Challenge
Our legacy monolithic message queue (RabbitMQ) was becoming a single point of failure. It struggled with cross-region replication and had high operational overhead. As the company expanded to 3 continents, we needed a “Global Nervous System” that was self-healing and infinitely scalable.
The Solution
I designed Nebula, a fully serverless event bus built on AWS EventBridge.
Architecture Highlights
- Event Schema Registry: Enforced strictly typed JSON schemas for all events using Protobuf definitions, catching bad data at the door.
- Global Routing: Leveraged EventBridge archive and replay features to sync state across us-east-1, eu-central-1, and ap-southeast-1.
- Cost Optimization: Implemented specific filtering rules to only route necessary events, reducing cross-region data transfer costs by 40%.
Detailed Implementation
We used Terraform to define the entire infrastructure.
resource "aws_cloudwatch_event_bus" "nebula" {
name = "nebula-global-bus"
}
resource "aws_cloudwatch_event_rule" "cross_region" {
event_bus_name = aws_cloudwatch_event_bus.nebula.name
event_pattern = jsonencode({
"detail-type": ["order.created", "payment.processed"]
})
}
Results
- Zero Maintenance: No servers to patch or cluster to manage.
- Scalability: Handled a Black Friday peak of 50M events without a single dropped message.
- Latency: Reduced end-to-end latency from 1.5s to 200ms globally.