Work in Progress: This project is under active development. Core components are implemented, but not all features are production-ready.
Relayr is a high-performance, distributed message broker written in Go, inspired by the principles of Service-Oriented Architecture (SoA) and modern messaging patterns. Built with Go's powerful concurrency primitives, Relayr provides both queue (point-to-point) and topic (publish-subscribe) messaging patterns with a focus on performance, reliability, and simplicity.
- Dual Messaging Patterns: Supports both FIFO queues for point-to-point messaging and topics for publish-subscribe patterns
- High Performance: Leverages Go's goroutines and channels for efficient concurrent message processing
- Custom Wire Protocol: Text-based protocol similar to Redis/NATS for easy debugging and client implementation
- Efficient Storage: Linked-list based storage with O(1) enqueue/dequeue operations
- Message TTL Support: Automatic expiration of old messages
- Metrics & Monitoring: Built-in performance metrics and health checks
- Thread-Safe Operations: All operations are fully concurrent-safe
- Multiple Serialization Formats: JSON, Gob, and custom binary formats supported
- Defines the core
Messagetype with headers, metadata, and body - Supports message TTL, priority, and correlation IDs
- Provides multiple serialization formats for different use cases
- Implements message validation and error handling
- LinkedListQueue: FIFO queue implementation using linked lists
- O(1) enqueue and dequeue operations
- Automatic expired message cleanup
- Configurable capacity limits
- SharedListTopic: Topic storage with multiple subscriber support
- Reference counting for efficient garbage collection
- Independent read positions for each subscriber
- Automatic cleanup of consumed messages
- Custom text-based wire protocol
- Command-based communication (PUB, SUB, ACK, NACK, etc.)
- Support for headers and metadata
- Efficient frame parsing and encoding
- Defines core broker abstractions
- Queue and Topic interfaces
- Publisher/Consumer patterns
- Subscription management
- Health monitoring and statistics
Relayr uses a human-readable, text-based protocol inspired by Redis and NATS:
# Connect to broker
CONNECT {"version":"1.0","client_id":"client123"}
OK
# Publish to queue
PUB queue:orders 123
{"order_id":"12345","amount":99.99}
OK msg_id:abc123
# Subscribe to topic
SUB topic:events sub_id:sub456
OK
# Receive message
MSG topic:events sub_id:sub456 msg_id:xyz789 45
{"event":"order_placed","order_id":"12345"}
# Acknowledge message
ACK msg_id:xyz789
OK
Relayr/
├── pkg/ # Public packages
│ ├── broker/ # Core broker interfaces
│ │ └── interfaces.go # Broker, Queue, Topic interfaces
│ ├── message/ # Message definitions
│ │ ├── message.go # Core message types
│ │ ├── errors.go # Error definitions
│ │ └── serialization.go # Message serialization
│ ├── protocol/ # Wire protocol
│ │ ├── protocol.go # Protocol definitions
│ │ ├── parser.go # Frame parsing
│ │ └── encoder.go # Frame encoding
│ └── storage/ # Storage interfaces
│ └── storage.go # Storage abstractions
├── internal/ # Private packages
│ ├── queue/ # Queue implementation
│ │ └── queue.go # Linked-list FIFO queue
│ └── topic/ # Topic implementation
│ └── topic.go # Shared-list topic with ref counting
├── go.mod # Go module definition
└── go.sum # Dependency checksums
- Core message types and metadata structures
- Message serialization (JSON, Gob, Binary)
- Storage interfaces and abstractions
- Linked-list queue implementation with TTL support
- Shared-list topic with reference counting
- Wire protocol definition
- Protocol parser and encoder
- Broker interfaces and abstractions
- Comprehensive error handling
- Transport layer (TCP/UDP servers)
- Connection management
- Broker orchestration
- Client SDK
- Persistent storage backend
- Clustering and replication
- Message routing and filtering
- Authentication and authorization
- WebSocket support
- gRPC interface
- Admin dashboard
- Docker deployment
- Go 1.24.1 or higher
- Make (for build automation)
# Clone the repository
git clone https://github.com/zde37/relayr.git
cd relayr
# Download dependencies
go mod download
# Build the project
make build// Example client code (not yet implemented)
client, err := relayr.Connect("localhost:9092")
if err != nil {
log.Fatal(err)
}
// Publish to a queue
err = client.Publish("queue:orders", []byte(`{"order_id":"12345"}`))
// Subscribe to a topic
sub, err := client.Subscribe("topic:events")
for msg := range sub.Messages() {
fmt.Printf("Received: %s\n", msg.Body)
msg.Ack()
}- Queue Operations: O(1) enqueue and dequeue
- Topic Publish: O(n) where n is the number of subscribers
- Message Storage: Efficient linked-list structure minimizes memory fragmentation
- Garbage Collection: Reference counting ensures minimal memory overhead
- Concurrency: Lock-free operations where possible, fine-grained locking elsewhere
- Linked Lists over Arrays: Chosen for O(1) operations and to avoid array resizing overhead
- Reference Counting: Enables efficient garbage collection in topics without stop-the-world pauses
- Text Protocol: Human-readable for easier debugging while maintaining performance
- Goroutine-per-Connection: Leverages Go's lightweight threads for scalability
- Modular Architecture: Clear separation between protocol, storage, and broker layers
Contributions are welcome! As this is a work in progress, please:
- Check existing issues or create a new one before starting work
- Follow Go best practices and conventions
- Add tests for new functionality
- Update documentation as needed
MIT Liscense
For questions, issues, or suggestions, please open an issue on GitHub or feel free to reach out anytime hi@zde37.com
Note: This README will be updated as the project evolves and more features are implemented.