Apache Iggy (Incubating) | Hyper-Efficient Message Streaming | Apache Iggy Hyper-Efficient Message Streaming at Laser Speed Apache Iggy™ (Incubating) is a high-performance, persistent message streaming platform written in Rust, capable of processing millions of messages per second with ultra-low latency. Millions Messages/Second/Node ~1 ms Avg Write Latency 7 Language SDKs 100% Free & Open Source Get StartedView on GitHub Built for performance Designed from the ground up with io_uring and thread-per-core, shared nothing architecture. Each CPU core runs its own shard, pinned and NUMA-aware. No locks on the hot path, no GC pauses, no thread contention. ExperimentalVSR clustering is coming soon Viewstamped Replication Revisited is already implemented in the server-ng module on main. Its deterministic simulation testing (DST) exercises failures, delays, restarts and network partitions to validate consensus. Together, VSR and DST provide the foundation for highly available, fault-tolerant and reliable Iggy clusters. Explore VSR → In developmentKafka Gateway for compatible clients and easier migrations The upcoming Kafka wire-protocol proxy will bridge existing Kafka producers and consumers to Iggy, enabling gradual migration with minimal client-side changes. Core APIs, consumer groups, admin operations and authentication are tracked in a phased public roadmap. Track the epic ↗ 2M+msg/s Throughput Single node 1GB/s Producer throughput Persisted writes 2GB/s Consumer throughput Persistent log reads 0.976ms Producer P99 0.466 ms average 0.495ms Consumer P99 0.357 ms average Producer 0.466 ms avg Consumer 0.357 ms avg Apache Iggy 0.8.0 · 40M messages ms 1.0 0.8 0.6 0.4 0.2 0.0 Latency breakdown (ms) Percentile Producer Consumer Avg 0.466 0.357 Median 0.349 0.351 P95 0.886 0.446 P99 0.976 0.495 P99.9 1.114 0.566 Machine: AWS i4i.4xlarge · persistent log workloadProducer result →Consumer result → Ultra-High Performance Process millions of messages per second with predictable low latency thanks to Rust, combined with io_uring and thread-per-core, shared nothing architecture. Zero-Copy Serialization Custom zero-copy (de)serialization for improved performance and reduced memory usage, working directly with binary data. Multiple Transport Protocols Support for QUIC, TCP, WebSocket, and HTTP protocols with TLS encryption, giving you flexibility in how clients connect. Multi-Language SDKs Client libraries available for Rust, C#, Java, Go, Python, Node.js and PHP, with C++ coming soon. Consumer Groups & Partitioning Built-in support for consumer groups with cooperative rebalancing, partitioning, and horizontal scaling across connected clients. Security & Access Control TLS on all transports, per-stream and per-topic permissions, Personal Access Tokens for programmatic access, optional AES-256-GCM encryption at rest. Built-in Monitoring OpenTelemetry logs & traces, Prometheus metrics, and built-in benchmarking tools for performance monitoring. Multi-Tenant Support Stream abstraction for multi-tenancy, configurable message retention policies, and tiered storage coming in the future. How it works Messages flow from producers through streams and topics into partitioned, append-only segment files on disk. Pick your language and start streaming in minutes. 1 Producers send messages Connect via TCP, QUIC, WebSocket or HTTP. Messages are routed to the target partition using balanced, key-based or explicit partitioning. 2 Shard receives and buffers Each partition is owned by exactly one CPU-pinned shard. Messages are buffered in a memory journal, then flushed to disk via vectored I/O through io_uring. 3 Segments store on disk Data lands in append-only .log files with .index files for offset and timestamp lookups. Segments are sealed at 1 GiB and rotated automatically. 4 Consumers poll at any offset Read from the beginning, a specific offset, a timestamp, or continue from the last committed position. Consumer groups distribute partitions for horizontal scaling. RustPythonJavaGoNode.jsC#PHPC++ (WIP) producer.rs SDK docs → use iggy::prelude::*; let client = IggyClient::from_connection_string( "iggy://iggy:iggy@localhost:8090" )?; client.connect().await?; let producer = client .producer("orders", "events")? .direct( DirectConfig::builder() .batch_length(100) .build() ) .partitioning(Partitioning::balanced()) .build(); producer.init().await?; let msg = IggyMessage::from_str("order-123")?; producer.send(vec![msg]).await?; cargo add iggycrates.io → Complete ecosystem Iggy is more than a server. Integrate with external systems, manage everything from your browser or terminal, and connect LLMs to your streaming infrastructure. Connectors Dynamically loaded Rust plugins for data integration. Source from PostgreSQL, Elasticsearch, or sink to MongoDB, Elasticsearch, Apache Iceberg, Quickwit. Built-in data transforms. PostgreSQLMongoDBElasticsearchIcebergQuickwit MCP Server Model Context Protocol server exposing 40+ tools for LLM integration. Works with Claude Desktop and other MCP clients via stdio and HTTP transports. ClaudeLLM40+ toolsstdioHTTP Web UI SvelteKit dashboard for stream/topic management, message browsing with JSON/string/XML decoders, user management, server logs, and real-time terminal. DashboardMessage browserUser mgmt CLI Full-featured command-line interface with named connection contexts, session-based login, and shell completions for bash/zsh/fish/powershell. ContextsCompletionsLogin sessions