Verification against real brokers

The map is not the territory. Unit tests prove kacrab is self-consistent — they do not prove it is Kafka-consistent: a codec framed the wrong way, or a SASL round skipped, passes a Rust-only round trip because kacrab is on both ends. Most of the discoveries in this book — the OAUTHBEARER error round, the GSSAPI security layer, the burst wedge, the IPv6 hang, the topic-id codec traps — were found here, walking the actual territory: every path exercised end-to-end against real Apache Kafka 4.3.0, with the environments captured as docker-compose.*.yml files.

Quote

“A few minutes of docker compose proves a path works; proving it holds up needs hours of load.” — see Benchmarks and the production acceptance plan.

SASL — every mechanism

docker-compose.auth.yml (a single broker with PLAIN + SCRAM-256/512 + OAUTHBEARER) and docker-compose.gssapi.yml (a real MIT Kerberos KDC + a GSSAPI broker). Each mechanism authenticates and runs ApiVersions; negative cases confirm rejection is fast and clear.

TLS — SSL, SASL_SSL, mutual TLS

docker-compose.tls.yml publishes three listeners; certs are generated by scripts/gen-tls-certs.sh. One-way SSL, SASL/SCRAM over TLS, and mutual TLS all authenticate; an untrusted server certificate is rejected fast with the rustls reason.

Compression — every codec, proven on disk

The decisive test: produce gzip/snappy/lz4/zstd batches, then verify three things end to end — the broker accepts the produce, kafka-dump-log shows the on-disk batch stored with the correct codec (so a silently-uncompressed send would fail), and kafka-console-consumer decompresses the exact payloads back.

on-disk batch codecs: [gzip×3, snappy×3, lz4×3, zstd×3]
all 12 compressed records round-tripped through the broker and CLI consumer

Consumer — groups, offsets, and both protocols

kacrab/tests/real_kafka_consumer.rs (12 tests, --test-threads=1) walks the consumer surface against a real broker: manual assignment and commit round-trips; one subscriber owning a whole topic; two consumers splitting it via a real rebalance; cooperative-sticky converging with no double-owned partition; the roundrobin assignor; pattern subscription; interceptors; offset queries and lag; async commits landing in call order; seek-past-end recovering; and a KIP-848 (group.protocol=consumer) member joining, consuming, and committing through ConsumerGroupHeartbeat. See the consumer chapters for what each proves.

Multi-broker — dispatch & leadership-change

docker-compose.cluster.yml runs a 3-broker KRaft cluster. The tests confirm records route to every broker’s leaders, and that a broker loss re-routes affected partitions to their new leaders without wedging the healthy partitions batched alongside them — the bug described in Idempotency & transactions and Failure modes. A 6-partition burst across a broker loss delivers 6/6.

Admin — the full operation surface

docker-compose.kafka.yml (core ops), docker-compose.kafka-admin.yml (a broker with StandardAuthorizer and the share/streams features, for ACLs and the Kafka 4.x group families), and docker-compose.auth.yml (SASL, for delegation tokens) back kacrab/tests/real_kafka_admin*.rs. Every one of the 62 admin operations is exercised across all four routing paths (controller, coordinator, per-leader, broadcast). Operations that need cluster state the test cannot create are asserted at the wire layer — a well-formed broker error code proves correct encode/decode. This pass caught real wire bugs the unit tests could not: an empty ApiVersions client name, the OffsetCommit/OffsetFetch v10 topic-name-vs-id switch, and the missing transient-coordinator-error retry (see The admin client).

Note

These are #[ignore] integration tests — they need the compose environments and (for the cluster/compression cases) the docker CLI. They are not part of the default cargo test run; CI runs the in-process suite.