Challenge #5c: Efficient Kafka-Style Log

In this challenge, you’ll need to take your Multi-Node Kafka system and make it more efficient. Storing data in lin-kv has a drawback in that with multiple leaders and a high request rate, you’re likely to see frequent compare-and-swap errors due to contention on shared keys.

Specification

Using the Lamport diagrams (messages.svg) and network stats (results.edn) as your guide, reduce the probability of CaS failures and improve overall efficiency. Your solution should exchange significantly fewer messages per operation. Improve your latency & availability.

All correctness checks should still pass.

Evaluation

Build your Go binary as maelstrom-kafka and run it against Maelstrom with the following command:

./maelstrom test -w kafka --bin ~/go/bin/maelstrom-kafka --node-count 2 --concurrency 2n --time-limit 20 --rate 1000

This challenge has a looser definition of success. See how much you can improve your performance and availability. Post your results or swap notes with folks on the Fly.io Community forum

Once you’re done, then you’ve completed the Kafka challenge! Continue on to the Totally-Available Transactions challenge.