Please provide a short (approximately 100 word) summary of the following web Content, written in the voice of the original author. If there is anything controversial please highlight the controversy. If there is something surprising, unique, or clever, please highlight that as well. Content: Title: Implementing a distributed key-value store on top of implementing Raft in Go Site: notes.eatonphil.com As part of bringing myself up-to-speed after joining TigerBeetle , I wanted some background on how distributed consensus and replicated state machines protocols work. TigerBeetle uses Viewstamped Replication . But I wanted to understand all popular protocols and I decided to start with Raft . We'll implement two key components of Raft in this post (leader election and log replication). Around 1k lines of Go. It took me around 7 months of sporadic studying to come to (what I hope is) an understanding of the basics. Disclaimer : I'm not an expert. My implementation isn't yet hooked up to Jepsen . I've run it through a mix of manual and automated tests and it seems generally correct. This is not intended to be used in production. It's just for my education. All code for this project is available on GitHub . Let's dig in! The algorithm The Raft paper itself is quite readable. Give it a read and you'll get the basic idea. The gist is that nodes in a cluster conduct elections to pick a leader. Users of the Raft cluster send messages to the leader. The leader passes the message to followers and waits for a majority to store the message. Once the message is committed (majority consensus has been reached), the message is applied to a state machine the user supplies. Followers learn about the latest committed message from the leader and apply each new committed message to their local user-supplied state machine. There's more to it including reconfiguration and snapshotting, which I won't get into in this post. But you can get the gist of Raft by thinking about 1) leader election and 2) replicated logs powering replicated state machines. Modeling with state machines and key-value stores I've written before about how you can build a key-value store on top of Raft . How you can build a SQL database on top of a key-value store . And how you can build a distributed SQL database on top of Raft . This post will start quite similarly to that first post except for that we won't stop at the Raft layer. A distributed key-value store To build on top of the Raft library we'll build, we need to create a state machine and commands that are sent to the state machine. Our state machine will have two operations: get a value from a key, and set a key to a value. This will go in cmd/kvapi/main.go . package main import ( "bytes" crypto "crypto/rand" "encoding/binary" "fmt" "log" "math/rand" "net/http" "os" "strconv" "strings" "sync" "github.com/eatonphil/goraft" ) type statemachine struct { db *sync.Map server int } type commandKind uint8 const ( setCommand commandKind = iota getCommand ) type command struct { kind commandKind key string value string } func (s *statemachine) Apply(cmd []byte) ([]byte, error) { c := decodeCommand(cmd) switch c.kind { case setCommand: s.db.Store(c.key, c.value) case getCommand: value, ok := s.db.Load(c.key) if !ok { return nil, fmt.Errorf("Key not found") } return []byte(value.(string)), nil default: return nil, fmt.Errorf("Unknown command: %x", cmd) } return nil, nil } But the Raft library we'll build needs to deal with various state machines. So commands passed from the user into the Raft cluster must be serialized to bytes. func encodeCommand(c command) []byte { msg := bytes.NewBuffer(nil) err := msg.WriteByte(uint8(c.kind)) if err != nil { panic(err) } err = binary.Write(msg, binary.LittleEndian, uint64(len(c.key))) if err != nil { panic(err) } msg.WriteString(c.key) err = binary.Write(msg, binary.LittleEndian, uint64(len(c.value))) if err != nil { panic(err) } msg.WriteString(c.value) return msg.Bytes() } And the Apply() function from above needs to be able to decode the bytes: func decodeCommand(msg []byte) command { var c command c.kind = commandKind(msg[0]) keyLen := binary.LittleEndian.Uint64(msg[1:9]) c.key = string(msg[9 : 9+keyLen]) if c.kind == setCommand { valLen := binary.LittleEndian.Uint64(msg[9+keyLen : 9+keyLen+8]) c.value = string(msg[9+keyLen+8 : 9+keyLen+8+valLen]) } return c } HTTP API Now that we've modeled the key-value store as a state machine. Let's build the HTTP endpoints that allow the user to operate the state machine through the Raft cluster. First, let's implement the set operation. We need to grab the key and value the user passes in and call Apply() on the Raft cluster. Calling Apply() on the Raft cluster will eventually call the Apply() function we just wrote, but not until the message sent to the Raft cluster is actually replicated. type httpServer struct { raft *goraft.Server db *sync.Map } // Example: // // curl http://localhost:2020/set?key=x&value=1 func (hs httpServer) setHandler(w http.ResponseWriter, r *http.Request) { var c command c.kind = setCommand c.key = r.URL.Query().Get("key") c.value = r.URL.Query().Get("value") _, err := hs.raft.Apply([][]byte{encodeCommand(c)}) if err != nil { log.Printf("Could not write key-value: %s", err) http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest) return } } To reiterate, we tell the Raft cluster we want this message replicated. The message contains the operation type ( set ) and the operation details ( key and value ). These messages are custom to the state machine we wrote. And they will be interpreted by the state machine we wrote, on each node in the cluster. Next we handle get -ing values from the cluster. There are two ways to do this. We already embed a local copy of the distributed key-value map. We could just read from that map in the current process. But it might not be up-to-date or correct. It would be fast to read though. And convenient for debugging. But the only correct way to read from a Raft cluster is to pass the read through the log replication too. So we'll support both. // Example: // // curl http://localhost:2020/get?key=x // 1 // curl http://localhost:2020/get?key=x&relaxed=true # Skips consensus for the read. // 1 func (hs httpServer) getHandler(w http.ResponseWriter, r *http.Request) { var c command c.kind = getCommand c.key = r.URL.Query().Get("key") var value []byte var err error if r.URL.Query().Get("relaxed") == "true" { v, ok := hs.db.Load(c.key) if !ok { err = fmt.Errorf("Key not found") } else { value = []byte(v.(string)) } } else { var results []goraft.ApplyResult results, err = hs.raft.Apply([][]byte{encodeCommand(c)}) if err == nil { if len(results) != 1 { err = fmt.Errorf("Expected single response from Raft, got: %d.", len(results)) } else if results[0].Error != nil { err = results[0].Error } else { value = results[0].Result } } } if err != nil { log.Printf("Could not encode key-value in http response: %s", err) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) return } written := 0 for written < len(