Jepsen Testing Results
Jepsen is a tool, written by Kyle Kingsbury, designed to test the partition tolerance of distributed systems. It creates network partitions while fuzzing the system with random operations. The results are analyzed to see if the system violates any of the consistency properties it claims to have.
As part of our Consul testing, we ran a Jepsen test to determine if any consistency issues could be uncovered. In our testing, Consul gracefully recovered from partitions without introducing any consistency issues.
Running the tests
At the moment, testing with Jepsen is rather complex as it requires setting up multiple virtual machines, SSH keys, DNS configuration, and a working Clojure environment. We hope to contribute our Consul testing code upstream and to provide a Vagrant environment for Jepsen testing soon.
Output
Below is the output captured from Jepsen. We ran Jepsen multiple times, and it passed each time. This output is only representative of a single run and has been edited for length. Please reach out on Consul's Discuss if you would like to reproduce the Jepsen results.
$ lein test :only jepsen.system.consul-testlein test jepsen.system.consul-testINFO jepsen.os.debian - :n5 setting up debianINFO jepsen.os.debian - :n3 setting up debianINFO jepsen.os.debian - :n4 setting up debianINFO jepsen.os.debian - :n1 setting up debianINFO jepsen.os.debian - :n2 setting up debianINFO jepsen.os.debian - :n4 debian set upINFO jepsen.os.debian - :n5 debian set upINFO jepsen.os.debian - :n3 debian set upINFO jepsen.os.debian - :n1 debian set upINFO jepsen.os.debian - :n2 debian set upINFO jepsen.system.consul - :n1 consul nukedINFO jepsen.system.consul - :n4 consul nukedINFO jepsen.system.consul - :n5 consul nukedINFO jepsen.system.consul - :n3 consul nukedINFO jepsen.system.consul - :n2 consul nukedINFO jepsen.system.consul - Running nodes: {:n1 false, :n2 false, :n3 false, :n4 false, :n5 false}INFO jepsen.system.consul - :n2 consul nukedINFO jepsen.system.consul - :n3 consul nukedINFO jepsen.system.consul - :n4 consul nukedINFO jepsen.system.consul - :n5 consul nukedINFO jepsen.system.consul - :n1 consul nukedINFO jepsen.system.consul - :n1 starting consulINFO jepsen.system.consul - :n2 starting consulINFO jepsen.system.consul - :n4 starting consulINFO jepsen.system.consul - :n5 starting consulINFO jepsen.system.consul - :n3 starting consulINFO jepsen.system.consul - :n3 consul readyINFO jepsen.system.consul - :n2 consul readyINFO jepsen.system.consul - Running nodes: {:n1 true, :n2 true, :n3 true, :n4 true, :n5 true}INFO jepsen.system.consul - :n5 consul readyINFO jepsen.system.consul - :n1 consul readyINFO jepsen.system.consul - :n4 consul readyINFO jepsen.core - Worker 0 startingINFO jepsen.core - Worker 2 startingINFO jepsen.core - Worker 1 startingINFO jepsen.core - Worker 3 startingINFO jepsen.core - Worker 4 startingINFO jepsen.util - 2 :invoke :read nilINFO jepsen.util - 3 :invoke :cas [4 4]INFO jepsen.util - 0 :invoke :write 4INFO jepsen.util - 1 :invoke :write 1INFO jepsen.util - 4 :invoke :cas [4 0]INFO jepsen.util - 2 :ok :read nilINFO jepsen.util - 4 :fail :cas [4 0](Log Truncated...)INFO jepsen.util - 4 :invoke :cas [3 3]INFO jepsen.util - 4 :fail :cas [3 3]INFO jepsen.util - :nemesis :info :stop nilINFO jepsen.util - :nemesis :info :stop "fully connected"INFO jepsen.util - 0 :fail :read nilINFO jepsen.util - 1 :fail :write 0INFO jepsen.util - :nemesis :info :stop nilINFO jepsen.util - :nemesis :info :stop "fully connected"INFO jepsen.core - nemesis doneINFO jepsen.core - Worker 3 doneINFO jepsen.util - 1 :invoke :read nilINFO jepsen.core - Worker 2 doneINFO jepsen.core - Worker 4 doneINFO jepsen.core - Worker 0 doneINFO jepsen.util - 1 :ok :read 3INFO jepsen.core - Worker 1 doneINFO jepsen.core - Run complete, writingINFO jepsen.core - Analyzing(Log Truncated...)INFO jepsen.core - Analysis completeINFO jepsen.system.consul - :n3 consul nukedINFO jepsen.system.consul - :n2 consul nukedINFO jepsen.system.consul - :n4 consul nukedINFO jepsen.system.consul - :n1 consul nukedINFO jepsen.system.consul - :n5 consul nuked1964 element history linearizable. :DRan 1 tests containing 1 assertions.0 failures, 0 errors.