What is a log

A log is just a immutable sequence of records wih strong ordering semantics that can be used to provide durability, replication and to model consensus. It is usually a 0 indexed file that new entries are appended to because expensive disk seeks can usually be avoided when appending to a file1.

Not to be confused with the type of logs most people are used to: application logs that are meant to be read by humans although application logs are a degenerative case of the log we are talking about2.

What i called a record is a entry in the log. The entry can be anything in any format.

You have seen a log before

Databases

Postgres uses a write-ahead log to ensure data is not lost if a crash happens3, to enable replication and change data capture. Tables and indexes are modified only after the change been written to the log in which case if a crash happens, the log can be used to go back to a valid state.

Datomic takes it to the next level by being a log-centric database4.

File systems

Some file systems known as journaling file systems5 write changes to a log before actually applying them to the internal file system structures to enable crash recovery and avoid data corruption.

Distributed systems

Distributed systems such as Kafka which considers a message as accepted by the cluster after the quorum of in-sync replicas(configuration dependent) have written the message to their log6

Consensus

Consensus algorithms such as Raft7 aka replicated state machines8.

  1. Operating systems: Three easy pieces - Hard Disk Drives ↩︎

  2. I Love logs ↩︎

  3. Postgres write-ahead log ↩︎

  4. Rich Hickey: Deconstructing the Database ↩︎

  5. Journaling file-system ↩︎

  6. Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale ↩︎

  7. In Search of an Understandable Consensus Algorithm ↩︎

  8. Replicated state machines ↩︎