sectionChubby [777f756c]

Part of paxos

Google locking system that predates Zookeeper
Used by GFS (Google File System) to appoint a manager server and stores small amounts of data
Used by BigTable to elect bigtable amanger, discover servers, enable clients to locate Bigtable manager, and low volume storage

1. Requirements

Provides 3 APIs: whole file reads and writes, advisory locks, and notification of events
Coarse grained locking service, you can lock with minimal overhead, and is a reliable low volume storage
Also needs to be available and reliable, and have decent throughput

Chubby cell is multiple serbers (usually 5), which replicate with each other
Each server has a namespace composed of directions and files, with ACLs
One replica is always elected as the primary, which initiates read and write operations
Replicas copy the database using a consensus protocol and elect new primaries as needed
Clients discover the primary by asking any server within the cell which is the primary, and caches it on the client side. It uses this as the primary until the server stops being the primary
Two types of requests
- Writes: propagated to all the replica servers, are async and only acked when a quorum responds
- Reads: serviced by the primary replica
Each file or directory within a chubby server is knownas a node (similar to a inodes)
- Each node has a unique name, there's emphemeral and permanent nodes
- Ephemeral nodes are similar to temp files, they get deleted when no client has them open
Uses ACLs in the directory that keeps track of authorized names

multiple clients can hold a lock in reader mode, only a single client can hold a writer lock
Locks are adivsory, which means you don't need a lock to read a file. Clients are required to cooperate for conflicts, but advisory locks are more scalable rather than mandatory locks, and clients can emulate strict locking easily (just have them explicitly check for locks)
If clients are holding a read lock, a write lock cannot be acquired

If you hold a lock and never release it, then we have issues
Sequence numbers are introduced after a lock is acquired
Locks that are not released are expired, and we use sequence numbers to note the current state (similar to a lamport clock)
If a lock becomes free after an expiration, the lock server does not let any other client claim it for a specific time period. This is used to create a backoff from faulty clients claiming a lock, then releasing, then claiming it again.
To allow systems to know what is happening, Chubby sends events: modifying file contents, modifying nodes, replica failover, handles, locks, conflicts, etc

Primaries keeps a list of data that clients are caching and sends invalidations to the clients to keep them consistent
If data or metadata needs to be changed, the primary blocks modifications and sends invalidations to clients with the relevant data cached
When a client receives an invalidation, it flushes the invalid state and acks it with a keepalive
- If there's no acks for invalidations, then the primary keeps the node uncachable

A client and a cell keep track of each other with sessions that are held with keepalives
A session comes with a lease, which is defined as a time period where the primary will tell the client with updates and will not terminate the connection unilaterally
The lease is used by the local client to know if something has gone wrong, if it's missing keepalives
- If a local lease times out, then the client
  - Empties and disables the cache
  - Waits for a grace period, and tries keepalives
  - If it connects it, then enable the cache
  - If nothing, then assume terminated

If a node cannot communicate with the primary after the lease ends, it starts and election
Periodically, the primaries keep a keepalive for the leadership
Each cell's primary takes snapshots and backs it up
Mirroring can also occur across different regions, and a mirror that cannot be accessed remains unaltered until communication is reestablished. Updated files are located by contrasting their checksums
Failure steps for replicas that do not recover after a few hours
- A replacement system is used to provision a new replica
- It initaties the lock server binary
- DNS tables are updated
- Current primary replica also has to have this info, which it polls for
- Cell DB is updated
- Replica servers update themselves with the new member
- New server recieves information
- Afterwards, it is permitted to vote in an election after it processes a write

Proxies act as LB's by allowing KeepAlives to be decreased to the main server
Partitioning allows us to shard chubby's namespace, where different namespaces set different replicas

Use a lock service for centrally managed things
Permit a huge number of clients to access a Chubby file without using many servers
Use event notification instead of polling because clients and replicas may want to know when things have changed
Cache the data on the client side
Use consistent caching because developers get confused by non-intuitive caching semantics
Redirecting all reads and writes via a single node was the way to provide strong consistentcy. Caches and proxies make up for the loss in R/W throughput

Chubby's aspects (lock system, small file storage, and session/lease management) are one thing, whereas boxwood had three different peices
Chubby has a more advanced interface than boxwood
Boxwood had a 200ms lease period, whereas Chubby's is 12
Chubby has a grace period to prevent losing locks

Coarse grained locks will need much less load on the lock server, they are rarely acquired
However, transferring locks from one client to another needs expensive recovery procedures
Fine grained locks are frequently accessed, which means availability would become critical
Time penalty for dropping locks would not be severe, since locks are only held for a short period