Keyboard shortcuts

/ or ⌘/Ctrl K
Find a note
j / k
Next / previous section or linked note
h / l
Collapse or go to parent / expand or enter
e or Alt-click
Read a linked note here
o
Open focused note on its own
g g / G
First / last section or linked note
g h / g a
Home / all notes
g b / g t
Backlinks / table of contents
t
Cycle System, Light, Dark
? / Esc
Show / close this reference

Search: ↑/↓ or Ctrl N/P, Enter to open. Shortcuts pause while typing.

sectionTectonic [cf37f28d]

Part of storage
  • Facebook version of GFS, uses ZippyDB to handle async metadata management and block management
  • Only allows one single writer at a time via locking
  • Each block is 80MB
  • Organizes the shards in zippy so that all the ones with the same sharding ID go on the same replica, which uses the directory_id
  • Remote chunk store that actually stores the physical bytes
  • Has a cache note called "sealing", which restraints updates, allowing caches to be stored longer
  • Has a 2pc for rebalancing when nodes need to go down for maintanence
  • Divides traffic groups into 3, depending on priority
  • Uses the Leaky Bucket Algorithm for traffic control
  • Fat client library, client lib actually does RS encoding

1. Multitenancy

  • Tectonic has quota limits for every user
  • Also allows applications to mark itself in the traffic groups, similar to nice values in linux

  • Tectonic distinguishes between emphermal and non-emphemeral resources for traffic sharding

    • Emphemeral is IOPS and metadata query capacity, stuff that changes in real time
    • Non-emphemeral is non-sharable resources that are once allocated to a tenant and will not be allocated to another, like storage
  • TrafficGroups is an emphemeral sharing group, which is a collection of clients that have the same latency and IOPS requirements
  • There's 50 TrafficGroups per cluster, although this is configurable.

  • Each tenant allocates the require emphermal resources based on TraffiCgroup and TrafficClasses (gold, silver, bronze)

    • unused emphemeral resources are shared with the traffic group within the tenant of a high traffic class
  • Client is actually responsible for this, the client library will check for spare capacity that is available within the same traffic group, then different traffic group of the same tenant, and finally spare capacity in the different tenants with the same TrafficClass priority

  • Optimizations used by storage nodes

    • Avoiding local hotspots is done with a weight round robin
    • use a greedy optimiation for allowing low-latency requests to give up their turn for the high traffic class if the request can be completed after the completion of the high traffic class
    • limits are set for all non-gold requests
    • we have enough disks to rearrange the IO requests
  • Authentication is done via a token based system

2. Tenant Specific Optimizations

  • Data warehouse: write once, read many

    • RS (reed-solomon encoding) encoded async writes

      • RS encoding happens out of sync after the stuff is written

    • Hedged quorum writes - generating reservation requests

      • Improves tail latency
      • Rather than sending the chunk to write further on the storage nodes, it sends only reservation requests, which then tell the nodes to accept the request of the reservation
      • So if you have 5 nodes, you send out to 5, and just write to the 3 that respond first
      • It's like RS but for transactions

  • Blob storage (low latency, write many, read many, typically smaller than block size)

    • Consistent append on partial blocks

      • Whenever a small blob comes in, since they're usually smaller than a block, we can just append to the block and perform replication on the partial block only
      • To prevent partial block issues, there's a lock on the writer

    • Re-encoding blocks

      • RS encoding on full blocks is IO efficient, but partial block is inefficient. Save the RS encoding until a full block has been written, and seal the block from further mutation, similar to SS tables