Module forest_filecoin::db::car::plain

Varint frames

CARs are made of concatenations of varint frames. Each varint frame is a concatenation of the body length as an varint, and the frame body itself. unsigned_varint::codec::UviBytes can be used to read frames piecewise into memory.

       varint frame
│◄───────────────────────►│
│                         │
├───────────┬─────────────┤
│varint:    │             │
│body length│frame body   │
└───────────┼─────────────┤
            │             │
frame body ►│◄───────────►│
    offset     =body length

CARv1 layout and seeking

The first varint frame is a header frame, where the frame body is a CarHeader encoded using ipld_dagcbor.

Subsequent varint frames are block frames, where the frame body is a concatenation of a Cid and the block data addressed by that CID.

block frame ►│
body offset  │
             │  =body length
             │◄────────────►│
 ┌───────────┼───┬──────────┤
 │body length│cid│block data│
 └───────────┴───┼──────────┤
                 │◄────────►│
                 │  =block data length
     block data  │
         offset ►│

Block ordering

… a filecoin-deterministic car-file is currently implementation-defined as containing all DAG-forming blocks in first-seen order, as a result of a depth-first DAG traversal starting from a single root.

• CAR documentation

Future work

• fadvise-based APIs to pre-fetch parts of the file, to improve random access performance.
• Use an inner Blockstore for writes.
• Use safe arithmetic for all operations - a malicious frame shouldn’t cause a crash.
• Theoretically, file-backed blockstores should be clonable (or even Sync) with very low overhead, so that multiple threads could perform operations concurrently.
• CARv2 support
• A wrapper that abstracts over car formats for reading.

Structs

• CompressedBlockDataLocation
• CountRead 🔒
  A reader that keeps track of how many bytes it has read.
• PlainCar
  Note that all operations on this store are blocking.
• UncompressedBlockDataLocation
  If you seek to offset (from the start of the file), and read length bytes, you should get data that corresponds to a Cid (but NOT the Cid itself).

Functions

• cid_error_to_io_error 🔒
• get_roots_from_v1_header 🔒
• handle_write_cache 🔒
  Panics
• read_block_data_location_and_skip 🔒
  Returns (Cid, the block data offset and block data length)
• read_header 🔒
• read_varint_body_length_or_eof 🔒
  Reads body length, leaving the reader at the start of a varint frame, or returns [Ok(None)] if we’ve reached EOF
• write_skip_frame_header_async