Module forest_filecoin::db::car::forest

source ·
Expand description

§Forest CAR format

See crate::db::car::plain for details on the CAR format.

The forest.car.zst format wraps multiple CAR blocks in small (usually 8 KiB) zstd frames, and has an index in a skippable zstd frame. At the end of the data, there has to be a fixed-size skippable frame containing magic numbers and meta information about the archive. CAR blocks may not span multiple z-frames and the CAR header is kept it a separate z-frame.

Imagine a forest.car.zst archive with 5 blocks. They could be arranged in z-frames as drawn below:

 Z-Frame 1   Z-Frame 2   Z-Frame 3   Skip Frame    Skip Frame
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌───────────┐ ┌────────────┐
│┌──────┐ │ │┌───────┐│ │┌───────┐│ │Offsets    │ │Index offset│
││Header│ │ ││Block 1││ ││Block 4││ │ Z-Frame 2 │ │Magic number│
│└──────┘ │ │└───────┘│ │└───────┘│ │ Z-Frame 2 │ │Version info|
└─────────┘ │┌───────┐│ │┌───────┐│ │ Z-Frame 2 │ └────────────┘
            ││Block 2││ ││Block 5││ │ Z-Frame 3 │
            │└───────┘│ │└───────┘│ │ Z-Frame 3 │
            │┌───────┐│ └─────────┘ └───────────┘
            ││Block 3││
            │└───────┘│
            └─────────┘

Looking up a block uses an index::Reader to find the right z-frame. The frame is then decoded and each block is linearly scanned until a match is found. Decoded (and scanned) z-frames are stored in a lru-cache for faster repeat retrievals.

forest.car.zst files are backward compatible with Lotus (and all other tools that consume compressed CAR files). All Forest-specifc information is encoded as skippable frames that are (as the name suggests) skipped by tools that don’t understand them.

§Additional reading

zstd frame format: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md

CARv1 specification: https://ipld.io/specs/transport/car/carv1/

Modules§

  • index 🔒
    Embedded index for the .forest.car.zst format.

Structs§

Constants§

Functions§