Module forest_filecoin::db::car::forest
source · Expand description
§Forest CAR format
See crate::db::car::plain
for details on the CAR format.
The forest.car.zst
format wraps multiple CAR blocks in small (usually 8 KiB)
zstd frames, and has an index in a skippable zstd frame. At the end of the
data, there has to be a fixed-size skippable frame containing magic numbers
and meta information about the archive. CAR blocks may not span multiple
z-frames and the CAR header is kept it a separate z-frame.
Imagine a forest.car.zst
archive with 5 blocks. They could be arranged in
z-frames as drawn below:
Z-Frame 1 Z-Frame 2 Z-Frame 3 Skip Frame Skip Frame
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌───────────┐ ┌────────────┐
│┌──────┐ │ │┌───────┐│ │┌───────┐│ │Offsets │ │Index offset│
││Header│ │ ││Block 1││ ││Block 4││ │ Z-Frame 2 │ │Magic number│
│└──────┘ │ │└───────┘│ │└───────┘│ │ Z-Frame 2 │ │Version info|
└─────────┘ │┌───────┐│ │┌───────┐│ │ Z-Frame 2 │ └────────────┘
││Block 2││ ││Block 5││ │ Z-Frame 3 │
│└───────┘│ │└───────┘│ │ Z-Frame 3 │
│┌───────┐│ └─────────┘ └───────────┘
││Block 3││
│└───────┘│
└─────────┘
Looking up a block uses an index::Reader
to find
the right z-frame. The frame is then decoded and each block is linearly
scanned until a match is found. Decoded (and scanned) z-frames are stored in
a lru-cache for faster repeat retrievals.
forest.car.zst
files are backward compatible with Lotus (and all other
tools that consume compressed CAR files). All Forest-specifc information is
encoded as skippable frames that are (as the name suggests) skipped by tools
that don’t understand them.
§Additional reading
zstd
frame format: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
CARv1 specification: https://ipld.io/specs/transport/car/carv1/
Modules§
- index 🔒Embedded index for the
.forest.car.zst
format.