Changes to walk_snapshot to match Forest and Lotus snapshots

[jdjaustin]

Summary of changes

Changes introduced in this pull request:

• Changes to walk_snapshot to match Lotus logic (i.e., don't include identity CIDs in snapshot export).

Reference issue to close (if applicable)

Closes#1884

Other information and links

Change checklist

• I have performed a self-review of my own code,
• I have made corresponding changes to the documentation,
• I have added tests that prove my fix is effective or that my feature works (if possible),
• I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

[jdjaustin]

Calibnet snapshots are matching up now but I'm so far unable to verify mainnet. @LesnyRumcajs

[LesnyRumcajs]

@jdjaustin, what's the issue with mainnet snapshots? Is it because of the hardware limitations? If so, have you tried installing Forest with an alternative, turbo fast, pioneer & bleeding edge backend with make install-with-jemalloc

[jdjaustin]

@jdjaustin, what's the issue with mainnet snapshots? Is it because of the hardware limitations? If so, have you tried installing Forest with an alternative, turbo fast, pioneer & bleeding edge backend with make install-with-jemalloc

Yes, I tried make install-with-jemalloc and I'm still getting No space left on device errors. I can clear out some disk space and try again.

[LesnyRumcajs]

Ah, so your issue is with disk space, not RAM. In this case, you need around 250G of free disk space.

1. clean up forest directory
2. download and import the snapshot - this would require around 250G at least - 110G snapshot + 130G DB, more or less.
3. Delete the snapshot to get back some disk space.
4. Export the snapshot - again, you will need around 250G plus some more because of the additional chain progression.

[lemmih]

@jdjaustin If you still don't have enough disk space after deleting the old files, ask Hubert to set up a droplet for you.

[jdjaustin]

Good news. Verified that the Lotus and Forest mainnet snapshots also match!

[LesnyRumcajs]

@jdjaustin does the shasum of both match?

[jdjaustin]

@jdjaustin does the shasum of both match?

Yepp!

[LesnyRumcajs]

Rock solid! Now it would be fantastic to keep it this way. One approach would be to modify the test a bit in our workflow https://github.com/ChainSafe/forest/blob/main/.github/workflows/rust.yml#L150-L172

1. Download the Lotus snapshot.
2. Import it into Forest
3. Export the snapshot at height specified in Lotus snapshot.
4. Compare the checksums.

We can do it in a separate issue if it's too complex; I may be missing some gotchas.

[LesnyRumcajs]

Linking the "source" PR for future reference if needed. filecoin-project/lotus#8691

[LesnyRumcajs]

@jdjaustin Perhaps we should match the logic in Lotus for the snapshot, just in case. I am not entirely sure we need it but better safe than sorry. So this:

// We only include raw and dagcbor, for now.// Raw for "code" CIDs.switchprefix.Codec {
casecid.Raw, cid.DagCBOR:
default:
continue
}

Should also get included.

[jdjaustin]

@jdjaustin Perhaps we should match the logic in Lotus for the snapshot, just in case. I am not entirely sure we need it but better safe than sorry. So this:

// We only include raw and dagcbor, for now.// Raw for "code" CIDs.switchprefix.Codec {
casecid.Raw, cid.DagCBOR:
default:
continue
}

Should also get included.

This block of code doesn't seem to be affecting snapshot exports currently, so my biggest question/concern with adding this is: how will I test the Rust code is working as intended?

[LesnyRumcajs]

@jdjaustin Perhaps we should match the logic in Lotus for the snapshot, just in case. I am not entirely sure we need it but better safe than sorry. So this:

// We only include raw and dagcbor, for now.// Raw for "code" CIDs.switchprefix.Codec {
casecid.Raw, cid.DagCBOR:
default:
continue
}

Should also get included.

This block of code doesn't seem to be affecting snapshot exports currently, so my biggest question/concern with adding this is: how will I test the Rust code is working as intended?

I believe just checking if the snapshots are still the same will be okay.

[jdjaustin]

I believe just checking if the snapshots are still the same will be okay.

Got it, I thought it was just going to be dead code, but I was just confused by the Go code; once I started writing the Rust code, I understood what was going on. And the snapshots still match: