MinerX FF Edition: Curio Storage - Testing Assignment Week 2

[Reiers]

Hello MinerX Fellows !

Week 2 is focused on getting hands-on experience with Curio Storage.
Everyone should either deploy or migrate an existing miner on calibration.
Follow the guide below. Looking forward to the feedback and answers. Thank you!

Requirement

• OS: Linux
• Yugabyte

Hardware	Specification
CPU	8-core processor
CPU Support	Models with support for Intel SHA Extensions (AMD since Zen microarchitecture or Intel since Ice Lake) will significantly speed up the processes.
RAM	256 GiB RAM + Swap
GPU	Nvidia GPU with at least 11GB VRAM
Disk	2 TB NVMe disk

🚀 This post if you are participating in this testing round.
If you aren't able to join for some reason, please reach out to me @Reiers here or on Slack!

Task assignment

1. Install Curio

git clone https://github.com/filecoin-project/lotus.git curio-calibration
cd curio-calibration
git checkout master
make clean && RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make calibnet
sudo make install

2. Install Boost

git clone https://github.com/filecoin-project/boost boost-calibration
cd boost-calibration
git checkout feat/curio
make clean build
sudo make install

3. Install Yugabyte (if you haven't already)

❗If you have already set up a YugabyteDB for Boost then you can reuse the same YugabyteDB instance for Curio. You can skip directly to migrating from Lotus-Miner to Curio or Initializing new Curio Miner.

• https://docs.yugabyte.com/

4. Start syncing Lotus

aria2c -x5 https://forest-archive.chainsafe.dev/latest/calibnet/
lotus daemon --import-snapshot <snapshot-file>

5. Create or use existing addresses

• Owner: lotus wallet new
• Worker: lotus wallet new bls

For tFIL and DC:

• https://faucet.calibnet.chainsafe-fil.io/

6. Migrating from lotus-miner to curio

• curio guided-setup
• curio test window-post task

7. (Optional) Deploy a new Curio Storage Miner

• curio guided-setup

Docs: https://lotus.filecoin.io/storage-providers/curio/setup/#migrating-from-lotus-miner-to-curio

Reply with:

• - 🦭 If you have successfully sealed some CC sectors.
• - 🤝 If you have successfully made some storage deals.
• - 📁 Made a f05/market Deal.
• - 📁 Made a FIL+ Market Deal.
• - 📁 Made a FIL+ DDO Deal.
• - 🗄️ Put everything above in one sector.
• - 📂 If you have successfully had some retrieval deals.
• - 💻 If you passed your first WindowPoSt.

❗ Create an issue if you think there is a bug, don't forget to attach the logs! 🐛

If you have any questions or need help, join:

• For dev questions : #fil-curio-dev
• For general help : #fil-curio-help
• Calibraion net help : #fil-net-calibration-discuss

[johansealstorage]

@Reiers , are there any hints or advise on deploying and configuring curio on ESPA type hardware? In other words, CPU servers with SHA (no gpu) and GPU servers with no SHA but with multiple GPU's.

[Reiers]

Great question!

You shouldn't have to do any tweaking with flags or add environment to your sealing pipeline - add the server to the curio cluster, let's say you have a "worker", point it at the DB and run curio run --layers <the ones you want>

CPU Servers with SHA (no GPU):

Focus on tasks that don't require GPU, such as sealing and post tasks.
Example command: curio run --layers=seal.

GPU Servers with Multiple GPUs (no SHA):

Use for GPU-intensive tasks like SDR trees and PoRep proof generation.
Example command: curio run --layers=seal-gpu,seal-snark.

https://lotus.filecoin.io/storage-providers/curio/config/#layer-stacking

You can control everything from the UI, things that used to be environment:
https://lotus.filecoin.io/storage-providers/curio/config/#pre-built-layers

[johansealstorage]

Still getting the hang of layers 😁

If EnableSealSDRTrees is true in the layer seal and in seal-gpu, would it be executed on CPU Server or GPU server - or any/both?
I guess the question is which curio node will grab that task?

[magik6k]

If you enable that task on both, the machines will have a ~50/50 chance of grabbing the task (assuming one machine with each layer)

(Once we land #11796 the machine with data locally will be much more likely to get the Trees task)

[magik6k]

Note that you likely don't want the SDRTrees task enabled on the CPU machines, that task is equal to PreCommit2 in lotus-miner (except we also do TreeD here instead of in PreCommit1), so it really benefits from GPUs.

[SBudo]

• 🦭 If you have successfully sealed some CC sectors.
• 🤝 If you have successfully made some storage deals.
• 📁 Made a f05/market Deal.
• 📁 Made a FIL+ Market Deal.
• 📁 Made a FIL+ DDO Deal.
• 🗄️ Put everything above in one sector.
• 📂 If you have successfully had some retrieval deals.
• 💻 If you passed your first WindowPoSt.

[SBudo]

It would be good to add the following features:

• Abort sealing jobs
• Update a sector state (to send it back in earlier stage of sealing process)
• Remove sectors (before they are on the chain)
• Add a filter on the PoRep pipeline page to filter out sectors that are proving, as to only view the sectors that are in the process of being sealed or in error

[stuberman]

• - 🦭 If you have successfully sealed some CC sectors.
• - 🤝 If you have successfully made some storage deals.
• - 📁 Made a f05/market Deal.
• - 📁 Made a FIL+ Market Deal.
• - 📁 Made a FIL+ DDO Deal.
• - 🗄️ Put everything above in one sector.
• - 📂 If you have successfully had some retrieval deals.
• - 💻 If you passed your first WindowPoSt.

[stuberman]

EDIT: I was fixed this by using default paths and not trying to set a custom path for data directories.

Having trouble integrating Boost, keep getting this error:

I run: boostd-data run yugabyte --hosts 192.168.1.96 --connect-string="postgresql://yugabyte:[email protected]:5433?sslmode=disable" --addr 0.0.0.0:8044

I get a response:

2024-06-01T23:15:45.783Z INFO boostd-data-yb yugabyte/setup.go:22 creating cassandra keyspace idx
2024-06-01T23:15:46.162Z INFO boostd-data-yb yugabyte/setup.go:38 creating cassandra tables
Error: starting yugabyte store: starting local index directory service: creating cassandra tables: executing
CREATE TABLE IF NOT EXISTS PayloadToPieces (
PayloadMultihash BLOB,
PieceCid BLOB,
-- PayloadMultihash is the partition key: the key that determines which
-- node the row is sent to.
-- PieceCid is the clustering key: the data is sorted by this key.
-- We want both of these to be in the primary key because
-- the piece cids should be a unique set.
PRIMARY KEY (PayloadMultihash, PieceCid)
)
code=2200; message=Invalid Table Definition. Error creating table idx.payloadtopieces on the master: Not enough tablet servers in the requested placements. Need at least 1, have 0
CREATE TABLE IF NOT EXISTS PayloadToPieces (
^^^^^^^^^^^^^^^
PayloadMultihash BLOB,
PieceCid BLOB,
-- PayloadMultihash is the partition key: the key that determines which
-- node the row is sent to.
-- PieceCid is the clustering key: the data is sorted by this key.
-- We want both of these to be in the primary key because
-- the piece cids should be a unique set.
PRIMARY KEY (PayloadMultihash, PieceCid)
)
(ql error -302)

git show

commit cd7c946232662ddc6ff8f5be13dc68ea743c7880 (HEAD -> feat/curio, origin/feat/curio)

[stuberman]

I am finally able to run Boost now.
Now I need to get some deals, any suggestions?

[stuberman]

As of June 5, I see these significant problems with the code:

• WinningPosts fail
• Inability to store Yugabyte data outside of /home directory (even though Yugabyte has flags for data paths)
• Incomplete documentation and/or features (how to create control addresses in Curio as example)

[stuberman]

With latest commits able to seal deals but commit message fails on Calibnet when multiple deals are in a sector of type verified and unverified due to gas error. I am told this is not an issue on mainnet.

[stuberman]

Next, planning to do a reinstall from scratch with new GA build and test.

[herrehesse]

• - 🦭 If you have successfully sealed some CC sectors.
• - 🤝 If you have successfully made some storage deals.
• - 📁 Made a f05/market Deal.
• - 📁 Made a FIL+ Market Deal.
• - 📁 Made a FIL+ DDO Deal.
• - 🗄️ Put everything above in one sector.
• - 📂 If you have successfully had some retrieval deals.
• - 💻 If you passed your first WindowPoSt.

[NDLABS-Leo]

• - 🦭 If you have successfully sealed some CC sectors.

  • - 🤝 If you have successfully made some storage deals.

  • - 📁 Made a f05/market Deal.

  • - 📁 Made a FIL+ Market Deal.

  • - 📁 Made a FIL+ DDO Deal.

  • - 🗄️ Put everything above in one sector.

  • - 📂 If you have successfully had some retrieval deals.

  • - 💻 If you passed your first WindowPoSt.

[lei7337]

• - 🦭 If you have successfully sealed some CC sectors.
• 🤝 If you have successfully made some storage deals.
• 📁 Made a f05/market Deal.
• 📁 Made a FIL+ Market Deal.
• 📁 Made a FIL+ DDO Deal.
• 🗄️ Put everything above in one sector.
• 📂 If you have successfully had some retrieval deals.
• - 💻 If you passed your first WindowPoSt.
  Miner ID t0120833