Elgato_dark/datahaven
1
0
Fork 0
mirror of https://github.com/datahaven-xyz/datahaven synced 2026-05-24 09:50:01 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

chore: ♻️ Updated StorageHub nodes docs

Browse source
This commit is contained in:
Steve Degosserie 2025-12-01 12:28:46 +01:00
parent b6cfd5eae1
commit 03e78679e6
No known key found for this signature in database
GPG key ID: 9455A9F1AD80CE80
4 changed files with 229 additions and 67 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

82
docs/storagehub-bsp.md
View file

@ -17,10 +17,32 @@ Backup Storage Providers (BSPs) provide redundant storage for files in the Stora
- DataHaven node binary or Docker image
- Funded account with sufficient balance for deposits
- Storage capacity (minimum 2 data units, recommended 10+ GiB)
- Storage capacity (minimum 1 TB, recommended 2+ TB)
- Stable network connection
- Open network ports (30333, optionally 9944)
## Hardware Requirements
BSPs have similar hardware requirements to MSPs as they store backup data and must reliably submit proofs of storage.
### Specifications
| Component | Requirement |
|-----------|-------------|
| **CPU** | 8 physical cores @ 3.4 GHz (Intel Ice Lake+ or AMD Zen3+) |
| **RAM** | 32 GB DDR4 ECC |
| **Storage (System)** | 500 GB NVMe SSD (chain data) |
| **Storage (User Data)** | 1 TB NVMe SSD or HDD (minimum) |
| **Network** | 500 Mbit/s symmetric |
### Important Considerations
- **Separate storage volumes**: Keep chain data and user data on separate volumes for better I/O performance
- **Storage expansion**: Plan for growth; user data storage should be easily expandable
- **max-storage-capacity**: Set this CLI flag to **80% of available physical disk space** to leave headroom for filesystem overhead and temporary files
- **Cloud compatible**: BSPs can run effectively on cloud VPS with dedicated storage volumes
- **Proof submission**: Ensure reliable network connectivity for timely proof submissions
## Key Requirements
### BCSV Key (ECDSA - 1 Required)
@ -67,13 +89,22 @@ The entrypoint script automatically injects the BCSV key.
- **Purpose**: BSP registration, transaction fees, and deposits
- **Required Balance**:
- Minimum deposit: 100 HAVE (SpMinDeposit)
- Deposit per data unit: 2 HAVE per unit
- Base deposit: 100 HAVE (`SpMinDeposit`)
- Deposit per GiB: 2 HAVE (`DepositPerData`)
- Transaction fees: ~10 HAVE
- **Recommended**: 200+ HAVE for initial setup
- **Funding**: Must be funded **before** BSP registration
- **Account Type**: Ethereum-style 20-byte address (AccountId20)
**Deposit Calculation by Capacity:**
| Storage Capacity | Deposit Required | Recommended Balance |
|------------------|------------------|---------------------|
| 800 GiB (1 TB disk) | ~1,700 HAVE | 1,800+ HAVE |
| 1.6 TiB (2 TB disk) | ~3,400 HAVE | 3,600+ HAVE |
| 4 TiB (5 TB disk) | ~8,300 HAVE | 8,500+ HAVE |
Formula: `100 + (capacity_in_gib × 2) + buffer`
### Generate Provider Account
```bash
@ -110,8 +141,11 @@ datahaven-node \
| `--storage-path <PATH>` | Storage path (required if rocksdb) | No | None |
**Example Values:**
- `--max-storage-capacity 10737418240` (10 GiB)
- `--jump-capacity 1073741824` (1 GiB)
- `--max-storage-capacity 858993459200` (800 GiB = 80% of 1 TB disk)
- `--max-storage-capacity 1717986918400` (1.6 TiB = 80% of 2 TB disk)
- `--jump-capacity 107374182400` (100 GiB)
**Note**: Set `--max-storage-capacity` to approximately **80% of your available physical disk space** to leave headroom for filesystem overhead and temporary files.
### BSP-Specific Task Flags
@ -174,9 +208,11 @@ datahaven-node key insert \
```bash
# Transfer funds to BSP account
# Minimum: 200 HAVE (100 deposit + 100 for operations)
# For 800 GiB capacity: ~1,800 HAVE (1,700 deposit + 100 buffer)
# For 1.6 TiB capacity: ~3,600 HAVE (3,400 deposit + 200 buffer)
# Using Polkadot.js or a funded account, send HAVE tokens to $BSP_ACCOUNT
# Formula: 100 + (capacity_in_gib × 2) + buffer
```
### 3. Start BSP Node
@ -188,8 +224,8 @@ datahaven-node \
--base-path /data/bsp \
--provider \
--provider-type bsp \
--max-storage-capacity 10737418240 \
--jump-capacity 1073741824 \
--max-storage-capacity 858993459200 \
--jump-capacity 107374182400 \
--storage-layer rocksdb \
--storage-path /data/bsp/storage \
--bsp-upload-file-task \
@ -235,8 +271,8 @@ services:
- "--keystore-path=/data/keystore"
- "--provider"
- "--provider-type=bsp"
- "--max-storage-capacity=10737418240"
- "--jump-capacity=1073741824"
- "--max-storage-capacity=858993459200"
- "--jump-capacity=107374182400"
- "--storage-layer=rocksdb"
- "--storage-path=/data/storage"
- "--bsp-upload-file-task"
@ -304,8 +340,8 @@ spec:
- "--chain=stagenet-local"
- "--provider"
- "--provider-type=bsp"
- "--max-storage-capacity=10737418240"
- "--jump-capacity=1073741824"
- "--max-storage-capacity=858993459200"
- "--jump-capacity=107374182400"
- "--storage-layer=rocksdb"
- "--storage-path=/data/storage"
- "--bsp-upload-file-task"
@ -319,14 +355,14 @@ spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 100Gi
storage: 500Gi
- metadata:
name: storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 500Gi
storage: 1000Gi
```
## On-Chain Registration
@ -359,7 +395,7 @@ const typedApi = client.getTypedApi(datahaven);
const bspSigner = /* your polkadot-api signer */;
// BSP configuration
const capacity = BigInt(10_737_418_240); // 10 GiB in bytes
const capacity = BigInt(858_993_459_200); // 800 GiB (80% of 1 TB disk)
const multiaddresses = [
'/ip4/127.0.0.1/tcp/30333',
'/dns/bsp01.example.com/tcp/30333'
@ -467,7 +503,7 @@ const sudoSigner = /* sudo account signer */;
const bspCall = typedApi.tx.Providers.force_bsp_sign_up({
who: bspAccount,
bsp_id: /* pre-generated provider ID */,
capacity: BigInt(10_737_418_240),
capacity: BigInt(858_993_459_200), // 800 GiB
multiaddresses: multiaddresses,
payment_account: bspAccount,
weight: undefined // Optional weight parameter
@ -481,17 +517,21 @@ await sudoTx.signAndSubmit(sudoSigner);
| Parameter | Type | Description | Example |
|-----------|------|-------------|---------|
| `capacity` | StorageDataUnit | Storage capacity in bytes | `10737418240` (10 GiB) |
| `capacity` | StorageDataUnit | Storage capacity in bytes | `858993459200` (800 GiB) |
| `multiaddresses` | Vec<Bytes> | P2P network addresses | `[Binary.fromText("/ip4/...")]` |
| `payment_account` | AccountId | Account receiving payments | `0x...` (20-byte) |
### Deposit Requirements
- **Base Deposit**: 100 HAVE (`SpMinDeposit`)
- **Per Data Unit**: 2 HAVE per unit (`DepositPerData`)
- **Total for 10 GiB**: ~100 HAVE + (10 GiB in units × 2 HAVE)
- **Per GiB**: 2 HAVE (`DepositPerData`)
- **Formula**: `100 + (capacity_in_gib × 2)`
The deposit is **held (reserved)** from your account when you call `request_bsp_sign_up` and remains held while you operate as a BSP.
**Examples:**
- 800 GiB capacity: `100 + (800 × 2) = 1,700 HAVE`
- 1.6 TiB capacity: `100 + (1,638 × 2) = 3,376 HAVE`
The deposit is **held (reserved)** from your account when you call `request_bsp_sign_up` and remains held while you operate as a BSP. The deposit is returned when you deregister as a BSP.
## Monitoring

41
docs/storagehub-fisherman.md
View file

@ -18,10 +18,31 @@ Fisherman nodes monitor and validate storage provider behavior, detecting violat
- DataHaven node binary or Docker image
- Funded account with sufficient balance for challenges
- PostgreSQL 14+ database (can share with Indexer)
- Sufficient storage for database (100+ GB recommended)
- Sufficient storage for chain data
- Stable network connection
- Open network ports (30333, optionally 9944)
## Hardware Requirements
Fisherman nodes have moderate hardware requirements. They rely on a PostgreSQL database (typically shared with an Indexer node) to monitor provider behavior.
### Specifications
| Component | Requirement |
|-----------|-------------|
| **CPU** | 4 physical cores @ 2.5 GHz |
| **RAM** | 8 GB DDR4 |
| **Storage (Chain Data)** | 200 GB NVMe SSD |
| **Storage (Database)** | Shared with Indexer |
| **Network** | 100 Mbit/s symmetric |
### Important Considerations
- **Database dependency**: Fisherman requires a running Indexer node in `fishing` or `full` mode
- **Shared database**: Can share PostgreSQL with Indexer to reduce resource overhead
- **Network reliability**: Stable connection required for timely challenge submissions
- **Cloud compatible**: Works well on cloud VPS
## Key Requirements
### BCSV Key (ECDSA - 1 Required)
@ -365,11 +386,11 @@ spec:
mountPath: /data
resources:
requests:
memory: "4Gi"
cpu: "2"
limits:
memory: "8Gi"
cpu: "4"
limits:
memory: "16Gi"
cpu: "8"
args:
- "--chain=stagenet-local"
- "--name=Fisherman-Gustavo"
@ -386,7 +407,7 @@ spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 100Gi
storage: 200Gi
```
## On-Chain Registration
@ -540,16 +561,6 @@ docker logs storagehub-fisherman 2>&1 | grep -i "challenge.*success"
## Performance Considerations
### Resource Requirements
| Component | Minimum | Recommended |
|-----------|---------|-------------|
| CPU | 2 cores | 4 cores |
| RAM | 4 GB | 8 GB |
| Storage (Chain Data) | 100 GB | 200 GB |
| Storage (Database) | Shared with Indexer | Shared with Indexer |
| Network | 100 Mbps | 1 Gbps |
### Tuning Parameters
```bash

50
docs/storagehub-indexer.md
View file

@ -16,10 +16,42 @@ Indexer nodes index blockchain data into a PostgreSQL database, enabling efficie
- DataHaven node binary or Docker image
- PostgreSQL 14+ database server
- Sufficient storage for database (100+ GB recommended)
- Sufficient storage for chain data and database
- Stable network connection
- Open network ports (30333, optionally 9944)
## Hardware Requirements
Indexer nodes have varying requirements depending on the indexing mode. Full mode requires more resources for complete historical data indexing.
### Lite/Fishing Mode Specifications
| Component | Requirement |
|-----------|-------------|
| **CPU** | 4 physical cores @ 2.5 GHz |
| **RAM** | 16 GB DDR4 |
| **Storage (Chain Data)** | 100 GB NVMe SSD |
| **Storage (Database)** | 100 GB NVMe SSD |
| **Network** | 100 Mbit/s symmetric |
### Full Mode Specifications (Recommended)
| Component | Requirement |
|-----------|-------------|
| **CPU** | 8 physical cores @ 3.0 GHz (Intel Ice Lake+ or AMD Zen3+) |
| **RAM** | 32 GB DDR4 |
| **Storage (Chain Data)** | 300 GB NVMe SSD |
| **Storage (Database)** | 500 GB NVMe SSD |
| **Network** | 500 Mbit/s symmetric |
### Important Considerations
- **Archive mode**: Full indexers should run with `--pruning archive` for complete historical data
- **Database performance**: Use NVMe SSD for PostgreSQL data directory
- **Separate volumes**: Keep chain data and database on separate volumes for better I/O
- **Database growth**: Plan for database growth; full mode can grow significantly over time
- **Cloud compatible**: Indexer nodes work well on cloud VPS with dedicated storage
## Key Requirements
### No Session Keys Required
@ -372,10 +404,10 @@ spec:
mountPath: /data
resources:
requests:
memory: "8Gi"
memory: "16Gi"
cpu: "4"
limits:
memory: "16Gi"
memory: "32Gi"
cpu: "8"
args:
- "--chain=stagenet-local"
@ -394,9 +426,11 @@ spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 300Gi
storage: 500Gi
```
**Note**: Database storage (200Gi in PostgreSQL StatefulSet) should be increased to 500Gi for full mode in production.
## On-Chain Registration
### Not Required
@ -563,14 +597,6 @@ datahaven-node \
--max-runtime-instances 8
```
### Resource Requirements
| Mode | CPU | RAM | Storage (Chain) | Storage (DB) | Network |
|------|-----|-----|-----------------|--------------|---------|
| Full | 4-8 cores | 16-32 GB | 200 GB | 200-500 GB | 100 Mbps |
| Lite | 2-4 cores | 8-16 GB | 100 GB | 50-100 GB | 100 Mbps |
| Fishing | 2-4 cores | 8-16 GB | 100 GB | 50-100 GB | 100 Mbps |
## Security Considerations
1. **Database Security**: Use strong passwords, restrict network access

123
docs/storagehub-msp.md
View file

@ -17,11 +17,43 @@ Main Storage Providers (MSPs) are primary storage providers in the StorageHub ne
- DataHaven node binary or Docker image
- Funded account with sufficient balance for deposits
- Storage capacity (minimum 2 data units, recommended 10+ GiB)
- Storage capacity (minimum 1 TB, recommended 2+ TB)
- Stable network connection
- Open network ports (30333, optionally 9944)
- Optional: PostgreSQL database for advanced features
## Hardware Requirements
MSPs have validator-level hardware requirements plus additional storage capacity for user data. Single-threaded CPU performance is important for block processing.
### Minimum Specifications
| Component | Requirement |
|-----------|-------------|
| **CPU** | 8 physical cores @ 3.4 GHz (Intel Ice Lake+ or AMD Zen3+) |
| **RAM** | 32 GB DDR4 ECC |
| **Storage (System)** | 500 GB NVMe SSD (chain data) |
| **Storage (User Data)** | 1 TB NVMe SSD or HDD |
| **Network** | 500 Mbit/s symmetric |
### Recommended Specifications
| Component | Requirement |
|-----------|-------------|
| **CPU** | Intel Xeon E-2386/E-2388 or AMD Ryzen 9 5950x/5900x |
| **RAM** | 64 GB DDR4 ECC |
| **Storage (System)** | 1 TB NVMe SSD (chain data) |
| **Storage (User Data)** | 2+ TB NVMe SSD (expandable) |
| **Network** | 1 Gbit/s symmetric |
### Important Considerations
- **Disable Hyper-Threading/SMT**: Single-threaded performance is prioritized over core count
- **Separate storage volumes**: Keep chain data and user data on separate volumes for better I/O performance
- **Storage expansion**: Plan for growth; user data storage should be easily expandable
- **max-storage-capacity**: Set this CLI flag to **80% of available physical disk space** to leave headroom for filesystem overhead and temporary files
- **Bare metal preferred**: Cloud VPS may have inconsistent performance; bare metal provides better I/O predictability
## Key Requirements
### BCSV Key (ECDSA - 1 Required)
@ -68,13 +100,22 @@ The entrypoint script automatically injects the BCSV key.
- **Purpose**: MSP registration, transaction fees, and deposits
- **Required Balance**:
- Minimum deposit: 100 HAVE (SpMinDeposit)
- Deposit per data unit: 2 HAVE per unit
- Base deposit: 100 HAVE (`SpMinDeposit`)
- Deposit per GiB: 2 HAVE (`DepositPerData`)
- Transaction fees: ~10 HAVE
- **Recommended**: 200+ HAVE for initial setup
- **Funding**: Must be funded **before** MSP registration
- **Account Type**: Ethereum-style 20-byte address (AccountId20)
**Deposit Calculation by Capacity:**
| Storage Capacity | Deposit Required | Recommended Balance |
|------------------|------------------|---------------------|
| 800 GiB (1 TB disk) | ~1,700 HAVE | 1,800+ HAVE |
| 1.6 TiB (2 TB disk) | ~3,400 HAVE | 3,600+ HAVE |
| 4 TiB (5 TB disk) | ~8,300 HAVE | 8,500+ HAVE |
Formula: `100 + (capacity_in_gib × 2) + buffer`
### Generate Provider Account
```bash
@ -113,10 +154,13 @@ datahaven-node \
| `--storage-path <PATH>` | Storage path (required if rocksdb) | No | None |
**Example Values:**
- `--max-storage-capacity 10737418240` (10 GiB)
- `--jump-capacity 1073741824` (1 GiB)
- `--max-storage-capacity 858993459200` (800 GiB = 80% of 1 TB disk)
- `--max-storage-capacity 1717986918400` (1.6 TiB = 80% of 2 TB disk)
- `--jump-capacity 107374182400` (100 GiB)
- `--msp-charging-period 100` (100 blocks)
**Note**: Set `--max-storage-capacity` to approximately **80% of your available physical disk space** to leave headroom for filesystem overhead and temporary files.
### MSP-Specific Task Flags
| Flag | Description | Default |
@ -175,9 +219,11 @@ datahaven-node key insert \
```bash
# Transfer funds to MSP account
# Minimum: 200 HAVE (100 deposit + 100 for operations)
# For 800 GiB capacity: ~1,800 HAVE (1,700 deposit + 100 buffer)
# For 1.6 TiB capacity: ~3,600 HAVE (3,400 deposit + 200 buffer)
# Using Polkadot.js or a funded account, send HAVE tokens to $MSP_ACCOUNT
# Formula: 100 + (capacity_in_gib × 2) + buffer
```
### 3. Start MSP Node
@ -360,19 +406,23 @@ const typedApi = client.getTypedApi(datahaven);
const mspSigner = /* your polkadot-api signer */;
// MSP configuration
const capacity = BigInt(10_737_418_240); // 10 GiB in bytes
const capacity = BigInt(858_993_459_200); // 800 GiB (80% of 1 TB disk)
const multiaddresses = [
'/ip4/127.0.0.1/tcp/30333',
'/dns/msp01.example.com/tcp/30333'
].map(addr => Binary.fromText(addr));
// Pricing: $0.20 / 50 GiB / month at HAVE = $0.01
// See "Calculating Storage Pricing" section for formula
const pricePerGibPerBlock = BigInt(926_000_000_000);
// Step 1: Request MSP sign up
const requestTx = typedApi.tx.Providers.request_msp_sign_up({
capacity: capacity,
multiaddresses: multiaddresses,
value_prop_price_per_giga_unit_of_data_per_block: BigInt(18_520_000_000),
value_prop_price_per_giga_unit_of_data_per_block: pricePerGibPerBlock,
commitment: Binary.fromText('msp01'),
value_prop_max_data_limit: BigInt(1_073_741_824),
value_prop_max_data_limit: BigInt(53_687_091_200), // 50 GiB
payment_account: mspSigner.publicKey // Account receiving payments
});
@ -469,11 +519,11 @@ const sudoSigner = /* sudo account signer */;
const mspCall = typedApi.tx.Providers.force_msp_sign_up({
who: mspAccount,
msp_id: /* pre-generated provider ID */,
capacity: BigInt(10_737_418_240),
value_prop_price_per_giga_unit_of_data_per_block: BigInt(18_520_000_000),
capacity: BigInt(858_993_459_200), // 800 GiB
value_prop_price_per_giga_unit_of_data_per_block: BigInt(926_000_000_000),
multiaddresses: multiaddresses,
commitment: Binary.fromText('msp01'),
value_prop_max_data_limit: BigInt(1_073_741_824),
value_prop_max_data_limit: BigInt(53_687_091_200), // 50 GiB
payment_account: mspAccount
});
@ -485,20 +535,55 @@ await sudoTx.signAndSubmit(sudoSigner);
| Parameter | Type | Description | Example |
|-----------|------|-------------|---------|
| `capacity` | StorageDataUnit | Storage capacity in bytes | `10737418240` (10 GiB) |
| `capacity` | StorageDataUnit | Storage capacity in bytes | `858993459200` (800 GiB) |
| `multiaddresses` | Vec<Bytes> | P2P network addresses | `[Binary.fromText("/ip4/...")]` |
| `value_prop_price_per_giga_unit_of_data_per_block` | Balance | Price per GiB per block | `18520000000` |
| `value_prop_price_per_giga_unit_of_data_per_block` | Balance | Price per GiB per block (18 decimals) | `926_000_000_000` |
| `commitment` | Bytes | Service commitment identifier | `Binary.fromText("msp01")` |
| `value_prop_max_data_limit` | StorageDataUnit | Max data per value prop | `1073741824` (1 GiB) |
| `value_prop_max_data_limit` | StorageDataUnit | Max data per value prop | `53687091200` (50 GiB) |
| `payment_account` | AccountId | Account receiving payments | `0x...` (20-byte) |
### Calculating Storage Pricing
The `value_prop_price_per_giga_unit_of_data_per_block` parameter sets your price per GiB of data stored per block. This value is in HAVE with 18 decimals.
**Formula:**
```
price_per_gib_per_block = (target_monthly_price / storage_gb / blocks_per_month) / have_price × 10^18
```
**Example Calculation:**
Given:
- HAVE token price: **$0.01**
- Target monthly revenue: **$0.20 per 50 GiB per month**
- Block time: 6 seconds → **432,000 blocks per month** (30 days)
Step-by-step:
1. Price per GiB per month: `$0.20 / 50 GiB = $0.004 per GiB/month`
2. Price per GiB per block: `$0.004 / 432,000 = $9.26 × 10⁻⁹ per GiB/block`
3. Convert to HAVE: `$9.26 × 10⁻⁹ / $0.01 = 9.26 × 10⁻⁷ HAVE`
4. Apply 18 decimals: `9.26 × 10⁻⁷ × 10¹⁸ = 926,000,000,000`
**Result:** `value_prop_price_per_giga_unit_of_data_per_block: BigInt(926_000_000_000)`
| Target Price | HAVE @ $0.01 | Value (18 decimals) |
|-------------|--------------|---------------------|
| $0.10 / 50 GiB / month | 0.463 µHAVE/GiB/block | `463_000_000_000` |
| $0.20 / 50 GiB / month | 0.926 µHAVE/GiB/block | `926_000_000_000` |
| $0.50 / 50 GiB / month | 2.315 µHAVE/GiB/block | `2_315_000_000_000` |
| $1.00 / 50 GiB / month | 4.630 µHAVE/GiB/block | `4_630_000_000_000` |
### Deposit Requirements
- **Base Deposit**: 100 HAVE (`SpMinDeposit`)
- **Per Data Unit**: 2 HAVE per unit (`DepositPerData`)
- **Total for 10 GiB**: ~100 HAVE + (10 GiB in units × 2 HAVE)
- **Per GiB**: 2 HAVE (`DepositPerData`)
- **Formula**: `100 + (capacity_in_gib × 2)`
The deposit is **held (reserved)** from your account when you call `request_msp_sign_up` and remains held while you operate as an MSP.
**Examples:**
- 800 GiB capacity: `100 + (800 × 2) = 1,700 HAVE`
- 1.6 TiB capacity: `100 + (1,638 × 2) = 3,376 HAVE`
The deposit is **held (reserved)** from your account when you call `request_msp_sign_up` and remains held while you operate as an MSP. The deposit is returned when you deregister as an MSP.
## Monitoring