datahaven/test
Steve Degosserie 67f375860b
feat: Performance-Based Validator Rewards and Inflation Scaling (#306)
## Summary

Building on #304, this PR implements two complementary mechanisms to
improve validator incentives and network performance:

1. **Performance-Based Validator Rewards** (session-level)
2. **Inflation Scaling** (era-level)

## Reward Model Comparison

### Old Model (main branch) vs New Model

| Metric | Old Model (20 pts/block) | New Model (320 pts/block pool) |
|--------|--------------------------|--------------------------------|
| **Per Block** | Author: 20 pts, Others: 0 | Author: ~196 pts, Others:
~4 pts each |
| **Formula** | Direct author reward | 60% authoring + 30% liveness +
10% base |
| **Per Session** (600 blocks, 32 validators) | 12,000 total pts |
192,000 total pts |
| **Per Validator/Session** (uniform) | ~375 pts | ~6,000 pts |
| **Per Validator/Era** (6 sessions) | ~2,250 pts | ~36,000 pts |
| **Offline Validator** | 0 pts | ~600 pts/session (base only) |
| **Over-performer (150% blocks)** | 150% of fair share | Up to 130%
reward (soft cap) |

### Key Differences
- **Pool-based**: New model adds 320 points to a shared pool per block,
distributed via formula
- **Liveness rewarded**: 30% of rewards go to validators who are online
(heartbeat OR block authorship)
- **Base guarantee**: 10% ensures all active validators receive minimum
rewards
- **Soft cap**: Prevents extreme over-performance rewards (max 150% of
fair share credited)

## Performance-Based Validator Rewards

Introduces a **60/30/10 reward formula** that rewards validators based
on their contribution during each session:

- **60%** based on block production (with soft cap allowing up to 150%
of fair share)
- **30%** based on liveness (ImOnline heartbeat OR block authorship)
- **10%** guaranteed base reward for all active validators

### Key Features
- Tracks individual validator block authorship per session
- Calculates fair share dynamically: `fair_share = total_blocks /
total_validator_count`
- Fair share uses **total** validator count (including whitelisted)
since all validators occupy block slots
- **Soft cap**: Over-performers can earn credit up to 150% of their fair
share (configurable via `OperatorRewardsFairShareCap` at 50%)
- With 60% BlockAuthoringWeight, this gives over-performers up to **30%
bonus reward**
- **BasePointsPerBlock**: Defines points added to pool per block
produced (default: 320)
- Integrates with SessionManager for automatic point awards at session
end
- Excludes whitelisted validators from rewards (but includes them in
fair share calculation)
- Slashing check disabled but hook retained for future use
- Points accumulate across sessions within an era

### Dynamic Parameters (Governance-Adjustable)
- `OperatorRewardsBlockAuthoringWeight`: Weight for block authoring
(default: 60%)
- `OperatorRewardsLivenessWeight`: Weight for liveness (default: 30%)
- `OperatorRewardsFairShareCap`: Soft cap percentage above fair share
(default: 50%)

## Inflation Scaling

Implements **dynamic inflation scaling** that adjusts total inflation
based on network block production:

- **Minimum**: 20% of base inflation (network halt protection)
- **Maximum**: 100% of base inflation (caps at expected blocks)
- **Linear scaling** between minimum and maximum based on performance

### Scaling Examples
- 0% blocks produced → 20% inflation (safety floor)
- 50% blocks produced → 60% inflation  
- 100% blocks produced → 100% inflation
- >100% blocks produced → capped at 100%

### Configuration
- **ExpectedBlocksPerEra**: Computed as `SessionsPerEra ×
EpochDurationInBlocks`
- **MinInflationPercent**: 20%
- **MaxInflationPercent**: 100%

## Combined Effect

These mechanisms work together to create a comprehensive incentive
structure:

1. **Session rewards** encourage individual validator performance and
uptime
2. **Era inflation scaling** incentivizes collective network health
3. **Minimum inflation floor** protects against network halt
4. **Soft cap** allows over-performers to earn up to 30% bonus while
preventing extreme centralization

## Implementation Details

### Pallet Changes
- Add `BlocksAuthoredInSession` storage for per-validator tracking
- Add `BlocksProducedInEra` storage for total network tracking (cleaned
up with HistoryDepth)
- Add `note_block_author()` function called on block production
- Add `award_session_performance_points()` function with configurable
60/30/10 formula
- Add `calculate_scaled_inflation()` function for era-level scaling
- Update `on_era_end()` to use scaled inflation
- Integrate with SessionManager via wrapper types
- Defensive weight validation: proportionally scales if sum > 100%

### Configuration Parameters
- `ValidatorSet`: Provides active validator list
- `LivenessCheck`: Uses `ImOnline::is_online()` (heartbeat OR block
authorship)
- `SlashingCheck`: Integration with slashing pallet (currently disabled)
- `BasePointsPerBlock`: Points added to pool per block (default: 320)
- `BlockAuthoringWeight`: Dynamic parameter (60%)
- `LivenessWeight`: Dynamic parameter (30%)
- `FairShareCap`: Dynamic parameter (50%)
- `ExpectedBlocksPerEra`: Computed from session/epoch config
- `MinInflationPercent`: 20%
- `MaxInflationPercent`: 100%

### Runtime Updates
- Full configuration added to mainnet, testnet, and stagenet runtimes
- Dynamic parameters added to `runtime_params.rs` for governance control
- Uses `prod_or_fast!()` macro for environment-specific parameters
- `ValidatorIsOnline` uses `ImOnline::is_online()` for accurate liveness
detection

## Testing

- **76 tests passing** 
- Comprehensive coverage of both mechanisms

### Test Coverage
- Inflation scaling at 0%, 25%, 50%, 75%, 100%, >100% blocks
- Session performance with 60/30/10 formula
- Fair share calculations with soft cap (150%)
- Whitelisted validator exclusion from rewards (with correct fair share
using total count)
- Total points verification (sum of individual = total)
- Whitelisted over-producer scenarios
- Overflow protection (large block counts, near-u32::MAX)
- End-to-end session to era flow
- MockLivenessCheck mirrors ImOnline behavior (block authorship =
online)
- Multiple eras with different performance levels
- Edge cases (zero participation, single validator, large numbers)
- BlocksProducedInEra cleanup on era start

## ⚠️ Breaking Changes ⚠️

### Reward Distribution

Previously, rewards were distributed equally among all validators
regardless of their contribution. Now:

- **Performance-based**: Validators earn rewards proportional to their
block production (60%), liveness (30%), and a guaranteed base (10%)
- **Pool-based**: `BasePointsPerBlock` defines points added to pool per
block (320), distributed via formula
- **Fair share uses total validators**: Ensures non-whitelisted aren't
penalized for whitelisted validators' block slots
- **Soft cap**: Block production rewards allow up to 150% of fair share
(50% cap = 30% bonus with 60% weight)
- **Slashing check disabled**: Hook retained for future use, but
currently not applied

### Inflation Mechanism

Previously, the full calculated inflation was minted each era. Now:

- **Scaled by performance**: Total inflation scales between 20%-100%
based on actual blocks produced vs expected
- **Safety floor**: Even with zero blocks, 20% of inflation is still
minted to prevent complete halt
- **Network incentive**: Collective block production directly impacts
total rewards available

### Pallet Configuration

The `pallet-external-validators-rewards` Config now requires additional
types:
- `BlockAuthoringWeight`, `LivenessWeight`, `FairShareCap` for reward
formula
- `ValidatorSet`, `LivenessCheck`, `SlashingCheck` for validator
tracking
- `ExpectedBlocksPerEra`, `MinInflationPercent`, `MaxInflationPercent`
for inflation scaling

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ahmad Kaouk <56095276+ahmadkaouk@users.noreply.github.com>
2025-12-16 16:27:03 +01:00
..
.papi feat: Performance-Based Validator Rewards and Inflation Scaling (#306) 2025-12-16 16:27:03 +01:00
cli feat: enable AVS owner workflow (#332) 2025-12-10 17:38:21 +01:00
configs fix: 🛡️ Check origin for validator set messages (#343) 2025-12-15 14:11:08 +01:00
contract-bindings refactor: remove BSP and MSP operator sets (#323) 2025-11-28 14:01:28 +01:00
datahaven feat: set POV gas limit ratio to zero for solo chain (#313) 2025-11-27 11:14:41 +01:00
docker refactor: Consolidate and optimize Docker image architecture (#233) 2025-10-15 01:33:20 +02:00
docs feat: enable AVS owner workflow (#332) 2025-12-10 17:38:21 +01:00
framework test: Update validator set e2e test (#126) 2025-10-02 11:23:40 +00:00
launcher feat: enable AVS owner workflow (#332) 2025-12-10 17:38:21 +01:00
resources fix: 🐛 Use lodestar instead of lighthouse CL client (#91) 2025-06-09 12:29:31 -03:00
scripts fix: 🛡️ Check origin for validator set messages (#343) 2025-12-15 14:11:08 +01:00
suites fix: 🛡️ Check origin for validator set messages (#343) 2025-12-15 14:11:08 +01:00
utils fix: 🛡️ Check origin for validator set messages (#343) 2025-12-15 14:11:08 +01:00
.bun-version chore: pin Bun version and migrate to bun.lock (#290) 2025-11-10 22:37:39 +01:00
.gitignore test: Integrate moonwall (#185) 2025-09-30 14:47:39 +00:00
.nvmrc fix: 🔧 Fix publish runtime draft release (#226) 2025-10-12 23:59:32 +02:00
biome.json fix: 🚨 Add error in TS for missing awaits (#81) 2025-05-19 22:28:43 +00:00
bun.lock test: port ethereum tests from moonbeam (#278) 2025-11-22 10:02:05 +01:00
bunfig.toml test: 🧙 Generate Type Bindings for Contracts (#58) 2025-05-01 11:14:19 +01:00
Makefile feat: injecting contracts feature for e2e testing (#295) 2025-11-20 12:42:12 +01:00
moonwall.config.json fix: 🔨 Don't run Moonwall tests in parallel (#316) 2025-11-22 01:07:54 +01:00
package.json test: port ethereum tests from moonbeam (#278) 2025-11-22 10:02:05 +01:00
README.md feat: enable AVS owner workflow (#332) 2025-12-10 17:38:21 +01:00
tsconfig.json test: port ethereum tests from moonbeam (#278) 2025-11-22 10:02:05 +01:00
wagmi.config.ts test: 🧙 Generate Type Bindings for Contracts (#58) 2025-05-01 11:14:19 +01:00

DataHaven E2E Testing

End-to-end testing framework for DataHaven, providing automated network deployment, contract interaction, and cross-chain scenario testing. This directory contains all tools needed to launch a complete local DataHaven network with Ethereum, Snowbridge relayers, and run comprehensive integration tests.

For comprehensive documentation, see E2E Testing Guide.

Pre-requisites

  • Kurtosis: For launching test networks
  • Bun v1.3.2 or higher: TypeScript runtime and package manager
  • Docker: For container management
  • Foundry: To deploy contracts
  • Helm: The Kubernetes Package Manager

MacOS

If you are running this on a Mac, zig is a pre-requisite for crossbuilding the node. Instructions for installation can be found here. You may also need to install libpq for PostgreSQL connectivity and set the appropriate Rust flags.

# Install libpq using Homebrew
brew install zig

# Install libpq using Homebrew
brew install libpq

# Set environment variables for Rust compilation
export PKG_CONFIG_PATH="/opt/homebrew/opt/libpq/lib/pkgconfig"
export CPPFLAGS="-I$(brew --prefix libpq)/include"
export LDFLAGS="-L$(brew --prefix libpq)/lib"
export PKG_CONFIG_PATH="$(brew --prefix libpq)/lib/pkgconfig"

# Add to your shell profile (~/.zshrc or ~/.bash_profile) to persist
echo 'export PKG_CONFIG_PATH="/opt/homebrew/opt/libpq/lib/pkgconfig"' >> ~/.zshrc
echo 'export CPPFLAGS="-I$(brew --prefix libpq)/include"' >> ~/.zshrc
echo 'export LDFLAGS="-L$(brew --prefix libpq)/lib"' >> ~/.zshrc
echo 'export PKG_CONFIG_PATH="$(brew --prefix libpq)/lib/pkgconfig"' >> ~/.zshrc

Quick Start

# Install dependencies
bun i

# Interactive CLI to launch a full local DataHaven network
bun cli launch

# Run all the e2e tests
bun test:e2e

# Run all the e2e tests with limited concurrency
bun test:e2e:parallel

# Run a specific test suite
bun test suites/some-test.test.ts

NOTES: Adding the environment variable INJECT_CONTRACTS=true will inject the contracts when starting the tests to speed up setup.

AVS Owner Parameters & Tx Execution

Our deployment tooling now separates “who becomes the ServiceManager owner” from “who executes the privileged post-deployment calls.” The knobs are:

Flag / Env Purpose Default
--avs-owner-address / AVS_OWNER_ADDRESS Address set as avsOwner in the ServiceManager initializer. Required when targeting testnet/mainnet (Safe multisig). Falls back to config/<network>.json only for local/anvil. Local uses config value; non-local must supply.
--avs-owner-key / AVS_OWNER_PRIVATE_KEY Private key used to sign owner-only calls the script performs (e.g. setSlasher). Only read when tx execution is enabled. Anvil default key if unset.
--execute-owner-transactions (CLI) / `TX_EXECUTION=true false` (env) Controls whether the script actually broadcasts owner calls. When disabled, we skip sending transactions and instead print ABI-encoded payloads that a Safe can execute.

Examples

# Local/anvil developer run (executes owner txs immediately)
bun cli contracts deploy --chain anvil --avs-owner-key $LOCAL_OWNER_KEY --execute-owner-transactions

# Testnet deployment where ownership is a Safe (prints multisig payloads)
AVS_OWNER_ADDRESS=0x... bun cli contracts deploy --chain hoodi

# Force execution during launch/deploy automation (already the default)
bun cli launch --deploy-contracts --execute-owner-transactions

When tx execution is off, the CLI prints a list of {to, data, value} objects for:

  1. updateAVSMetadataURI("")
  2. setSlasher(vetoableSlasher)
  3. setRewardsRegistry(validatorsSetId, rewardsRegistry)
  4. setRewardsAgent(validatorsSetId, rewardsAgent)

Copy each object into your safe transaction builder (or preferred multisig workflow) to finalize the deployment.

Generating Ethereum state

To avoid deploying contracts everytime for each tests, you can generate and then inject state in the Ethereum client.

Generate state

$ bun cli launch --all
$ make generate-ethereum-state
$ bun cli stop --all

What Gets Launched

The bun cli launch command deploys a complete local environment:

  1. Ethereum Network (via Kurtosis):

    • 2x Execution Layer clients (reth)
    • 2x Consensus Layer clients (lodestar)
    • Blockscout Explorer (optional: --blockscout)
    • Dora Consensus Explorer
  2. DataHaven Network:

    • 2x Validator nodes (Alice & Bob) with keys (babe, grandpa, imonline, beefy)
    • EVM compatibility via Frontier
    • Fast block times (2-3s in dev mode)
    • Fast churn settings (--fast-runtime gives 1-minute epochs and 3-session eras while block time stays 6s)
  3. Smart Contracts:

    • EigenLayer AVS contracts deployed to Ethereum
    • Optional Blockscout verification (--verified)
  4. Snowbridge Relayers:

    • Beacon relay (Ethereum → DataHaven)
    • BEEFY relay (DataHaven → Ethereum)
    • Execution relay (Ethereum → DataHaven)
    • Solochain relay (DataHaven → Ethereum)
  5. StorageHub Components (optional: --storagehub):

    • 1x MSP (Main Storage Provider) node with bcsv ecdsa key
    • 1x BSP (Backup Storage Provider) node with bcsv ecdsa key
    • 1x Indexer node with PostgreSQL database
    • 1x Fisherman node
    • Automatic provider registration via force_msp_sign_up / force_bsp_sign_up
  6. Network Configuration:

    • Validator registration and funding
    • Parameter initialization
    • Validator set updates

For more information on the E2E testing framework, see the E2E Testing Framework Overview.

Common Commands

Command Description
Network Management
bun cli Interactive CLI menu for all operations
bun cli launch Launch full local network (interactive options)
bun cli launch --all Launch all components including StorageHub
bun cli launch --storagehub Launch with StorageHub nodes (MSP, BSP, Indexer, Fisherman)
bun start:e2e:local Launch local network (non-interactive)
bun start:e2e:verified Launch with Blockscout and contract verification
bun start:e2e:ci CI-optimized network launch
bun cli stop Stop all services (interactive)
bun stop:dh Stop DataHaven only
bun stop:sb Stop Snowbridge relayers only
bun stop:eth Stop Ethereum network only
Testing
bun test:e2e Run all E2E test suites
bun test:e2e:parallel Run tests with limited concurrency
bun test <file> Run specific test file
Code Generation
bun generate:wagmi Generate TypeScript contract bindings (after contract changes)
bun generate:types Generate Polkadot-API types from runtime
bun generate:types:fast Generate types with fast-runtime feature
Code Quality
bun fmt:fix Fix TypeScript formatting with Biome
bun typecheck TypeScript type checking
Deployment
bun cli deploy Deploy to Kubernetes cluster (interactive)
bun build:docker:operator Build local Docker image (datahavenxyz/datahaven:local)

Local Network Deployment

Follow these steps to set up and interact with your local network:

  1. Deploy a minimal test environment

    bun cli launch
    

    This script will:

    1. Check for required dependencies.
    2. Launch a DataHaven solochain.
    3. Start a Kurtosis network which includes:
      • 2 Ethereum Execution Layer clients (reth)
      • 2 Ethereum Consensus Layer clients (lodestar)
      • Blockscout Explorer services for EL (if enabled with --blockscout)
      • Dora Explorer service for CL
    4. Deploy DataHaven smart contracts to the Ethereum network. This can optionally include verification on Blockscout if the --verified flag is used (requires Blockscout to be enabled).
    5. Perform validator setup and funding operations.
    6. Set parameters in the DataHaven chain.
    7. Launch Snowbridge relayers.
    8. Perform validator set update.

    Note

    If you want to also have the contracts verified on Blockscout, you can pass the --verified flag to the bun cli launch command, along with the --blockscout flag. This will do all the previous, but also verify the contracts on Blockscout. However, note that this takes some time to complete.

  2. Explore the network

Troubleshooting

E2E Network Launch doesn't work

Script halts unexpectedly

When running bun cli launch the script appears to halt after the following:

## Setting up 1 EVM.

==========================

Chain 3151908

Estimated gas price: 2.75 gwei

Estimated total gas used for script: 71556274

Estimated amount required: 0.1967797535 ETH

==========================

This is due to how forge streams output to stdout, but is infact still deploying contracts to the chain. You should be able to see in blockscout the deploy script is indeed still working.

Errors with deploying forge scripts on kurtosis network

Try running forge clean to clear any spurious build artefacts, and running forge build again. Also try deploying manually to the still running kurtosis network.

Blockscout is empty

If you look at the browser console, if you see the following:

Content-Security-Policy: The page's settings blocked the loading of a resource (connect-src) at http://127.0.0.1:3000/node-api/proxy/api/v2/stats because it violates the following directive: "connect-src ' ...

this is a result of CORS and CSP errors due to running this as a local docker network.

Make sure you are connected directly to http://127.0.0.1:3000 (not localhost).

Alternatively, you can try installing a browser addon such as anti-CORS / anti-CSP to circumvent this problem.

Weird forge Errors

In the /contracts directory, you can try to run forge clean and forge build to see if it fixes the issue.

Linux: See if disabling ipV6 helps

I have found that ipV6 on Arch Linux does not play very nicely with Kurtosis networks. Disabling it completely fixed the issue for me.

macOS: Verify Docker networking settings

Docker Network Settings

If using Docker Desktop, make sure settings have permissive networking enabled.

Polkadot-API types don't match expected runtime types

If you've made changes to the runtime types, you need to re-generate the TS types for the Polkadot-API. Don't worry, this is fully automated.

From the ./test directory run the following command:

bun generate:types

This script will:

  1. Compile the runtime using cargo build --release in the ../operator directory.
  2. Re-generate the Polkadot-API types using the newly built WASM binary.

Note

The script uses the --release flag by default, meaning it uses the WASM binary from ./operator/target/release. If you need to use a different build target, you may need to adjust the script or run the steps manually.

Project Structure

test/
├── suites/                              # E2E test suites
│   ├── contracts.test.ts               # Contract deployment & configuration
│   ├── cross-chain.test.ts             # Cross-chain message passing
│   ├── datahaven-substrate.test.ts     # Block production & finality
│   ├── ethereum-basic.test.ts          # Ethereum network validation
│   ├── native-token-transfer.test.ts   # Cross-chain token transfers
│   ├── rewards-message.test.ts         # Validator rewards distribution
│   └── validator-set-update.test.ts    # Dynamic validator set updates
├── framework/                           # Test utilities & helpers
│   ├── connectors.ts                   # Network connectors
│   ├── manager.ts                      # Test environment manager
│   ├── suite.ts                        # Test suite utilities
│   └── index.ts                        # Framework exports
├── launcher/                            # Network deployment tools
│   ├── kurtosis/                       # Ethereum network launcher
│   ├── snowbridge/                     # Relayer management
│   └── datahaven/                      # DataHaven node management
├── generated/                           # Generated types
│   ├── wagmi/                          # Contract bindings
│   └── polkadot-api/                   # Runtime types
└── docs/                                # Testing documentation
    ├── E2E_TESTING_GUIDE.md
    └── E2E_FRAMEWORK_OVERVIEW.md

Test Suites

  • contracts.test.ts: Contract deployment and configuration validation
  • cross-chain.test.ts: Cross-chain message passing between Ethereum and DataHaven
  • datahaven-substrate.test.ts: Block production, finalization, and consensus
  • ethereum-basic.test.ts: Ethereum network health and basic functionality
  • native-token-transfer.test.ts: Cross-chain token transfers via Snowbridge
  • rewards-message.test.ts: Validator reward distribution from Ethereum to DataHaven
  • validator-set-update.test.ts: Dynamic validator registration/deregistration via EigenLayer

Run individual suites:

bun test suites/rewards-message.test.ts
bun test suites/native-token-transfer.test.ts
bun test suites/validator-set-update.test.ts

Further Information

  • Kurtosis: Ethereum network orchestration
  • Zombienet: Polkadot-SDK network testing
  • Bun: TypeScript runtime and tooling
  • Foundry: Solidity development framework
  • Polkadot-API: Type-safe Substrate interactions