test: new commandline argument -tinyblk to use blk size just 64kb instead 16Mb

[knst]

Issue being fixed or feature implemented

Even trivial functional tests that have only genesis blocks without any funds still creates 2 files 17 Mbs in total: 16Mb files for blk00000.dat and 1Mb for rev00000.dat for each node:

$ test/functional/rpc_help.py --nocleanup
<succeed>
$ find /tmp/dash_func_test_5coohz_8 -ls | grep -E '/rev|blk'
   436679   1024 -rw-------   1 knst     knst      1048576 Nov 17 02:08 /tmp/dash_func_test_5coohz_8/node0/regtest/blocks/rev00000.dat
   436673  16384 -rw-------   1 knst     knst     16777216 Nov 17 02:08 /tmp/dash_func_test_5coohz_8/node0/regtest/blocks/blk00000.dat

It doesn't look like a lot for a single test, but we have 300 functional tests, in average ~3 nodes -> more than 10Gb of mostly zeroes written on disk for each CI run. Files are not sparse and indeed takes IO and disk space.

What was done?

Introduced a new command line argument -tinyblk which reduce allocated space for empty blk files to 64kb only.
It works similar as -fastprune but doesn't affect pruning settings; only block size.

Also minor refactoring for functional tests to avoid copy-past code when writting dash.conf for case of statically and dynamically added new nodes; duplicated code is unified.

How Has This Been Tested?

CI succeed for this PR: https://github.com/dashpay/dash/actions/runs/19412323566/job/55536154684?pr=6986

While develop fails: https://github.com/dashpay/dash/actions/runs/19392476511

Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/actions-runner/cached/_diag/Worker_20251116-201117-utc.log'

Local run:

  1602527     64 -rw-------   1 knst     knst        65536 Nov 17 04:12 /tmp/dash_func_test_bw8hmeiz/node0/regtest/blocks/blk00000.dat
  1602533     64 -rw-------   1 knst     knst        65536 Nov 17 04:12 /tmp/dash_func_test_bw8hmeiz/node0/regtest/blocks/rev00000.dat

64kb only.
Some other tests that have more blocks and transactions in blocks have increased data file correctly as expected, for example:

  1505724    320 -rw-------   1 knst     knst       327680 Nov 17 04:05 ./feature_llmq_simplepose_268/node0/regtest/blocks/blk00000.dat

Local run with 30 parallel jobs (-j30) become ~10seconds faster (195s -> 180+); for 20 jobs - no difference as IO is no more limiting factor.

Breaking Changes

N/A

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation
• I have assigned this pull request to a milestone

[github-actions]

✅ No Merge Conflicts Detected

This PR currently has no conflicts with other open PRs.

[coderabbitai]

Walkthrough

A new -tinyblk debug flag was added to the server arguments. Block and undo file chunk-size selection was updated: if -fastprune is set use 16KB for block chunks; else if -tinyblk is set use 64KB for block chunks; undo (rev) chunks use 64KB when -tinyblk is set, otherwise the default undo chunk size. Test framework refactors replace port-based datadir initialization with an index-based API, centralize DIP-3 config via a new append_dip3_config helper, and add tinyblk=1 to generated dash.conf in tests.

Sequence Diagram(s)

sequenceDiagram
    participant Init as init.cpp (arg parsing)
    participant Storage as blockstorage.cpp (chunk selection)
    participant Test as test_framework/util.py (test setup)

    note over Init,Storage: Flag registration
    Init->>Init: register -tinyblk (DEBUG_TEST)

    note over Init,Storage: Runtime flag decision
    Init->>Storage: process flags (-fastprune?, -tinyblk?)
    alt -fastprune set
        Storage->>Storage: blk_chunk = 16KB
    else -tinyblk set
        Storage->>Storage: blk_chunk = 64KB
        Storage->>Storage: rev_chunk = 64KB
    else
        Storage->>Storage: blk_chunk = BLOCKFILE_CHUNK_SIZE
        Storage->>Storage: rev_chunk = UNDOFILE_CHUNK_SIZE
    end

    note over Test: Test datadir/config flow
    Test->>Test: dynamically_initialize_datadir(mnidx)
    Test->>Test: write_config(datadir, tinyblk=1, ...)
    Test->>Test: append_dip3_config(datadir)

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• review src/init.cpp: flag registration and category
• review src/node/blockstorage.cpp: conditional chunk-size logic and precedence (fastprune vs tinyblk)
• review test/functional/test_framework/test_framework.py and util.py: signature changes, new append_dip3_config, write_config changes (ensure datadir generation and DIP-3 injection are correct)
• verify all call sites updated for changed method signatures and that tests produce correct dash.conf entries (tinyblk=1)

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 8.33% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: introducing a -tinyblk command-line argument that reduces block file size to 64KB, which is the primary objective across all file modifications.
Description check	✅ Passed	The description is directly related to the changeset, providing clear context about the disk space issue being addressed, the implementation approach with -tinyblk, testing results, and refactoring details.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/node/blockstorage.cpp (1)

628-628: Missing -tinyblk check in max file size logic.

Line 628 determines when to roll over to a new block file, but it only checks for -fastprune. When -tinyblk is set without -fastprune, files could grow up to MAX_BLOCKFILE_SIZE instead of the intended 64 KB. This defeats the purpose of the -tinyblk flag, which is to keep files small for testing.

Apply this diff to check for -tinyblk as well:

-        while (m_blockfile_info[nFile].nSize + nAddSize >= (gArgs.GetBoolArg("-fastprune", false) ? 0x10000 /* 64kb */ : MAX_BLOCKFILE_SIZE)) {
+        while (m_blockfile_info[nFile].nSize + nAddSize >= (gArgs.GetBoolArg("-tinyblk", false) || gArgs.GetBoolArg("-fastprune", false) ? 0x10000 /* 64kb */ : MAX_BLOCKFILE_SIZE)) {

🧹 Nitpick comments (1)

src/init.cpp (1)

538-538: Consider enhancing the help text with the specific file size.

The description "Use smaller block files for testing purposes" is generic. Consider specifying the actual size (64 KB) to help users understand the behavior without needing to examine the code.

Example:

-    argsman.AddArg("-tinyblk", "Use smaller block files for testing purposes", ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::DEBUG_TEST);
+    argsman.AddArg("-tinyblk", "Use smaller block files (64 KB) for testing purposes", ArgsManager::ALLOW_ANY | ArgsManager::DEBUG_ONLY, OptionsCategory::DEBUG_TEST);

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between caf5010 and 375bf9d.

📒 Files selected for processing (4)

• src/init.cpp (1 hunks)
• src/node/blockstorage.cpp (1 hunks)
• test/functional/test_framework/test_framework.py (6 hunks)
• test/functional/test_framework/util.py (1 hunks)

🧰 Additional context used

🧠 Learnings (3)

📓 Common learnings

Learnt from: kwvg
Repo: dashpay/dash PR: 6543
File: src/wallet/receive.cpp:240-251
Timestamp: 2025-02-06T14:34:30.466Z
Learning: Pull request #6543 is focused on move-only changes and refactoring, specifically backporting from Bitcoin. Behavior changes should be proposed in separate PRs.

Learnt from: kwvg
Repo: dashpay/dash PR: 6718
File: test/functional/test_framework/test_framework.py:2102-2102
Timestamp: 2025-06-09T16:43:20.996Z
Learning: In the test framework consolidation PR (#6718), user kwvg prefers to limit functional changes to those directly related to MasternodeInfo, avoiding scope creep even for minor improvements like error handling consistency.

📚 Learning: 2025-06-09T16:43:20.996Z

Learnt from: kwvg
Repo: dashpay/dash PR: 6718
File: test/functional/test_framework/test_framework.py:2102-2102
Timestamp: 2025-06-09T16:43:20.996Z
Learning: In the test framework consolidation PR (#6718), user kwvg prefers to limit functional changes to those directly related to MasternodeInfo, avoiding scope creep even for minor improvements like error handling consistency.

Applied to files:

• test/functional/test_framework/test_framework.py

📚 Learning: 2025-08-10T13:52:46.289Z

Learnt from: kwvg
Repo: dashpay/dash PR: 6666
File: test/functional/rpc_netinfo.py:73-81
Timestamp: 2025-08-10T13:52:46.289Z
Learning: In the Bitcoin/Dash test framework, `self.node.extra_args` contains the original startup arguments for a TestNode and is not modified by `test.restart_node()`. Each restart with new `extra_args` doesn't update the stored `self.node.extra_args`, so copying from it always provides a clean slate without accumulated modifications from previous restarts.

Applied to files:

• test/functional/test_framework/test_framework.py

🧬 Code graph analysis (2)

src/node/blockstorage.cpp (1)

src/flatfile.cpp (1)

• FlatFileSeq (13-21)

test/functional/test_framework/test_framework.py (1)

test/functional/test_framework/util.py (3)

• get_chain_folder (492-501)
• write_config (396-428)
• append_config (435-438)

🪛 Flake8 (7.3.0)

test/functional/test_framework/test_framework.py

[error] 628-628: continuation line under-indented for visual indent

(E128)

---

[error] 1480-1480: over-indented

(E117)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)

• GitHub Check: mac-build / Build source
• GitHub Check: linux64_tsan-build / Build source
• GitHub Check: linux64_nowallet-build / Build source
• GitHub Check: arm-linux-build / Build source
• GitHub Check: linux64_sqlite-build / Build source
• GitHub Check: linux64-build / Build source
• GitHub Check: linux64_ubsan-build / Build source
• GitHub Check: linux64_fuzz-build / Build source
• GitHub Check: win64-build / Build source
• GitHub Check: Lint / Run linters

🔇 Additional comments (6)

test/functional/test_framework/util.py (1)

422-423: LGTM!

The addition of tinyblk=1 to the test framework configuration is well-placed and clearly documented. This will help reduce I/O and disk usage during functional tests, directly supporting the performance improvements mentioned in the PR objectives.

test/functional/test_framework/test_framework.py (5)

58-59: LGTM! Import additions support the config writing refactor.

The addition of get_chain_folder and write_config imports aligns with the refactoring goal to consolidate config writing logic and introduce the tinyblk=1 setting.

---

1465-1465: LGTM! Config writing consolidation improves maintainability.

Moving the DIP3 config writing to a dedicated method reduces code duplication and aligns with the PR's refactoring objectives.

---

1600-1600: LGTM! Caller correctly updated for new signature.

The call to dynamically_initialize_datadir correctly uses the new signature with the masternode index parameter.

---

631-633: The directory creation is intentional and necessary.

The verification confirms that stderr and stdout directories must exist before test nodes attempt to create temporary log files within them (test_node.py lines 231-235, 615-616). The same pattern appears in util.py lines 391-392, indicating this is an established practice. The exist_ok=True flag ensures safety even if both util.py and test_framework.py functions are called.

---

610-610: All callers confirmed to be updated to the new signature.

Verification complete:

• Only one caller exists at line 1600: self.dynamically_initialize_datadir(mn_idx) — already using the new signature
• No external test scripts call this method
• All references are internal to the test framework and properly updated

The signature change is safe and all callers have been updated accordingly.

[UdjinM6]

Looks good! Pls apply coderabbitai's suggestion re indentation 3338a6f

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

test/functional/test_framework/test_framework.py (1)

610-633: Use the mnidx parameter consistently throughout the method.

Line 612 computes the datadir path using len(self.nodes) while Line 627 uses the mnidx parameter. For consistency and clarity, Line 612 should use mnidx instead:

Apply this diff to use the parameter consistently:

 def dynamically_initialize_datadir(self, mnidx):
     source_data_dir = get_datadir_path(self.options.tmpdir, 0)  # use node0 as a source
-    new_data_dir = get_datadir_path(self.options.tmpdir, len(self.nodes))
+    new_data_dir = get_datadir_path(self.options.tmpdir, mnidx)

Note: The method now returns new_data_dir, but this value is not used at the call site (Line 1598). While this doesn't cause issues currently, consider either using the return value or removing it if not needed.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7a93394 and def5d0a.

📒 Files selected for processing (1)

• test/functional/test_framework/test_framework.py (8 hunks)

🧰 Additional context used

🧠 Learnings (3)

📓 Common learnings

Learnt from: kwvg
Repo: dashpay/dash PR: 6718
File: test/functional/test_framework/test_framework.py:2102-2102
Timestamp: 2025-06-09T16:43:20.996Z
Learning: In the test framework consolidation PR (#6718), user kwvg prefers to limit functional changes to those directly related to MasternodeInfo, avoiding scope creep even for minor improvements like error handling consistency.

Learnt from: UdjinM6
Repo: dashpay/dash PR: 6786
File: ci/test/04_install.sh:99-101
Timestamp: 2025-08-01T07:46:37.840Z
Learning: In backport PRs like #6786, UdjinM6 prefers to defer non-critical fixes (such as shell command expansion issues) to separate commits/PRs to maintain focus on the primary backport objectives, consistent with the project's pattern of avoiding scope creep.

📚 Learning: 2025-06-09T16:43:20.996Z

Learnt from: kwvg
Repo: dashpay/dash PR: 6718
File: test/functional/test_framework/test_framework.py:2102-2102
Timestamp: 2025-06-09T16:43:20.996Z
Learning: In the test framework consolidation PR (#6718), user kwvg prefers to limit functional changes to those directly related to MasternodeInfo, avoiding scope creep even for minor improvements like error handling consistency.

Applied to files:

• test/functional/test_framework/test_framework.py

📚 Learning: 2025-06-16T17:59:55.669Z

Learnt from: kwvg
Repo: dashpay/dash PR: 6665
File: src/evo/deterministicmns.cpp:1284-1287
Timestamp: 2025-06-16T17:59:55.669Z
Learning: In Dash masternode ProRegTx validation, platform ports (platformHTTPPort and platformP2PPort) are mandatory and must be non-zero, while netInfo (ipAndPort) is optional. This means that even if an empty netInfo returns 0 from GetPrimary().GetPort(), it won't cause false positives in port duplication checks since platform ports cannot be 0.

Applied to files:

• test/functional/test_framework/test_framework.py

🧬 Code graph analysis (1)

test/functional/test_framework/test_framework.py (1)

test/functional/test_framework/util.py (5)

• get_chain_folder (492-501)
• write_config (396-428)
• append_config (435-438)
• softfork_active (526-528)
• p2p_port (359-361)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)

• GitHub Check: win64-build / Build source
• GitHub Check: mac-build / Build source
• GitHub Check: arm-linux-build / Build source
• GitHub Check: linux64_tsan-build / Build source
• GitHub Check: linux64_nowallet-build / Build source
• GitHub Check: linux64_ubsan-build / Build source
• GitHub Check: linux64_sqlite-build / Build source
• GitHub Check: linux64-build / Build source
• GitHub Check: linux64_fuzz-build / Build source
• GitHub Check: Lint / Run linters

🔇 Additional comments (4)

test/functional/test_framework/test_framework.py (4)

58-59: LGTM! Imports support the refactoring.

The new imports get_chain_folder and write_config are correctly added and used in the refactored configuration logic.

---

1461-1468: LGTM! Proper use of centralized DIP-3 config.

The refactoring correctly uses the new append_dip3_config helper to inject DIP-3 configuration for dynamically added nodes, eliminating code duplication.

---

1479-1484: LGTM! Good refactoring to centralize DIP-3 configuration.

The new append_dip3_config method eliminates code duplication by centralizing DIP-3 related configuration injection. The implementation is clean and the indentation is correct.

---

1617-1648: LGTM! Addresses prior review feedback.

The node_p2p_port calculation has been moved into dynamically_prepare_masternode (Line 1621) as suggested by UdjinM6, simplifying the method signature and improving encapsulation.

[UdjinM6]

test folder size:
develop: 13G
this PR: 3.9G

👍

light ACK def5d0a

[kwvg]

utACK def5d0a