Refact test runner #671

joaozinhom · 2025-10-12T22:33:24Z

What is the purpose of this pull request?

Which crates are being modified?

Description and Notes

Summary

This PR replaces our homegrown functional testing harness with pytest and removes the need to scrape runtime data (like ports) from log files. The goal is to make tests reliable, faster, and easier to maintain without reinventing what mature tooling already provides.

Motivation

The custom test framework started to feel like we were rebuilding an existing wheel:

We tried to mirror Bitcoin Core’s style while repeatedly fixing generic bugs and re-implementing features test frameworks already solve well.
Tests depended on a single log file per test to share state (e.g., ports), which led to brittle parsing, noisy logs, and timeouts when logs grew large.
CI became flaky and slow, with random failures and long runs that weren’t adding confidence.

Approach

Adopt pytest as the test runner.
Stop reading ports and other runtime data from logs; tests now rely on explicit, structured hand-offs and configuration.
Remove the “multiple Python processes per task” pattern and delegate parallelism to pytest (and its ecosystem, e.g., pytest-xdist if/when we enable it).
Centralize configuration in pyproject.toml and use pytest’s built-in reporting, fixtures, and plugin ecosystem.
Keep the initial refactor focused: we didn’t convert every single test in this PR to minimize risk and surface early feedback sooner.

Why this helps

Stability: No more log scraping or log-size-related timeouts; fewer sources of nondeterminism.
Maintainability: Standard fixtures, clearer test boundaries, and better failure diagnostics.
Performance: Leaner process model and optional parallelism via pytest’s ecosystem.
Community alignment: Leverages widely used, well-documented tooling instead of a bespoke framework.

Goals

Eliminate random test failures tied to log parsing and timeouts.
Avoid spawning extra Python processes during test runtime.
Improve overall performance and CI throughput.
Control nodes imperatively via explicit APIs/fixtures instead of logs.

Scope and collaboration

This work was done with input and decision-making support from @jaoleal, and from @qlrd to understand the test framework. We intentionally did not refactor all tests at once; feedback from the team will help guide the remaining migration and any follow-up improvements.

How to run

Locally: to run everithing (uv run tests/test_runner.py), to run just pytest you can does (uv run pytest tests/ -n auto)
Configuration lives in pyproject.toml.

Open questions and next steps

Are there remaining tests that still depend on log-derived state? Let’s list and plan conversions.
Do we want to enable pytest-xdist by default in CI, or keep it opt-in for now?
Any fixtures we should promote to shared utilities?

Feedback welcome—especially on migration strategy, fixture design, and CI defaults.

jaoleal · 2025-10-14T03:19:49Z

The command to run is actually uv run pytest tests/example/* -n auto, correct ?

Also, a rebase would be good

jaoleal

Nice!!! I was able to run once but when i ran it again i got a

Failed to start florestad: Failed to bind Electrum server: Address already in use (os error 98)

This is a partial review btw, we should focus on organizing nodes to avoid such errors.

tests/node_manager.py

joaozinhom · 2025-10-15T14:11:34Z

Nice!!! I was able to run once but when i ran it again i got a
Failed to start florestad: Failed to bind Electrum server: Address already in use (os error 98)
This is a partial review btw, we should focus on organizing nodes to avoid such errors.

Solved with a cleanup on the temporary dir of the nodes data and logs.

pyproject.toml

Davidson-Souza · 2025-10-29T19:50:54Z

Floresta is logging to stdout. I think this is too polluting and unnecessary

moisesPompilio

Mark some lines that exceed 80 characters. In this case, it would be interesting if the changes you made followed a pattern where each line has a limit of 80 characters per line.

tests/node_manager.py

tests/test_runner.py

tests/example/bitcoin.py

pyproject.toml

joaozinhom · 2025-11-20T20:03:21Z

@Davidson-Souza I've already started writing the florestad tests, but I'm worried this PR is becoming too long. What do you think about just solving the issues @moisesPompilio pointed out (and any following ones), and leaving the florestad and floresta-cli test conversions to pytest for another PR?

Davidson-Souza · 2025-11-24T22:31:50Z

I've already started writing the florestad tests, but I'm worried this PR is becoming too long. What do you think about just solving the issues @moisesPompilio pointed out (and any following ones), and leaving the florestad and floresta-cli test conversions to pytest for another PR?

Sure!

moisesPompilio

The PR description should include instructions on how to run the tests. If pytest requires specific commands, they need to be listed.

It should also mention whether the migrated tests will be integrated into test_runner or propose a new strategy for running all tests.

moisesPompilio · 2025-11-25T21:24:45Z

bin/florestad/src/main.rs

        log_to_file: params.log_to_file,
        assume_valid: params.assume_valid,
-        log_to_stdout: true,
+        log_to_stdout: false,


I’m not sure if changing this part is appropriate, what do you think @Davidson-Souza ?

I think the logging issue comes from Floresta, Utreexod, and Bitcoind all printing to the terminal, which makes analysis harder. It might be useful to have a flag to enable terminal logs when you want to see what the node is doing. But for tests, ideally only the information relevant to the test should appear, like whether a node connected or disconnected, and that should be handled by the test itself.

think the logging issue comes from Floresta, Utreexod, and Bitcoind all printing to the terminal

+1

qlrd · 2025-11-25T20:47:03Z

tests/example/bitcoin.py

@@ -1,64 +1,11 @@
-"""
-bitcoin-test.py
+import pytest


IMO, i think we should keep, at least the module docstrings. Class docstrings and method docstrings are welcomed too, since one of the reasons on every file on tests (including the framework).

In case of this test, it have an educational approach: it should explain well what is happening so developers could be guided.

qlrd · 2025-11-25T20:47:17Z

tests/example/electrum.py

-"""
-
-from test_framework import FlorestaTestFramework
+import pytest


qlrd · 2025-11-25T20:51:35Z

tests/test_framework/__init__.py

-    `run_test, but DOES NOT override `__init__` or `main`. If those standards
-    are violated, a `TypeError` is raised.
-    """
+    """Metaclass for enforcing test framework contract."""


Why remove the bigtext on docstring?

qlrd · 2025-11-25T20:53:06Z

tests/test_framework/__init__.py

-    test/test_framework/{crypto,daemon,rpc,electrum}/*.py to see
-    how the test framework was structured.
-    """
+    """Base class for Floresta integration tests."""


Yeah, I agree that is a very big text. But i still against remove a big block of docstrings on tests. Maybe they could be rewriten, but they should guide the new testers.

This init.py will be removed we can create some documentation work on the node_manager.py, i don't agree too much of write docs for a dead marked file...

I think we can work more on the documentation when everything is integrated with pytest's ideas and we have a good overview of what will be discarded and what will be merged with what.

qlrd · 2025-11-25T20:54:45Z

tests/test_framework/__init__.py

-        This is used to create a log file for the test.
-        """
-        tempdir = str(FlorestaTestFramework.get_integration_test_dir())
+        """Get the path for the test log file.


IMO it's important to explain that the class name will come in lowercase.

qlrd · 2025-11-25T22:40:36Z

tests/test_framework/__init__.py

-
-        If the node not exists raise a IndexError. At the time
-        the tests will only run nodes configured to run on regtest.
+        """Start a node and initialize its RPC connection.


qlrd · 2025-11-25T22:46:15Z

tests/conftest.py

+
+# pylint: disable=redefined-outer-name  # Pytest fixtures pattern
+
+sys.path.insert(0, os.path.dirname(__file__))


for windows could be worth to use os.path.normpath

qlrd · 2025-11-25T22:47:12Z

tests/conftest.py

+    return node
+
+
+@pytest.fixture


I think could be nice to add an utreexod_with_tls to test electrum stuffs

qlrd · 2025-11-25T22:49:30Z

tests/test_runner.py

@@ -1,330 +1,272 @@
-"""
-test_runner.py
+"""i disable all pylint here because this is a temporary file, we will replace


Here too, about the comments in the docstrings at module level

qlrd · 2025-11-25T22:54:52Z

tests/utilities.py

@@ -0,0 +1,116 @@
+"""Utility functions for Floresta integration tests.


Very nice! IMO this docstring at module level could receive more love and a little more verbose explanation of which types of utilities we can expect here.

qlrd · 2025-11-26T10:31:54Z

The PR description should include instructions on how to run the tests. If pytest requires specific commands, they need to be listed.

The new instructions should be on doc/running-tests.md.

Also, neet to add an tests/requirements.txt equals to the dependencies entry on pyproject.toml so someone could run uv pip install --requirements tests/requirements.txt.

qlrd · 2025-11-26T10:23:03Z

bin/florestad/src/main.rs

        log_to_file: params.log_to_file,
        assume_valid: params.assume_valid,
-        log_to_stdout: true,
+        log_to_stdout: false,


think the logging issue comes from Floresta, Utreexod, and Bitcoind all printing to the terminal

+1

qlrd · 2025-11-26T17:52:35Z

tests/test_runner.py

-    results_queue = Queue()
-    workers = []
+    print("=" * 60)
+    print(f"TEST SUMMARY")


Suggested change

print(f"TEST SUMMARY")

print("TEST SUMMARY")

qlrd · 2025-11-26T17:53:51Z

tests/test_runner.py

+
+    # Argument parser
+    parser = argparse.ArgumentParser(
+        prog="multi_test_runner", description="Run Floresta integration tests"


Suggested change

prog="multi_test_runner", description="Run Floresta integration tests"

prog="test_runner", description="Run Floresta integration tests"

qlrd · 2025-11-26T18:03:11Z

tests/test_runner.py

-    ("example", "utreexod"),
-]
-
-# Before running the tests, we check if the number of tests


Why remove these check? When someone add a test and the test isn't registered, it will not run the test in CI and give some false sense of success.

qlrd · 2025-11-26T18:05:40Z

tests/test_framework/__init__.py

 import contextlib
 from datetime import datetime, timezone
-from typing import Any, Dict, List, Literal, Pattern, TextIO
+from typing import Any, Dict, List, Pattern


Unused Dict:

Suggested change

from typing import Any, Dict, List, Pattern

from typing import Any, List, Pattern

qlrd · 2025-11-26T18:09:00Z

tests/test_framework/__init__.py

-        Check if an option is set in extra_args
+        return any(arg.startswith(option) for arg in extra_args)
+
+    def extract_port_from_args(self, extra_args: list[str], option: str) -> int:


IMO we can rename this method since it's not extracting anymore, but instead, giving a random available one:

Suggested change

def extract_port_from_args(self, extra_args: list[str], option: str) -> int:

def get_available_random_port(self, extra_args: list[str], option: str) -> int:

I think the suggested name doesn’t fit. The function is extracting the port directly from the args, so extract_port_from_args is the clearer and more accurate name...

qlrd · 2025-11-26T18:09:34Z

tests/test_framework/__init__.py

+            ports["rpc"] = 18443 + port_index
+            default_args.append(f"--rpc-address=127.0.0.1:{ports['rpc']}")
+        else:
+            ports["rpc"] = self.extract_port_from_args(extra_args, "--rpc-address")


Replace the name with what you renamed

qlrd · 2025-11-26T20:42:22Z

tests/conftest.py

+"""
+Pytest configuration and fixtures for node testing.
+
+This module provides fixtures for creating and managing test nodes
+(florestad, bitcoind, utreexod) in various configurations.


I think it's worth to teach rustceans how to use created marks with examples in docstring at module level.

pyproject.toml

qlrd · 2025-11-26T20:51:03Z

tests/conftest.py

+@pytest.fixture
+def electrum_setup(node_manager) -> Dict[str, Node]:
+    """Setup for electrum tests"""
+    florestad = node_manager.create_node(
+        variant="florestad",
+        extra_args=["--electrum-address=127.0.0.1:50001"],
+        testname="pytest_electrum",
+    )
+    node_manager.start_node(florestad)
+    return {"florestad": florestad}


IMO we can change the name in favor to add another implementation in a follow-up (electrs, esplora, etc...):

Suggested change

@pytest.fixture

def electrum_setup(node_manager) -> Dict[str, Node]:

"""Setup for electrum tests"""

florestad = node_manager.create_node(

variant="florestad",

extra_args=["--electrum-address=127.0.0.1:50001"],

testname="pytest_electrum",

)

node_manager.start_node(florestad)

return {"florestad": florestad}

@pytest.fixture

def floresta_electrum_setup(node_manager) -> Dict[str, Node]:

"""Setup for electrum tests"""

florestad = node_manager.create_node(

variant="florestad",

extra_args=["--electrum-address=127.0.0.1:50001"],

testname="pytest_electrum",

)

node_manager.start_node(florestad)

return {"florestad": florestad}

tests/conftest.py

qlrd · 2025-11-26T20:57:35Z

tests/node_manager.py

+# I disable the following because i literally will delete this file too later
+# its a middle way to the __init__.py from test framework for the pytest


IMO this could be in tests/test_framework/node/{__init__.py,node.py,node_manager.py}.

qlrd · 2025-11-26T21:20:09Z

tests/test_runner.py

-# Scripts that are run by default.
-# Longest test should go first,
-# to favor running tests in parallel.
-# We use this like those in  the
-# Bitcoin Core tests/functional/test_runner.py:89
-
-# Tests that are ran by default. The longest running tests should be ran first,
-# so parallelization is used most effectively. This structure is copied from
-# Bitcoin Core's functional tests:
-# https://github.com/bitcoin/bitcoin/blob/master/test/functional/test_runner.py#L89


IMO, this explanation is important to be above if TEST_REGISTRY declaration on previous commented review request.

qlrd · 2025-11-26T21:24:23Z

tests/test_runner.py

+TEST_REGISTRY = {
+    "addnodev2": import_test_class("floresta-cli.addnode-v2", "AddnodeTestV2"),
+    "addnodev1": import_test_class("floresta-cli.addnode-v1", "AddnodeTestV1"),
+    "reorg_chain": import_test_class("florestad.reorg-chain", "ChainReorgTest"),
+    "getbestblockhash": import_test_class(
+        "floresta-cli.getbestblockhash", "GetBestblockhashTest"
+    ),
+    "getblockcount": import_test_class(
+        "floresta-cli.getblockcount", "GetBlockCountTest"
+    ),
+    "uptime": import_test_class("floresta-cli.uptime", "UptimeTest"),
+    "restart": import_test_class("florestad.restart", "TestRestart"),
+    "connect": import_test_class("florestad.connect", "CliConnectTest"),
+    "stop": import_test_class("floresta-cli.stop", "StopTest"),
+    "ping": import_test_class("floresta-cli.ping", "PingTest"),
+    "getrpcinfo": import_test_class("floresta-cli.getrpcinfo", "GetRpcInfoTest"),
+    "getblockhash": import_test_class("floresta-cli.getblockhash", "GetBlockhashTest"),
+    "tls": import_test_class("florestad.tls", "TestSslInitialization"),
+    "getroots": import_test_class("floresta-cli.getroots", "GetRootsIDBLenZeroTest"),
+    "getblock": import_test_class("floresta-cli.getblock", "GetBlockTest"),
+    "getmemoryinfo": import_test_class(
+        "floresta-cli.getmemoryinfo", "GetMemoryInfoTest"
+    ),
+    "getblockheader": import_test_class(
+        "floresta-cli.getblockheader", "GetBlockheaderHeightZeroTest"
+    ),
+    "getpeerinfo": import_test_class("floresta-cli.getpeerinfo", "GetPeerInfoTest"),
+    "tls_fail": import_test_class("florestad.tls-fail", "TestSslFailInitialization"),
+    "getblockchaininfo": import_test_class(
+        "floresta-cli.getblockchaininfo", "GetBlockchaininfoTest"
+    ),
+}


There are some forgotten tests here as well IMO this could be cleaner:

Suggested change

TEST_REGISTRY = {

"addnodev2": import_test_class("floresta-cli.addnode-v2", "AddnodeTestV2"),

"addnodev1": import_test_class("floresta-cli.addnode-v1", "AddnodeTestV1"),

"reorg_chain": import_test_class("florestad.reorg-chain", "ChainReorgTest"),

"getbestblockhash": import_test_class(

"floresta-cli.getbestblockhash", "GetBestblockhashTest"

),

"getblockcount": import_test_class(

"floresta-cli.getblockcount", "GetBlockCountTest"

),

"uptime": import_test_class("floresta-cli.uptime", "UptimeTest"),

"restart": import_test_class("florestad.restart", "TestRestart"),

"connect": import_test_class("florestad.connect", "CliConnectTest"),

"stop": import_test_class("floresta-cli.stop", "StopTest"),

"ping": import_test_class("floresta-cli.ping", "PingTest"),

"getrpcinfo": import_test_class("floresta-cli.getrpcinfo", "GetRpcInfoTest"),

"getblockhash": import_test_class("floresta-cli.getblockhash", "GetBlockhashTest"),

"tls": import_test_class("florestad.tls", "TestSslInitialization"),

"getroots": import_test_class("floresta-cli.getroots", "GetRootsIDBLenZeroTest"),

"getblock": import_test_class("floresta-cli.getblock", "GetBlockTest"),

"getmemoryinfo": import_test_class(

"floresta-cli.getmemoryinfo", "GetMemoryInfoTest"

),

"getblockheader": import_test_class(

"floresta-cli.getblockheader", "GetBlockheaderHeightZeroTest"

),

"getpeerinfo": import_test_class("floresta-cli.getpeerinfo", "GetPeerInfoTest"),

"tls_fail": import_test_class("florestad.tls-fail", "TestSslFailInitialization"),

"getblockchaininfo": import_test_class(

"floresta-cli.getblockchaininfo", "GetBlockchaininfoTest"

),

}

# Scripts that are run by default.

# Longest test should go first,

# to favor running tests in parallel.

# We use this like those in the

# Bitcoin Core tests/functional/test_runner.py:89

# Tests that are ran by default. The longest running tests should be ran first,

# so parallelization is used most effectively. This structure is copied from

# Bitcoin Core's functional tests:

# https://github.com/bitcoin/bitcoin/blob/master/test/functional/test_runner.py#L89

TEST_REGISTRY = {}

REGISTRY_LIST = [

("floresta-cli.addnode-v2", "AddnodeTestV2"),

("floresta-cli.addnode-v1", "AddnodeTestV1"),

("florestad.reorg-chain", "ChainReorgTest"),

("floresta-cli.getbestblockhash", "GetBestblockhashTest"),

("floresta-cli.getblockcount", "GetBlockCountTest"),

("floresta-cli.uptime", "UptimeTest"),

("florestad.restart", "TestRestart"),

("florestad.connect", "CliConnectTest"),

("floresta-cli.stop", "StopTest"),

("floresta-cli.ping", "PingTest"),

("floresta-cli.getrpcinfo", "GetRpcInfoTest"),

("floresta-cli.getblockhash", "GetBlockhashTest"),

("florestad.tls", "TestSslInitialization"),

("floresta-cli.getroots", "GetRootsIDBLenZeroTest"),

("floresta-cli.getblock", "GetBlockTest"),

("floresta-cli.getmemoryinfo", "GetMemoryInfoTest"),

("floresta-cli.getblockheader", "GetBlockheaderHeightZeroTest"),

("floresta-cli.getpeerinfo", "GetPeerInfoTest"),

("florestad.tls-fail", "TestSslFailInitialization"),

( "floresta-cli.getblockchaininfo", "GetBlockchaininfoTest"),

]

for (module_name, class_name) in REGISTRY_LIST:

key_template = module_name.split(".")[1]

key = key_template.replace("-", "")

TEST_REGISTRY[key] = import_test_class(module_name, class_name)

qlrd · 2025-11-26T21:27:35Z

tests/test_runner.py

+
+    # Get integration test directory
+    try:
+        temp_dir = FlorestaTestFramework.get_integration_test_dir()


No need to declare the variable temp_dir:

Suggested change

temp_dir = FlorestaTestFramework.get_integration_test_dir()

FlorestaTestFramework.get_integration_test_dir()

qlrd · 2025-11-26T21:29:33Z

tests/test_runner.py

-        default=[],
-        help="Test name to be tested in a suite. May be used more than once.",
+        "-v", "--verbose", action="store_true", help="Show verbose output on failure"
    )


This remotion will cause issues to test individually while developing

qlrd · 2025-11-26T21:34:32Z

tests/test_runner.py

+    # Handle --list-tests
+    if args.list_tests:
+        print(f"{INFO_EMOJI} Available tests:")
+        for test_name in sorted(TEST_REGISTRY.keys()):
+            print(f"  • {test_name}")
+        print(f"\nTotal: {len(TEST_REGISTRY)} tests")
+        return

-    test_dir = os.path.abspath(os.path.dirname(__file__))
+    # Determine which tests to run
+    if args.test == "all":
+        tests_to_run = TEST_REGISTRY
+        print(f"{INFO_EMOJI} Running ALL tests ({len(tests_to_run)} total)")
+    else:
+        tests_to_run = {args.test: TEST_REGISTRY[args.test]}
+        print(f"{INFO_EMOJI} Running single test: {args.test}")

-    if args.list_suites:
-        list_test_suites(test_dir)
-        return
+    # Run the tests
+    overall_start_time = time.time()

-    task_queue = setup_test_suite(args, test_dir)
-    results = run_test_workers(task_queue, args)
+    if len(tests_to_run) == 1:
+        # Single test mode - use original behavior
+        test_name = list(tests_to_run.keys())[0]
+        test_class = list(tests_to_run.values())[0]

-    passed = [(name, log, start, end) for (name, ok, log, start, end) in results if ok]
-    failed = [
-        (name, log, start, end) for (name, ok, log, start, end) in results if not ok
-    ]
+        result = run_test_direct(test_name, test_class, args.verbose)

-    print("\nTest Summary:")
-    print(f"\n{len(passed)} test(s) passed:")
-    for name, log, start, end in passed:
-        print(f"\n  {SUCCESS_EMOJI} {name}: {log} (took {end - start:.2f}s)")
+        if result.success:
+            print(f"{SUCCESS_EMOJI} {test_name} PASSED in {result.duration:.2f}s")
+            print(f"{ALLDONE_EMOJI} Test completed successfully!")
+        else:
+            print(f"{FAILURE_EMOJI} {test_name} FAILED in {result.duration:.2f}s")
+            if args.verbose and result.error_msg:
+                print(f"Error: {result.error_msg}")

-    if failed:
-        print(f"\n{len(failed)} test(s) failed:")
-        for name, log, start, end in failed:
-            print(f"\n  {FAILURE_EMOJI} {name} failed: {log} (took {end - start:.2f}s)")
-        raise SystemExit(
-            f"\n{FAILURE_EMOJI} Some tests failed. Check the logs in {args.log_dir}."
+        if not result.success:
+            sys.exit(1)
+    else:
+        # Multi-test mode
+        results, overall_duration = run_all_tests(
+            tests_to_run,
+            args.verbose,
+            continue_on_failure=not args.stop_on_failure,
+        )
+
+        # Print summary
+        all_passed = print_summary(results, overall_duration, args.verbose)
+
+        if not all_passed:
+            sys.exit(1)
+
+    overall_end_time = time.time()
+    print(f"Total runtime: {overall_end_time - overall_start_time:.2f}s")
+
+    # Run pytest after all tests complete
+    print("\n" + "=" * 60)
+    print(f"{RUNNING_EMOJI} Running pytest tests...")
+    print("=" * 60)
+
+    try:
+        result = subprocess.run(
+            ["uv", "run", "pytest", "tests/", "-n=4"], capture_output=False, text=True
+        )
+
+        if result.returncode == 0:
+            print(f"\n{SUCCESS_EMOJI} Pytest tests completed successfully!")
+        else:
+            print(
+                f"\n{FAILURE_EMOJI} Pytest tests failed with exit code: {result.returncode}"
+            )
+            sys.exit(result.returncode)
+    except FileNotFoundError:
+        print(
+            f"\n{FAILURE_EMOJI} Error: 'uv' command not found. Make sure uv is installed."
        )
-    print(f"\n{ALLDONE_EMOJI} ALL TESTS PASSED! GOOD JOB!")
+        sys.exit(1)
+    except Exception as e:
+        print(f"\n{FAILURE_EMOJI} Error running pytest: {e}")
+        sys.exit(1)


The filtering feature of individual tests shorten the development phase time. I know this will be resolved with pytest but while the old test_runner isn't fully removed i think could be good to maintain it.

jaoleal

This is some old review, feel free to just resolve them if you think they are outdated.

jaoleal · 2025-11-25T01:59:44Z

bin/florestad/src/main.rs

        log_to_file: params.log_to_file,
        assume_valid: params.assume_valid,
-        log_to_stdout: true,
+        log_to_stdout: false,


This is to avoid logging during test execution, correct ? It could be a parameter, but it seems out-of-scope for this PR.

tests/test_framework/__init__.py

jaoleal · 2025-11-25T02:24:35Z

tests/test_framework/__init__.py

-    This metaclass ensures that any subclass of `FlorestaTestFramework`
-    adheres to a standard whereby the subclass overrides `set_test_params` and
-    `run_test, but DOES NOT override `__init__` or `main`. If those standards
-    are violated, a `TypeError` is raised.


This part of the doc appears to be important, no ?

This is outdated and this file will be removed, new tests should be in pytest format that don't need this form and old ones will be refactored to pytest too so i don't think this should be mantained.

jaoleal · 2025-11-25T02:28:02Z

tests/test_framework/__init__.py


 # pylint: disable=too-many-public-methods
 class FlorestaTestFramework(metaclass=FlorestaTestMetaClass):
-    """


This whole doc explains how we expect FlorestaTestFramework to be consumed, it should be kept and apply changes where needed

tests/test_framework/__init__.py

joaozinhom · 2025-12-04T20:58:41Z

Hi @Davidson-Souza @qlrd @moisesPompilio @jaoleal — this PR currently mixes several independent changes, which makes review and merging slower and riskier. I propose splitting it into smaller, focused PRs in this order:

Fix: reading ports from logs and direct-run behavior.
Introduce: pytest for example test cases (add pytest support/examples and a minimal guide).
Migrate: convert the existing tests to pytest.
Cleanup: remove old test-framework files (including replacing test_framework/init.py with node_manager.py) after the migration is complete.

Because some files will be removed/replaced (for example test_framework/init.py → node_manager.py), adding documentation to files that are about to be replaced may not be the best use of time. If the purpose of those docs is to help new testers, I recommend encouraging new testers to implement tests directly in pytest.

jaoleal · 2025-12-07T12:03:39Z

Hi @Davidson-Souza @qlrd @moisesPompilio @jaoleal — this PR currently mixes several independent changes, which makes review and merging slower and riskier. I propose splitting it into smaller, focused PRs in this order:

Happens!!! I really got lost here.

Davidson-Souza · 2025-12-08T20:01:43Z

I think it can be done here, but with this commit structure you've mentioned. Right now this git history is kinda messed, full of "this file will be removed", for people going commit-by-commit is weird to review something and it be removed next commit.

Creates the direct run, a way to test running directly like the old test_runner.py, soon pytest will replace it, so its a temporary file.

Refact from the framework avoiding the python calling python paralelism and read the ports from log ideas, and made the nescessary configurations and fixtures for pytest in the conftest and node_manager file.

This commit add a customizable test framework (`--test-runner` for old test-framework and `--pytest` for new test-framework).

Refact the integrations tests, add example dir example/ for pytest.

feat: add arguments to tests/run.sh

Update tests/test_framework/__init__.py Co-authored-by: Shigeru Myamoto <[email protected]> Update tests/test_framework/__init__.py Co-authored-by: Shigeru Myamoto <[email protected]> Update tests/conftest.py Co-authored-by: qlrd <[email protected]> Update tests/test_framework/__init__.py Co-authored-by: Shigeru Myamoto <[email protected]>

joaozinhom · 2025-12-15T17:29:00Z

Hi @Davidson-Souza @qlrd @moisesPompilio @jaoleal — this PR currently mixes several independent changes, which makes review and merging slower and riskier. I propose splitting it into smaller, focused PRs in this order:

Fix: reading ports from logs and direct-run behavior.

Introduce: pytest for example test cases (add pytest support/examples and a minimal guide).

Migrate: convert the existing tests to pytest.

Cleanup: remove old test-framework files (including replacing test_framework/init.py with node_manager.py) after the migration is complete.

Because some files will be removed/replaced (for example test_framework/init.py → node_manager.py), adding documentation to files that are about to be replaced may not be the best use of time. If the purpose of those docs is to help new testers, I recommend encouraging new testers to implement tests directly in pytest.

#731

Davidson-Souza added code quality Generally improves code readability and maintainability functional tests CI A change or issue related to CI labels Oct 12, 2025

jaoleal mentioned this pull request Oct 13, 2025

[Enhancement] functional tests: Add a max_retry option #628

Open

jaoleal reviewed Oct 14, 2025

View reviewed changes

joaozinhom force-pushed the refact_test_runner branch 2 times, most recently from d399e3e to 7f1c9d3 Compare October 14, 2025 18:08

joaozinhom force-pushed the refact_test_runner branch from 7f1c9d3 to cdb4ba3 Compare October 15, 2025 18:41

joaozinhom requested a review from jaoleal October 15, 2025 18:47

joaozinhom force-pushed the refact_test_runner branch 3 times, most recently from c2dce29 to 8859071 Compare October 23, 2025 16:32

Davidson-Souza reviewed Oct 29, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

joaozinhom force-pushed the refact_test_runner branch from 8859071 to 42147e9 Compare November 17, 2025 17:13

joaozinhom requested a review from Davidson-Souza November 17, 2025 17:13

moisesPompilio reviewed Nov 18, 2025

View reviewed changes

joaozinhom force-pushed the refact_test_runner branch 2 times, most recently from 45c62ef to 4581127 Compare November 24, 2025 20:32

joaozinhom requested a review from moisesPompilio November 24, 2025 20:33

joaozinhom force-pushed the refact_test_runner branch from 4581127 to dca5f58 Compare November 24, 2025 23:35

moisesPompilio reviewed Nov 25, 2025

View reviewed changes

qlrd suggested changes Nov 25, 2025

View reviewed changes

joaozinhom force-pushed the refact_test_runner branch from dca5f58 to 6c2ea05 Compare November 26, 2025 12:45

qlrd reviewed Nov 26, 2025

View reviewed changes

moisesPompilio mentioned this pull request Nov 26, 2025

refactor(tests): Improve performance when obtaining bitcoind binary #716

Open

qlrd suggested changes Nov 26, 2025

View reviewed changes

joaozinhom force-pushed the refact_test_runner branch from ac759f8 to 19dcdf1 Compare November 27, 2025 16:40

jaoleal reviewed Dec 2, 2025

View reviewed changes

joaozinhom force-pushed the refact_test_runner branch 2 times, most recently from 7f1d391 to fb8a6ec Compare December 4, 2025 19:21

jaoleal mentioned this pull request Dec 7, 2025

Adjust get_block RPC to match Bitcoin Core output #682

Open

16 tasks

joaozinhom and others added 7 commits December 15, 2025 12:46

refact: reducing paralalism and stop with read ports from the logfiles

61cc91c

Creates the direct run, a way to test running directly like the old test_runner.py, soon pytest will replace it, so its a temporary file.

fix: turn off stdout

455e377

refact(test_framework) to pytest form

e51bd98

Refact from the framework avoiding the python calling python paralelism and read the ports from log ideas, and made the nescessary configurations and fixtures for pytest in the conftest and node_manager file.

feat: add arguments to tests/run.sh

0a9f646

This commit add a customizable test framework (`--test-runner` for old test-framework and `--pytest` for new test-framework).

test: refact integration tests

1e4f11b

Refact the integrations tests, add example dir example/ for pytest.

Merge pull request #2 from qlrd/feat/refact-test-runner-justfile

901f154

feat: add arguments to tests/run.sh

joaozinhom force-pushed the refact_test_runner branch from fb8a6ec to 2a2e5b5 Compare December 15, 2025 15:59

joaozinhom marked this pull request as draft December 15, 2025 17:14

moisesPompilio mentioned this pull request Dec 20, 2025

Feat(functional): Add pytest #742

Open


		# pylint: disable=redefined-outer-name # Pytest fixtures pattern

		sys.path.insert(0, os.path.dirname(__file__))

		@@ -0,0 +1,116 @@
		"""Utility functions for Floresta integration tests.

	prog="multi_test_runner", description="Run Floresta integration tests"
	prog="test_runner", description="Run Floresta integration tests"

	from typing import Any, Dict, List, Pattern
	from typing import Any, List, Pattern

	def extract_port_from_args(self, extra_args: list[str], option: str) -> int:
	def get_available_random_port(self, extra_args: list[str], option: str) -> int:

		# I disable the following because i literally will delete this file too later
		# its a middle way to the __init__.py from test framework for the pytest

	temp_dir = FlorestaTestFramework.get_integration_test_dir()
	FlorestaTestFramework.get_integration_test_dir()

Refact test runner #671

Are you sure you want to change the base?

Refact test runner #671

Uh oh!

Conversation

joaozinhom commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of this pull request?

Which crates are being modified?

Description and Notes

Summary

Motivation

Approach

Goals

Scope and collaboration

How to run

Open questions and next steps

Uh oh!

jaoleal commented Oct 14, 2025

Uh oh!

jaoleal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joaozinhom commented Oct 15, 2025

Uh oh!

Uh oh!

Davidson-Souza commented Oct 29, 2025

Uh oh!

moisesPompilio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joaozinhom commented Nov 20, 2025

Uh oh!

Davidson-Souza commented Nov 24, 2025

Uh oh!

moisesPompilio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

joaozinhom commented Oct 12, 2025 •

edited

Loading

qlrd commented Nov 26, 2025 •

edited

Loading