Skip to content

Conversation

@pdobacz
Copy link
Member

@pdobacz pdobacz commented Oct 7, 2025

Tackles the point (3.) from github.com/ethereum/execution-spec-tests/pull/2243 (consume direct using evmone in EEST).

Here we switch statetest and blockchaintest runners to discover gtests by parsing the JSON files and recognizing each test.

The downside is that the invocation we use in CI (just ...evmone-statetest ... fixtures/state_tests) became very chatty, so --gtest_brief=1 was added. There is an option to only go inside the JSONs when a single .json file path is given (as consume direct does), but this would complicate things quite a bit, requiring us to have more code, but not terribly much more.

Makes consume direct for evmone version in PR mentioned 👇 work correctly

@codecov
Copy link

codecov bot commented Oct 7, 2025

Codecov Report

❌ Patch coverage is 92.06349% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.09%. Comparing base (0591d81) to head (70f14a6).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
test/blockchaintest/blockchaintest.cpp 87.87% 2 Missing and 2 partials ⚠️
test/statetest/statetest.cpp 96.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1331      +/-   ##
==========================================
+ Coverage   87.07%   87.09%   +0.02%     
==========================================
  Files         169      169              
  Lines       24820    24864      +44     
  Branches     4076     4081       +5     
==========================================
+ Hits        21611    21656      +45     
+ Misses        548      546       -2     
- Partials     2661     2662       +1     
Flag Coverage Δ
eest-develop 84.17% <ø> (ø)
eest-develop-gmp 15.83% <19.04%> (-0.02%) ⬇️
eest-legacy 11.10% <15.87%> (-0.01%) ⬇️
eest-legacy-silkpre 17.56% <20.63%> (-0.02%) ⬇️
evmone-unittests 83.89% <77.77%> (+0.19%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
core 93.55% <ø> (ø)
tooling 86.50% <92.06%> (+0.36%) ⬆️
tests 84.13% <ø> (ø)
Files with missing lines Coverage Δ
test/statetest/statetest.cpp 92.85% <96.66%> (-2.02%) ⬇️
test/blockchaintest/blockchaintest.cpp 87.05% <87.87%> (+8.37%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pdobacz
Copy link
Member Author

pdobacz commented Oct 8, 2025

It seems that discovering tests individually has more problems than I thought initially, if you apply it always. Problems which arise:

  1. When calling with a dir with many tests it becomes quite unresponsive, as it discovers the tests for a long time before giving any output
  2. It becomes problematic to filter out tests

Would it work for us if discover was done as it used to be (per file) when provided with a dir path (e.g. like we do in our CI) and discover per test when provided with a single .json file path (e.g. like what consume direct uses)? @chfast

EDIT: We seem to have decided it would work, this PR takes this approach

@pdobacz pdobacz force-pushed the granular-test branch 4 times, most recently from 97e3872 to b8468ed Compare October 8, 2025 15:44
@pdobacz pdobacz marked this pull request as ready for review October 8, 2025 15:45
@pdobacz pdobacz requested a review from chfast October 8, 2025 15:45
@pdobacz pdobacz requested a review from chfast October 9, 2025 12:37
@pdobacz pdobacz enabled auto-merge (squash) October 9, 2025 12:52
@pdobacz pdobacz merged commit 89bfa23 into master Oct 9, 2025
21 checks passed
@pdobacz pdobacz deleted the granular-test branch October 9, 2025 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants