Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] Limit Result Format and QueryMetricProvider total unexpected records #10432

Merged
merged 35 commits into from
Oct 2, 2024

Conversation

NathanFarmer
Copy link
Contributor

@NathanFarmer NathanFarmer commented Sep 20, 2024

  • Limit the number of records allowed in Result Format and QueryMetricProvider unexpected records
  • Update docs to indicate that you can only get 200 unexpected rows
  • Refactor common logic from sqlalchemy query metrics into QueryMetricProvider
  • Annotate return types on all auxiliary methods
  • Remove unused code

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

@NathanFarmer NathanFarmer self-assigned this Sep 20, 2024
Copy link

netlify bot commented Sep 20, 2024

Deploy Preview for niobium-lead-7998 ready!

Name Link
🔨 Latest commit cc9c5c5
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/66fc5fec4de777000806d888
😎 Deploy Preview https://deploy-preview-10432.docs.greatexpectations.io
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link

codecov bot commented Sep 20, 2024

Codecov Report

Attention: Patch coverage is 78.50467% with 23 lines in your changes missing coverage. Please review.

Project coverage is 80.07%. Comparing base (69709c0) to head (cc9c5c5).
Report is 4 commits behind head on develop.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...expectations/metrics/query_metrics/query_column.py 22.22% 7 Missing ⚠️
...tations/metrics/query_metrics/query_column_pair.py 14.28% 6 Missing ⚠️
...ions/expectations/metrics/query_metric_provider.py 90.69% 4 Missing ⚠️
...ns/metrics/query_metrics/query_multiple_columns.py 33.33% 4 Missing ⚠️
...etric_provider/map_condition_auxilliary_methods.py 88.88% 1 Missing ⚠️
.../expectations/metrics/query_metrics/query_table.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop   #10432      +/-   ##
===========================================
+ Coverage    79.97%   80.07%   +0.10%     
===========================================
  Files          461      461              
  Lines        40000    39993       -7     
===========================================
+ Hits         31988    32023      +35     
+ Misses        8012     7970      -42     
Flag Coverage Δ
3.10 67.67% <48.59%> (+0.09%) ⬆️
3.10 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.10 aws_deps ?
3.10 big ?
3.10 filesystem ?
3.10 mssql ?
3.10 mysql ?
3.10 postgresql ?
3.10 spark_connect ?
3.10 trino ?
3.11 67.68% <48.59%> (+0.11%) ⬆️
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.11 aws_deps ?
3.11 big ?
3.11 filesystem ?
3.11 mssql ?
3.11 mysql ?
3.11 postgresql ?
3.11 spark_connect ?
3.11 trino ?
3.12 66.30% <48.59%> (+0.09%) ⬆️
3.12 aws_deps 45.93% <20.56%> (+0.03%) ⬆️
3.12 big 54.56% <28.97%> (+0.04%) ⬆️
3.12 filesystem 61.34% <28.97%> (+0.04%) ⬆️
3.12 mssql 50.03% <37.38%> (+0.04%) ⬆️
3.12 mysql 50.10% <37.38%> (+0.04%) ⬆️
3.12 postgresql 54.36% <37.38%> (+0.04%) ⬆️
3.12 spark 57.88% <48.59%> (+0.05%) ⬆️
3.12 spark_connect 46.23% <23.36%> (+0.03%) ⬆️
3.12 trino 52.49% <37.38%> (+0.04%) ⬆️
3.8 67.71% <48.59%> (+0.08%) ⬆️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 55.17% <42.05%> (+0.05%) ⬆️
3.8 aws_deps 45.95% <20.56%> (+0.03%) ⬆️
3.8 big 54.58% <28.97%> (+0.04%) ⬆️
3.8 databricks 47.66% <21.49%> (+0.03%) ⬆️
3.8 filesystem 61.36% <28.97%> (+0.04%) ⬆️
3.8 mssql 50.02% <37.38%> (+0.04%) ⬆️
3.8 mysql 50.09% <37.38%> (+0.04%) ⬆️
3.8 postgresql 54.35% <37.38%> (+0.04%) ⬆️
3.8 snowflake 48.52% <21.49%> (+0.03%) ⬆️
3.8 spark 57.85% <48.59%> (+0.05%) ⬆️
3.8 spark_connect 46.24% <23.36%> (+0.03%) ⬆️
3.8 trino 52.48% <37.38%> (+0.04%) ⬆️
3.9 67.71% <48.59%> (+0.09%) ⬆️
3.9 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.9 aws_deps ?
3.9 big ?
3.9 filesystem ?
3.9 mssql ?
3.9 mysql ?
3.9 postgresql ?
3.9 spark_connect ?
3.9 trino ?
cloud 0.00% <0.00%> (ø)
docs-basic 52.56% <39.25%> (+0.04%) ⬆️
docs-creds-needed 52.78% <37.38%> (+0.04%) ⬆️
docs-spark 52.18% <36.44%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@NathanFarmer NathanFarmer changed the title [BUGFIX] UnexpectedRowsQuery limit metric result set [BUGFIX] Limit Result Format and QueryMetricProvider total unexpected records Oct 1, 2024
@NathanFarmer NathanFarmer changed the title [BUGFIX] Limit Result Format and QueryMetricProvider total unexpected records [MAINTENANCE] Limit Result Format and QueryMetricProvider total unexpected records Oct 1, 2024
@NathanFarmer NathanFarmer marked this pull request as ready for review October 1, 2024 16:39
Comment on lines -12 to -14
:::note
`include_unexpected_rows` returns EVERY row for each unexpected value. In large tables, this could result in an unmanageable amount of data.
:::
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All I did was remove this note. Not sure why the diffs are hard to read.

@NathanFarmer NathanFarmer added this pull request to the merge queue Oct 2, 2024
Merged via the queue into develop with commit 09bdbcc Oct 2, 2024
69 checks passed
@NathanFarmer NathanFarmer deleted the b/ph-1599/unexpected-rows-query-oom branch October 2, 2024 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants