[FEATURE] Update `Batch.validate()` API to accept expectation parameters #10456

cdkini · 2024-10-02T15:53:55Z

Description of PR changes above includes a link to an existing GitHub issue
PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
Code is linted - run invoke lint (uses ruff format + ruff check)
Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

netlify · 2024-10-02T15:54:11Z

✅ Deploy Preview for niobium-lead-7998 ready!

Name	Link
🔨 Latest commit	`5e53490`
🔍 Latest deploy log	https://app.netlify.com/sites/niobium-lead-7998/deploys/66feb9ff5a9ad600082f9e32
😎 Deploy Preview	https://deploy-preview-10456.docs.greatexpectations.io
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

codecov · 2024-10-02T15:57:18Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
4798	2	4796	280

View the top 2 failed tests by shortest run time

tests.expectations.core.test_core_model_schemas test_all_core_model_schemas_are_serializable

Stack Traces | 0.018s run time

@pytest.mark.unit
    def test_all_core_model_schemas_are_serializable():
        all_models = [
            expectation
            for expectation in expectation_dictionary.values()
            if isinstance(expectation, MetaExpectation)
        ]
        # are they still there?
        assert len(all_models) &gt; 50
        for model in all_models:
&gt;           model.schema_json()

.../expectations/core/test_core_model_schemas.py:22: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/main.py:675: in schema_json
    cls.schema(by_alias=by_alias, ref_template=ref_template), default=pydantic_encoder, **dumps_kwargs
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/main.py:664: in schema
    s = model_schema(cls, by_alias=by_alias, ref_template=ref_template)
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:188: in model_schema
    m_schema, m_definitions, nested_models = model_process_schema(
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:581: in model_process_schema
    m_schema, m_definitions, nested_models = model_type_schema(
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:622: in model_type_schema
    f_schema, f_definitions, f_nested_models = field_schema(
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:255: in field_schema
    f_schema, f_definitions, f_nested_models = field_type_schema(
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:527: in field_type_schema
    f_schema, f_definitions, f_nested_models = field_singleton_schema(
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:852: in field_singleton_schema
    return field_singleton_sub_fields_schema(
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:747: in field_singleton_sub_fields_schema
    sub_schema, sub_definitions, sub_nested_models = field_type_schema(
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:527: in field_type_schema
    f_schema, f_definitions, f_nested_models = field_singleton_schema(
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:929: in field_singleton_schema
    sub_schema, sub_definitions, sub_nested_models = model_process_schema(
.../hostedtoolcache/Python/3.8.18..................................../x64/lib/python3.8.../pydantic/v1/schema.py:595: in model_process_schema
    schema_extra(s, model)
.../expectations/core/expect_table_row_count_to_equal.py:162: in schema_extra
    BatchExpectation.Config.schema_extra(schema, model)
great_expectations/expectations/expectation.py:1549: in schema_extra
    Expectation.Config.schema_extra(schema, model)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

schema = {'additionalProperties': False, 'properties': {'$PARAMETER': {'title': '$Parameter'}}, 'title': 'Expect table row count to equal', 'type': 'object'}
model = &lt;class 'great_expectations.core.suite_parameters.SuiteParameterDict'&gt;

    @staticmethod
    def schema_extra(schema: Dict[str, Any], model: Type[Expectation]) -&gt; None:
        schema["properties"]["metadata"] = {
            "type": "object",
            "properties": {
                "expectation_class": {
                    "title": "Expectation Class",
                    "type": "string",
                    "const": model.__name__,
                },
                "expectation_type": {
                    "title": "Expectation Type",
                    "type": "string",
&gt;                   "const": model.expectation_type,
                },
            },
        }
E       AttributeError: type object 'SuiteParameterDict' has no attribute 'expectation_type'

great_expectations/expectations/expectation.py:309: AttributeError

tests.expectations.core.test_core_model_schemas test_all_core_model_schemas_are_serializable

Stack Traces | 0.019s run time

@pytest.mark.unit
    def test_all_core_model_schemas_are_serializable():
        all_models = [
            expectation
            for expectation in expectation_dictionary.values()
            if isinstance(expectation, MetaExpectation)
        ]
        # are they still there?
        assert len(all_models) &gt; 50
        for model in all_models:
&gt;           model.schema_json()

.../expectations/core/test_core_model_schemas.py:22: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/main.py:675: in schema_json
    cls.schema(by_alias=by_alias, ref_template=ref_template), default=pydantic_encoder, **dumps_kwargs
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/main.py:664: in schema
    s = model_schema(cls, by_alias=by_alias, ref_template=ref_template)
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:188: in model_schema
    m_schema, m_definitions, nested_models = model_process_schema(
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:581: in model_process_schema
    m_schema, m_definitions, nested_models = model_type_schema(
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:622: in model_type_schema
    f_schema, f_definitions, f_nested_models = field_schema(
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:255: in field_schema
    f_schema, f_definitions, f_nested_models = field_type_schema(
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:527: in field_type_schema
    f_schema, f_definitions, f_nested_models = field_singleton_schema(
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:852: in field_singleton_schema
    return field_singleton_sub_fields_schema(
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:747: in field_singleton_sub_fields_schema
    sub_schema, sub_definitions, sub_nested_models = field_type_schema(
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:527: in field_type_schema
    f_schema, f_definitions, f_nested_models = field_singleton_schema(
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:929: in field_singleton_schema
    sub_schema, sub_definitions, sub_nested_models = model_process_schema(
.../hostedtoolcache/Python/3.11.10..................................../x64/lib/python3.11.../pydantic/v1/schema.py:595: in model_process_schema
    schema_extra(s, model)
.../expectations/core/expect_table_row_count_to_equal.py:162: in schema_extra
    BatchExpectation.Config.schema_extra(schema, model)
great_expectations/expectations/expectation.py:1549: in schema_extra
    Expectation.Config.schema_extra(schema, model)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

schema = {'additionalProperties': False, 'properties': {'$PARAMETER': {'title': '$Parameter'}}, 'title': 'Expect table row count to equal', 'type': 'object'}
model = &lt;class 'great_expectations.core.suite_parameters.SuiteParameterDict'&gt;

    @staticmethod
    def schema_extra(schema: Dict[str, Any], model: Type[Expectation]) -&gt; None:
        schema["properties"]["metadata"] = {
            "type": "object",
            "properties": {
                "expectation_class": {
                    "title": "Expectation Class",
                    "type": "string",
                    "const": model.__name__,
                },
                "expectation_type": {
                    "title": "Expectation Type",
                    "type": "string",
&gt;                   "const": model.expectation_type,
                },
            },
        }
E       AttributeError: type object 'SuiteParameterDict' has no attribute 'expectation_type'

great_expectations/expectations/expectation.py:309: AttributeError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

billdirks · 2024-10-02T22:54:22Z

great_expectations/checkpoint/checkpoint.py

@@ -60,6 +60,7 @@
 from great_expectations.render.renderer.renderer import Renderer

 if TYPE_CHECKING:
+    from great_expectations.core.suite_parameters import SuiteParameterDict


SuiteParameterDict is a dict vs a dict[str, Any]. I'm wondering if we could be more restrictive here.

billdirks

LGTM, thanks!

tyler-hoffman

Sweet!

…_expectations into f/core-314/batch_validate_suite_params

init commit

a1fc4af

cdkini added 5 commits October 2, 2024 12:15

start tests

7752543

add tests

1839536

use suiteparameterdict

85e4dc1

Merge branch 'develop' into f/core-314/batch_validate_suite_params

d4aa165

Merge branch 'develop' into f/core-314/batch_validate_suite_params

83052ec

billdirks reviewed Oct 2, 2024

View reviewed changes

billdirks approved these changes Oct 2, 2024

View reviewed changes

tyler-hoffman approved these changes Oct 3, 2024

View reviewed changes

cdkini added 2 commits October 3, 2024 10:55

update docs

26829ca

Merge branch 'develop' of https://github.com/great-expectations/great…

5e53490

…_expectations into f/core-314/batch_validate_suite_params

cdkini enabled auto-merge October 3, 2024 15:16

cdkini force-pushed the f/core-314/batch_validate_suite_params branch from b8ccd90 to 5e53490 Compare October 3, 2024 15:36

cdkini added this pull request to the merge queue Oct 3, 2024

Merged via the queue into develop with commit e3e3b32 Oct 3, 2024
116 of 120 checks passed

cdkini deleted the f/core-314/batch_validate_suite_params branch October 3, 2024 16:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Update `Batch.validate()` API to accept expectation parameters #10456

[FEATURE] Update `Batch.validate()` API to accept expectation parameters #10456

cdkini commented Oct 2, 2024

netlify bot commented Oct 2, 2024 •

edited

Loading

codecov bot commented Oct 2, 2024 •

edited

Loading

billdirks Oct 2, 2024

billdirks left a comment

tyler-hoffman left a comment

[FEATURE] Update Batch.validate() API to accept expectation parameters #10456

[FEATURE] Update Batch.validate() API to accept expectation parameters #10456

Conversation

cdkini commented Oct 2, 2024

netlify bot commented Oct 2, 2024 • edited Loading

✅ Deploy Preview for niobium-lead-7998 ready!

codecov bot commented Oct 2, 2024 • edited Loading

❌ 2 Tests Failed:

billdirks Oct 2, 2024

Choose a reason for hiding this comment

billdirks left a comment

Choose a reason for hiding this comment

tyler-hoffman left a comment

Choose a reason for hiding this comment

[FEATURE] Update `Batch.validate()` API to accept expectation parameters #10456

[FEATURE] Update `Batch.validate()` API to accept expectation parameters #10456

netlify bot commented Oct 2, 2024 •

edited

Loading

codecov bot commented Oct 2, 2024 •

edited

Loading