Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Improve context.data_sources autocomplete #10447

Merged
merged 3 commits into from
Oct 1, 2024

Conversation

tyler-hoffman
Copy link
Contributor

@tyler-hoffman tyler-hoffman commented Oct 1, 2024

Adds method signatures to sources.pyi for some methods we missed.

Here's how to find the missing methods:

import ast
from pathlib import Path
from typing import Set, Tuple


def extract_class_methods(filename: Path, include_private: bool = False) -> Set[Tuple[str, str]]:
    with open(filename) as file:
        tree = ast.parse(file.read())

    class_methods: Set[Tuple[str, str]] = set()

    for node in ast.walk(tree):
        if isinstance(node, ast.ClassDef):
            methods = {n for n in node.body if isinstance(n, ast.FunctionDef)}
            if not include_private:
                methods = {m for m in methods if not m.name.startswith("_")}

            for method_name in {m.name for m in methods}:
                class_methods.add((node.name, method_name))

    return class_methods


def diff_modules(superset_file: Path, subset_file: Path, include_private: bool = False):
    superset_class_methods = extract_class_methods(superset_file, include_private=include_private)
    subset_class_methods = extract_class_methods(subset_file, include_private=include_private)

    missing_defs = superset_class_methods - subset_class_methods

    return sorted(missing_defs)


if __name__ == "__main__":
    py_file = Path("great_expectations/datasource/fluent/sources.py")
    pyi_file = Path("great_expectations/datasource/fluent/sources.pyi")

    missing_defs = diff_modules(py_file, pyi_file)

    for class_name, method_name in missing_defs:
        print(f"{class_name}.{method_name}")

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

For more information about contributing, see Contribute.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

Copy link

netlify bot commented Oct 1, 2024

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit c945314
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/66fbf0316f08ef000887ddf1

Copy link

codecov bot commented Oct 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.97%. Comparing base (4680ce2) to head (c945314).
Report is 2 commits behind head on develop.

✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop   #10447   +/-   ##
========================================
  Coverage    79.97%   79.97%           
========================================
  Files          460      460           
  Lines        40006    40006           
========================================
  Hits         31996    31996           
  Misses        8010     8010           
Flag Coverage Δ
3.10 67.58% <ø> (ø)
3.10 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.10 aws_deps ?
3.10 big ?
3.10 filesystem ?
3.10 mssql ?
3.10 mysql ?
3.10 postgresql ?
3.10 spark_connect ?
3.11 67.58% <ø> (-0.02%) ⬇️
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.11 aws_deps ?
3.11 big ?
3.11 filesystem ?
3.11 mssql ?
3.11 mysql ?
3.11 postgresql ?
3.11 spark_connect ?
3.11 trino ?
3.12 66.21% <ø> (-0.02%) ⬇️
3.12 aws_deps 45.91% <ø> (ø)
3.12 big 54.53% <ø> (ø)
3.12 filesystem 61.31% <ø> (ø)
3.12 mssql 50.00% <ø> (ø)
3.12 mysql 50.06% <ø> (ø)
3.12 postgresql 54.31% <ø> (ø)
3.12 spark 57.83% <ø> (ø)
3.12 spark_connect 46.20% <ø> (ø)
3.12 trino 52.45% <ø> (ø)
3.8 67.64% <ø> (+0.01%) ⬆️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 55.12% <ø> (ø)
3.8 aws_deps 45.93% <ø> (ø)
3.8 big 54.55% <ø> (ø)
3.8 databricks 47.64% <ø> (ø)
3.8 filesystem 61.32% <ø> (ø)
3.8 mssql 49.98% <ø> (ø)
3.8 mysql 50.05% <ø> (ø)
3.8 postgresql 54.30% <ø> (ø)
3.8 snowflake 48.49% <ø> (ø)
3.8 spark 57.80% <ø> (ø)
3.8 spark_connect 46.22% <ø> (ø)
3.8 trino 52.44% <ø> (ø)
3.9 67.61% <ø> (-0.02%) ⬇️
3.9 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.9 aws_deps ?
3.9 big ?
3.9 filesystem ?
3.9 mssql ?
3.9 mysql ?
3.9 postgresql ?
3.9 spark_connect ?
cloud 0.00% <ø> (ø)
docs-basic 52.51% <ø> (ø)
docs-creds-needed 52.74% <ø> (ø)
docs-spark 52.14% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tyler-hoffman tyler-hoffman requested a review from a team October 1, 2024 13:35
@tyler-hoffman tyler-hoffman added this pull request to the merge queue Oct 1, 2024
Merged via the queue into develop with commit 90ab234 Oct 1, 2024
71 checks passed
@tyler-hoffman tyler-hoffman deleted the m/_/improve_sources_pyi_file branch October 1, 2024 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants