Skip to content

Refactor handling of ignored_http_status_codes and SessionError #708

Closed

Description

@janbuchar

The reasoning behind the change was that some errors (such as 401) are automatically considered SessionError and http-based crawlers don't take ignore_htttp_error_status_codes into account for them. While uncommon, it should be possible to explicitly ignore any status code.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    debtCode quality improvement or decrease of technical debt.
    on Nov 18, 2024
  2. vdusek commented on Nov 19, 2024

    @vdusek
    Collaborator

    rather then

    if (
        context.session
        and status_code not in self._http_client._ignore_http_error_status_codes  # noqa: SLF001
        and context.session.is_blocked_status_code(status_code=status_code)
    ):

    use something like

    if context.session and context.session.is_blocked_status_code(
        status_code=status_code,
        additional_blocked_status_codes=self._http_client.additional_blocked_status_codes,
        ignore_http_error_status_codes=self._http_client.ignore_http_error_status_codes,
    ):

    or come up with something better

  3. added a commit that references this issue on Dec 18, 2024
    8daf4bd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    debtCode quality improvement or decrease of technical debt.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions