Skip to content

Fix the usage of Configuration class #670

Closed

Description

@vdusek

Our Configuration class currently contains many fields that are unused and/or undocumented. Review each field, decide whether it should be used, removed, or consolidated, and add missing doc strings.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Nov 8, 2024
  2. janbuchar commented on Nov 8, 2024

    @janbuchar
    Collaborator

    Thanks for bringing this up! If you could write out the attributes in question here, we can discuss the appropriate steps together.

  3. vdusek commented on Nov 8, 2024

    @vdusek
    CollaboratorAuthor

    The fields that are used:

    • internal_timeout
    • verbose_log (but it just sets the DEBUG level, and we have another field exactly for that)
    • default_dataset_id
    • default_key_value_store_id
    • default_request_queue_id
    • purge_on_start
    • write_metadata
    • persist_storage
    • memory_mbytes
    • storage_dir

    The fields that are not used:

    • default_browser_path
    • disable_browser_sandbox
    • persist_state_interval
    • system_info_interval
    • max_used_cpu_ratio
    • available_memory_ratio
    • chrome_executable_path
    • headless
    • xvfb

    The inputKey field from JS is missing completely.

  4. janbuchar commented on Nov 11, 2024

    @janbuchar
    Collaborator

    The fields that are not used:

    • default_browser_path

    In JS, this is used by PlaywrightLauncher

    • disable_browser_sandbox

    In JS, this is used by BrowserLauncher to add an argument to playwright/puppeteer

    • chrome_executable_path

    In JS, this is used by BrowserLauncher, and set in https://github.com/apify/apify-actor-docker/blob/master/node-playwright/Dockerfile or by the actor developer in environment variables

    • xvfb

    This may be set in https://github.com/apify/apify-actor-docker/blob/master/node-playwright/Dockerfile or by the actor developer in environment variables, but I don't think it actually has any effect 🤔 @vladfrangu do you know more?

    • headless
      We should probably implement this - it's convenient for local debugging
    • persist_state_interval
    • system_info_interval

    These should probably be consumed by LocalEventManager. Currently, the SDK passes that, which may not be optimal.

    • max_used_cpu_ratio
    • available_memory_ratio

    These should probably be handled by LocalEventManager, SystemInfo or Snapshotter.

    The inputKey field from JS is missing completely.

    I must have missed that one. The only use case I can think of is when you want to debug something locally against multiple inputs and don't want to bother with renaming files... which is valid

  5. vdusek commented on Nov 11, 2024

    @vdusek
    CollaboratorAuthor

    Agreed, thanks @janbuchar for the investigation. We should also note, that Configuration values should be overridable by component-specific values.

    E.g.

    config = Configuration(headless=False)
    crawler = PlaywrightCrawler(config=config, headless=True)

    then True is used for headless.

    Also, we should split this issue into many smaller ones.

  6. self-assigned this
    on Nov 22, 2024
  7. changed the title [-]Improve the Configuration class[/-] [+]Fix the usage of Configuration class[/+] on Jan 7, 2025
  8. added a commit that references this issue on Jan 15, 2025
    0f1cf6f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions