Skip to content

feat: support custom profile in playwright - #410

Closed
sherpya wants to merge 1 commit into
apify:masterfrom
sherpya:master
Closed

sherpya wants to merge 1 commit into
apify:masterfrom
sherpya:master

Conversation

@sherpya

@sherpya sherpya commented Aug 8, 2024 •

Copy link
Copy Markdown

if user_data_dir option is found in browser_option, then the launch function used is launch_persistent_context instead of launch and the user_data_dir option is passed to playwright

Description

it makes possible to use custom user provided profile directory with playwright
without changing the api

Issues

Testing

def create_crawler(profile: Path):
    browser_options = {
        'user_data_dir': profile
    }

    plugin = PlaywrightBrowserPlugin(browser_type='chromium', browser_options=browser_options)
    browser_pool = BrowserPool(plugins=[plugin])
    crawler = PlaywrightCrawler(browser_pool=browser_pool)

    return crawler

Checklist

  • CI passed

@sherpya

sherpya commented Aug 8, 2024 •

Copy link
Copy Markdown
Author

it would be better to add a Literal check for the browser with typing.get_args, but this would unify the typevar used in __init__ so I think it's better to have a new PR

Update: ok, I don't like the idea of leaving this commit with an arbitrary getattr, so I added a tuple check for now

@janbuchar

Copy link
Copy Markdown
Collaborator

Hello @sherpya and thanks for the contribution! We should have time to review it next week.

@janbuchar
janbuchar requested a review from vdusek August 12, 2024 15:40
@fnesveda fnesveda added the t-tooling Issues with this label are in the ownership of the tooling team. label Aug 14, 2024

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems good, thanks! Could you please also add a test covering launch_persistent_context case?

@sherpya

sherpya commented Aug 22, 2024

Copy link
Copy Markdown
Author

Hi, I can make requested changes, perhaps I have no idea how to make a test for a similiar feature, do I just need to see if the call work? Or I may check if somehow the profile is created in the given directory

@janbuchar

Copy link
Copy Markdown
Collaborator

Hi, I can make requested changes, perhaps I have no idea how to make a test for a similiar feature, do I just need to see if the call work? Or I may check if somehow the profile is created in the given directory

The bare minimum would be to just run it and make sure it doesn't crash 🙂 But if you could make a profile directory somewhere and find a way to check that it is actually being used by the headless browser, that would be awesome!

@sherpya

sherpya commented Aug 23, 2024 •

Copy link
Copy Markdown
Author

Hi, I've got a problem, launch with profile returns a BrowserContext instead of a Browser and browser_controller._browser is <BrowserContext browser=None>, browser_controller.is_browser_connected fails because of missing property in _browser

I've updated the PR but commented out the is_connected check for now

if user_data_dir option is found in browser_option,
then the launch function used is launch_persistent_context
instead of launch and the user_data_dir option is passed to playwright
@sherpya

sherpya commented Aug 24, 2024

Copy link
Copy Markdown
Author

I've tried with something like (also with assignment), (not proud of):

        if isinstance(browser, BrowserContext):
            setattr(browser, 'is_connected', lambda: not browser._impl_obj._close_was_called)  # noqa: SLF001, B010

but mypy says:

assert not browser_controller.is_browser_connected

is unreachable

I think it should be implemented upstream, but profile dir support would be very nice to have before (in some way)

@vdusek

vdusek commented Aug 27, 2024

Copy link
Copy Markdown
Collaborator

The fact that launch_persistent_context returns a BrowserContext while the launch returns a Browser is a problem and our current abstraction is not designed to handle this difference. Addressing it would require deeper intervention into it, which I'm hesitant about.

Isn't there a way to extract a Browser instance from a BrowserContext? Alternatively, isn't there another approach to using a custom profile while still obtaining the Browser instance? I would rather go this way.

@sherpya

sherpya commented Aug 30, 2024

Copy link
Copy Markdown
Author

I'll close for now, I will open a new PR if I get a decent way to implement it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for launching Pw with user provided profile

4 participants