Skip to content

Mask Playwright's "headless" headers #401

Closed

Description

@vdusek

Description

  • When running headless browsers with Playwright, certain HTTP headers (like User-Agent and Sec-Ch-Ua) reveal the browser is operating in headless mode by including the substring "headless". This makes it easy for anti-scraping systems to detect and block automated browsers.
  • To prevent detection, these headers should be replaced with more realistic, browser-like values that mimic typical, non-headless browser behavior.
  • Example of header values for headless Chromium:
{
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "Accept-Encoding": "gzip, deflate, br, zstd",
    "Host": "httpbin.org",
    "Priority": "u=0, i",
    "Sec-Ch-Ua": "\"Chromium\";v=\"128\", \"Not;A=Brand\";v=\"24\", \"HeadlessChrome\";v=\"128\"",
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": "\"Linux\"",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/128.0.6613.18 Safari/537.36",
    "X-Amzn-Trace-Id": "Root=1-66d04117-141b301674c02e4e2136f1f1"
}

Relevant links

Activity

  1. added
    enhancementNew feature or request.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Aug 6, 2024
  2. self-assigned this
    on Aug 26, 2024
  3. changed the title [-]Add fingerprint injector for Playwright crawler[/-] [+]Add fingerprint injector and integrate it into Playwright crawler[/+] on Sep 17, 2024
  4. added 5 commits that reference this issue on Sep 17, 2024
    ff85a62
    77f1c5a
    ab8bd22
    ccd37e6
    421d710
  5. added 4 commits that reference this issue on Sep 23, 2024
    70ade19
    4e6e1a6
    e043b5c
    81a2cca
  6. changed the title [-]Add fingerprint injector and integrate it into Playwright crawler[/-] [+]Mask Playwright's "headless" headers[/+] on Sep 26, 2024
  7. added 2 commits that reference this issue on Sep 26, 2024
    e0730fd
    68fc490
  8. added a commit that references this issue on Sep 30, 2024
    d1445e4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions