Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions docs/examples/fill-and-submit-web-form.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
---
id: fill-and-submit-web-form
title: Fill and submit web form
---

import ApiLink from '@site/src/components/ApiLink';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

This example demonstrates how to fill and submit a web form using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> crawler. The same approach applies to any crawler that inherits from the <ApiLink to="class/HttpCrawler">`HttpCrawler`</ApiLink>.

We are going to use the [httpbin.org](https://httpbin.org) website to demonstrate how it works.

## Investigate the form fields

First, we need to examine the form fields and the form's action URL. You can do this by opening the [httpbin.org/forms/post](https://httpbin.org/forms/post) page in a browser and inspecting the form fields.

In Chrome, right-click on the page and select "Inspect" or press `Ctrl+Shift+I`.
Use the element selector (`Ctrl+Shift+C`) to click on the form element you want to inspect.

![HTML input element name](/img/fill-and-submit-web-form/00.jpg 'HTML input element name.')

Identify the field names. For example, the customer name field is `custname`, the email field is `custemail`, and the phone field is `custtel`.

Now navigate to the "Network" tab in developer tools and submit the form by clicking the "Submit order" button.

![Submitting the form](/img/fill-and-submit-web-form/01.jpg 'Submitting the form.')

Find the form submission request and examine its details. The "Headers" tab will show the submission URL, in this case, it is `https://httpbin.org/post`.

![Network request investigation](/img/fill-and-submit-web-form/02.jpg 'Network request investigation.')

The "Payload" tab will display the form fields and their submitted values. This method could be an alternative to inspecting the HTML source code directly.

![Network payload investigation](/img/fill-and-submit-web-form/03.jpg 'Network payload investigation.')

## Preparing a POST request

Now, let's create a POST request with the form fields and their values using the <ApiLink to="class/Request">`Request`</ApiLink> class, specifically its <ApiLink to="class/Request#from_url">`Request.from_url`</ApiLink> constructor:

```python
from crawlee.models import Request

# Prepare a POST request to the form endpoint.
request = Request.from_url(
url='https://httpbin.org/post',
method='POST',
data={
'custname': 'John Doe',
'custtel': '1234567890',
'custemail': 'johndoe@example.com',
'size': 'large',
'topping': ['bacon', 'cheese', 'mushroom'],
'delivery': '13:00',
'comments': 'Please ring the doorbell upon arrival.',
},
)
```

Alternatively, you can send form data as URL parameters using the `query_params` argument. It depends on the form and how it is implemented. However, sending the data as a POST request body using the `data` parameter is generally a better approach.

## Implementing the crawler

Finally, let's implement the crawler and run it with the prepared request. Although we are using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>, the process is the same for any crawler that inherits from the <ApiLink to="class/HttpCrawler">`HttpCrawler`</ApiLink>.

```python
import asyncio

from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.models import Request


async def main() -> None:
crawler = BeautifulSoupCrawler()

# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
context.log.info(f'Processing {context.request.url} ...')
response = context.soup.contents
context.log.info(f'Response: {response}') # To see the response in the logs.

# Prepare a POST request to the form endpoint.
request = Request.from_url(
url='https://httpbin.org/post',
method='POST',
data={
'custname': 'John Doe',
'custtel': '1234567890',
'custemail': 'johndoe@example.com',
'size': 'large',
'topping': ['bacon', 'cheese', 'mushroom'],
'delivery': '13:00',
'comments': 'Please ring the doorbell upon arrival.',
},
)

# Run the crawler with the initial list of requests.
await crawler.run([request])


if __name__ == '__main__':
asyncio.run(main())
```

## Running the crawler

Finally, run your crawler. Your logs should show something like this:

```plaintext
[crawlee.beautifulsoup_crawler.beautifulsoup_crawler] INFO Processing https://httpbin.org/post ...
[crawlee.beautifulsoup_crawler.beautifulsoup_crawler] INFO Response: [<html><body><p>{
"args": {},
"data": "",
"files": {},
"form": {
"comments": "Please ring the doorbell upon arrival.",
"custemail": "johndoe@example.com",
"custname": "John Doe",
"custtel": "1234567890",
"delivery": "13:00",
"size": "large",
"topping": "['bacon', 'cheese', 'mushroom']"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Content-Length": "204",
"Content-Type": "application/x-www-form-urlencoded",
"Host": "httpbin.org",
"X-Amzn-Trace-Id": "Root=1-66c6fd46-1990133c0fecf44a38027b3b"
},
"json": null,
"origin": "78.80.81.196",
"url": "https://httpbin.org/post"
}
</p></body></html>]
[crawlee.autoscaling.autoscaled_pool] INFO Waiting for remaining tasks to finish
[crawlee.beautifulsoup_crawler.beautifulsoup_crawler] INFO Final request statistics:
┌───────────────────────────────┬──────────┐
│ requests_finished │ 1 │
│ requests_failed │ 0 │
│ retry_histogram │ [1] │
│ request_avg_failed_duration │ None │
│ request_avg_finished_duration │ 0.518298 │
│ requests_finished_per_minute │ 111 │
│ requests_failed_per_minute │ 0 │
│ request_total_duration │ 0.518298 │
│ requests_total │ 1 │
│ crawler_runtime │ 0.539099 │
└───────────────────────────────┴──────────┘
```

This log output confirms that the crawler successfully submitted the form and processed the response. Congratulations! You have successfully filled and submitted a web form using the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> crawler.
4 changes: 2 additions & 2 deletions docs/introduction/04-real-world-project.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -157,11 +157,11 @@ Let's open DevTools by going to https://warehouse-theme-metal.myshopify.com/coll

In the DevTools, choose the **Select an element** tool and try hovering over one of the Actor cards.

![select an element](/img/getting-started/select-an-element.png 'Finding the select an element tool.')
![select an element](/img/getting-started/select-an-element.jpg 'Finding the select an element tool.')

You'll see that you can select different elements inside the card. Instead, select the whole card, not just some of its contents, such as its title or description.

![selected element](/img/getting-started/selected-element.png 'Selecting an element by hovering over it.')
![selected element](/img/getting-started/selected-element.jpg 'Selecting an element by hovering over it.')

Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.

Expand Down
10 changes: 8 additions & 2 deletions src/crawlee/http_clients/curl_impersonate.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,9 +90,11 @@ async def crawl(

try:
response = await client.request(
method=request.method.upper(), # curl-cffi requires uppercase method
url=request.url,
method=request.method.upper(), # curl-cffi requires uppercase method
headers=request.headers,
params=request.query_params,
data=request.data,
cookies=session.cookies if session else None,
allow_redirects=True,
)
Expand Down Expand Up @@ -128,6 +130,8 @@ async def send_request(
*,
method: HttpMethod = 'GET',
headers: HttpHeaders | None = None,
query_params: dict[str, Any] | None = None,
data: dict[str, Any] | None = None,
session: Session | None = None,
proxy_info: ProxyInfo | None = None,
) -> HttpResponse:
Expand All @@ -136,9 +140,11 @@ async def send_request(

try:
response = await client.request(
method=method.upper(), # curl-cffi requires uppercase method
url=url,
method=method.upper(), # curl-cffi requires uppercase method
headers=headers,
params=query_params,
data=data,
cookies=session.cookies if session else None,
allow_redirects=True,
)
Expand Down
8 changes: 7 additions & 1 deletion src/crawlee/http_clients/httpx.py
Original file line number Diff line number Diff line change
Expand Up @@ -103,9 +103,11 @@ async def crawl(
client = self._get_client(proxy_info.url if proxy_info else None)

http_request = client.build_request(
method=request.method,
url=request.url,
method=request.method,
headers=request.headers,
params=request.query_params,
data=request.data,
cookies=session.cookies if session else None,
extensions={'crawlee_session': session if self._persist_cookies_per_session else None},
)
Expand Down Expand Up @@ -144,6 +146,8 @@ async def send_request(
*,
method: HttpMethod = 'GET',
headers: HttpHeaders | None = None,
query_params: dict[str, Any] | None = None,
data: dict[str, Any] | None = None,
session: Session | None = None,
proxy_info: ProxyInfo | None = None,
) -> HttpResponse:
Expand All @@ -153,6 +157,8 @@ async def send_request(
url=url,
method=method,
headers=headers,
params=query_params,
data=data,
extensions={'crawlee_session': session if self._persist_cookies_per_session else None},
)

Expand Down
4 changes: 4 additions & 0 deletions src/crawlee/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ class BaseRequestData(BaseModel):

headers: Annotated[dict[str, str] | None, Field(default_factory=dict)] = None

query_params: Annotated[dict[str, Any] | None, Field(default_factory=dict)] = None

data: Annotated[dict[str, Any] | None, Field(default_factory=dict)] = None

user_data: Annotated[dict[str, Any], Field(alias='userData', default_factory=dict)]
"""Custom user data assigned to the request. Use this to save any request related data to the
request's scope, keeping them accessible on retries, failures etc.
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/static/img/getting-started/current-price.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/static/img/getting-started/scraping-practice.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file modified website/static/img/getting-started/sku.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified website/static/img/getting-started/title.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.