Everything you need to integrate WebScraping.AI into your applications.
Get started with WebScraping.AI in under 5 minutes. Here's a simple example to extract data from any webpage.
Sign up at webscraping.ai to get your free API key with 2,000 credits.
Try the AI question endpoint to ask a question about any webpage:
curl -G "https://api.webscraping.ai/ai/question" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode "question=What is this page about?"
import requests
response = requests.get("https://api.webscraping.ai/ai/question", params={
"api_key": "YOUR_API_KEY",
"url": "https://example.com",
"question": "What is this page about?"
})
print(response.text)
const params = new URLSearchParams({
api_key: 'YOUR_API_KEY',
url: 'https://example.com',
question: 'What is this page about?'
});
const response = await fetch(
`https://api.webscraping.ai/ai/question?${params}`
);
const answer = await response.text();
console.log(answer);
$params = http_build_query([
'api_key' => 'YOUR_API_KEY',
'url' => 'https://example.com',
'question' => 'What is this page about?'
]);
$response = file_get_contents(
"https://api.webscraping.ai/ai/question?{$params}"
);
echo $response;
require 'net/http'
require 'uri'
params = {
api_key: 'YOUR_API_KEY',
url: 'https://example.com',
question: 'What is this page about?'
}
uri = URI('https://api.webscraping.ai/ai/question')
uri.query = URI.encode_www_form(params)
response = Net::HTTP.get(uri)
puts response
The API returns the AI-generated answer as plain text:
All API requests require an API key. Pass your key as a query parameter:
GET https://api.webscraping.ai/html?url=https://example.com&api_key=YOUR_API_KEY
All API endpoints use this base URL:
timeout parameter to control duration.| Configuration | Credits | Notes |
|---|---|---|
| Basic (no JS, datacenter proxy) | 1 | Fastest, for static sites |
| With JS rendering | 2 | Default setting, headless Chrome |
| Residential proxy (no JS) | 5 | For anti-bot protected sites |
| Residential proxy + JS | 10 | Maximum compatibility |
| AI endpoints (/ai/question, /ai/fields) | 5 | Per request, plus proxy costs |
All costs are per successful request. Failed requests are free.
Our API typically achieves 80%+ success rate for most websites. If you encounter failures:
timeout to 20000-30000ms for slow-loading websitesproxy=residential if datacenter proxies are blockedjs_timeout for pages with slow-loading dynamic contentJavaScript Rendering
JS rendering is enabled by default (js=true) using headless Chrome. Keep enabled for SPAs (React, Vue, Angular), AJAX content, and dynamic pages. Disable (js=false) for static sites, faster responses, or server-rendered content.
Proxy Strategy
Start with datacenter proxies (default) for speed and cost. Switch to residential proxies if: website blocks datacenter IPs, getting 403 errors, need to bypass anti-bot protection, or scraping geo-restricted content.
Use our official SDKs for easier integration in your preferred language.
Need an SDK for another language? Let us know!
Integrate WebScraping.AI with AI assistants and LLM platforms.
Our open-source MCP server integrates WebScraping.AI directly with AI assistants that support the Model Context Protocol:
Use our OpenAPI specification to integrate with AI tools that support API schemas, such as GPT Actions or custom agents.
Use WebScraping.AI as a proxy server for your existing tools. Route requests through our infrastructure without changing your code.
| Host | proxy.webscraping.ai |
| Port | 8888 |
| Username | Your API key |
| Password | Parameters (e.g., js=true&proxy=residential) |
curl -x "http://YOUR_API_KEY:js=true&proxy=residential@proxy.webscraping.ai:8888" \
-k "https://example.com"import requests
proxy_url = "http://YOUR_API_KEY:js=true&proxy=residential@proxy.webscraping.ai:8888"
response = requests.get(
"https://example.com",
proxies={"http": proxy_url, "https": proxy_url},
verify=False # Required for self-signed certificate
)
print(response.text)const HttpsProxyAgent = require('https-proxy-agent');
const proxyUrl = 'http://YOUR_API_KEY:js=true&proxy=residential@proxy.webscraping.ai:8888';
const agent = new HttpsProxyAgent(proxyUrl);
const response = await fetch('https://example.com', {
agent,
rejectUnauthorized: false
});
console.log(await response.text());$ch = curl_init('https://example.com');
curl_setopt($ch, CURLOPT_PROXY, 'proxy.webscraping.ai:8888');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'YOUR_API_KEY:js=true&proxy=residential');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
echo $response;require 'net/http'
proxy = Net::HTTP::Proxy('proxy.webscraping.ai', 8888, 'YOUR_API_KEY', 'js=true&proxy=residential')
uri = URI('https://example.com')
response = proxy.start(uri.host, uri.port, use_ssl: true, verify_mode: OpenSSL::SSL::VERIFY_NONE) do |http|
http.get(uri)
end
puts response.body-k flag in cURL or verify=False in Python.
Use AI to answer questions about any webpage. Perfect for extracting specific information without parsing HTML.
/ai/question
| Parameter | Type | Description |
|---|---|---|
| url required | string | URL of the webpage to analyze |
| question required | string | Question to ask about the page content |
| api_key required | string | Your API key |
curl -G "https://api.webscraping.ai/ai/question" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://news.ycombinator.com" \
--data-urlencode "question=What are the top 3 stories on this page?"import requests
response = requests.get("https://api.webscraping.ai/ai/question", params={
"api_key": "YOUR_API_KEY",
"url": "https://news.ycombinator.com",
"question": "What are the top 3 stories on this page?"
})
print(response.text)const params = new URLSearchParams({
api_key: 'YOUR_API_KEY',
url: 'https://news.ycombinator.com',
question: 'What are the top 3 stories on this page?'
});
const response = await fetch(`https://api.webscraping.ai/ai/question?${params}`);
console.log(await response.text());$params = http_build_query([
'api_key' => 'YOUR_API_KEY',
'url' => 'https://news.ycombinator.com',
'question' => 'What are the top 3 stories on this page?'
]);
$response = file_get_contents("https://api.webscraping.ai/ai/question?{$params}");
echo $response;require 'net/http'
params = { api_key: 'YOUR_API_KEY', url: 'https://news.ycombinator.com', question: 'What are the top 3 stories on this page?' }
uri = URI('https://api.webscraping.ai/ai/question')
uri.query = URI.encode_www_form(params)
puts Net::HTTP.get(uri)Extract specific data fields from any webpage as structured JSON. Ideal for scraping product details, articles, profiles, and more.
/ai/fields
| Parameter | Type | Description |
|---|---|---|
| url required | string | URL of the webpage to extract from |
| fields required | object | Object with field names as keys and extraction instructions as values |
| api_key required | string | Your API key |
curl -G "https://api.webscraping.ai/ai/fields" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://amazon.com/dp/B08N5WRWNW" \
--data-urlencode "fields[title]=Product title" \
--data-urlencode "fields[price]=Current price with currency" \
--data-urlencode "fields[rating]=Average star rating"import requests
response = requests.get("https://api.webscraping.ai/ai/fields", params={
"api_key": "YOUR_API_KEY",
"url": "https://amazon.com/dp/B08N5WRWNW",
"fields[title]": "Product title",
"fields[price]": "Current price with currency",
"fields[rating]": "Average star rating"
})
print(response.json())const params = new URLSearchParams({
api_key: 'YOUR_API_KEY',
url: 'https://amazon.com/dp/B08N5WRWNW',
'fields[title]': 'Product title',
'fields[price]': 'Current price with currency',
'fields[rating]': 'Average star rating'
});
const response = await fetch(`https://api.webscraping.ai/ai/fields?${params}`);
console.log(await response.json());$params = http_build_query([
'api_key' => 'YOUR_API_KEY',
'url' => 'https://amazon.com/dp/B08N5WRWNW',
'fields' => ['title' => 'Product title', 'price' => 'Current price with currency', 'rating' => 'Average star rating']
]);
$response = file_get_contents("https://api.webscraping.ai/ai/fields?{$params}");
print_r(json_decode($response, true));require 'net/http'
require 'json'
params = { api_key: 'YOUR_API_KEY', url: 'https://amazon.com/dp/B08N5WRWNW', 'fields[title]' => 'Product title', 'fields[price]' => 'Current price with currency' }
uri = URI('https://api.webscraping.ai/ai/fields')
uri.query = URI.encode_www_form(params)
puts JSON.parse(Net::HTTP.get(uri))Fetch the full HTML content of any webpage. Includes JavaScript rendering via headless Chrome and automatic proxy rotation.
/html
| Parameter | Type | Description |
|---|---|---|
| url required | string | URL of the webpage to fetch |
| api_key required | string | Your API key |
| js optional | boolean | Enable JavaScript rendering (default: true) |
| proxy optional | string | Proxy type: "datacenter" (default) or "residential" |
curl -G "https://api.webscraping.ai/html" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode "js=true"import requests
response = requests.get("https://api.webscraping.ai/html", params={
"api_key": "YOUR_API_KEY",
"url": "https://example.com",
"js": "true"
})
print(response.text)const params = new URLSearchParams({
api_key: 'YOUR_API_KEY',
url: 'https://example.com',
js: 'true'
});
const response = await fetch(`https://api.webscraping.ai/html?${params}`);
console.log(await response.text());$params = http_build_query([
'api_key' => 'YOUR_API_KEY',
'url' => 'https://example.com',
'js' => 'true'
]);
$html = file_get_contents("https://api.webscraping.ai/html?{$params}");
echo $html;require 'net/http'
params = { api_key: 'YOUR_API_KEY', url: 'https://example.com', js: 'true' }
uri = URI('https://api.webscraping.ai/html')
uri.query = URI.encode_www_form(params)
puts Net::HTTP.get(uri)/html
Use POST requests to send data to the target page (e.g., form submissions, API calls).
| body optional | string | Request body to send to the target URL |
# Submit form data to a target page
curl -X POST "https://api.webscraping.ai/html" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "api_key=YOUR_API_KEY" \
-d "url=https://httpbin.org/post" \
-d "body=username=test&password=demo"import requests
# Submit form data to a target page
response = requests.post("https://api.webscraping.ai/html", data={
"api_key": "YOUR_API_KEY",
"url": "https://httpbin.org/post",
"body": "username=test&password=demo"
})
print(response.text)const response = await fetch('https://api.webscraping.ai/html', {
method: 'POST',
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
body: new URLSearchParams({
api_key: 'YOUR_API_KEY',
url: 'https://httpbin.org/post',
body: 'username=test&password=demo'
})
});
console.log(await response.text());Extract only the visible text content from a webpage. Perfect for feeding content to LLMs or text analysis.
/text
| Parameter | Type | Description |
|---|---|---|
| url required | string | URL of the webpage |
| text_format optional | string | "plain" (default), "json", or "xml" |
| return_links optional | boolean | Include links in JSON response (default: false) |
curl -G "https://api.webscraping.ai/text" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode "text_format=json"import requests
response = requests.get("https://api.webscraping.ai/text", params={
"api_key": "YOUR_API_KEY",
"url": "https://example.com",
"text_format": "json"
})
print(response.json())const params = new URLSearchParams({
api_key: 'YOUR_API_KEY',
url: 'https://example.com',
text_format: 'json'
});
const response = await fetch(`https://api.webscraping.ai/text?${params}`);
console.log(await response.json());$params = http_build_query([
'api_key' => 'YOUR_API_KEY',
'url' => 'https://example.com',
'text_format' => 'json'
]);
$response = file_get_contents("https://api.webscraping.ai/text?{$params}");
print_r(json_decode($response, true));require 'net/http'
require 'json'
params = { api_key: 'YOUR_API_KEY', url: 'https://example.com', text_format: 'json' }
uri = URI('https://api.webscraping.ai/text')
uri.query = URI.encode_www_form(params)
puts JSON.parse(Net::HTTP.get(uri))Extract HTML from specific page elements using CSS selectors. Useful when you only need a portion of the page.
/selected
| Parameter | Type | Description |
|---|---|---|
| url required | string | URL of the webpage |
| selector required | string | CSS selector (e.g., "h1", ".price", "#main") |
curl -G "https://api.webscraping.ai/selected" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode "selector=h1"import requests
response = requests.get("https://api.webscraping.ai/selected", params={
"api_key": "YOUR_API_KEY",
"url": "https://example.com",
"selector": "h1"
})
print(response.text)const params = new URLSearchParams({
api_key: 'YOUR_API_KEY',
url: 'https://example.com',
selector: 'h1'
});
const response = await fetch(`https://api.webscraping.ai/selected?${params}`);
console.log(await response.text());$params = http_build_query([
'api_key' => 'YOUR_API_KEY',
'url' => 'https://example.com',
'selector' => 'h1'
]);
$html = file_get_contents("https://api.webscraping.ai/selected?{$params}");
echo $html;require 'net/http'
params = { api_key: 'YOUR_API_KEY', url: 'https://example.com', selector: 'h1' }
uri = URI('https://api.webscraping.ai/selected')
uri.query = URI.encode_www_form(params)
puts Net::HTTP.get(uri)/selected-multiple endpoint with a selectors[] array parameter to extract multiple elements in one request.
Check your remaining API credits and account status.
/account
curl "https://api.webscraping.ai/account?api_key=YOUR_API_KEY"import requests
response = requests.get("https://api.webscraping.ai/account", params={
"api_key": "YOUR_API_KEY"
})
print(response.json())const response = await fetch(
'https://api.webscraping.ai/account?api_key=YOUR_API_KEY'
);
console.log(await response.json());$response = file_get_contents(
"https://api.webscraping.ai/account?api_key=YOUR_API_KEY"
);
print_r(json_decode($response, true));require 'net/http'
require 'json'
uri = URI('https://api.webscraping.ai/account?api_key=YOUR_API_KEY')
puts JSON.parse(Net::HTTP.get(uri))Every successful API response includes helpful headers with request metadata.
| Header | Description | Example |
|---|---|---|
X-Credits-Used |
Number of credits consumed by this request | 2 |
X-Credits-Remaining |
Your remaining credit balance | 1850 |
X-Target-Status |
HTTP status code from the target website | 200 |
X-Target-Url |
Final URL after redirects (if any) | https://example.com/page |
Content-Type |
Response content type | text/html; charset=utf-8 |
# Use -i flag to include headers in output
curl -i "https://api.webscraping.ai/html?api_key=YOUR_API_KEY&url=https://example.com"
# Response includes:
# X-Credits-Used: 2
# X-Credits-Remaining: 1850
# X-Target-Status: 200import requests
response = requests.get("https://api.webscraping.ai/html", params={
"api_key": "YOUR_API_KEY",
"url": "https://example.com"
})
# Access response headers
credits_used = response.headers.get('X-Credits-Used')
credits_remaining = response.headers.get('X-Credits-Remaining')
target_status = response.headers.get('X-Target-Status')
print(f"Credits used: {credits_used}")
print(f"Credits remaining: {credits_remaining}")const response = await fetch(
'https://api.webscraping.ai/html?api_key=YOUR_API_KEY&url=https://example.com'
);
// Access response headers
const creditsUsed = response.headers.get('X-Credits-Used');
const creditsRemaining = response.headers.get('X-Credits-Remaining');
const targetStatus = response.headers.get('X-Target-Status');
console.log(`Credits used: ${creditsUsed}`);X-Credits-Remaining to monitor your quota and implement alerts when credits are running low.
Understanding API error responses and how to handle them.
| Code | Description | Solution |
|---|---|---|
| 400 | Invalid parameters | Check parameter values and format |
| 402 | Insufficient credits | Upgrade your plan or wait for credit reset |
| 403 | Invalid API key | Verify your API key is correct |
| 429 | Too many concurrent requests | Reduce request rate or upgrade for higher concurrency |
| 500 | Target website error | Try again or check if target site is available |
| 504 | Request timeout | Increase timeout parameter value |
Complete documentation for all API parameters. Click on any parameter to see detailed information and examples.
| Parameter | Type | Default | Description |
|---|---|---|---|
| url | string | required | Target webpage URL |
| api_key | string | required | Your API key for authentication |
| js | boolean | true | Enable JavaScript rendering |
| js_timeout | integer | 2000 | JavaScript rendering timeout (ms) |
| timeout | integer | 10000 | Total request timeout (ms) |
| wait_for | string | - | CSS selector to wait for |
| proxy | string | datacenter | Proxy type |
| country | string | us | Proxy country |
| device | string | desktop | Device emulation |
| headers | object | - | Custom HTTP headers |
| js_script | string | - | Custom JavaScript to execute |
| custom_proxy | string | - | Your own proxy URL |
| error_on_404 | boolean | false | Return error for 404 pages |
| error_on_redirect | boolean | false | Return error on redirects |
The URL of the target webpage to scrape or analyze. Must be a valid HTTP or HTTPS URL.
Detailsurl=https://example.com/page?id=123
Your unique API key for authentication. Get your key from the dashboard.
Enable JavaScript rendering using a headless Chromium browser. Required for SPAs, dynamic content, and modern web applications.
When to use js=true# With JS rendering (default)
curl "https://api.webscraping.ai/html?api_key=KEY&url=https://spa-app.com&js=true"
# Without JS rendering (faster, cheaper)
curl "https://api.webscraping.ai/html?api_key=KEY&url=https://static-site.com&js=false"
Maximum time in milliseconds to wait for JavaScript execution after the page loads. Increase this value if you see loading indicators instead of actual content.
Detailsjs=truewait_for for more precisionMaximum total time in milliseconds for the entire request, including page retrieval, JavaScript rendering, and processing.
Detailstimeout=15000
Sets a 15-second timeout for slow-loading pages
CSS selector to wait for before returning the page content. The request will wait until this element appears in the DOM, then return the content. This overrides js_timeout.
wait_for=.product-pricewait_for=#search-resultswait_for=[data-loaded="true"]wait_for=.reviews-container# Wait for product grid to load before scraping
curl -G "https://api.webscraping.ai/html" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://shop.com/products" \
--data-urlencode "wait_for=.product-grid"
Type of proxy to use for the request. Choose based on your target website's anti-bot measures.
| Value | Description | Best for | Cost |
|---|---|---|---|
datacenter |
Fast datacenter proxies with rotating IPs | Most websites, APIs, general scraping | 1 credit |
residential |
Real residential IPs from ISPs | Anti-bot protected sites, sneaker sites, social media | 5 credits |
Country code for geo-targeting. The request will be made from a proxy in the specified country.
| Code | Country | Code | Country |
|---|---|---|---|
us |
United States | ru |
Russia |
gb |
United Kingdom | jp |
Japan |
de |
Germany | kr |
South Korea |
fr |
France | in |
India |
ca |
Canada | it |
Italy |
es |
Spain |
# Scrape from a UK IP address
curl "https://api.webscraping.ai/html?api_key=KEY&url=https://uk-shop.com&country=gb"
Device type emulation. Affects viewport size, user agent, and touch capabilities.
| Value | Viewport | Use case |
|---|---|---|
desktop |
1920x1080 | Desktop websites, full layouts |
mobile |
375x812 (iPhone X) | Mobile sites, responsive layouts, AMP pages |
tablet |
768x1024 (iPad) | Tablet-optimized layouts |
Custom HTTP headers to send with the request. Useful for authentication, cookies, or custom user agents.
Format optionsheaders={"Cookie":"session=abc"}headers[Cookie]=session=abcCookie - Session cookiesAuthorization - Bearer tokensReferer - Referrer URL# Pass custom headers as JSON
curl -G "https://api.webscraping.ai/html" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode 'headers={"Cookie":"session=abc123","Authorization":"Bearer token"}'
Custom JavaScript code to execute on the page after it loads. Useful for clicking buttons, scrolling, filling forms, or extracting data.
Use casesreturn_script_result=true to get the return value of your script instead of the page HTML.
# Click a button
js_script=document.querySelector('.load-more-btn').click()
# Scroll to bottom
js_script=window.scrollTo(0, document.body.scrollHeight)
# Extract data and return it
js_script=return JSON.stringify(window.__INITIAL_DATA__)
Use your own proxy server instead of our built-in proxy pool. Useful if you have specific proxy requirements or existing proxy subscriptions.
Formathttp://username:password@host:port
Example
custom_proxy=http://user:pass@proxy.example.com:8080
Return an error response when the target page returns a 404 status code, instead of returning the 404 page content.
Return an error when the target page redirects, instead of following the redirect.
For complete API specification including all endpoints, parameters, and response schemas, see our OpenAPI documentation.
View OpenAPI Reference Download OpenAPI Spec