Skip to content

Latest commit

 

History

History

python

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 

README.md

serpscraper: Official Python SDK for SerpScraper.Dev (Google SERP API, AI Overview & Trends)

SerpScraper.Dev Logo

Official Python client library for SerpScraper.Dev — a fast SERP API for real-time Google search rankings, Google AI Overviews, and Google Trends.

PyPI Version Python 3.8+ Zero Dependencies MIT License Free API Key


⚡ Why Use serpscraper for Python?

  • 🐍 Zero External Dependencies: Built 100% on Python standard library (urllib.request, json). Zero conflict with requests, httpx, or aiohttp versions in your virtual environment.
  • 🤖 Native Google AI Overview Extraction: Directly inspect synthesized AI answers, citation cards, and markdown content via the Google AI Overview API with results.summary and results.sources.
  • 📈 Real-Time Google Trends: Monitor live trending searches and historical search interest with the Google Trends API.
  • 🧠 AI & LLM Ready: Perfect for LangChain, LlamaIndex, AutoGen, and CrewAI web-search grounding and RAG pipelines.
  • 🌍 Global Geolocation Targeting: Target searches from 190+ countries (gl) and all languages (hl).
  • 🛡️ Bypass Bot Protection: Rotating residential IP pool completely bypasses Cloudflare, Akamai, and Google CAPTCHAs.

📦 Installation

Install via pip:

pip install serpscraper-dev

Or install from source in editable mode:

pip install -e .

🚀 Quick Start

Get your free API key at serpscraper.dev.

1. Google Web Search & AI Overview Extraction

from serpscraper import SerpScraperClient

client = SerpScraperClient("YOUR_API_KEY")

# Scrape Google organic results + AI Overview targeting US English
results = client.google_search("best vector database for rag", gl="US", hl="en")

print(f"Fetched {len(results)} results in {results.in_seconds}s:\n")

# 1. Iterate through Organic Search Rankings
for item in results:
    print(f"#{item['position']} {item['title']}")
    print(f"    URL: {item['link']}")
    print(f"    Snippet: {item.get('snippet', '')}\n")

# 2. Extract Google AI Overview (SGE) & Sources
if results.summary:
    print("──────────────────────────────────────────")
    print(f"🤖 [Google AI Overview]:\n{results.summary}")
    print("──────────────────────────────────────────")
    print("Cited Sources:")
    for src in results.sources:
        print(f" - {src['title']} ({src['link']})")

2. Google Trends Real-Time & Interest Over Time

Track market trends and trending searches in real time via the Google Trends API:

from serpscraper import SerpScraperClient

client = SerpScraperClient("YOUR_API_KEY")

# 1. Fetch live trending queries in the US
trends = client.trends_now(gl="US")

print("🔥 Top Trending Searches:")
for t in trends:
    traffic = t.get("traffic", "N/A")
    print(f"#{t['position']} {t['keyword']} (Search volume: {traffic})")

# 2. Fetch Google Trends Interest Over Time
interest = client.trends_interest("deepseek", gl="US", time="today 12-m")

print("\n📊 Interest Timeline Points:")
for point in interest.timeline_data:
    print(f"{point['date']}: {point['value']}")

🤖 Use Case: LangChain & LLM Tool Integration

Turn SerpScraper into a live web search tool for OpenAI, Anthropic, or local LLMs:

from serpscraper import SerpScraperClient

client = SerpScraperClient("YOUR_API_KEY")

def serp_search_tool(query: str) -> str:
    """Useful for searching the live web and getting real-time information."""
    results = client.google_search(query, gl="US", hl="en")
    
    output = []
    if results.summary:
        output.append(f"AI Overview Summary:\n{results.summary}\n")
        
    output.append("Top Web Results:")
    for item in results[:5]:
        output.append(f"- {item['title']}: {item['snippet']} ({item['link']})")
        
    return "\n".join(output)

# Ready to pass as a tool into LangChain, LlamaIndex, or AutoGen!

🌐 Supported API Methods

All methods return a clean SerpScraperResponse object with intuitive helpers:

Python Method API Endpoint Documentation & Landing Page
client.google_search(kw, **opts) webs-search (engine=google) Google Search API
client.web_search(kw, **opts) webs-search Web Search API
client.aio_search(kw, **opts) aio Google AI Overview API
client.trends_now(**opts) trends-now-search Google Trends API (Real-Time)
client.trends_interest(kw, **opts) trends-interest-search Google Trends Interest Over Time
client.image_search(kw, **opts) images-search Image Search API
client.video_search(kw, **opts) videos-search Video Search API
client.news_search(kw, **opts) news-search News Search API
client.shopping_search(kw, **opts) shopping-search Shopping Search API
client.scholar_search(kw, **opts) scholar-search Google Scholar API
client.maps_search(kw, **opts) maps-search Google Maps API
client.reviews_search(kw, **opts) reviews-search Reviews Search API
client.autocomplete(kw, **opts) autocomplete SERP Autocomplete Suggestions
client.extract_webpage(url) webpage Webpage Content Extractor API
client.ip_lookup(ip) ip-lookup IP Geolocation & Threat API
client.exchange_rate(from_c, to_c) exchange-rate Exchange Rate API
client.crypto_price(symbols, vs) crypto-price Cryptocurrency Price API
client.domain_info(domain) domain-info Domain Intelligence & WHOIS API
client.seo_domain_overview(domain) seo/domain-overview SEO Domain Overview API
client.seo_backlinks(domain) seo/backlinks SEO Backlinks API
client.seo_keyword_metrics(keywords) seo/keywords/metrics Keyword Metrics API
client.seo_keyword_suggestions(keyword) seo/keywords/suggestions Keyword Suggestions API
client.proxy_session(country) proxy/session Proxy API
client.proxy_rotate(session_id) proxy/rotate Proxy API
client.proxy_locations() proxy/locations Proxy API

🛡️ Error Handling

Catch specific exceptions from serpscraper.exceptions:

from serpscraper import SerpScraperClient
from serpscraper.exceptions import (
    AuthenticationError,
    RateLimitError,
    NetworkError,
    SerpScraperError,
)

client = SerpScraperClient("YOUR_API_KEY")

try:
    results = client.google_search("query")
except AuthenticationError as e:
    print("Invalid API Key:", e)
except RateLimitError as e:
    print("Credit limit exceeded:", e)
except NetworkError as e:
    print("Connection failed:", e)
except SerpScraperError as e:
    print("API Error:", e)

🧪 Testing

Run standard unit tests:

python -m unittest discover -s tests

📚 Resources & Documentation


📄 License

The SerpScraper Python SDK is open-sourced software licensed under the MIT License.