Python Web Scraping Tutorial: Collect Search Data with TalorData SERP API

Learn the practical difference between manually parsing search HTML and collecting structured search data with the official TalorData Python SDK.

Web scraping is the practice of collecting information from pages or online services. Python is popular for this work because it is readable, has a large ecosystem, and connects easily to data, AI, and automation tools.

When the target is a search-results page, however, downloading HTML and guessing at CSS selectors can be fragile. Search engines render different layouts, require JavaScript, apply localization, and change markup without notice. TalorData provides a SERP API that returns structured search data through one authenticated request, so your Python code works with data fields instead of page markup.

In this tutorial you will: compare manual HTML scraping with a SERP API, install talordata-serp, send a search request, read structured results, handle JSON and HTML response modes, paginate through large result sets, export data to JSON and CSV, and test your integration without spending a live API request.

Tools for collecting search data with Python

There are two broad approaches to search-data collection. You can fetch a page yourself and parse its HTML, or you can use a search API that performs collection and normalization on the server side.

This tutorial focuses on the third approach. The SDK uses Python’s standard data structures, so you can pass its results directly to a database, a dataframe library, an LLM workflow, or a CSV writer.

Manual HTML scraping versus a SERP API

With manual scraping, an application typically follows this sequence:

1. Build a URL and send an HTTP request.
2. Handle status codes, headers, JavaScript rendering, and rate limits.
3. Parse HTML and locate the result blocks with CSS selectors.
4. Normalize fields into an application-specific schema.
5. Maintain everything when the search engine changes its markup.

With TalorData, your application sends search parameters to /serp/v1/request. The SDK adds Bearer authentication, sends form-encoded data, parses the response, and returns a dictionary-like result object. You focus on what to do with the data, not how to obtain it.

import os

from talordata_serp import Client

client = Client(api_token=os.getenv("TALORDATA_API_TOKEN"))
results = client.search(engine="google", q="python data pipelines")

Step-by-step: basic TalorData workflow

The examples below use the public interface in the Talordata/talordata-serp-python repository.

Prerequisites

• Python 3.8 or newer.
• A TalorData account and API token.
• Basic Python and dictionary knowledge.
• A virtual environment for your project dependencies.

Step 1: Install the SDK

python -m pip install talordata-serp

The package depends on requests and exposes the client and exception classes from the talordata_serp module.

Step 2: Configure the API token

Keep credentials outside your source files. Set the environment variable before running a script:

# macOS / Linux
export TALORDATA_API_TOKEN="paste-your-token-here"

# PowerShell
$env:TALORDATA_API_TOKEN = "paste-your-token-here"

Client also accepts an explicit api_token parameter, which is useful when a secret manager injects the value at runtime.

Step 3: Send a first search

import os

from talordata_serp import Client

client = Client(api_token=os.getenv("TALORDATA_API_TOKEN"))
result = client.search(
    engine="google",
    q="python web scraping",
)

print("status:", result.status)
print("metadata:", result.search_metadata)

client.search() uses json=1 by default. When the response contains a JSON object, the SDK wraps it in SerpResults, which behaves like a normal Python dictionary.

Step 4: Add request parameters

Pass parameters as keyword arguments or as a mapping. Values set to None are omitted automatically. Python booleans are normalized to "1" and "0" for the form request.

params = {
    "engine": "google",
    "q": "python RAG tutorial",
    "location": "Austin, Texas",
    "safe": True,
}

result = client.search(params)
print(result.as_dict())

Read structured results

Use dictionary access for engine-specific result blocks and the helper properties for common metadata:

print(result.status)
print(result.search_metadata)

search_info = result.get("search_information", {})
print(search_info.get("query_displayed"))

# Convert the wrapper to a plain dict for another library.
payload = result.as_dict()

Result blocks are optional. Prefer get() or a default value when a field may not be returned for a particular engine or query type.

Static HTML and dynamic search pages

Traditional Python scraping tutorials distinguish between static HTML and JavaScript-heavy pages. A static page can sometimes be fetched with requests, while a dynamic page may require a browser session and explicit wait logic.

A SERP API changes this trade-off entirely. Your application stays at the structured API boundary and never inspects the search page’s DOM to extract titles, links, snippets, or metadata.

# The application consumes structured data instead of CSS selectors.
organic_results = result.get("organic_results", [])
for item in organic_results:
    print(item.get("title"), item.get("link"))

This does not mean every website becomes automatically crawlable or that an API replaces all browser automation. It means TalorData is the right abstraction when your goal is search-result data rather than arbitrary interaction with a private web application.

Choose a response mode

The SDK offers helpers for each output format exposed by the TalorData endpoint.

Structured JSON

result = client.search(engine="google", q="pizza")
plain = client.search_json(engine="google", q="pizza")

print(result.status)
print(plain["search_metadata"])

Combined JSON and HTML

Mode json=2 returns HTML together with a nested JSON string. The SDK attempts to decode that nested JSON automatically when it is valid.

combined = client.search(engine="google", q="pizza", json=2)
print(combined.get("html"))
print(combined.get("json", {}).get("search_metadata"))

HTML or raw response text

html = client.search_html(
    engine="google",
    q="pizza",
)
print(html[:500])

raw_body = client.raw_search(engine="google", q="pizza")
print(raw_body[:500])

Use HTML mode when you explicitly need the markup. Use the default JSON mode for application logic whenever possible — it is faster and avoids brittle DOM parsing.

Work with pagination and repeated searches

Pagination is controlled by the parameters supported by the selected engine. For a Google-style query, an offset such as start is commonly used:

for start in (0, 10, 20):
    page = client.search(
        engine="google",
        q="python web scraping",
        start=start,
    )
    print(start, page.status)

For a small batch, a normal loop is clear and easy to monitor:

queries = ["Python RAG", "Python SEO", "Python search API"]

for query in queries:
    page = client.search(engine="google", q=query)
    print(query, page.search_metadata.get("status"))

For larger workloads, add bounded concurrency, logging, and application-level rate limiting that respect your account limits. The SDK does not silently retry requests or provide a built-in paginator — this keeps the integration explicit and predictable.

Export search data

Since as_dict() returns ordinary Python data, standard-library exporters work without any TalorData-specific adapter.

Export to JSON

import json

with open("search-result.json", "w", encoding="utf-8") as file:
    json.dump(result.as_dict(), file, ensure_ascii=False, indent=2)

Export selected fields to CSV

import csv

organic = result.get("organic_results", [])
with open("organic-results.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=["title", "link", "snippet"])
    writer.writeheader()
    for item in organic:
        writer.writerow({
            "title": item.get("title", ""),
            "link": item.get("link", ""),
            "snippet": item.get("snippet", ""),
        })

Handle errors in production

The package exports named exceptions for missing credentials and common transport failures. Catch them at the application boundary so your worker, CLI tool, or web service can report a useful status.

from talordata_serp import (
    APITokenNotProvided,
    HTTPConnectionError,
    HTTPError,
    TimeoutError,
)

try:
    result = client.search(engine="google", q="pizza")
except APITokenNotProvided:
    print("Set TALORDATA_API_TOKEN before running the job.")
except TimeoutError:
    print("The request timed out.")
except HTTPConnectionError:
    print("TalorData could not be reached.")
except HTTPError as error:
    print("HTTP status:", error.status_code)
    print("API error:", error.error)

Set a client-wide timeout or override it for a single request:

client = Client(
    api_token=os.getenv("TALORDATA_API_TOKEN"),
    timeout=15,
)
result = client.search(engine="google", q="pizza", timeout=10)

Test the integration without a live request

The SDK accepts a custom requests.Session. This lets you assert the request method, URL, headers, and form payload with a fake session — no credentials exposed, no live API request spent.

from talordata_serp import Client

class FakeSession:
    def request(self, **kwargs):
        assert kwargs["method"] == "POST"
        assert kwargs["url"].endswith("/serp/v1/request")
        assert kwargs["data"]["json"] == "1"
        raise AssertionError("Return a fixture response in a real test")

client = Client(api_token="test-token", session=FakeSession())

The repository also includes runnable examples under examples/: basic_search.py and html_output.py.

Reuse the client for a job

Use one client for related requests and close its session deterministically with a context manager:

import os
from talordata_serp import Client

with Client(api_token=os.getenv("TALORDATA_API_TOKEN")) as client:
    first = client.search(engine="google", q="Python")
    second = client.search(engine="google", q="TalorData")

print(first.status, second.status)

Why use Python with TalorData?

1. Readable integration code: a search request is a small, ordinary Python function call.
2. Strong data ecosystem: JSON results move directly into CSV writers, databases, dataframes, and AI pipelines.
3. Reusable workflows: one client powers search copilots, RAG retrieval, SEO monitoring, research jobs, and scheduled automations.
4. Clear failure handling: token, timeout, connection, and HTTP failures are exposed as named exceptions.
5. Less page-maintenance work: your application consumes response fields rather than coupling business logic to search-page CSS selectors.

TalorData is not a general-purpose browser automation framework. It is a focused search-data API. That boundary makes it the right fit when the output you need is structured SERP data.

Frequently asked questions

Is using a SERP API the same as scraping a website?

It is a different integration layer. Manual scraping downloads and parses a page in your application. A SERP API exposes search data through an authenticated API response. Your application should still follow the API terms, account limits, and applicable laws.

Do I need Beautiful Soup or Selenium for this tutorial?

No. Those tools are useful for other kinds of web automation, but this tutorial uses the TalorData SDK to obtain search results directly. It does not parse the search page DOM in your process.

Where should the API token live?

Use TALORDATA_API_TOKEN as an environment variable locally, and a deployment secret manager in production. Never commit a real token to a repository or place it in browser-side JavaScript.

Which output mode should I choose?

Use the default JSON mode for structured application data. Use search_json() when you want a plain dictionary explicitly, search_html() for HTML output, and raw_search() for the unprocessed response body.

Where are the complete examples?

See the TalorData Python SDK on GitHub for the README, package source, tests, and runnable examples. Use the official SERP API documentation for engine-specific parameters.

That is the complete workflow: install the SDK, protect the token, send structured search requests, export only the fields your application needs, and keep the API boundary separate from your business logic. From here, the natural next steps are to explore engine-specific parameters or browse the SDK source on GitHub.