# Crawlbase Documentation — Full Corpus Complete product documentation for Crawlbase (web data infrastructure: Crawling API, Smart AI Proxy, Enterprise Crawler, Cloud Storage, and the Web MCP Server) as a single Markdown file for LLMs and agents. Per-page exports exist at https://crawlbase.com/docs/.md; a curated site overview lives at https://crawlbase.com/llms.txt; the blog index lives at https://crawlbase.com/blog/llms.txt. Sections below are separated by `---` and each carries its canonical Source URL. --- Source: https://crawlbase.com/docs # Crawlbase developer documentation The web, structured for builders. Crawl, scrape, and parse any website at scale with a single API. Production-ready endpoints, native SDKs, and an MCP server that plugs straight into Claude, Cursor, and your agent stack. [Get started](/docs/quick-start)[API Reference](/docs/api-reference)[MCP for AI](/docs/ai) Up to 20,000 free requests195 countries51M req/month per tokenNo credit card ~/crawlbase $ curl'https://api.crawlbase.com/?' \'token=YOUR\_TOKEN'\'&url=https://github.com/crawlbase' → 200 OK // 4.2s · cb\_status: 200 · 14.8 KB# JS-rendered, geo-routed, anti-bot bypassed \\…\ $ curl'https://api.crawlbase.com/?' \'token=YOUR\_TOKEN&format=json'\'&url=https://github.com/crawlbase' → {"original\_status": 200,"cb\_status": 200,"url": "https://github.com/crawlbase","body": "\…"} $ curl'https://api.crawlbase.com/?' \'token=YOUR\_TOKEN&format=md'\'&url=https://github.com/crawlbase' → # CrawlbaseWeb crawling & scraping API - Python, Node.js, Ruby, PHP, Go SDKs. # Or via the MCP server (same result, agent-native)\> tool\_use:crawl\_markdown(url="https://github.com/crawlbase") HTMLJSONMarkdown APIs ## Pick the surface that fits your stack [Browse all APIs](/docs/api-reference) [ Crawling API General purpose crawling with full headless-browser rendering, residential proxies, and built-in anti-bot bypass. The Swiss army knife. JSONHTMLMarkdown ](/docs/crawling-api)[ Enterprise Crawler Push millions of URLs at high concurrency, get results streamed back to your webhook. We handle the queueing, retries, and storage. WebhooksAsyncQueues ](/docs/crawler)[ Smart AI Proxy Residential & datacenter proxies with rotation. A single endpoint that picks the right exit node, retries failures, works with any HTTP client. HTTPSOCKS5Sticky IP ](/docs/smart-proxy)[ Cloud Storage Store, manage and serve scraped data. Persist crawled HTML and parsed JSON - fetch later by URL or RID, no infrastructure to operate. S3 CompatibleCDN ](/docs/cloud-storage) Use cases ## What can I build? [Browse scrapers](/docs/scrapers) [ Price & availability monitoring Poll Amazon, Walmart, Best Buy or any retailer's product pages on a schedule. Snapshot price, stock, and rating fields into a database - alert when they move. E-commerce scrapersCrawler queues ](/docs/scrapers/ecommerce)[ SEO & rank tracking Daily SERP snapshots for your target keywords. Track domain position in organic, watch People-Also-Ask coverage, build a SERP-feature presence dashboard. Google SERPBing SERP ](/docs/scrapers/search-engines)[ AI agents & training data Real-time web access for Claude, Cursor, and any MCP-compatible agent. Or batch-crawl a corpus and pipe Markdown into a retrieval index for grounded LLM answers. MCP serverMarkdown export ](/docs/ai)[ Lead enrichment & prospecting Walk a list of company domains, pull every visible email address with the email-extractor scraper, enrich with LinkedIn company / profile data. Lead lists ready for CRM upsert. email-extractorLinkedIn scrapers ](/docs/scrapers/email-extractor)[ Competitor & brand monitoring Track competitor product launches, social engagement, and review sentiment over time. Diff scraped JSON week-over-week to flag pricing, copy, or feature changes. Social scrapersGeneric extractor ](/docs/scrapers/social-media) Quickstart ## Your first crawl in 60 seconds [ 1 Grab your token Sign up free, no credit card. You'll get a Normal token (TCP) and a JavaScript token. ](/login)[ 2 Make your first request Encode a URL, hit the endpoint with curl or your SDK. The response includes the crawled page plus metadata. ](/docs/crawling-api#quickstart)[ 3 Scale up Switch to async mode, push to a Crawler queue, or wire up the MCP server to your AI agent. ](/docs/crawler) GEThttps://api.crawlbase.com/ ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fgithub.com%2Fcrawlbase' ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) response = api.get('https://github.com/crawlbase') if response['status_code'] == 200: print(response['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); api.get('https://github.com/crawlbase') .then(res => console.log(res.statusCode, res.body)) .catch(err => console.error(err)); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') response = api.get('https://github.com/crawlbase') puts response.status_code puts response.body ``` ``` 'YOUR_TOKEN']); $response = $api->get('https://github.com/crawlbase'); echo $response->statusCode; echo $response->body; ``` ``` package main import ( "fmt" "github.com/crawlbase/crawlbase-go" ) func main() { api := crawlbase.NewCrawlingAPI("YOUR_TOKEN") res, _ := api.Get("https://github.com/crawlbase") fmt.Println(res.StatusCode, res.Body) } ``` AI & MCP ## Native plumbing for AI agents [Explore AI & MCP](/docs/ai) [Crawlbase MCP Server](/docs/ai-mcp) Expose every Crawlbase tool to Claude, Cursor, ChatGPT, and any MCP-compatible client. Read docs [Claude Desktop](/docs/ai-claude) One-click install in Claude. Crawlbase becomes a native tool your conversations can call. Setup [LangChain & LlamaIndex](/docs/integrations-langchain) Crawlbase document loaders, retrievers, and tools - drop-in for your agent graph. Integrate [Prompt patterns](/docs/ai-prompts) Battle-tested prompts for extraction, summarization, monitoring, and lead enrichment. Browse SDKs & Integrations ## Drop into any stack [ ![Python logo](/assets/images/python-logo-round-6275a9b1c7.png) Python pip install crawlbase ](/docs/sdk-python)[ ![Node.js logo](/assets/images/node-js-logo-round-dcd8bf0db6.png) Node.js npm i crawlbase ](/docs/sdk-node)[ ![Ruby logo](/assets/images/ruby-logo-round-42f0935ab8.png) Ruby gem install crawlbase ](/docs/sdk-ruby)[ ![PHP logo](/assets/images/php-logo-round-8d5ebe870d.png) PHP composer require crawlbase ](/docs/sdk-php)[ Go Golang go get crawlbase-go ](/docs/sdk-go)[ ![Zapier logo](/assets/images/zapier-logo-round-33c40eb7c3.png) Zapier no-code automation ](/docs/integrations-zapier)[ ![n8n logo](/assets/images/n8n-circle-2715fc8444.png) n8n self-hosted workflows ](/docs/integrations-n8n)[ Make visual scenarios ](/docs/integrations-make) [Status & Errors](/docs/status-codes) Every code, every meaning. Bookmark this page. [Rate Limits](/docs/rate-limits) 20 req/sec per token, with paths to higher limits. [Changelog](/docs/changelog) What shipped this week, with migration notes. [Talk to engineering](/docs/support) Got a hard problem? We're a Slack/email away. --- Source: https://crawlbase.com/docs/account-api # Account API Programmatic access to your monthly usage stats: successes, failures, due amount, remaining credits, and per-domain breakdowns. ## Overview The Account API returns monthly usage statistics for your Crawlbase products. Use it to monitor consumption, build internal dashboards, or alert when you're approaching your credit limit. This API is **actively supported**. It's grouped under "legacy" in the navigation only because it's an older endpoint shape - there's no replacement planned. **Endpoint:** `https://api.crawlbase.com/account` Rate limit: 1 request per 5 minutes This endpoint aggregates a lot of data on the server side. Cache the response - there is no need to call it more frequently. ## Quickstart ``` curl 'https://api.crawlbase.com/account?token=YOUR_TOKEN&product=crawling-api' ``` ## Parameters token stringrequired Your Crawlbase token. product stringrequired Which product's usage to retrieve. One of: `crawling-api`, `crawler`, `smartproxy`, `scraper-api`, `leads-api`, `screenshot-api`. previous\_month booleanfalse When `true`, includes previous-month statistics alongside the current month for trend comparison. ## Response fields totalSuccess Total successful requests this month. totalFailed Total failed requests this month. totalDue Total amount due in USD for successful requests this month. remainingCredits Credits still available this month. Only returned for subscription-based products. domainStats Array of per-domain breakdowns: `domain`, `totalRequests`, `success`, `failure`, `successRate`. ## Per-product examples ``` # Crawling API curl "https://api.crawlbase.com/account?token=YOUR_TOKEN&product=crawling-api" # Enterprise Crawler curl "https://api.crawlbase.com/account?token=YOUR_TOKEN&product=crawler" # Smart AI Proxy curl "https://api.crawlbase.com/account?token=YOUR_TOKEN&product=smartproxy" # Scraper API (legacy) curl "https://api.crawlbase.com/account?token=YOUR_TOKEN&product=scraper-api" # Leads API (legacy) curl "https://api.crawlbase.com/account?token=YOUR_TOKEN&product=leads-api" # Screenshots API (legacy) curl "https://api.crawlbase.com/account?token=YOUR_TOKEN&product=screenshot-api" # With previous month for trend comparison curl "https://api.crawlbase.com/account?token=YOUR_TOKEN&product=crawling-api&previous_month=true" ``` [← PreviousCloud Storage](/docs/cloud-storage)[Next →Overview](/docs/ai) --- Source: https://crawlbase.com/docs/ai # AI & MCP Plug Crawlbase into your AI agent stack. The MCP server gives any MCP-compatible client (Claude Desktop, Cursor, VS Code, OpenAI's Codex agent) live, structured access to the web - no model can browse without it, with it the model can crawl, scrape, and screenshot any URL on demand. What MCP is MCP (Model Context Protocol) is an open spec from Anthropic for connecting AI assistants to external tools and data. An MCP server exposes a set of named tools (like `crawl_url`, `crawl_markdown`, `crawl_screenshot`) that any MCP-aware client can discover and call. The Crawlbase MCP server gives your agent the same crawl + scrape capabilities the [Crawling API](/docs/crawling-api) provides - exposed as agent-callable tools instead of REST endpoints. ## Start here - [`ai-mcp`](/docs/ai-mcp) - MCP server reference. Install steps, tools exposed (with input/output schemas), client config, environment variables, security notes. The most thorough page in this section - read it first. ## Use it with your client Per-client setup guides for the IDEs and agents that already speak MCP: - [`ai-claude`](/docs/ai-claude) - Claude Desktop and Claude Code. Drop one block of JSON into your settings; Claude can crawl URLs you reference in chat. - [`ai-cursor`](/docs/ai-cursor) - Cursor. The MCP server appears as a tool the agent can call mid-conversation. - [`ai-vscode`](/docs/ai-vscode) - VS Code (with Continue, Claude Code, or any MCP-aware extension). - [`ai-codex`](/docs/ai-codex) - OpenAI Codex agent. Adds Crawlbase as a connected tool. - [`ai-opencode`](/docs/ai-opencode) - OpenCode terminal agent. Native MCP client (local + remote, with OAuth) - drops Crawlbase tools straight into the in-shell coding loop. ## Prompt patterns - [`ai-prompts`](/docs/ai-prompts) - recipes for asking the model to use the Crawlbase tools effectively. RAG retrieval, comparison flows, multi-step research, screenshot-grounded reasoning - proven prompt shapes for each. ## Why route AI through Crawlbase Most LLM clients can't browse the live web at all; the few that can (search-augmented chat, browser-using agents) hit the same walls human scrapers do - JS-heavy SPAs, anti-bot challenges, geo-restricted content, rate limits. The MCP server hands the agent a tool that _has already solved those problems_: every Crawlbase feature (residential proxies, JS rendering, bot bypass, geo-routing, the [scraper library](/docs/scrapers)) is reachable from the agent without exposing the model to any of the implementation detail. If you're shipping an agent product, this is the cheapest way to give it real-world web access without writing your own browser-automation layer. [← PreviousAccount API](/docs/account-api)[Next →MCP Server](/docs/ai-mcp) --- Source: https://crawlbase.com/docs/ai-claude # Use with Claude Drop a small JSON snippet into Claude Desktop, restart, and your conversations can crawl, scrape, and screenshot anything on the web. ## 1. Locate the config file Claude Desktop reads MCP servers from a single JSON file. Open it in your editor: | OS | Path | | --- | --- | | macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` | | Windows | `%APPDATA%\Claude\claude_desktop_config.json` | | Linux | `~/.config/Claude/claude_desktop_config.json` | If the file doesn't exist yet, create it with an empty `{}` object. ## 2. Add the Crawlbase server ``` { "mcpServers": { "crawlbase": { "type": "stdio", "command": "npx", "args": ["@crawlbase/mcp@latest"], "env": { "CRAWLBASE_TOKEN": "YOUR_TOKEN", "CRAWLBASE_JS_TOKEN": "YOUR_JS_TOKEN" } } } } ``` ``` // Claude Code reads the same shape from claude.json { "mcpServers": { "crawlbase": { "type": "stdio", "command": "npx", "args": ["@crawlbase/mcp@latest"], "env": { "CRAWLBASE_TOKEN": "YOUR_TOKEN", "CRAWLBASE_JS_TOKEN": "YOUR_JS_TOKEN" } } } } ``` Already have other servers? If your config already has an `mcpServers` block, just add the `crawlbase` entry alongside the others. Don't replace existing servers. ## 3. Restart Claude Desktop Quit fully (not just close the window - use Cmd+Q on macOS or right-click the system tray on Windows) and relaunch. The next conversation will have access to the Crawlbase tools. ## 4. Verify it's working Click the slider/tools icon in the chat input. You should see **crawlbase** listed with 9 tools available (3 crawl + 6 storage). Try a quick test: ``` Crawl https://anthropic.com and tell me what's in the navigation. ``` You'll see Claude call `crawl` (or `crawl_markdown`), then summarize the result. If you don't see the tool call, check the logs: | OS | Logs path | | --- | --- | | macOS | `~/Library/Logs/Claude/mcp*.log` | | Windows | `%APPDATA%\Claude\logs\mcp*.log` | ## Example prompts Things that just work once Crawlbase is connected: - "Compare the pricing pages of Stripe, Adyen, and Square. Make a table of plans and headline rates." - "Take screenshots of these 5 landing pages and tell me which has the strongest above-the-fold." - "What are the top 3 results for 'open source LLM observability' right now? Summarize each." - "Watch `news.ycombinator.com` and tell me when a post about my company hits the front page." ## Troubleshooting No "crawlbase" in tools list Config JSON is invalid or path is wrong. Validate with `jq . claude_desktop_config.json`. Check the logs for parse errors. Tool calls return 401 Token wrong or hasn't been pasted in `env`. Double-check the values from your [dashboard](https://crawlbase.com/dashboard). "npx not found" Node isn't installed or not in Claude Desktop's PATH. Install Node 18+ or use the absolute path: `"command": "/usr/local/bin/npx"`. [← PreviousMCP Server](/docs/ai-mcp)[Next →Use with Cursor](/docs/ai-cursor) --- Source: https://crawlbase.com/docs/ai-codex # Use with OpenAI A native plugin that brings Crawlbase MCP into OpenAI Codex. Crawl any URL, extract clean Markdown, take screenshots, and optionally push results to Cloud Storage - all without leaving Codex. ## What it does The Crawlbase Codex plugin wraps [Crawlbase MCP](/docs/ai-mcp) as a Codex-native plugin. Once installed, you can ask Codex to crawl a page, extract its content, or capture a screenshot in plain English - Codex picks the right tool, calls Crawlbase, and returns the result. Powered by Crawlbase's infrastructure: JavaScript rendering, automatic proxy rotation, and built-in anti-bot bypass. Same reliability you use in production, conversational interface in Codex. Source The plugin is open source: [github.com/crawlbase/crawlbase-codex-plugin](https://github.com/crawlbase/crawlbase-codex-plugin). Issues and PRs welcome. ## Prerequisites You need a Crawlbase account and two API tokens: CRAWLBASE\_TOKEN required Normal token - used for static pages. CRAWLBASE\_JS\_TOKEN required JavaScript token - used for JS-rendered pages and all screenshots. Grab both from your [dashboard](https://crawlbase.com/dashboard). See [Authentication](/docs/authentication) for the difference. ## Install from Codex Marketplace 1. Open Codex and go to **Plugins → Browse Marketplace**. 2. Search for **Crawlbase Web Scraper**. 3. Click **Install**. 4. Add your `CRAWLBASE_TOKEN` and `CRAWLBASE_JS_TOKEN` when prompted. Marketplace listing coming soon The marketplace listing is still in review. Use [manual installation](#install-manual) below in the meantime. ## Manual installation Clone into your Codex plugins directory and set environment variables: ``` # Clone the plugin into Codex's plugins directory git clone https://github.com/crawlbase/crawlbase-codex-plugin \ ~/.codex/plugins/crawlbase-mcp # Set your tokens export CRAWLBASE_TOKEN=YOUR_TOKEN export CRAWLBASE_JS_TOKEN=YOUR_JS_TOKEN # Restart Codex - the plugin auto-discovers ``` ## Usage Once installed, ask Codex naturally. It will pick the right tool and call Crawlbase under the hood. ``` # Crawling "Crawl https://example.com and return the HTML" "Get the markdown content of https://example.com/article" "Take a screenshot of https://example.com" # Device emulation "Fetch the page at https://example.com using a mobile browser" "Take a full-page screenshot of https://example.com and describe what you see" ``` ## Tools exposed The plugin registers three crawl tools and six storage tools. ### Crawl tools crawl tool Fetch any URL and return raw HTML. Accepts `store: true` to push the page to [Cloud Storage](/docs/cloud-storage) instead of returning inline. crawl\_markdown tool Crawl a URL and return clean Markdown - content extracted from HTML noise, optimized for LLM consumption. Supports `store: true`. crawl\_screenshot tool Render the URL as PNG. The screenshot is returned ephemerally via `screenshot_url`: the underlying HTML can be persisted with `store: true` but the image itself is not stored. ### Storage tools storage\_get tool Fetch one stored page by `rid` or `url`. Pass `as: "json"`, `"html"`, or `"markdown"` to choose the response shape. storage\_bulk\_get tool Fetch up to 100 RIDs in one call. Optional `delete_after` flag for fire-and-forget pipelines. storage\_list tool Enumerate stored RIDs with scroll pagination, up to 1,000 per call. storage\_count tool Total document count in your storage silo. storage\_delete tool Delete a single stored page by RID. storage\_bulk\_delete tool Delete up to 100 RIDs in one call. ## Storage usage examples ``` "Crawl https://example.com and store it in Crawlbase Cloud Storage" "List all stored pages in Crawlbase" "Fetch rid abc123 from storage as markdown" "Bulk-retrieve these 50 rids and delete them afterward" "How many pages do I have in Crawlbase storage?" ``` ## Per-token storage silos Storage is partitioned per token. Pages crawled with `CRAWLBASE_TOKEN` live in a separate silo from pages crawled with `CRAWLBASE_JS_TOKEN` (which covers JS-rendered pages and all screenshots). Every crawl response includes a `token_type` field - `"normal"` or `"js"`: that tells you which silo a result landed in. When calling any storage tool, pass `use_js_token: true` if the item lives in the JS silo. Otherwise omit it. Querying the wrong silo returns "Not found" If `storage_get` returns a not-found error for a RID you know exists, you're probably querying the wrong silo. Try again with `use_js_token: true` (or remove it if you had it set). ## Related - [Crawlbase MCP Server](/docs/ai-mcp)- the underlying MCP server the plugin wraps - [Cloud Storage](/docs/cloud-storage)- the storage backend - [Prompt patterns](/docs/ai-prompts)- battle-tested prompts you can adapt for Codex [← PreviousUse with VS Code](/docs/ai-vscode)[Next →Use with OpenCode](/docs/ai-opencode) --- Source: https://crawlbase.com/docs/ai-cursor # Use with Cursor Crawlbase as a Cursor MCP server. Pull live docs, scrape competitive code, and fetch reference content without leaving the editor. ## Setup Cursor reads MCP servers from a JSON file. Open **Cursor Settings → Tools and Integrations → Add Custom MCP** , or edit the file directly: | Scope | Path | | --- | --- | | Global (all projects) | `~/.cursor/mcp.json` | | Per project | `.cursor/mcp.json`in the project root | ## Configuration ``` { "mcpServers": { "crawlbase": { "type": "stdio", "command": "npx", "args": ["@crawlbase/mcp@latest"], "env": { "CRAWLBASE_TOKEN": "YOUR_TOKEN", "CRAWLBASE_JS_TOKEN": "YOUR_JS_TOKEN" } } } } ``` Save and reload Cursor (Cmd/Ctrl+Shift+P → "Reload Window"). The Crawlbase tools should now show as a green dot in the MCP settings panel. ## Usage in chat Open the Cursor chat panel (Cmd/Ctrl+L). The AI can now reach for Crawlbase tools when relevant. ``` Fetch the latest changelog from https://nextjs.org/blog and tell me what's new in the most recent release. Then update our package.json to match if our version is older. ``` ## Patterns that work well - **Live docs lookup:**"Pull the latest _library_ docs and explain how to do X" - beats whatever's in the model's training data. - **Competitive analysis:**"Look at how _competitor_ implements _feature_ on their public site" - pair with code edits. - **Build-time scraping:** add Crawlbase calls to a tool prompt for one-off data fetches you don't want in the codebase. Tag a project with .cursor/mcp.json Per-project configs let different repos use different Crawlbase tokens - useful if you have separate accounts for prod and dev workloads. [← PreviousUse with Claude](/docs/ai-claude)[Next →Use with VS Code](/docs/ai-vscode) --- Source: https://crawlbase.com/docs/ai-mcp # MCP Server Expose every Crawlbase tool to AI assistants through the Model Context Protocol. One install and your AI can crawl, scrape, screenshot, and search the web with the same reliability you use in production. ## What is MCP? The **Model Context Protocol** is an open standard for connecting AI assistants to external tools. The Crawlbase MCP server speaks MCP, so any compatible client - Claude Desktop, Cursor, Zed, Continue, the OpenAI Agents SDK - can use Crawlbase as a native capability. The result: your AI can fetch a page, parse a product, take a screenshot, or search the web during a conversation. No glue code, no copy-paste between windows, no proxy server. Same APIs, conversational interface The MCP server is a thin wrapper over the same APIs documented in [AI & MCP](/docs/ai). Your token, your concurrency limits, your usage. The only thing that changes is who's calling - your code, or your AI. ## Install The server runs as a small Node process. Most clients launch it on demand via `npx`: no global install required. ``` # No install - let your client launch it npx @crawlbase/mcp@latest ``` ``` # Or install globally if you prefer npm install -g @crawlbase/mcp crawlbase-mcp ``` ``` docker run -i --rm \ -e CRAWLBASE_TOKEN=YOUR_TOKEN \ -e CRAWLBASE_JS_TOKEN=YOUR_JS_TOKEN \ crawlbase/mcp ``` Source on [GitHub](https://github.com/crawlbase/crawlbase-mcp). Requires Node 18+ if running directly. ## Configure your client Every MCP client uses the same config shape - server name, command to run, environment variables. Drop this into your client's config file. ``` { "mcpServers": { "crawlbase": { "type": "stdio", "command": "npx", "args": ["@crawlbase/mcp@latest"], "env": { "CRAWLBASE_TOKEN": "YOUR_TOKEN", "CRAWLBASE_JS_TOKEN": "YOUR_JS_TOKEN" } } } } ``` Per-client setup guides: - [Claude Desktop](/docs/ai-claude) & Claude Code - config goes in `claude_desktop_config.json` / `claude.json` - [Cursor](/docs/ai-cursor) - Settings → Tools and Integrations → Add Custom MCP - [VS Code & Windsurf](/docs/ai-vscode) - via Continue, Cline, or Windsurf's built-in MCP support - [Codex plugin](/docs/ai-codex) - wraps this server as a native Codex plugin ## Tools exposed The server registers three crawl tools and six storage tools. Your AI sees each as a callable function. ### Crawl tools crawl tool Fetch any URL and return raw HTML. Maps to the [Crawling API](/docs/crawling-api). Accepts `store: true` to push results to [Cloud Storage](/docs/cloud-storage). crawl\_markdown tool Crawl a URL and return clean Markdown - content extracted from the HTML, optimized for LLM consumption. crawl\_screenshot tool Render the URL as PNG. Returned as image content the model can see directly. Accepts `store: true` to persist the underlying HTML page to Cloud Storage (the screenshot image itself is not stored - only the rendered HTML). ### Storage tools Six tools for retrieving and managing pages stored via `store: true`: storage\_get tool Fetch one stored page by `rid` or `url`. Choose response shape with `as: "json" | "html" | "markdown"`. storage\_bulk\_get tool Fetch up to 100 RIDs in one call. Pass `as: "metadata_only"` (default) to keep context lean - returns RID/URL/timestamps only - or `as: "json" | "html" | "markdown"` to include bodies. Optional `auto_delete: true` for fire-and-forget pipelines that drain the silo as they read. storage\_list tool Enumerate stored RIDs with scroll pagination, up to 1,000 per call. storage\_count tool Total document count in your storage silo. storage\_delete tool Delete one stored page by RID. storage\_bulk\_delete tool Delete up to 100 stored pages by RID in a single call. Useful for cleaning out the silo at the end of a pipeline. Per-token storage silos Storage is partitioned per token. Pages crawled with `CRAWLBASE_TOKEN` live in a different silo from pages crawled with `CRAWLBASE_JS_TOKEN`. The `token_type` field in crawl responses (`"normal"` or `"js"`) tells you which. Pass `use_js_token: true` to storage tools when retrieving items from the JS silo. ## Example session Once configured, your AI calls these tools naturally during conversation. A typical turn looks like: ``` # You What's the current price of "Web Scraping with Python" (3rd ed.) on Amazon US, UK, and DE? # AI (calls crawl_markdown three times in parallel) tool_use: crawl_markdown( url="https://www.amazon.com/dp/1098145356" ) tool_use: crawl_markdown( url="https://www.amazon.co.uk/dp/1098145356" ) tool_use: crawl_markdown( url="https://www.amazon.de/dp/1098145356" ) # AI "Web Scraping with Python" (3rd ed.) prices right now: - US: $59.99 (in stock) - UK: £52.99 (in stock) - DE: €57.99 (in stock) The US price is the lowest after currency conversion (~£47). ``` ## Environment variables CRAWLBASE\_TOKEN required Your Normal token. Used by default for the `crawl`, `crawl_markdown`, and storage tools. CRAWLBASE\_JS\_TOKEN recommended Your JavaScript token. Used for `crawl_screenshot` and any tool call that needs JS rendering (SPAs, client-rendered pages). CRAWLBASE\_DEFAULT\_COUNTRY optional Default country for geo-routing (ISO code). Tools can override per-call. CRAWLBASE\_LOG\_LEVEL info One of `error`, `warn`, `info`, `debug`. Logs go to stderr so they don't interfere with MCP stdio. ## Security notes - **Tokens never leave the server process.** The MCP client sees tool definitions and results, not your credentials. - **The model can request any URL.** If you're concerned about prompt injection driving outbound requests, run with `CRAWLBASE_ALLOWED_DOMAINS` set to an allowlist. - **Run locally.** The server is designed for local stdio transport. Don't expose it over the network without an auth layer. [← PreviousOverview](/docs/ai)[Next →Use with Claude](/docs/ai-claude) --- Source: https://crawlbase.com/docs/ai-opencode # Use with OpenCode Crawlbase as an OpenCode MCP server. Pull live docs, scrape competitive code, and fetch reference content while the terminal agent is mid-task - no copy-paste, no context-switch. ## About OpenCode [OpenCode](https://opencode.ai) is a terminal AI coding agent (similar in shape to Claude Code or Aider) with native support for the Model Context Protocol - both local stdio servers and remote HTTP servers, with OAuth and bearer-auth helpers for the remote variant. The Crawlbase MCP server is a local stdio server, so the local-config block below is what you want. ## Install OpenCode Skip this section if you already have OpenCode running. Otherwise the canonical install path is: ``` # macOS / Linux / WSL - one-liner installer curl -fsSL https://opencode.ai/install | bash # Or via npm / Homebrew / paru - see opencode.ai for details ``` ## Config file OpenCode reads MCP servers from a JSON config alongside its other settings. Pick the scope that fits - global means every project you open with OpenCode sees the Crawlbase tools; per-project means only the repo that contains the file. | Scope | Path | | --- | --- | | Global (all projects) | `~/.config/opencode/opencode.jsonc` | | Per project | `opencode.jsonc`(or `opencode.json`) in the project root | ## Configuration ``` { "$schema": "https://opencode.ai/config.json", "mcp": { "crawlbase": { "type": "local", "command": ["npx", "-y", "@crawlbase/mcp@latest"], "enabled": true, "environment": { "CRAWLBASE_TOKEN": "YOUR_TOKEN", "CRAWLBASE_JS_TOKEN": "YOUR_JS_TOKEN" } } } } ``` Note the OpenCode-specific keys: `mcp` (not `mcpServers`), `type: "local"` (not `"stdio"`), `command` as an array of strings, and `environment` (not `env`). The Crawlbase MCP package on npm is the same one used by every other client - only the wrapping config schema differs. Save the file and restart OpenCode. The Crawlbase tools surface as standard MCP tools the agent can pick up mid-conversation; you don't need to mention them by name in the prompt. ## Usage in chat Drop into an OpenCode session and ask for something that would benefit from live web context. The agent decides when to reach for the Crawlbase tools - you don't have to invoke them explicitly. ``` Pull the latest TanStack Query docs from https://tanstack.com/query/latest and refactor our hooks in src/hooks/useUser.ts to match the v5 API. ``` ## Patterns that work well - **Live docs lookup:**"Read the latest _library_ docs and update our usage" - beats whatever's in the model's training cut-off. - **Competitive analysis:**"Look at how _competitor_ implements _feature_ on their public site, then propose an equivalent in our codebase." - **Build-time scraping:** one-off data fetches from inside an agent task - useful when the data is too transient or too target-specific to belong in your codebase. Per-project config = per-project token Drop an `opencode.jsonc` into a repo's root and that project gets its own Crawlbase token - useful when prod and dev workloads sit on different accounts, or when you want a teammate's checkout to use a shared service token instead of yours. [← PreviousUse with OpenAI](/docs/ai-codex)[Next →Prompt patterns](/docs/ai-prompts) --- Source: https://crawlbase.com/docs/ai-prompts # Prompt patterns A small library of prompts that get reliable results from Crawlbase tools. Use them as system prompts, agent instructions, or starting templates. ## Extraction from a single page Use this when you want structured data out of a specific URL. Direct, no agent loop required. ``` You will be given a URL. Use the crawl_url tool to fetch it, then extract a JSON object matching this schema: { "title": string, "author": string | null, "published_date": ISO 8601 date | null, "main_image_url": string | null, "summary": string // 2-3 sentences } Return ONLY the JSON object, no commentary. URL: {url} ``` Tip: pin the model to JSON output mode if your client supports it. Otherwise, use a JSON parser that tolerates leading/trailing whitespace. ## Multi-source research For "what does the web say about X" tasks. Combines search and fetch. ``` You are a research assistant. Given a topic, you must: 1. Use search_web to find 5-8 high-quality recent sources. 2. Use crawl_url on the top 3-4 to read them in full. 3. Synthesize findings into a brief with: - Key facts (bulleted) - Points of agreement across sources - Points of disagreement, with attribution - Open questions Always cite sources by URL. Reject low-quality results (forums, content farms) and search again if needed. Topic: {topic} ``` ## Change detection For "tell me when X changes" workflows. Pair with a scheduled job. ``` You are monitoring this URL: {url} The previous snapshot is in ... tags below. Use crawl_url to fetch the current version. Compare them and report: - Has the page changed in any meaningful way? (Ignore timestamps, view counts, ad rotations.) - If yes, summarize what changed in 1-3 bullet points. - If no, respond with the single word "UNCHANGED". {previous_snapshot} ``` ## Visual QA Combine the screenshot tool with the model's vision capability for layout review. ``` Use the screenshot tool with mode=fullpage on this URL: {url}. Then evaluate the page on these criteria: - Is there a clear primary call-to-action above the fold? - Is the hero text scannable in under 3 seconds? - Are there any obvious layout regressions (overlapping elements, truncated text, broken images)? Be specific - point to coordinates or sections, not vague feelings. ``` ## Lead enrichment For sales/marketing - start from a name + company, end with a profile. ``` You will receive a name and company. Your job is to enrich them into a structured profile. 1. search_web for "{name} {company} linkedin" - find the LinkedIn URL. 2. scrape_structured with scraper=linkedin-profile on that URL. 3. search_web for "{company}" to find their domain. 4. crawl_url the company homepage and extract a 1-line description. Return: { "name": ..., "title": ..., "linkedin": ..., "company": ..., "company_domain": ..., "company_description": ... } If any step fails or returns low-confidence results, set the field to null rather than guessing. ``` Always include a refusal path AI tools fail more gracefully when you tell them what to do on failure. "Set to null rather than guessing" is much better than silently letting the model fabricate answers from training data. ## General tips - **Specify the schema.** Don't ask for "the data on this page" - describe the exact fields you want. - **Limit recursive crawling.** Tell the agent how many URLs maximum it should fetch in a single turn. - **Cache when you can.** Use `store=true` to avoid re-crawling the same URL across turns. - **Set `page_wait` for SPAs.** Mention this in the prompt: "for client-rendered sites, use page\_wait=2000". [← PreviousUse with OpenCode](/docs/ai-opencode)[Next →Overview](/docs/scrapers) --- Source: https://crawlbase.com/docs/ai-vscode # Use with VS Code Windsurf has first-class MCP support. VS Code itself doesn't, but Continue, Cline, and Copilot Chat all do - same Crawlbase config, four places to drop it. ## Windsurf (officially supported) Windsurf is an officially supported client for Crawlbase MCP. Configuration: 1. Open **Windsurf → File → Preferences → Windsurf Settings → General → MCP Servers → Manage MCPs → View raw config**. 2. Add the Crawlbase entry to `mcp_config.json`: ``` { "mcpServers": { "crawlbase": { "type": "stdio", "command": "npx", "args": ["@crawlbase/mcp@latest"], "env": { "CRAWLBASE_TOKEN": "YOUR_TOKEN", "CRAWLBASE_JS_TOKEN": "YOUR_JS_TOKEN" } } } } ``` Save and restart Windsurf. The Crawlbase tools appear in the MCP panel. ## VS Code (via assistant extensions) VS Code itself doesn't ship MCP support, but the popular AI assistants for it do. Pick whichever you already use. ### Continue [Continue](https://continue.dev) is a popular open-source AI assistant for VS Code. It supports MCP servers in its YAML config: ``` mcpServers: - name: crawlbase type: stdio command: npx args: - "@crawlbase/mcp@latest" env: CRAWLBASE_TOKEN: YOUR_TOKEN CRAWLBASE_JS_TOKEN: YOUR_JS_TOKEN ``` Reload Continue and the tools appear in the agent panel. ### Cline [Cline](https://github.com/cline/cline) (formerly Claude Dev) has a settings UI for MCP servers. Open Cline → Settings → MCP Servers → Add, and paste: ``` { "crawlbase": { "type": "stdio", "command": "npx", "args": ["@crawlbase/mcp@latest"], "env": { "CRAWLBASE_TOKEN": "YOUR_TOKEN", "CRAWLBASE_JS_TOKEN": "YOUR_JS_TOKEN" } } } ``` ### GitHub Copilot Chat Copilot Chat has MCP support. Open VS Code Settings, search for "MCP", enable the integration, then add via the same JSON shape used above to `~/.config/Code/User/mcp.json`. Pick whichever you already use All four options use identical Crawlbase config. Windsurf is the official upstream-supported client; the three VS Code options work just as well - Continue is most extensible, Cline is most agentic, Copilot is most native. ## Usage Once configured, the workflow is the same as in Cursor or Claude Desktop: ``` Crawl https://docs.example.com/openapi.json and generate a typed client for the /users endpoints in TypeScript. ``` The assistant calls `crawl` to fetch the spec, parses it, and edits files in your workspace. [← PreviousUse with Cursor](/docs/ai-cursor)[Next →Use with OpenAI](/docs/ai-codex) --- Source: https://crawlbase.com/docs/api-playground # API Playground Test any Crawlbase API directly from your browser. Paste your token, pick an endpoint, fill the form, hit Run. Token stays in your browser - never sent anywhere except `api.crawlbase.com`. **How this works.** Requests go directly from your browser to `api.crawlbase.com`. If your browser blocks the request due to CORS, the playground falls back to showing you a copy-paste curl command you can run from your terminal. Either way, you get the same result. ## Request GET api.crawlbase.com/ Token Get yours from your [dashboard](https://crawlbase.com/dashboard). APICrawling API - fetch any URLCrawling API + Scraper - JSON outputCloud Storage - read stored pagesAccount API - usage statsUser Agents API - random UA URL to crawl Scraperamazon-product-detailsamazon-serpamazon-best-sellersgoogle-serpgoogle-product-offersfacebook-pageinstagram-profilegeneric-extractoremail-extractor Country Devicedefaultdesktopmobile use JS renderingstore in Cloud Storageasync (linkedin.com only) RID or URL Productcrawling-apicrawler (Enterprise)smartproxyscraper-api (legacy)leads-api (legacy)screenshot-api (legacy) include previous month curlCopy The constructed curl command will appear here when you fill out the form. Runready ## Response Copy ResponsePreviewHeaders Hit **Run** to see the response here The response will appear here after you run the request. Response headers will appear here after a successful call. **No Credit Card** Get started for free **Powerful Infrastructure** Global proxies & browsers **Developer Friendly** Simple, flexible, reliable **Enterprise Ready** Scale with confidence [← PreviousAirbyte](/docs/integrations-airbyte)[Next →User Agents API](/docs/user-agents-api) ## LinkedIn agreement required Async crawling is restricted to public LinkedIn pages at a fixed rate of **$15 per 1,000 requests**. Accept the one-time agreement in your dashboard to enable it for your account. Cancel[Open agreement in dashboard](/dashboard?linkedin_agreement=true) --- Source: https://crawlbase.com/docs/api-reference # API Reference Endpoint specs and parameter references for every Crawlbase API. One token authenticates all of them; pricing and concurrency budgets are shared across the products you subscribe to. One platform, one token All APIs below authenticate against the same token (Normal or JavaScript variants - see [Authentication](/docs/authentication)). The Crawling API is the engine; everything else is a different surface on top of it (proxy interface, persistent storage, queue management) or a small specialized helper. ## Core APIs Three endpoints cover 95% of crawl + scrape workloads. Pick one based on how you want to address the API: - [Crawling API](/docs/crawling-api) - REST endpoint. Pass URL + parameters as a query string, get the page back. Powers JS rendering, anti-bot bypass, geo-routing, and the [scraper library](/docs/scrapers). The default choice for new integrations. - [Enterprise Crawler](/docs/crawler) - high-throughput async queue: push millions of URLs, get results streamed back to your webhook. Manages retries, rate, and persistence so your client doesn't have to. - [Smart AI Proxy](/docs/smart-proxy) - proxy interface. Same network, same feature surface as the Crawling API; configured once in your HTTP client instead of per-request. The right fit when you can't or don't want to change the request shape of an existing scraper. ## Data & storage - [Cloud Storage](/docs/cloud-storage) - durable storage for crawl results. S3-compatible, CDN-fronted; persists HTML or parsed JSON keyed by request ID so you can fetch later without re-crawling. ## Account & metadata - [Account API](/docs/account-api) - monthly usage, credits, success rates, per-domain stats. Useful for in-app metering displays and proactive backoff. - [User Agents API](/docs/user-agents-api) - randomized User-Agent strings tuned for crawling, free with a 1 req/s rate limit. Drop-in for clients that want to rotate UAs without maintaining their own pool. ## Legacy APIs These predate the modern endpoints above and are still operational for existing customers - closed to new sign-ups, no shutdown scheduled. New integrations should use the modern equivalents named in the migration callout at the top of each page. - [Scraper API](/docs/scraper-api) - standalone scraper endpoint. Migrate to: Crawling API + `&scraper=`. - [Screenshots API](/docs/screenshots-api) - standalone screenshot endpoint. Migrate to: Crawling API + screenshot params, or [MCP](/docs/ai-mcp)'s `crawl_screenshot`. - [Proxy API](/docs/proxy-api) - Proxy Backconnect. Migrate to: [Smart AI Proxy](/docs/smart-proxy). - [Leads API](/docs/leads-api) - domain-scoped email extraction. No direct replacement; closest workflows live in the [email-extractor](/docs/scrapers/email-extractor) scraper. Full overview of legacy options at [/docs/legacy](/docs/legacy). [← PreviousError Handling](/docs/errors)[Next →Crawling API](/docs/crawling-api) --- Source: https://crawlbase.com/docs/authentication # Authentication Crawlbase uses simple token-based authentication. No OAuth dance, no expiring credentials - just a token in your query string or SDK config. ## Overview Every Crawlbase account has **two tokens** , generated automatically when you sign up. Both authenticate the same account and share the same quota - they just route requests through different infrastructure. Normal token default For regular HTTP requests against APIs and static pages. Fastest, lowest cost. Use this unless you need JavaScript rendering. JavaScript token JS Routes requests through a real headless Chrome instance. Required for SPAs, infinite scroll, and any site that renders content client-side. Find your tokens Both tokens are visible on your [dashboard](https://crawlbase.com/dashboard) the moment you sign up. They look like `aBcD1234efGh5678`: about 22 characters of alphanumerics. ## How to authenticate Pass your token as the `token` query parameter on every request. That's the entire authentication scheme. GEThttps://api.crawlbase.com/?token=YOUR\_TOKEN&url=... ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fexample.com' ``` ``` from crawlbase import CrawlingAPI # Pass your token at construction; reuse the client across requests api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get('https://example.com') ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: process.env.CRAWLBASE_TOKEN }); const res = await api.get('https://example.com'); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: ENV['CRAWLBASE_TOKEN']) res = api.get('https://example.com') ``` ``` getenv('CRAWLBASE_TOKEN')]); $res = $api->get('https://example.com'); ``` ``` package main import ( "os" "github.com/crawlbase/crawlbase-go" ) func main() { api := crawlbase.NewCrawlingAPI(os.Getenv("CRAWLBASE_TOKEN")) res, _ := api.Get("https://example.com") _ = res } ``` ## Picking the right token Quick decision tree: | Site type | Token | Why | | --- | --- | --- | | REST/GraphQL APIs | `Normal` | Already returns JSON, no rendering needed | | Server-rendered HTML (Wordpress, classic e-com) | `Normal` | Content present in initial response | | SPA (React/Vue/Angular/Svelte) | `JavaScript` | Initial HTML is empty; needs JS execution | | Infinite scroll, lazy-load grids | `JavaScript` | Needs scroll/wait to populate | | Sites with bot challenges (Cloudflare, etc.) | `JavaScript` | Browser fingerprint required to pass | If you're not sure, try Normal first. If the response is missing content you can see in your browser, switch to JavaScript. ## Security & rotation Treat tokens like passwords. Don't commit them to version control, don't paste them in support tickets, and rotate them if you suspect a leak. - **Use environment variables** : never hard-code tokens in source files. - **Rotate from the dashboard** : the "Reset token" button generates a new value and immediately invalidates the old one. - **One token per environment** : keep development and production on separate accounts so a leak in one doesn't poison the other. - **Never expose tokens client-side** : Crawlbase calls must originate from your backend. A token in browser JavaScript is a token in everyone's hands. Don't ship tokens to the browser Browser-side tokens leak via DevTools, network logs, source maps, and shared screenshots. Always proxy Crawlbase calls through your own backend. ## Auth-related errors If something's wrong with your token, you'll get one of these responses: 401 Unauthorized auth Token is missing, malformed, or has been reset. Double-check the value and that you're using the right one for the request type. 402 Payment Required quota Account is out of credits or trial period has ended. Top up from the dashboard. 403 Forbidden auth Token is valid but doesn't have access to this product (e.g. using a Normal token on a JS-only endpoint). For the full set of status codes, see [Status Codes](/docs/status-codes). ## Next steps [Rate Limits](/docs/rate-limits) How many requests per second your token can handle. [Status Codes](/docs/status-codes) Every code Crawlbase returns and what it means. [Crawling API](/docs/crawling-api) Now that you can authenticate, learn the API. [← PreviousQuick start](/docs/quick-start)[Next →Rate Limits](/docs/rate-limits) --- Source: https://crawlbase.com/docs/changelog # Changelog Recent ships across the Crawlbase API, SDKs, and integrations. Source repos live under [github.com/crawlbase](https://github.com/crawlbase). ## August 2026 ### 2026-08-07 · New scrapers: LeetCode - Four new [scrapers](/docs/scrapers) for LeetCode, in the [Developer & Tech](/docs/scrapers/developer) category - [LeetCode Problem Set](/docs/scrapers/leetcode-serp) (a problem set listing as an array with id, difficulty, acceptance rate, and premium flag), [LeetCode Problem](/docs/scrapers/leetcode-problem) (a single problem with its statement, topic tags, hints, and starter code snippets), [LeetCode Solutions](/docs/scrapers/leetcode-solutions) (the community solutions tab as an array with author, tags, and engagement counts), and [LeetCode Solution](/docs/scrapers/leetcode-solution) (a single solution post with its write-up and extracted code blocks). ## July 2026 ### 2026-07-29 · New scrapers: Kaggle - Four new [scrapers](/docs/scrapers) for Kaggle, in the [Developer & Tech](/docs/scrapers/developer) category - [Kaggle Dataset Search](/docs/scrapers/kaggle-dataset-serp) (a dataset search or listing page as a ranked array with owner, size, usability rating, and downloads), [Kaggle Dataset](/docs/scrapers/kaggle-dataset) (a single dataset with its description, keywords, license, and file list), [Kaggle Notebook Search](/docs/scrapers/kaggle-notebook-serp) (a notebook search or listing page as a ranked array with author, co-authors, and competition context), and [Kaggle Notebook](/docs/scrapers/kaggle-notebook) (a single notebook with its language, runtime, version history, and attached inputs). ### 2026-07-24 · New scrapers: OLX - Two new [scrapers](/docs/scrapers) for OLX classifieds, in the [E-Commerce](/docs/scrapers/ecommerce) category, across the shared frontend (olx.pl, olx.ua, olx.pt, olx.ro, olx.bg, olx.kz, olx.uz) - [OLX SERP](/docs/scrapers/olx-serp) (a search or category results page as a ranked array of listings with pricing and location, plus pagination) and [OLX Item](/docs/scrapers/olx-item) (a single ad with its full attributes, price, seller, and images). ### 2026-07-22 · New scrapers: Exercism - Four new [scrapers](/docs/scrapers) for Exercism, in the [Developer & Tech](/docs/scrapers/developer) category - [Exercism Exercises](/docs/scrapers/exercism-serp) (a track exercises page as a structured array with difficulty and unlock state), [Exercism Exercise](/docs/scrapers/exercism-exercise) (a single exercise with its full instructions as text and HTML), [Exercism Solutions](/docs/scrapers/exercism-solutions) (an exercise community-solutions page as a paginated array of authors, languages, and stars), and [Exercism Solution](/docs/scrapers/exercism-solution) (a single published solution with its iteration history and full source code). ### 2026-07-20 · New scrapers: Stack Exchange - Two new [scrapers](/docs/scrapers) for the Stack Exchange network (Stack Overflow, Super User, Ask Ubuntu, Server Fault, MathOverflow, and every `*.stackexchange.com` site), in the [Reviews & Q&A](/docs/scrapers/reviews-qa) category - [Stack Exchange Questions](/docs/scrapers/stackexchange-serp) (a questions, tagged, or search-results page as a structured array with pagination) and [Stack Exchange Thread](/docs/scrapers/stackexchange-thread) (a single question with its full answer and comment threads). ### 2026-07-16 · New scrapers: Booking.com - Two new [scrapers](/docs/scrapers) for Booking.com, in the [Travel, Events & Real Estate](/docs/scrapers/travel-events) category - [Booking SERP](/docs/scrapers/booking-serp) (a search-results listing as a structured array of properties with pricing and review scores, plus pagination) and [Booking Hotel](/docs/scrapers/booking-hotel) (a single hotel page with pricing, review scores, and facilities). ### 2026-07-16 · New scrapers: Product Hunt - Two new [scrapers](/docs/scrapers) for Product Hunt, in the [Reviews & Q&A](/docs/scrapers/reviews-qa) category - [Product Hunt Leaderboard](/docs/scrapers/producthunt-leaderboard) (daily and weekly leaderboards as a ranked array of products) and [Product Hunt Product](/docs/scrapers/producthunt-product) (a single product page with upvotes, makers, topics, and reviews). ### 2026-07-15 · New scrapers: Reddit - Three new [scrapers](/docs/scrapers) for Reddit, in the [Social Media](/docs/scrapers/social-media) category - [Reddit Subreddit](/docs/scrapers/reddit-subreddit) (a subreddit listing as a ranked array of posts with pagination), [Reddit Search](/docs/scrapers/reddit-serp) (a Reddit search-results page as a structured array with pagination), and [Reddit Post](/docs/scrapers/reddit-post) (a single post with its full comment tree). ### 2026-07-14 · New scrapers: GitHub - Three new [scrapers](/docs/scrapers) for GitHub, grouped under the new [Developer](/docs/scrapers/developer) category - [GitHub Repository](/docs/scrapers/github-repository) (a single repository page: stars, forks, watchers, languages, topics, license, open issues and PRs, default branch, and latest release), [GitHub SERP](/docs/scrapers/github-serp) (a repository search-results page as a structured array with pagination), and [GitHub Profile](/docs/scrapers/github-profile) (a user or organization profile: bio, followers, following, public repos, pinned repos, and organizations). ## June 2026 ### 2026-06-30 · New scrapers: Google Trends - Two new [scrapers](/docs/scrapers) for Google Trends - [Google Trends](/docs/scrapers/google-trends) (the "Trending now" page, returning the top trending searches with search-volume metrics and trend breakdowns) and [Google Trends Explore](/docs/scrapers/google-trends-explore) (returning interest over time, interest by sub-region, related topics, and related queries for a given topic). ### 2026-06-29 · Crawler API: delete a crawler - New `POST /crawler///delete` endpoint [deletes a crawler](/docs/crawler#delete-crawler) entirely - it clears the queue and unregisters the crawler, so it no longer shows up in your dashboard or stats. Unlike [Purge](/docs/crawler#purge), which only empties the queue, delete removes the crawler itself. This action is permanent. ### 2026-06-06 · New scrapers: Galaxus - Three new [scrapers](/docs/scrapers) for Galaxus - [Product](/docs/scrapers/galaxus-product), [SERP](/docs/scrapers/galaxus-serp), and [Product Reviews](/docs/scrapers/galaxus-product-reviews) - returning structured product, search, and review data from galaxus.ch. ### 2026-06-04 · Crawling API: PDF output - New `pdf=true` parameter on the [Crawling API](/docs/crawling-api) returns the fully rendered page as a PDF (`Content-Type: application/pdf`) instead of HTML. - Combine it with the rendering parameters (`country`, `device`, `page_wait`); PDF requests are billed the same as a JavaScript-rendered request. ### 2026-06-02 · Crawler management API - Create, retrieve, and update [Crawlers](/docs/crawler) programmatically over REST, instead of only through the dashboard. - New endpoints: `POST /crawler/` to [create](/docs/crawler#create-a-crawler), `GET /crawler//` for [details](/docs/crawler#crawler-details), and `PUT /crawler//` to [update](/docs/crawler#update-crawler). ## May 2026 ### 2026-05-21 · Passwordless onboarding - Sign in with a one-time code emailed to you. No password to set, remember, or reset; existing accounts continue to work as before. - Redesigned onboarding for new accounts. We pre-select a workflow (scraping, MCP, or integrations) from your stated use case, show a live preview of your first Crawling API request, and route you into the matching dashboard section when you finish. ### 2026-05-08 · New docs site - Relaunched [/docs](/docs) end-to-end - restructured navigation, dark mode, and instant in-page transitions across every section. - Command palette search (`⌘K` / `Ctrl K`) jumps straight to pages, sections, and API parameters from anywhere. - New **Ask AI** button opens a Crawlbase-trained assistant in-page so you can ask anything about the docs without leaving them. - Interactive [API Playground](/docs/api-playground) runs real Crawling API calls right in the browser, with response headers and a rendered page preview alongside the body. - Append `.md` to any docs URL (e.g. `/docs/crawling-api.md`) to get a clean Markdown copy you can hand to an LLM. - Published [/llms.txt](/llms.txt) as an index for AI-assistant discovery. - Refreshed German, French, Russian, and Simplified Chinese translations across every page for sharper, more idiomatic copy. ### 2026-05-03 · Go SDK v0.1.0 - First official [Go SDK](/docs/sdk-go) for the Crawlbase API. Single `CrawlingAPI` client, dependency-free, idiomatic Go. - Source: [github.com/crawlbase/crawlbase-go](https://github.com/crawlbase/crawlbase-go). Reference docs published on pkg.go.dev. ### 2026-05-02 · LangChain integration v0.1.0 - [langchain-crawlbase](/docs/integrations-langchain) is now on PyPI - a document loader, tool, and retriever backed by the Crawling API. - Source: [github.com/crawlbase/langchain-crawlbase](https://github.com/crawlbase/langchain-crawlbase). ## April 2026 ### 2026-04-24 · Crawling API: Markdown output - New `format=md` parameter on the [Crawling API](/docs/crawling-api) returns clean Markdown instead of HTML. - Pair it with `md_readability=true` to strip nav, ads, and chrome before conversion - same idea as Reader Mode. ### 2026-04-23 · Tablet device option - [Crawling API](/docs/crawling-api) `device` parameter now accepts `tablet` alongside `desktop` and `mobile`. ### 2026-04-23 · MCP Server v1.3.0 - storage tools - The [Crawlbase MCP server](/docs/ai-mcp) gains six storage tools so agents can list, read, and clean up [Cloud Storage](/docs/cloud-storage) items between crawls - not just trigger new ones. - Published as `@crawlbase/mcp@1.3.0`. ### 2026-04-23 · Codex plugin - The [Crawlbase Codex plugin](/docs/ai-codex) brings Crawlbase MCP into OpenAI Codex. - Manual install today via `git clone` into `~/.codex/plugins/`; Codex Marketplace listing in review. - Source: [github.com/crawlbase/crawlbase-codex-plugin](https://github.com/crawlbase/crawlbase-codex-plugin) ## March 2026 ### 2026-03-25 · Enterprise Crawler: queue\_timeout - [Enterprise Crawler](/docs/crawler) push now accepts `queue_timeout`, so you can cap how long a request sits in queue before it's dropped instead of waiting forever. ## February 2026 ### 2026-02-10 · Crawler renamed to Enterprise Crawler - The asynchronous push/pull product is now branded [Enterprise Crawler](/docs/crawler) across the dashboard and docs to distinguish it from the synchronous Crawling API. - Endpoints, parameters, and tokens are unchanged. ## November 2025 ### 2025-11-25 · MCP Server v1.2.0 - auth & HTTP mode - The [MCP server](/docs/ai-mcp) now supports header-based authentication and an optional HTTP transport mode in addition to stdio - useful for shared/remote MCP setups. - Published as `@crawlbase/mcp@1.2.0`. ### 2025-11-04 · Crawlbase brand across all locales - The Crawlbase rename is now reflected in every translated edition of the docs (previously English-only). ## October 2025 ### 2025-10-20 · Storage API renamed to Cloud Storage - The retrieval-and-retention product is now [Cloud Storage](/docs/cloud-storage) across docs, dashboard, and SDK method names. Existing `/storage` endpoints continue to work. ## September 2025 ### 2025-09-26 · custom\_success\_code parameter - [Crawling API](/docs/crawling-api) gains `custom_success_code` so you can mark non-2xx responses as successful when scraping endpoints that legitimately return e.g. 404 or 451. ### 2025-09-16 · Smart Proxy → Smart AI Proxy - [Smart Proxy](/docs/smart-proxy) is now branded Smart AI Proxy, reflecting the AI-driven routing and retry logic that's been added underneath. No client-side changes. ## July 2025 ### 2025-07-10 · Crawlbase MCP Server v1.0 - First public release of the [Crawlbase MCP server](/docs/ai-mcp) as `@crawlbase/mcp` on npm - three crawl tools (`crawl`, `crawl_markdown`, `crawl_screenshot`) usable from [Claude Desktop & Claude Code](/docs/ai-claude), [Cursor](/docs/ai-cursor), and [VS Code / Windsurf](/docs/ai-vscode). ## June 2025 ### 2025-06-25 · Smart Proxy: header & cookie forwarding - [Smart Proxy](/docs/smart-proxy) now forwards custom request headers and cookies through to the target site - useful for authenticated crawls and session-pinned scraping. ### 2025-06-18 · scroll\_interval billing clarified - Crawling API docs now spell out exactly how `scroll_interval` counts toward billing on long-scroll pages, so you can predict the cost of an infinite-scroll crawl before issuing it. [← PreviousUser Agents API](/docs/user-agents-api)[Next →Support](/docs/support) --- Source: https://crawlbase.com/docs/cloud-storage # Cloud Storage Store crawled pages in Crawlbase's managed storage. Fetch them later by URL or RID. Skip the database, skip the S3 bucket, skip the cron job that wires them together. ## Endpoint GEThttps://api.crawlbase.com/storage ``` # Two operations: store (write) and retrieve (read). # Storage is implicit - set store=true on a Crawling API call to write. # Use this endpoint to read. ``` ## Storing pages You don't call this endpoint to store. Instead, add `store=true` to any [Crawling API](/docs/crawling-api) call. The page gets stored automatically and you receive an `rid` in the response. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://example.com' \ --data-urlencode 'store=true' -G # Response includes rid: a1B2c3D4e5F6 ``` ## Retrieving pages ### By RID ``` curl 'https://api.crawlbase.com/storage?token=YOUR_TOKEN&rid=a1B2c3D4e5F6' ``` ``` from crawlbase import StorageAPI api = StorageAPI({'token': 'YOUR_TOKEN'}) res = api.get(rid='a1B2c3D4e5F6') print(res['body']) ``` ``` const { StorageAPI } = require('crawlbase'); const api = new StorageAPI({ token: 'YOUR_TOKEN' }); const res = await api.get({ rid: 'a1B2c3D4e5F6' }); console.log(res.body); ``` ### By URL Look up a stored page by its original URL. Returns the most recent stored version. ``` curl 'https://api.crawlbase.com/storage?token=YOUR_TOKEN&url=https%3A%2F%2Fexample.com' ``` ## Parameters token stringrequired Your Crawlbase token. rid stringone of Request identifier returned when the page was stored. url stringone of Original URL. Returns the most recent stored version. URL-encode it. format html | jsonhtml Response envelope. `json` wraps body and metadata. ## Bulk retrieve Pull up to 100 stored pages in one round-trip by RID. POST a JSON body with the list and (optionally) ask the server to delete each entry as it's returned - useful for "drain the queue" pipelines that don't need to keep storage warm. POSThttps://api.crawlbase.com/storage/bulk ``` curl -X POST 'https://api.crawlbase.com/storage/bulk?token=YOUR_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "rids": ["RID1","RID2","RID3"], "auto_delete": true }' ``` rids string[]required Array of RIDs to fetch. Maximum 100 per request - anything past 100 is silently dropped. auto\_delete booleanfalse When `true` , each successfully-returned entry is deleted from storage in the same call. Use this when you're draining a queue of one-shot results and don't need them retained. The response is a JSON array, one object per returned RID. The `body` field is base64-encoded and gzip-compressed - base64-decode then gzip-inflate to get the original page. ``` [ { "stored_at": "2021-03-01T14:22:58+02:00", "original_status": 200, "cb_status": 200, "rid": "RID1", "url": "https://example.com/a", "body": "H4sIAAAA…" // base64(gzip(html)) }, { "stored_at": "2021-03-01T14:30:51+02:00", "original_status": 200, "cb_status": 200, "rid": "RID2", "url": "https://example.com/b", "body": "H4sIAAAA…" } ] ``` ## Bulk delete Delete up to a list of RIDs in one call. Returns a per-RID status so you can spot the ones that were already gone or failed. POSThttps://api.crawlbase.com/storage/bulk\_delete ``` curl -X POST 'https://api.crawlbase.com/storage/bulk_delete?token=YOUR_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "rids": ["RID1","RID2","RID3"] }' ``` Response is a JSON array, one entry per submitted RID. `status: true` means the entry was deleted; `status: false` with `result: "Not Found"` means the RID didn't exist (already cleaned up, expired, or never written). ``` [ { "rid": "RID1", "result": "Deleted", "status": true }, { "rid": "RID2", "result": "Not Found", "status": false }, { "rid": "RID3", "result": "Failed", "status": false } ] ``` Deletion is irreversible Double-check the RID list before sending. There is no soft-delete or undo - if you need a recoverable workflow, retrieve with `auto_delete=false` first and only call `/bulk_delete` once you've persisted the body locally. ## Delete a single page Drop one entry from storage by RID. Use `DELETE /storage` with the RID on the query string. DELETEhttps://api.crawlbase.com/storage ``` curl -X DELETE 'https://api.crawlbase.com/storage?token=YOUR_TOKEN&rid=RID' ``` Three response shapes: | Outcome | Body | | --- | --- | | Found and deleted | `{"success": "The Storage item has been deleted successfully"}` | | Found but delete failed | `{"error": "The Storage item could not be deleted"}` | | Not in storage | `{"error": "Not Found"}` | ## List RIDs Page through the RIDs in your storage area - the inventory call. For datasets larger than a single response, use scroll-based pagination ( `scroll=true` seeds a scroll session and returns a `scroll_id` you replay on subsequent calls). GEThttps://api.crawlbase.com/storage/rids ``` # First page curl 'https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&limit=100&scroll=true' # Next page - replay the scroll_id from the previous response curl 'https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&scroll_id=dXVlcnlUaGVuRmV0Y2g7…' ``` limit integeroptional Maximum number of RIDs to return per call. Cap is 10000. No default - set this explicitly. scroll booleanfalse When `true` , the response includes a `scroll_id` you can replay to fetch the next page. Without it you only get the first page. scroll\_id stringoptional Token from a previous response. Replay it to advance the scroll. Don't pass `scroll=true` on follow-ups - only on the first call. scroll\_order asc | descdesc Order RIDs by stored timestamp. Default is newest first. ``` { "rids": ["RID1", "RID2", "RID3", "..."], "scroll_id": "dXVlcnlUaGVuRmV0Y2g7NTs1NDpDV…" } ``` Scroll sessions expire A `scroll_id` is good for ~15 seconds of inactivity. If you see `"Scroll session has expired or is invalid"` , start over with a fresh `scroll=true` request - the cursor's gone. ## Total count Single integer: how many pages are currently in your storage area. GEThttps://api.crawlbase.com/storage/total\_count ``` curl 'https://api.crawlbase.com/storage/total_count?token=YOUR_TOKEN' # Response # { "totalCount": 5491078 } ``` ## Retention & pricing - Stored pages are kept for **14 days by default**. Extend on enterprise plans. - Each `store=true` call counts as a single request - no extra charge. - Retrieval (this endpoint) is **free**. Read as many times as you need. - If a page is re-crawled with `store=true` , the new version replaces the old. When storage shines Audit trails ("what did the page say when we crawled it?"), reprocessing pipelines (re-parse stored HTML when your scraper logic improves), and serving cached results to readers without re-crawling. [← PreviousSmart AI Proxy](/docs/smart-proxy)[Next →Account API](/docs/account-api) --- Source: https://crawlbase.com/docs/crawler # Enterprise Crawler Push URLs into a managed queue, let Crawlbase run them at high concurrency, get results delivered to your webhook or persisted to Cloud Storage for you to pull. No client-side scheduling, no retry logic, no concurrency math. The Crawler is a managed queue you push URLs into and read results out of. Three lifecycle steps - **Setup** (configure the queue), **Push** (enqueue URLs), **Pull** (receive results) - covered in order below. ## Setup Create a named queue in your [dashboard](https://crawlbase.com/dashboard). Each crawler holds up to 100K URLs. Create one queue per workload - they don't share state. At creation you pick: - A unique **name** (you choose it: `product-monitor`, `news-feed`, etc.) - A **delivery mode** : either a callback URL (Crawlbase POSTs each result to that webhook) or [Cloud Storage](/docs/cloud-storage) (results are persisted automatically and you fetch them via the Storage API on your own schedule). Picked once at creation; the two modes are exclusive - the same crawler doesn't do both. - A **token type** (Normal or JavaScript) - A **concurrency limit** (default 20, raisable on request) No per-request store flag Storage delivery is a property of the crawler, not the push. If the crawler was created in Storage mode, every result lands in Cloud Storage automatically - you don't need to set `store=true` on each push, and webhook-mode crawlers can't opt in per request. ## Push Send URLs to the crawler's queue. The push returns immediately with an `rid` so your client can move on; the actual crawl happens in the background at the crawler's configured concurrency. Pass `callback=true` to opt the request into queue delivery instead of running it inline. GEThttps://api.crawlbase.com/?token=…&crawler=NAME&callback=true&url=… ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN&crawler=product-monitor&callback=true&url=https%3A%2F%2Fexample.com%2Fp%2F12345' ``` ``` from crawlbase import CrawlerAPI api = CrawlerAPI({'token': 'YOUR_TOKEN'}) res = api.push( 'https://example.com/p/12345', {'crawler': 'product-monitor'} ) print(res['rid']) ``` ``` const { CrawlerAPI } = require('crawlbase'); const api = new CrawlerAPI({ token: 'YOUR_TOKEN' }); const res = await api.push( 'https://example.com/p/12345', { crawler: 'product-monitor' } ); console.log(res.rid); ``` Push response is small - just confirmation that the URL is queued. ``` { "rid": "a1B2c3D4e5F6" } ``` Push throughput and queue caps Push rate is capped at **30 URLs/sec** per token by default. Each crawler holds up to 100K URLs in its waiting queue, and the combined total across all of your crawlers is capped at **1,000,000** ; once you cross that, pushes pause and you get an email - drain the queue (or [purge](#purge)) and pushes resume automatically. ## Pull How completed crawls reach you. Two channels, picked once at crawler creation: ### Webhook When the crawler was created with a callback URL, Crawlbase POSTs each result to that webhook the moment the crawl finishes. Body in the request body, metadata in the request headers - no polling, no client-side state. ``` # POST https://your-app.com/webhook # Content-Type: text/html (or application/json if scraper used) # cb_status: 200 # original_status: 200 # rid: a1B2c3D4e5F6 # url: https://example.com/p/12345 … ``` Your webhook should: - Be publicly reachable from Crawlbase servers. - Accept `POST` and respond with `200`, `201`, or `204` within 200ms. - Be idempotent on `rid`: duplicate deliveries can happen on retry. - Acknowledge before processing - kick the work off async if it takes longer than the response window. The body shape follows the `format` parameter you set on push: | `format=html` | `format=json` | | --- | --- | | `Content-Type: text/plain` | `Content-Type: gzip/json` | | Body is the HTML of the page | Body is JSON: `{ cb_status, original_status, rid, url, body }` | | Headers carry metadata: `Original-Status`, `CB-Status`, `rid`, `url` | Headers carry the same metadata, fields are also in the body | Both shapes arrive **gzip-compressed** (`Content-Encoding: gzip`) - your handler needs to decompress before parsing. The exception is Zapier webhooks, which can't read gzipped bodies; Crawlbase detects Zapier callback URLs and skips compression. Failed deliveries are charged Every retry counts as a successful crawl for billing purposes - Crawlbase already paid the proxy / browser cost. Keep your webhook reliable; the cheapest way to reduce credit burn is to stop dropping deliveries, not to fight the retry policy. **Testing.** When you're wiring up the handler for the first time and want to inspect the exact payload shape for a real URL, create a Storage-mode crawler alongside your webhook one and push the same URLs to both. Pull from [Cloud Storage](/docs/cloud-storage) by RID and you have a frozen reference to compare your webhook receipts against - useful for catching decompression bugs and metadata-handling mistakes before they hit production traffic. ### Uptime monitoring **Monitoring bot.** Crawlbase polls your webhook on a schedule to detect outages. If the bot can't reach your endpoint or you stop returning 2xx, the crawler _pauses itself_ automatically and resumes once your endpoint comes back. The probe is a regular `POST` with a JSON body, distinguishable by its User-Agent: ``` POST https://your-app.com/webhook User-Agent: Crawlbase Monitoring Bot 1.0 Content-Type: application/json { "monitor": true } ``` Treat probes as a no-op and return `200`. Don't process them as crawl results - there's no real RID to act on. ### Securing the endpoint **Protecting the endpoint.** A random-string path (`yourdomain.com/2340JOiow43djoqe21rjosi`) is already most of the protection in practice - the URL is unlikely to be discovered. For belt-and-braces, layer one or more of: - A query-string token: `?token=…` the webhook checks before accepting the body. - A custom header sent via `callback_headers` on push (e.g. `X-Webhook-Token|s3kret`) and verified server-side. - Reject anything that isn't `POST`. - Reject anything missing the expected metadata headers (`CB-Status`, `Original-Status`, `rid`). We don't recommend IP allowlisting - Crawlbase pushes from many IPs and the set rotates without notice. ### Cloud Storage When the crawler was created with Storage as its delivery mode, every result is persisted to [Cloud Storage](/docs/cloud-storage) automatically - no per-push flag, no webhook. Your consumer fetches results on its own schedule via the Storage API. Use this when downstream is batched, when you can't run an HTTPS endpoint, or when you want a stable URL for each crawled page. Push the same way you would for a webhook-mode crawler - the only difference is where the result ends up. Once a URL finishes crawling, fetch by RID: ``` # Single fetch by RID curl 'https://api.crawlbase.com/storage?token=YOUR_TOKEN&rid=a1B2c3D4e5F6' # Or batch-drain up to 100 RIDs at once with auto_delete=true # so storage stays small. curl -X POST 'https://api.crawlbase.com/storage/bulk?token=YOUR_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "rids": ["RID1","RID2","RID3"], "auto_delete": true }' ``` To know _when_ a result is ready, either poll [Find Job](#find-job) for a specific RID, watch [Stats](#stats) for the queued / completed counters, or just batch-drain [/storage/bulk](/docs/cloud-storage#bulk) on a schedule and let "RID not found" tell you what's still pending. Delivery mode is set at creation The two modes are exclusive and bound to the crawler when you create it. A webhook crawler doesn't write to Storage; a Storage crawler doesn't fire a webhook. To switch, create a new crawler with the other mode and migrate your push traffic to it. ## Management API Create, inspect, update, and monitor your crawlers via REST. All endpoints live under `/crawler//...` and authenticate by token in the path (no query-string token needed). Token in path, not query string Unlike the Crawling API, these endpoints expect the token in the URL path. For JavaScript-token crawlers, swap the Normal token for your JS token in every example below. ### Create a crawler Create a named crawler queue. `callback_url` is optional — omit it to default to [Cloud Storage](/docs/cloud-storage) delivery, or provide your own webhook URL to receive results via POST. POSThttps://api.crawlbase.com/crawler/\ ``` curl -X POST 'https://api.crawlbase.com/crawler/YOUR_TOKEN' \ -H 'Content-Type: application/json' \ -d '{"name":"product-monitor"}' ``` ### Get crawler details Retrieve a crawler's current configuration, status (running / paused), and recent page activity. GEThttps://api.crawlbase.com/crawler/\/\ ``` curl 'https://api.crawlbase.com/crawler/YOUR_TOKEN/product-monitor' ``` ### Update crawler Change a crawler's callback URL or other live settings without restarting it. PUThttps://api.crawlbase.com/crawler/\/\ ``` curl -X PUT 'https://api.crawlbase.com/crawler/YOUR_TOKEN/product-monitor' \ -H 'Content-Type: application/json' \ -d '{"callback_url":"https://my-server.com/hook"}' ``` ### Stats Summary across all your crawlers - concurrency, queue depth, completed/failed counts, and a history breakdown. GEThttps://api.crawlbase.com/crawler/\/stats ``` # All-time summary curl 'https://api.crawlbase.com/crawler/YOUR_TOKEN/stats' # Same, filtered to a date range (YYYY-MM-DD bounds, inclusive) curl 'https://api.crawlbase.com/crawler/YOUR_TOKEN/stats?history_from=2026-04-01&history_to=2026-04-30' ``` ### Purge a crawler Empties the crawler's queue immediately - every still-pending URL is dropped. Use this to recover from a runaway producer or to clear a batch you no longer want to process. There's no undo. POSThttps://api.crawlbase.com/crawler/\/\/purge ``` curl -X POST 'https://api.crawlbase.com/crawler/YOUR_TOKEN/product-monitor/purge' ``` Purge is immediate and total Every queued URL in that crawler is dropped - there's no soft-delete or recovery. If you only need to drop a single URL, use [Delete Job](#delete-job) instead. ### Delete a crawler Removes the crawler completely - it clears the queue and unregisters the crawler, so it no longer shows up in your dashboard or stats. Unlike purge, which empties the queue but keeps the crawler around, delete takes the whole thing away. There's no undo. POSThttps://api.crawlbase.com/crawler/\/\/delete ``` curl -X POST 'https://api.crawlbase.com/crawler/YOUR_TOKEN/product-monitor/delete' ``` Delete is permanent The crawler and everything still queued in it are gone for good - there's no soft-delete or recovery. If you only want to empty the queue but keep the crawler, use [Purge](#purge) instead. ### Delete a single job Drop one URL from the queue by its RID - the request ID returned when you pushed the URL. POSThttps://api.crawlbase.com/crawler/\/\/delete\_job ``` curl -X POST 'https://api.crawlbase.com/crawler/YOUR_TOKEN/product-monitor/delete_job?rid=YOUR_RID' ``` ### Find a job by RID Look up where a request stands. Returns `QUEUED` with the queued metadata if it's still pending, or `NOT_QUEUED` if it's already crawled (or never made it onto the queue). GEThttps://api.crawlbase.com/crawler/\/\/find\_by\_rid/\ ``` curl 'https://api.crawlbase.com/crawler/YOUR_TOKEN/product-monitor/find_by_rid/YOUR_RID' ``` ``` { "status": "QUEUED", "request_info": { "rid": "YOUR_RID", "url": "YOUR_URL", "retry": 3, "created_at": 1600494969.189415 } } ``` ``` { "status": "NOT_QUEUED", "request_info": { "rid": "YOUR_RID" } } ``` ### Pause and unpause Stop a crawler from picking up new work without losing its queue. Pushed URLs continue to enqueue, but the crawler stops processing them until you unpause. Useful for maintenance windows or backing off when a downstream system is unhealthy. ``` # Pause - stops the crawler picking up new work curl -X POST 'https://api.crawlbase.com/crawler/YOUR_TOKEN/product-monitor/pause' # Unpause - resumes processing curl -X POST 'https://api.crawlbase.com/crawler/YOUR_TOKEN/product-monitor/unpause' ``` ## Parameters crawler stringrequired Name of the crawler from your dashboard. url stringrequired (push) URL to enqueue. URL-encode it. callback\_headers stringoptional Extra headers to include on the webhook delivery, format `name|value|name|value`. Useful for passing IDs back to your handler. Webhook-mode crawlers only - ignored when the crawler delivers to Storage. queue\_timeout integer (minutes)optional Maximum time the request may sit in the queue before a worker picks it up. Range `1` – `10080` (1 minute to 7 days). Once a worker starts the crawl, this timer no longer applies. If the request expires waiting, you get a callback with HTTP `504` and `cb_status=699`. Omit (or set to `0`) to disable. Aggressive values raise failure rate - pick what reflects how long the result is actually useful to you. All Crawling API params optional Pass `page_wait`, `scroll`, `country`, `scraper`, etc. - they're applied to each crawl. ## When to use Crawler vs the API - **Crawler:** any time you have more than a few hundred URLs to process, especially across long time horizons. The queue handles retries, scheduling, and concurrency. - **Direct Crawling API:** when you need the result inline - page rendering for a user-facing request, AI agent fetching context, etc. [← PreviousCrawling API](/docs/crawling-api)[Next →Smart AI Proxy](/docs/smart-proxy) --- Source: https://crawlbase.com/docs/crawling-api # Crawling API Fetch any URL through Crawlbase's residential proxy network with optional JavaScript rendering, bot challenge solving, and geo-routing. The general-purpose endpoint that powers everything else. ## How it works Every Crawling API request takes a target URL and returns the page that target would have served to a real browser at the right geography, with the right device profile, after any anti-bot challenges have been resolved. Three things happen in sequence on every call: 1. **Routing.** The request is sent through a residential or datacenter exit node - automatically by default, or in a specific country if you pass `country=`. Sticky sessions are available so a sequence of calls reuses the same IP. 2. **Rendering.** If you authenticate with a **JavaScript token** , the URL is loaded in a real headless browser. Page-wait, scroll, click, and AJAX-idle controls let you wait for the actual content rather than the initial HTML shell. 3. **Anti-bot bypass.** Cloudflare, PerimeterX, DataDome, hCaptcha, and other common challenges are solved server-side. You get the post-challenge HTML, not the challenge page. The same endpoint covers all three. Pass only the parameters you need - there's no separate "JS-rendering API" or "anti-bot API". If you don't pass JS-token-only parameters, the request takes the cheap, fast path; the moment you do, the request shifts to the rendering path. Pricing is the same per successful response either way. ### Tokens Authentication uses one of two token types - both live on a single account, both authenticate the same endpoint: - **Normal Token (TCP)**: for static HTML or JSON responses where you don't need a browser. Faster, cheaper, used for the majority of straightforward scrape targets. - **JavaScript Token** : for SPAs, React/Vue/Angular apps, lazy-loaded feeds, and any target that hides content behind client-side rendering. Required to use `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. If a Normal-token request returns an empty body or a `525` (challenge couldn't be solved), the standard fix is to retry on the JavaScript token - most modern targets need a browser even when their initial HTML looks complete. See [Authentication](/docs/authentication) for the full token-management flow. ### Concurrency & pricing Every request that returns `cb_status: 200` counts against your monthly quota. Failed requests (timeouts, blocks, 5xx from the target) are free - retries against a flaky upstream don't surprise your bill. Concurrency limits scale with your plan; the response includes a `remaining` header you can use to back off proactively before hitting the cap. Long-running crawls (heavy JS rendering, large `page_wait`) should use the async mode below to release the concurrency slot the moment the request is queued. **Client timeouts.** Average response time is **4–10 seconds** per request, but tail-latency requests (heavy SPAs, `scroll_interval=60`, slow upstream sites) can take longer. Set your client timeout to at least **90 seconds** so legitimate slow responses don't time out before they arrive. **Other client-side recommendations.** Send `Accept-Encoding: gzip` on every request - payloads are non-trivial (full HTML pages or markdown) and gzip typically cuts them to a third of the wire size. If you're using [Scrapy](https://scrapy.org/), [disable the DNS cache](https://scrapy-cluster.readthedocs.io/en/latest/topics/advanced/dnscache.html) so the API host stays resolvable across long-lived crawls. ## Endpoint GETPOSTPUThttps://api.crawlbase.com/?token=YOUR\_TOKEN&url=ENCODED\_URL - Methods: `GET` (query-only), `POST` (form or JSON body), `PUT` (raw payload). - The `url` parameter must be fully [URL-encoded](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent). - Body is returned as the target page's content (HTML, JSON, image, etc). - Metadata is returned as response headers (`cb_status`, `original_status`, `url`, `rid`). ## Quickstart ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fgithub.com%2Fanthropic' ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get('https://github.com/anthropic') print(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get('https://github.com/anthropic'); console.log(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://github.com/anthropic') puts res.body ``` ``` 'YOUR_TOKEN']); $res = $api->get('https://github.com/anthropic'); echo $res->body; ``` ``` package main import ( "fmt" "github.com/crawlbase/crawlbase-go" ) func main() { api, _ := crawlbase.NewCrawlingAPI("YOUR_TOKEN") res, _ := api.Get("https://github.com/anthropic", nil) fmt.Println(res.Body) } ``` ## Request Every Crawling API request is a single HTTP call to the endpoint. Most requests are `GET`s - pass the [query parameters](#request-parameters) below to control rendering, geo, output format, and async behavior. Use [POST](#request-post) when you need to send a form or JSON body, and [PUT](#request-put) for raw payload uploads. ### Request parameters All parameters are passed as query string values. Only `token` and `url` are required. #### Required token stringrequired Your Normal or JavaScript token. See [Authentication](/docs/authentication). url stringrequired The fully URL-encoded target URL. Must include the scheme (`http://` or `https://`). #### Routing & geo Pick where the request originates and what device the target sees. Routing matters for storefronts, SERPs, and any site that localises content by IP - the German Amazon catalog isn't reachable from a US exit even with the right URL, and Google SERPs are localised by both geography and the `hl`/`gl` URL params combined with the IP. Set `country` explicitly and the right currency, language, and inventory show up automatically. country stringoptional Two-letter ISO country code (`US`, `GB`, `DE`, `JP`, …) to route the crawl through that country's exit nodes. Defaults to automatic geo selection. device desktop | tablet | mobiledesktop Emulate the User-Agent and viewport of the chosen device class. user\_agent stringoptional Override the User-Agent header. Use sparingly - defaults are tuned for each target. tor\_network booleanfalse Route the request over the Tor network so you can crawl `.onion` sites. Leave off for any clearnet target - Tor exits are slower and noisier than the residential pool. Country may be auto-overridden Crawlbase may override the `country` parameter to auto-select a proxy based on the URL - this gives the best success rate on most sites. [Contact support](https://crawlbase.com/dashboard/support) if you need to disable automatic proxy selection. Specifying a country can reduce the number of successful requests, so use it only when geolocation actually matters for the page you're crawling. Some sites (notably Amazon) are routed via dedicated proxies regardless of the country you pass - every country is allowed for those domains even if it's not in the supported list below. You have access to the following countries: | United Arab Emirates (AE) | Brazil (BR) | Canada (CA) | | China (CN) | Germany (DE) | Denmark (DK) | | Spain (ES) | Finland (FI) | France (FR) | | United Kingdom (GB) | Israel (IL) | Japan (JP) | | Kazakhstan (KZ) | Moldova (MD) | Netherlands (NL) | | Pakistan (PK) | Poland (PL) | Russia (RU) | | Sweden (SE) | Turkey (TR) | Ukraine (UA) | | United States (US) | | | #### Headers & cookies Forward your own request headers and cookies through to the target site, or pin a sticky session so `Set-Cookie` values from one call replay on the next. Useful when the target needs an `Accept-Language`, a CSRF cookie, or a logged-in session that needs to survive across the requests in a flow. request\_headers stringoptional URL-encoded list of headers to forward, pipe-separated: `accept-language:en-GB|accept-encoding:gzip`. Pair with `get_headers=true` to also surface the target's response headers. set\_cookies stringoptional Cookies to forward to the target, in standard `Cookie`-header form: `key1=value1; key2=value2`. cookies\_session stringoptional Sticky cookie session - Crawlbase replays the cookies returned from previous calls on every subsequent call sharing the same value. Any string up to 32 chars; a new value starts a new session. Sessions expire 300 seconds after the last call. **Allowed headers.** Not every header you pass via `request_headers` will reach the target site - Crawlbase strips a small set by default. To verify what actually goes out, send a test request to `https://postman-echo.com/headers` and inspect what the echo service receives. If you need an additional header authorised for your token, [contact support](https://crawlbase.com/dashboard/support) with the header name(s). #### JavaScript rendering These parameters require a **JavaScript token**. They control how the headless browser waits for content before capturing the DOM. If you find yourself reaching for several at once, the order to think about is: `page_wait` first (a fixed delay for predictable animations), then `ajax_wait` (drop the fixed delay if the page emits network requests after mount), then `scroll` (only if the content you need is below the fold), then `css_click_selector` (only if a button or accordion gates the data). A common pitfall: setting `page_wait` too high "just in case". Every extra millisecond is concurrency you can't use elsewhere. Start at 0, increase only when you see truncated output, and consider `ajax_wait` as a smarter alternative - it returns as soon as the network goes idle rather than blocking on a fixed timeout. page\_wait int (ms)0 Wait this many milliseconds after page load before capturing. Useful for content that animates in. ajax\_wait booleanfalse Wait until the network is idle (no requests for ~500ms). Best for SPAs that fetch data after mount. css\_click\_selector stringoptional CSS selector - Crawlbase will click the matching element before capturing. URL-encode special characters. scroll booleanfalse Scroll to the bottom of the page before capturing. Triggers lazy-load. scroll\_interval int (s)10 Maximum seconds to spend scrolling. Combined with `scroll=true`. screenshot booleanfalse Capture a JPEG of the rendered page. The URL comes back as `screenshot_url` in the response headers (or the JSON body when `format=json`) and expires after one hour. For multi-shot or full-page workflows reach for the dedicated [Screenshots API](/docs/screenshots-api) instead. pdf booleanfalse Render the page to a PDF and return the **PDF file as the response body** (`Content-Type: application/pdf`, `Content-Disposition: inline`), instead of the HTML. Uses the same headless pipeline as `screenshot`, so the rendering parameters above apply. Cannot be combined with `screenshot`; takes precedence over `format`. **Screenshot output options.** When `screenshot=true`, the default capture is the full rendered page. To narrow it to just the viewport, append `mode=viewport`; pair it with `width` and `height` (pixels) to constrain the capture. Both default to the screen dimensions and only take effect with `mode=viewport`. Example: `&screenshot=true&mode=viewport&width=1200&height=800`. Need the page as a PDF rather than an image? Use `pdf=true` for the same headless render returned as a PDF file. **PDF output.** Unlike `screenshot` (which returns a temporary `screenshot_url`), `pdf=true` streams the PDF back as the response body, so write the response straight to a `.pdf` file. The rendering parameters that shape the page before capture still apply (`country`, `device`, `user_agent`, `page_wait`, `scroll`), and the standard metadata headers (`cb_status`, `original_status`, `url`) come back alongside it. It cannot be combined with `screenshot=true`: requesting both returns a `400` with a message like `pdf=true cannot be combined with screenshot=true`. If you also set `format=json`, `pdf=true` takes precedence and the PDF is still returned. If the render or PDF generation fails you get an error status, never a partial or blank PDF. PDF requests are billed the same as a JavaScript-rendered request, since they share the headless pipeline. **How `scroll` is billed.** Scroll-enabled requests are billed by total server-side processing time. The first **8 seconds** (page load + scrolling combined) count as 1 request; every additional **5 seconds** beyond that adds 1 more billed request. A 20s scroll = 1 (first 8s) + 1 (9–13s) + 1 (14–18s) + 1 (19–20s, partial blocks count in full) = **4 billed requests**. If the page completes before `scroll_interval`, only the actual processing time is billed. The maximum `scroll_interval` is **60** seconds - past 60s scrolling stops and the response is returned. When you set `scroll_interval=60`, keep the client-side connection open for at least **90 seconds** so the response has time to come back. Combining `scroll` with `page_wait` increases the total processing time and therefore the billed request count. The `css_click_selector` parameter only takes effect when you're using the **JavaScript token** (it runs inside the headless browser before the DOM is captured). It accepts any fully specified, valid CSS selector - for example an ID like `#some-button`, a class like `.some-other-button`, or an attribute selector like `[data-tab-item="tab1"]`. Always URL-encode the value so special characters survive the query string intact. If the selector is not found on the page the request fails with `cb_status` `595`. To still receive a response when the click target may be absent, append a universally-found selector as a fallback - comma-separated. For example `#some-button,body` falls back to clicking `body` when `#some-button` doesn't exist. **Multiple selectors.** To click several elements in sequence before the capture, separate them with a pipe (`|`) character. URL-encode the whole value, including the pipe. For example, clicking `#start-button` and then `.next-page-link` looks like `#start-button|.next-page-link` in raw form, or `%23start-button%7C.next-page-link` URL-encoded. The clicks happen in the order given. If any selector in the chain is missing the same `cb_status` `595` rule applies, so the `,body` fallback pattern works per selector. Need to run **custom JavaScript** inside the page before Crawlbase captures the DOM (e.g. dispatch a synthetic event, mutate state, force a fetch)? That's a per-account feature gated on your use case - [contact support](https://crawlbase.com/dashboard/support) with what you're trying to do and we'll wire it up. #### Async & storage Async mode flips the API from "block until I have your page" to "queue this and tell me when it's done." The endpoint returns immediately with an `rid`; the actual result is delivered to a webhook you specify, or stored in Cloud Storage and fetched later by the same `rid`. This is the right mode for batch jobs and slow targets - async releases your concurrency slot the moment the request is queued, so you can keep submitting while crawls are still running. For high-volume jobs (millions of URLs), use the [Enterprise Crawler](/docs/crawler) which sits in front of this same async pipeline with retries, rate management, and result delivery. Async mode is currently linkedin.com only The `async=true` flag is currently supported only for `linkedin.com` URLs. If you need async crawls on other domains, [contact support](https://crawlbase.com/dashboard/support) with the target domain so we can enable it for your token. async booleanfalse Return immediately with an `rid` instead of blocking. Result is delivered to `callback` if set, or available via [Cloud Storage](/docs/cloud-storage) by `rid`. callback URLoptional Webhook URL to receive the crawl result. Required when `async=true` if you don't want to poll. store booleanfalse Persist the crawled page in [Cloud Storage](/docs/cloud-storage). Returns an `rid` in addition to the body. #### Output format The default response is the raw page body - exactly what a browser would receive after rendering and anti-bot resolution. For most pipelines that's the right shape (your downstream parser handles HTML directly). Use `format=json` when you want metadata (status, final URL, RID, headers) bundled into a single envelope rather than split across response headers and the body. Use `scraper=` or `autoparse=true` when the target is one we already have a parser for - you skip the parsing step entirely and get clean structured fields back instead of raw markup. format html | json | mdhtml Choose the response envelope. `html` returns the raw page with metadata in the response headers. `json` wraps the page plus all metadata into a single JSON object. `md` converts the page to GitHub-Flavored Markdown - pair with `md_readability=true` to strip nav/sidebar/ads first. md\_readability booleanfalse Only meaningful with `format=md`. When `true`, Crawlbase runs a readability pass over the page before converting to Markdown - drops the chrome (nav, sidebar, footer, ad slots) and keeps the main article content. Best fit for converting blog posts and articles into clean LLM context. pretty booleanfalse Only meaningful with `format=json`. Pretty-prints the JSON envelope with indentation and newlines for human reading; leave off in production to keep responses small. scraper stringoptional Apply a built-in [scraper](/docs/scraper-api) to extract structured data instead of returning HTML. Example: `amazon-product-details`. autoparse booleanfalse Auto-detect the page type and apply the matching scraper. Convenience for "give me JSON when you can". #### Response control These parameters change what the response contains or how Crawlbase decides a request succeeded. Use `get_headers` and `get_cookies` when you need the target site's response headers or `Set-Cookie` values surfaced back to you (they're stripped by default). Use `custom_success_codes` when the target legitimately returns a non-2xx status that your pipeline should treat as a clean fetch - without it, Crawlbase will retry those responses on your behalf. get\_headers booleanfalse Surface the target site's response headers. They come back prefixed as `original_header_*` response headers, or grouped under `original_headers` when `format=json`. get\_cookies booleanfalse Surface the target site's `Set-Cookie` values. They come back as `original_set_cookie` in the response headers, or under the same key when `format=json`. custom\_success\_codes stringoptional Comma-separated list of HTTP status codes to treat as successful - e.g. `custom_success_codes=403,429,503`. Crawlbase won't retry these, and the original status is preserved in `original_status`. Use it when the target legitimately returns these codes for your endpoint (auth-gated APIs, region-blocked pages you still want the body of). ### POST requests Use POST when the target endpoint expects a request body - form submissions, JSON APIs, GraphQL, anything that doesn't fit in a query string. Same endpoint, same parameters, same response shape as GET; only the HTTP method and the body change. POST is Normal-token only POST requests work with the **Normal token** only. The JavaScript token (and the JS-rendering parameters `page_wait`, `ajax_wait`, `scroll`, `css_click_selector`) are GET-only - when you need to submit a form on a JS-rendered page, use the JavaScript token with `css_click_selector` to drive the form button instead of POSTing to the form URL directly. The default Content-Type is `application/x-www-form-urlencoded`. Pass the form fields as the request body - Crawlbase forwards them to the target unchanged. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://postman-echo.com/post' -G \ -F 'parameter1=testing some post data' \ -F 'parameter2=here goes some data' ``` ``` import requests from urllib.parse import quote_plus url = quote_plus('https://postman-echo.com/post') res = requests.post( f'https://api.crawlbase.com/?token=YOUR_TOKEN&url={url}', data={'parameter1': 'value', 'parameter2': 'another value'}, ) print(res.status_code, res.text) ``` ``` const url = encodeURIComponent('https://postman-echo.com/post'); const body = new URLSearchParams({ parameter1: 'value', parameter2: 'another' }); const res = await fetch(`https://api.crawlbase.com/?token=YOUR_TOKEN&url=${url}`, { method: 'POST', headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, body, }); console.log(res.status, await res.text()); ``` ``` require 'net/http' uri = URI('https://api.crawlbase.com') uri.query = URI.encode_www_form(token: 'YOUR_TOKEN', url: 'https://postman-echo.com/post') res = Net::HTTP.post_form(uri, 'parameter1' => 'value', 'parameter2' => 'another') puts res.code, res.body ``` ``` 'value', 'parameter2' => 'another']); $ch = curl_init('https://api.crawlbase.com/?token=YOUR_TOKEN&url=' . urlencode($url)); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $body); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); echo curl_exec($ch); ``` ``` package main import ( "fmt" "io" "net/http" "net/url" "strings" ) func main() { target := url.QueryEscape("https://postman-echo.com/post") body := strings.NewReader("parameter1=value¶meter2=another") res, _ := http.Post( "https://api.crawlbase.com/?token=YOUR_TOKEN&url="+target, "application/x-www-form-urlencoded", body, ) out, _ := io.ReadAll(res.Body) fmt.Println(string(out)) } ``` Don't abuse this POST cannot be used to spam or otherwise harm target websites. Crawlbase actively monitors for abusive patterns; accounts caught using POST for spam, credential stuffing, or other malicious traffic will be suspended and reported. #### POST with a JSON body Override the default form-urlencoded content type with `post_content_type`. URL-encode the value (e.g. `application/json` becomes `application%2Fjson`). The body is forwarded to the target unchanged - encode it as JSON yourself. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://postman-echo.com/post' \ --data-urlencode 'post_content_type=application/json;charset=UTF-8' -G \ --request POST \ --data '{"param1":"value","param2":"another"}' ``` ``` import json, requests from urllib.parse import quote_plus url = quote_plus('https://postman-echo.com/post') res = requests.post( f'https://api.crawlbase.com/?token=YOUR_TOKEN' f'&url={url}' f'&post_content_type=application/json', data=json.dumps({'param1': 'value', 'param2': 'another'}), headers={'Content-Type': 'application/json'}, ) print(res.status_code, res.text) ``` ``` const url = encodeURIComponent('https://postman-echo.com/post'); const ct = encodeURIComponent('application/json;charset=UTF-8'); const body = JSON.stringify({ param1: 'value', param2: 'another' }); const res = await fetch( `https://api.crawlbase.com/?token=YOUR_TOKEN&url=${url}&post_content_type=${ct}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body }, ); console.log(res.status, await res.text()); ``` ``` require 'net/http' require 'json' uri = URI('https://api.crawlbase.com') uri.query = URI.encode_www_form( token: 'YOUR_TOKEN', url: 'https://postman-echo.com/post', post_content_type: 'application/json' ) req = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json') req.body = { param1: 'value', param2: 'another' }.to_json res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |h| h.request(req) } puts res.code, res.body ``` ``` 'value', 'param2' => 'another']); $ch = curl_init( 'https://api.crawlbase.com/?token=YOUR_TOKEN' . '&url=' . urlencode($url) . '&post_content_type=' . $ct ); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $body); curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); echo curl_exec($ch); ``` ``` package main import ( "bytes" "fmt" "io" "net/http" "net/url" ) func main() { target := url.QueryEscape("https://postman-echo.com/post") ct := url.QueryEscape("application/json;charset=UTF-8") body := bytes.NewBufferString(`{"param1":"value","param2":"another"}`) res, _ := http.Post( "https://api.crawlbase.com/?token=YOUR_TOKEN&url="+target+"&post_content_type="+ct, "application/json", body, ) out, _ := io.ReadAll(res.Body) fmt.Println(string(out)) } ``` **Note:** the target site decides whether to accept the body. Crawlbase forwards the request honestly - if the target returns 4xx because the body shape is wrong, that surfaces in `original_status`, not in `cb_status`. See [Errors](#errors) for the branching pattern. ### PUT requests PUT works the same way as POST - same endpoint, same parameters, same body-encoding rules. The only difference is the HTTP method. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://api.example.com/resource/42' -G \ --request PUT \ --header 'Content-Type: application/json' \ --data '{"name":"updated","status":"active"}' ``` ``` import requests from urllib.parse import quote_plus url = quote_plus('https://api.example.com/resource/42') res = requests.put( f'https://api.crawlbase.com/?token=YOUR_TOKEN&url={url}&post_content_type=application/json', data='{"name":"updated","status":"active"}', headers={'Content-Type': 'application/json'}, ) print(res.status_code, res.text) ``` ``` const url = encodeURIComponent('https://api.example.com/resource/42'); const ct = encodeURIComponent('application/json'); const body = JSON.stringify({ name: 'updated', status: 'active' }); const res = await fetch( `https://api.crawlbase.com/?token=YOUR_TOKEN&url=${url}&post_content_type=${ct}`, { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body }, ); console.log(res.status, await res.text()); ``` ``` require 'net/http' require 'json' uri = URI('https://api.crawlbase.com') uri.query = URI.encode_www_form( token: 'YOUR_TOKEN', url: 'https://api.example.com/resource/42', post_content_type: 'application/json' ) req = Net::HTTP::Put.new(uri, 'Content-Type' => 'application/json') req.body = { name: 'updated', status: 'active' }.to_json res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |h| h.request(req) } puts res.code, res.body ``` ``` 'updated', 'status' => 'active']); $ch = curl_init( 'https://api.crawlbase.com/?token=YOUR_TOKEN' . '&url=' . urlencode($url) . '&post_content_type=' . $ct ); curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT'); curl_setopt($ch, CURLOPT_POSTFIELDS, $body); curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); echo curl_exec($ch); ``` ``` package main import ( "bytes" "fmt" "io" "net/http" "net/url" ) func main() { target := url.QueryEscape("https://api.example.com/resource/42") ct := url.QueryEscape("application/json") body := bytes.NewBufferString(`{"name":"updated","status":"active"}`) req, _ := http.NewRequest( "PUT", "https://api.crawlbase.com/?token=YOUR_TOKEN&url="+target+"&post_content_type="+ct, body, ) req.Header.Set("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) out, _ := io.ReadAll(res.Body) fmt.Println(string(out)) } ``` Like POST, PUT requires the Normal token. Use `post_content_type` to control the body's media type if it isn't form-urlencoded. Don't use POST/PUT to spam Crawlbase actively monitors POST and PUT traffic. Sending request bodies that target third-party sites you don't own - comment spam, fraudulent form submissions, scripted account creation - gets the originating account suspended on first detection. Use these verbs for legitimate API integrations, your own staging and production endpoints, and explicitly permitted automation. ## Response Successful responses return the target page in the body. Metadata lives in the response headers. ### Headers | Header | Description | | --- | --- | | `cb_status` | Crawlbase status code. `200` = success. Formerly named `pc_status`. | | `original_status` | HTTP status from the target site. | | `url` | Final URL after redirects. | | `rid` | Request ID. Returned when `async=true` or `store=true`. | | `content-type` | MIME type of the body (`text/html`, `application/json`, `image/png`, etc). | | `original_header_*` | Returned when `get_headers=true`. Each header from the target site arrives with an `original_header_` prefix (e.g. `original_header_x_frame_options`). Grouped under `original_headers` when `format=json`. | | `screenshot_url` | Returned when `screenshot=true`. Temporary JPEG URL for the rendered page; expires one hour after the crawl. | | `original_set_cookie` | Returned when `get_cookies=true`. Concatenated `Set-Cookie` values from the target site's response. | | `domain_complexity` also `X-Domain-Complexity` | The complexity tier of the crawled domain - one of `standard`, `moderate`, or `complex`. Reflects the resources required to bypass the site's protections and maps directly onto the pricing tier billed for the request. See [complexity tiers](#domain-complexity) below. | | `storage_url` | Returned when the request was made with `store=true`. Pointer to the stored copy of the response in [Crawlbase Cloud Storage](https://crawlbase.com/dashboard/storage); pair with `rid` to retrieve later. | | `Content-Type` | `text/markdown; charset=utf-8` when the request was made with `format=md`; the standard `text/html` or `application/json` otherwise. | | `X-Markdown-Flavor` | Markdown dialect of the response body - currently `GitHub Flavored Markdown (GFM)`. Only emitted when `format=md`. | | `X-Markdown-Features` | Comma-separated list of GFM features used in the body (e.g. `tables,lists`). Lets you pick a parser with the right extensions enabled. Only emitted when `format=md`. | | `X-Markdown-Base-URL` | Host of the resolved URL (after any redirects). Useful for resolving relative links in the markdown body. Only emitted when `format=md`. | | `X-Markdown-Generator` | Identifies the converter - value is `ProxyCrawl-API`. Only emitted when `format=md`. | ### HTML response The default. `format=html` (or no `format` at all) returns the raw page body in the HTTP body, with metadata in the response headers (`url`, `original_status`, `cb_status`, `X-Domain-Complexity`, plus any `original_header_*` entries you opted into via `get_headers=true`). ``` GET 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fgithub.com%2Fcrawlbase&format=html' Response: Headers: url: https://github.com/crawlbase original_status: 200 cb_status: 200 X-Domain-Complexity: standard Body: ... ... (full page HTML) ... ``` ### JSON response Set `format=json` to get the same data as a single JSON object instead: ``` GET 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fgithub.com%2Fcrawlbase&format=json' Response: { "original_status": 200, "cb_status": 200, "url": "https://github.com/crawlbase", "domain_complexity": "standard", "body": "... (full page HTML) ..." } ``` ### Markdown response `format=md` returns the page already converted to **GitHub Flavored Markdown** in the body, with `Content-Type: text/markdown; charset=utf-8` and a block of `X-Markdown-*` metadata headers (`Flavor`, `Features`, `Base-URL`, `Generator`) alongside the usual `url` / `original_status` / `cb_status`. Pair it with `md_readability=true` when you want main-content extraction (article body, no chrome) before the conversion runs - see the [`md_readability`](#request-params-format) parameter. ``` GET 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fgithub.com%2Fcrawlbase&format=md' Response: Headers: Content-Type: text/markdown; charset=utf-8 X-Markdown-Flavor: GitHub Flavored Markdown (GFM) X-Markdown-Features: tables,lists X-Markdown-Base-URL: github.com X-Markdown-Generator: ProxyCrawl-API url: https://github.com/crawlbase original_status: 200 cb_status: 200 Body: # crawlbase ... (markdown text of the page) ... ``` ### Billable requests Crawlbase only charges requests where `cb_status` is `200` **and** `original_status` is one of: | Code | Meaning | | --- | --- | | `200` | OK | | `201` | Created | | `204` | No Content | | `301` | Moved Permanently | | `302` | Found - only when the redirect was followed and returned content | | `404` | Not Found | | `410` | Gone | Any other `original_status` is free, and so is any non-`200` `cb_status`. Use this list when reconciling a usage invoice against your application logs. ### Domain complexity tiers The `domain_complexity` field (also returned as the `X-Domain-Complexity` response header) tells you how hard it was to crawl the target domain - and what pricing tier the request fell into. - **`standard`** : easy to crawl, minimal protection. Lowest pricing tier. - **`moderate`** : moderate anti-bot protection that needs specialised handling. Intermediate pricing tier. - **`complex`** : advanced protection requiring specialised resources. Highest pricing tier. For tier-specific pricing see your subscription plan or [contact sales](https://crawlbase.com/contact). ## Common patterns End-to-end recipes that combine the request parameters above into the workflows teams reach for most often. ### JS-rendered SPA with scroll ``` curl 'https://api.crawlbase.com/?token=JS_TOKEN' \ --data-urlencode 'url=https://feed.example.com' \ --data-urlencode 'page_wait=2000' \ --data-urlencode 'scroll=true' \ --data-urlencode 'scroll_interval=15' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'JS_TOKEN'}) res = api.get('https://feed.example.com', { 'page_wait': 2000, 'scroll': True, 'scroll_interval': 15, }) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'JS_TOKEN' }); const res = await api.get('https://feed.example.com', { page_wait: 2000, scroll: true, scroll_interval: 15, }); console.log(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'JS_TOKEN') res = api.get('https://feed.example.com', page_wait: 2000, scroll: true, scroll_interval: 15 ) puts res.body ``` ``` 'JS_TOKEN']); $res = $api->get('https://feed.example.com', [ 'page_wait' => 2000, 'scroll' => true, 'scroll_interval' => 15, ]); echo $res->body; ``` ``` package main import ( "fmt" "log" "github.com/crawlbase/crawlbase-go" ) func main() { api, err := crawlbase.NewCrawlingAPI("JS_TOKEN") if err != nil { log.Fatal(err) } res, _ := api.Get("https://feed.example.com", map[string]string{ "page_wait": "2000", "scroll": "true", "scroll_interval": "15", }) fmt.Println(res.Body) } ``` ### Render a page as PDF Pass `pdf=true` with a JavaScript token to get the fully rendered page back as a PDF file. Write the response body straight to disk; the rendering parameters (`country`, `device`, `page_wait`) shape the page before it is captured. Handy for exporting pages for reporting, archiving, or compliance. ``` # Basic - save the fully rendered page as a PDF curl 'https://api.crawlbase.com/?token=JS_TOKEN' \ --data-urlencode 'url=https://example.com' \ --data-urlencode 'pdf=true' -G -o example.pdf # With rendering controls (geo, device, wait) curl 'https://api.crawlbase.com/?token=JS_TOKEN' \ --data-urlencode 'url=https://en.wikipedia.org/wiki/Taylor_Swift' \ --data-urlencode 'pdf=true' \ --data-urlencode 'country=us' \ --data-urlencode 'device=desktop' \ --data-urlencode 'page_wait=5000' -G -o article.pdf ``` ### Geo-routed request ``` # Route through Germany; the echoed JSON confirms the exit country curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://ipinfo.io/json' \ --data-urlencode 'country=DE' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get('https://ipinfo.io/json', {'country': 'DE'}) print(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get('https://ipinfo.io/json', { country: 'DE' }); console.log(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://ipinfo.io/json', country: 'DE') puts res.body ``` ``` 'YOUR_TOKEN']); $res = $api->get('https://ipinfo.io/json', ['country' => 'DE']); echo $res->body; ``` ``` package main import ( "fmt" "log" "github.com/crawlbase/crawlbase-go" ) func main() { api, err := crawlbase.NewCrawlingAPI("YOUR_TOKEN") if err != nil { log.Fatal(err) } res, _ := api.Get("https://ipinfo.io/json", map[string]string{ "country": "DE", }) fmt.Println(res.Body) } ``` ### Async crawl with webhook When to use async Async releases your concurrency slot the moment the request is queued, so a long crawl doesn't tie up budget. Use it for slow targets (heavy JS, long `page_wait`) when you need to push high volume. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://example.com' \ --data-urlencode 'async=true' \ --data-urlencode 'callback=https://your-app.com/webhook' -G # → returns immediately: { "rid": "a1B2c3D4e5F6" } # → result POSTed to your callback when ready ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get('https://example.com', { 'async': 'true', 'callback': 'https://your-app.com/webhook', }) print(res['rid']) # → returned immediately; result POSTed to callback later ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get('https://example.com', { async: true, callback: 'https://your-app.com/webhook', }); console.log(res.rid); // → returned immediately; result POSTed to callback later ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://example.com', async: true, callback: 'https://your-app.com/webhook' ) puts res.rid # → returned immediately; result POSTed to callback later ``` ``` 'YOUR_TOKEN']); $res = $api->get('https://example.com', [ 'async' => 'true', 'callback' => 'https://your-app.com/webhook', ]); echo $res->rid; // → returned immediately; result POSTed to callback later ``` ``` package main import ( "fmt" "log" "github.com/crawlbase/crawlbase-go" ) func main() { api, err := crawlbase.NewCrawlingAPI("YOUR_TOKEN") if err != nil { log.Fatal(err) } res, _ := api.Get("https://example.com", map[string]string{ "async": "true", "callback": "https://your-app.com/webhook", }) fmt.Println(res.RID) // → returned immediately; result POSTed to callback later } ``` ## Errors & retries The Crawling API surfaces two status codes on every response: `original_status` (what the target site returned) and `cb_status` (what Crawlbase made of it after applying anti-bot, redirect, and content-validation rules). They can disagree - a target might return `200` with an empty body, in which case `original_status` is `200` but `cb_status` is `520`. Always branch on `cb_status` when deciding whether to retry. The most common Crawling-API-specific failures: | Code | Meaning | Action | | --- | --- | --- | | `422` | `url` missing or not URL-encoded | Encode the URL before sending. Most clients (libcurl --data-urlencode, Python requests, Node fetch) handle this automatically - but hand-built query strings often miss it. | | `520` | Empty response from target | Retry once. If still empty, switch from Normal to JS token - many sites serve an empty shell to non-browser user agents and rely on JS to populate. | | `521` | Target site is down / unreachable | Treat like a transient upstream error. Backoff + retry; if persistent over minutes, the site is genuinely down. | | `522` | Connection timed out reaching the target | Retry with backoff. Try a different `country` if the target is geo-flaky. | | `523` | Origin unreachable from the chosen exit | Retry without `country` (let auto-routing pick) or with a different country. | | `525` | Anti-bot challenge couldn't be solved | Switch from Normal to JS token. If already on JS, retry; if persistent, escalate to support - usually means the target rolled out a new challenge variant. | | `595` | Selector not found. The page loaded successfully but the CSS selector you passed via `css_click_selector` didn't match any element. | Append a fallback to the selector (`#start-button,body`) so the click still lands on a known element. See the [`css_click_selector`](#request-params-js) notes for the full pattern. | | `599` | Internal Crawlbase error | Retry. If a request hits this consistently, contact support with the `rid`. | Full HTTP + `cb_status` reference is in [Status Codes](/docs/status-codes); [Error handling](/docs/errors) covers the recommended retry-with-backoff loop and the SDK helpers that implement it for you in each language. **Anchoring example.** The most common reason `cb_status` diverges from `original_status` is a CAPTCHA: the target site returns a `200` (the captcha page rendered fine) but Crawlbase recognises the response as an interstitial and surfaces `cb_status: 503` so you can route around it instead of treating the captcha HTML as your data. **Non-standard `cb_status` codes.** Codes outside the usual HTTP range - `601`, `999`, and similar - are internal markers used by the Crawlbase engineering team. They're surfaced in the response only to help you debug when contacting support; you don't need to handle them in application code. ### Retry strategy The simple version: retry transient errors (5xx) with exponential backoff up to a cap (typically 3-5 attempts), don't retry client errors (4xx - they won't fix themselves), and switch token type once on the first 520/525 before retrying further. The [SDK helpers](/docs/sdks) implement this loop with sensible defaults; for a custom client, the rule of thumb is: - First retry: ~1s after failure - Second retry: ~3s after failure - Third retry: ~10s after failure - After that: log + alert; persistent failures usually mean a target-side change rather than transient networking All retries against this API are free - only successful responses (`cb_status: 200`) count against your quota. That makes aggressive backoff cheap; the only real cost of retrying is the latency you add to your pipeline. ## Performance & best practices A few patterns recur across customers running this API at scale. Adopting them up front avoids the most common support-ticket categories. - **Use the cheapest token that works.** Don't default to the JavaScript token "just in case" - Normal token requests are faster and use less concurrency. Promote to JS only when the Normal response is empty or challenge-blocked. - **Prefer `ajax_wait` over `page_wait`.** Fixed delays burn concurrency on every request, even fast ones. `ajax_wait` returns the moment the page goes network-idle - typically faster on average and only slower on truly long-loading pages. - **Push high volume through async + webhook.** Synchronous mode is the right default for ad-hoc and interactive use. For batch jobs over a few hundred URLs, the async mode (or the [Enterprise Crawler](/docs/crawler)) keeps your concurrency budget free for new submissions while existing crawls finish. - **Reuse sessions for stateful flows.** If your target requires a logged-in session or cart cookies, hold a session ID and pass it on subsequent requests so the same exit IP and cookie jar are reused. See [Authentication](/docs/authentication) for the session-cookie pattern. - **Watch the `remaining` header.** Backoff before you hit your concurrency cap rather than discovering it through 429s - the response carries the number of slots left, so a healthy client sleeps proactively instead of reacting to errors. [← PreviousOverview](/docs/api-reference)[Next →Enterprise Crawler](/docs/crawler) --- Source: https://crawlbase.com/docs/errors # Error Handling Crawling at scale means errors happen. Build for them up front and you'll spend your time shipping features, not babysitting retries. ## Three classes of error Every Crawlbase error falls into one of three buckets, and each needs a different response. Transient retry Network blip, brief upstream outage, rate limit. `429`, `500`, `503`, `522`, `599`. **Always retry with backoff.** Site-side handle The target site returned a real error: `404`, `410`, `451`. Don't retry - the page genuinely doesn't exist or isn't accessible. Mark the URL as failed and move on. Configuration fix code Your fault. `401`, `402`, `403`, `422`. Retrying won't help - fix the request, the token, or the account. ## Production retry pattern The pattern that holds up under load: **exponential backoff with full jitter** , capped attempts, and a dead-letter destination for terminal failures. ``` import time, random, logging from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) log = logging.getLogger('crawler') TRANSIENT = {429, 500, 503, 522, 599} TERMINAL = {401, 402, 403, 404, 410, 422, 451} def crawl(url, max_attempts=5, base=0.5, cap=30): for attempt in range(max_attempts): res = api.get(url) status = res['status_code'] if status == 200 and res['cb_status'] == 200: return res if status in TERMINAL or res['cb_status'] in TERMINAL: log.warning(f'Terminal error {status}/{res['cb_status']} for {url}') raise PermanentFailure(url, status) # Transient - sleep with full jitter, then retry wait = min(cap, base * (2 ** attempt)) wait = random.uniform(0, wait) log.info(f'Attempt {attempt+1} got {status}; sleeping {wait:.2f}s') time.sleep(wait) raise RuntimeError(f'Exhausted retries for {url}') class PermanentFailure(Exception): pass ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: process.env.CRAWLBASE_TOKEN }); const TRANSIENT = new Set([429, 500, 503, 522, 599]); const TERMINAL = new Set([401, 402, 403, 404, 410, 422, 451]); async function crawl(url, { maxAttempts = 5, base = 500, cap = 30000 } = {}) { for (let attempt = 0; attempt < maxAttempts; attempt++) { const res = await api.get(url); const status = res.statusCode; if (status === 200 && res.pcStatus === 200) return res; if (TERMINAL.has(status) || TERMINAL.has(res.pcStatus)) { throw new Error(`Permanent failure ${status} for ${url}`); } const wait = Math.random() * Math.min(cap, base * 2 ** attempt); await new Promise(r => setTimeout(r, wait)); } throw new Error(`Exhausted retries for ${url}`); } ``` ## Dead-letter queue When retries exhaust, don't drop the URL silently. Push it somewhere a human can review. - **For Crawler API users:** failures are automatically retried up to your configured count, then delivered to your webhook with the failure metadata. No DLQ to build. - **For direct API users:** on terminal failure, write the URL + status + last response body to a separate queue or table. Review weekly. Don't retry forever Cap retries at 5 or so. A URL that fails 5 times in a row almost certainly will fail 50 times. Save the cycles for new work. ## What to monitor The four signals every Crawlbase-using system should chart: | Signal | Where it comes from | Alert when | | --- | --- | --- | | Success rate | `cb_status == 200` / total | \< 95% sustained for 10 min | | P95 latency | request duration | \> 15s sustained | | 429 rate | HTTP status histogram | \> 5% sustained - bump concurrency | | Retry count distribution | your retry loop | P95 \> 2 - something's degrading upstream | Tag every metric with the target domain so you can spot when a single site is poisoning your overall numbers. ## Making retries safe Crawlbase requests are inherently idempotent - a `GET` on the same URL with the same token returns the same kind of result every time. You can retry freely without worrying about duplicate side effects. Two notes: - **Async + store:** if you used `&async=true&store=true`, each retry consumes a credit and creates a new `rid`. Dedupe on your end if needed. - **Webhooks:** Crawler API webhooks may be delivered more than once on failure. Make your webhook handler idempotent on `rid`. [← PreviousStatus Codes](/docs/status-codes)[Next →Overview](/docs/api-reference) --- Source: https://crawlbase.com/docs/get-started # Get Started Five short pages that take you from sign-up to a first successful crawl, then cover the operational details - authentication, quotas, errors - you'll need once you start sending real traffic. Read top-to-bottom, or jump to whichever piece you came here for. New here? Start with [Quick Start](/docs/quick-start) - it gets you a working request in under five minutes. The other pages in this section are reference material you can come back to as questions arise. ## Your first request - [Quick start](/docs/quick-start) - sign up, grab your token, and send a working crawl in five minutes. Code samples in curl, Python, Node.js, Ruby, PHP, Go, Java, and C#. Read this first. ## Authentication & limits Once requests are flowing, the next questions are usually "how does auth work?" and "how much can I send?". Two short reference pages cover both. - [Authentication](/docs/authentication) - Normal vs. JavaScript tokens, why there are two, when to use each, how to keep them out of your repo. Tokens authenticate every Crawlbase API the same way, so this applies platform-wide. - [Rate limits](/docs/rate-limits) - concurrency budgets per plan tier, the difference between request throughput and concurrent connections, and the pattern for backing off when you hit the ceiling. ## Status codes & errors Real traffic means real failures - captchas, geo-blocks, target sites going down, your own client misconfiguring a parameter. Two pages explain what comes back and what to do about it. - [Status codes](/docs/status-codes) - every HTTP status the platform returns and what it means. Crawlbase splits the response into two status fields (`cb_status` for our side, `original_status` for the target site) so you can tell apart the two failure modes. - [Error handling](/docs/errors) - recoverable vs. terminal errors, retry strategy, and the specific error envelopes the platform returns so your client can branch on them. ## What's next Once you're past Get Started, the platform splits along two axes: what you're building and how you want to integrate. - **By API surface** : the [API Reference](/docs/api-reference) covers Crawling API, Smart AI Proxy, Cloud Storage, Enterprise Crawler, and the smaller helpers (Account API, User Agents API). - **By integration shape** : [SDKs](/docs/sdks) for the seven major languages, [Integrations](/docs/integrations) for low-code platforms (LangChain, Zapier, n8n, Make, Airbyte), and the [AI & MCP](/docs/ai) section for agent-driven access through Claude, Cursor, VS Code, and other MCP-aware clients. - **By task** : the [Scraper Library](/docs/scrapers) offers ready-made scrapers that return structured JSON for common sites - usually faster than parsing HTML yourself. - **To experiment** : the [API Playground](/docs/api-playground) lets you build and run live requests in the browser without writing any client code. [Next →Quick start](/docs/quick-start) --- Source: https://crawlbase.com/docs/integrations # Integrations Native connectors for the orchestration and data-pipeline tools most teams already run. The same Crawling API, packaged as a node / module / app / source for each platform - so you don't have to wire HTTP requests by hand. When to use an integration vs the API If your workflow lives in one of the tools below, the integration is almost always the right call - it gives you typed inputs, structured outputs, and the platform's native error handling for free. If you're building a custom pipeline in your own codebase, point the [SDKs](/docs/sdks) or the [Crawling API](/docs/crawling-api) directly. Both end up calling the same endpoints; the integrations just remove the wiring. ## Available today - [`langchain`](/docs/integrations-langchain) - LangChain provider for Python and JS/TS. Drop a Crawlbase tool into an agent's toolbelt and the agent fetches live web content with one call. - [`zapier`](/docs/integrations-zapier) - Zapier app. Trigger a crawl from any Zap; the parsed result feeds into the next step (Sheets, Airtable, Slack, anything Zapier connects to). - [`n8n`](/docs/integrations-n8n) - n8n community node for self-hosted workflows. A single Crawlbase node calls the Crawling API natively - method, options, and outputs mapped to native n8n fields, no HTTP wiring. ## In development The integrations below have docs preview-published so you can see the shape of what's coming. The actual node / module / connector hasn't shipped yet - the page tells you the workaround for today (usually the platform's built-in HTTP client + the Crawling API directly) and the email link to be notified when the dedicated version lands. - [`make`](/docs/integrations-make) - visual scenario builder (formerly Integromat). Workaround: Make's HTTP app + Crawling API. - [`airbyte`](/docs/integrations-airbyte) - open-source data pipelines. Workaround: HTTP API source against the Crawling API, or push to [Cloud Storage](/docs/cloud-storage) and ingest the bucket via Airbyte's S3 source. ## Don't see your tool? Most platforms with an HTTP-action primitive can call Crawlbase directly - the [Crawling API](/docs/crawling-api) is a regular HTTPS endpoint with token authentication. The [API Playground](/docs/api-playground) produces request templates you can paste into any platform's HTTP step. If you want a dedicated integration for a tool that isn't on this page, write to [support](/docs/support) with the use case - the roadmap is partly demand-driven. [← PreviousC# / .NET](/docs/sdk-csharp)[Next →LangChain](/docs/integrations-langchain) --- Source: https://crawlbase.com/docs/integrations-airbyte # Airbyte Pipe Crawlbase output directly into Snowflake, BigQuery, Redshift, or Postgres. The Airbyte source connector handles incremental sync and schema management. Coming soon - preview of how it will work The dedicated Crawlbase Airbyte source connector is in development. The setup + streams below are a preview of the shipped flow. [Email us](/docs/support) to be notified when it lands. Need it today? Use Airbyte's **HTTP API** source against the [Crawling API](/docs/crawling-api), or push results to [Cloud Storage](/docs/cloud-storage) and ingest the bucket via Airbyte's S3 source - both work end-to-end without the dedicated connector. ## Setup 1. In your Airbyte instance, go to **Sources → New Source**. 2. Search for **Crawlbase** and select it. 3. Configure: paste your token, choose a Crawler (the queue you push URLs to), pick which streams to sync. 4. Test the connection, save, and connect to a destination. ## Streams crawl\_results incremental Every completed crawl, one row per URL. Columns: `rid`, `url`, `cb_status`, `original_status`, `completed_at`, `body`, `headers`. scraper\_outputs incremental Structured scraper results, with per-scraper schemas (Amazon, Google, etc.) automatically inferred and exposed as nested columns. crawler\_status full refresh Snapshot of crawler queue health: queued, in-progress, completed/failed counts per crawler. ## Patterns - **Hourly product price warehouse:** push product URLs to a Crawler with the Amazon scraper. Sync every hour. Build a dbt model on top to flag price drops. - **Compliance archive:** daily full-page crawls of regulated sites, synced to S3 via Airbyte. Time-stamped, schemaed, queryable. - **SEO competitive monitoring:** SERPs scraped weekly, synced to BigQuery, dashboarded in Looker. [← PreviousMake](/docs/integrations-make)[Next →API Playground](/docs/api-playground) --- Source: https://crawlbase.com/docs/integrations-langchain # LangChain Drop-in document loaders, retrievers, and agent tools for LangChain. Crawl any URL straight into your retrieval pipeline or expose Crawlbase as a tool the agent can call. ## Install ``` pip install langchain-crawlbase ``` Lightweight install - only `langchain-core` and `requests` come along, no other LangChain extras required. Tested on Python 3.9+. ## Document loader Use `CrawlbaseLoader` anywhere LangChain expects a loader - RAG pipelines, vectorstore ingestion, agent context. ``` from langchain_crawlbase import CrawlbaseLoader loader = CrawlbaseLoader( urls=["https://example.com/blog/post-1", "https://example.com/blog/post-2"], token="YOUR_TOKEN", ) docs = loader.load() from langchain_community.vectorstores import Chroma from langchain_openai import OpenAIEmbeddings vs = Chroma.from_documents(docs, OpenAIEmbeddings()) ``` ## Agent tool Expose Crawlbase as an agent tool so the LLM can fetch URLs on demand. ``` from langchain_openai import ChatOpenAI from langchain_crawlbase import CrawlbaseTool tool = CrawlbaseTool(token="YOUR_TOKEN") llm = ChatOpenAI(model="gpt-4o").bind_tools([tool]) llm.invoke("What's on the homepage of anthropic.com today?") ``` ## Retriever `CrawlbaseRetriever` fetches a fixed set of seed URLs and returns documents matching a query. Useful when you want live results without standing up a vector store. ``` from langchain_crawlbase import CrawlbaseRetriever retriever = CrawlbaseRetriever( token="YOUR_TOKEN", urls=[ "https://crawlbase.com/docs/crawling-api", "https://crawlbase.com/docs/crawling-api#parameters", ], ) docs = retriever.invoke("how do I render JavaScript pages") ``` v0.1 uses case-insensitive substring matching against the fetched Markdown. For semantic retrieval, pair `CrawlbaseLoader` with the vector store of your choice. ## JavaScript-rendered pages For SPAs and pages whose content loads via JavaScript, pass your **JavaScript token** in the same `token` parameter - Crawlbase routes the request based on which token you send. No extra flag needed. ## Extra Crawlbase parameters Forward any [Crawlbase API parameter](/docs/crawling-api#parameters) (`country`, `device`, `page_wait`, `scroll`, `css_click_selector`, cookies, screenshots, etc.) via `extra_params`. ``` loader = CrawlbaseLoader( token="YOUR_TOKEN", urls=["https://example.com"], extra_params={"country": "US", "device": "mobile"}, ) ``` ## Document metadata Each `Document` returned by the loader / retriever carries response metadata from Crawlbase: - `source`: the URL you requested - `resolved_url`: the final URL after any redirects (when different from source) - `cb_status`: Crawlbase's final status code - `original_status`: HTTP status returned by the target site - `content_type`: response content-type header ## Common patterns - **RAG over a fresh crawl:** use `CrawlbaseLoader` to grab a few seed URLs, split into chunks, embed, query. - **Live web research agent:** register `CrawlbaseTool` alongside a search tool - the agent searches first, then crawls relevant results. - **Site monitoring:** schedule the loader to re-fetch the same URLs daily and diff into your vector store. [← PreviousOverview](/docs/integrations)[Next →Zapier](/docs/integrations-zapier) --- Source: https://crawlbase.com/docs/integrations-make # Make Visual scenario builder. Drop in a Crawlbase module, drag connections to your other apps, run on schedule or on demand. Coming soon - preview of how it will work The dedicated Crawlbase Make module is in development. The setup steps below are a preview of the shipped flow. [Email us](/docs/support) to be notified when it lands. Need it today? Use Make's built-in **HTTP** app and call the [Crawling API](/docs/crawling-api) directly - outputs map cleanly to downstream modules with one extra wiring step. ## Setup 1. In your Make scenario editor, click the **+** button and search for **Crawlbase**. 2. Pick a module (Crawl, Scrape, Screenshot, Crawler). 3. Click **Add** next to _Connection_ and paste your token. Save. 4. Configure module fields. Outputs are structured (status, body, headers as named items) so downstream modules can map them directly. ## Available modules Crawl a URL module Standard Crawling API with all parameters as form fields. Scrape Structured Data module Pick a scraper from a dropdown - output bundles all extracted fields. Capture Screenshot module Returns the image as a binary that other modules (Drive, S3, email) can save or attach. ## Example scenarios - **Watch and notify:** Schedule (every 6h) → Scrape product → Comparator → Email if price changed - **Lead pipeline:** Webhook → Scrape LinkedIn → Hubspot upsert - **Visual archive:** Schedule → Screenshot → Google Drive folder by date [← Previousn8n](/docs/integrations-n8n)[Next →Airbyte](/docs/integrations-airbyte) --- Source: https://crawlbase.com/docs/integrations-n8n # n8n Open-source automation, your servers. The Crawlbase n8n community node gives you the same APIs in a self-hosted workflow with no SaaS lock-in. ## Install The Crawlbase node is published as a community node. Install it from your n8n instance: 1. Go to **Settings → Community Nodes → Install a community node**. 2. Enter `n8n-nodes-crawlbase` and click Install. 3. Restart n8n if prompted. The Crawlbase node now shows up in the canvas search. ## Credentials Add a **Crawlbase API** credential under **Settings → Credentials** : 1. Paste your **API Token** from the [Crawlbase dashboard](https://crawlbase.com/dashboard). 2. Click **Test connection** to confirm the token is valid before running a workflow. Use your **Normal Token** for HTML targets and your **JavaScript Token** for SPAs and JS-rendered pages - create one credential per token tier and pick the right one per node. ## The Crawlbase node A single **Crawlbase** node wraps the [Crawling API](/docs/crawling-api). Drop it into a workflow, point it at a credential, and configure the request fields below. Method field GET, POST, or PUT. Use POST/PUT when the target needs a request body. Response format field HTML (default), JSON (parsed scraper output), or Markdown (clean text for LLM pipelines). Options field Optional Crawling API parameters - `page_wait`, `country`, `device`, `request_headers`, cookies, scraper, screenshot, store, async, and JS-rendering helpers. See the [Crawling API parameters](/docs/crawling-api) reference for the full list. Output field Each item returns `statusCode`, `headers`, `body`, and `metadata` (with `originalStatus`, `cbStatus`, and the resolved `url`). ## Item-list mode Set **URL Source** to **From input item field** and name the field that carries the URL (for example `url`). The node runs one Crawling API request per input item and emits one output item per input - pipe a Read-from-Sheet, Split-In-Batches, or any list-producing node straight in. ## Rate limits and retries Crawlbase rate limits depend on your plan. To keep workflows resilient: - Enable n8n's **Retry On Fail** on the Crawlbase node (Settings tab on the node). - Set **Wait Between Tries** to at least 1 second - higher if you hit limits. - For large URL lists, batch the work with **Loop Over Items** or **Split In Batches** rather than firing all requests at once. ## Common workflows - **Schedule → Crawlbase → Postgres:** daily snapshot of a competitor's pricing page into a database. - **Webhook → Crawlbase → Email:** on-demand product enrichment. - **RSS → Crawlbase → Vector DB:** populate a self-hosted retrieval index. [← PreviousZapier](/docs/integrations-zapier)[Next →Make](/docs/integrations-make) --- Source: https://crawlbase.com/docs/integrations-zapier # Zapier No-code automation. Trigger a Crawlbase call from any Zapier event and pipe the result into 6,000+ apps - Sheets, Slack, Notion, Airtable, your CRM. ## Setup 1. Open your [Zapier dashboard](https://zapier.com/apps) and search for **Crawlbase**. 2. Click **Connect Account**. Paste your Normal token (and JS token if you need rendering). 3. The Crawlbase app is now available as an Action in any Zap. ## Available actions Crawl URL action Run the Crawling API on any URL. Returns body, status, and headers as separate Zapier fields. Scrape Structured Data action Run a built-in scraper (Amazon, Google, LinkedIn, etc.) and get the parsed JSON back as fields. Take Screenshot action Render a URL as a PNG. The image is uploaded to Zapier's storage and available as a file field for downstream actions. ## Example Zaps - **Daily price tracker:** Schedule → Scrape Amazon Product → Slack message if price drops - **Lead enrichment:** New row in Sheets → Scrape LinkedIn Profile → Update row with title, company, location - **Visual monitoring:** Schedule → Take Screenshot → Email to yourself if the diff exceeds threshold - **Competitor watch:** RSS new item → Crawl URL → Append to Notion database Pair with Filter and Formatter Zapier's Filter and Formatter steps work great with Crawlbase output - extract a price with regex, normalize a date, branch on stock status. [← PreviousLangChain](/docs/integrations-langchain)[Next →n8n](/docs/integrations-n8n) --- Source: https://crawlbase.com/docs/leads-api # Leads API Domain-scoped email extraction. Useful for lead generation pipelines, contact research, and outreach list enrichment. Closed to new sign-ups since Oct 1, 2024 The Leads API has no direct modern replacement - for similar workflows, see the [email-extractor scraper](/docs/scrapers/email-extractor) (any URL → emails) or the [google-serp scraper](/docs/scrapers/google-serp) for domain-scoped contact discovery. Existing integrations continue to work, no shutdown is scheduled. ## Overview The Leads API extracts publicly visible email addresses associated with a domain. Useful for lead generation, contact research, and verifying outreach lists. **Endpoint:** `https://api.crawlbase.com/leads` ## Quickstart ``` curl 'https://api.crawlbase.com/leads?token=YOUR_TOKEN&domain=slack.com' ``` ## Parameters token stringrequired Your Crawlbase token. domain stringrequired Bare domain (no scheme, no path). Example: `slack.com`. limit integer10 Maximum number of emails to return. Defaults to 10. Billing is **1 credit per 10 emails** (or fewer) per domain - a `limit=100` call that returns 100 emails consumes 10 credits. ## Response shape JSON with discovered email addresses, source URLs where each was found, and contact metadata where available. ## Modern equivalent Two paths: - For full LinkedIn-style enrichment (titles, companies, profiles): use [Crawling API](/docs/crawling-api) with `scraper=linkedin-profile` or `scraper=linkedin-company`. - For raw email extraction: use `scraper=email-extractor` on any page, or fetch the page and run regex over the body. The dedicated endpoint became hard to keep correct as websites evolved their contact-page structures. [← PreviousScraper API](/docs/scraper-api)[Next →Screenshots API](/docs/screenshots-api) --- Source: https://crawlbase.com/docs/legacy # Legacy APIs APIs that predate the modern Crawlbase platform but are still operational. Some are closed to new sign-ups; some are simply older shapes being gradually replaced. None are scheduled for shutdown. What does "legacy" mean here? These APIs are still fully operational for existing customers. Some are closed to new sign-ups (Scraper, Leads, Screenshots) and some are simply older products being gradually replaced by newer ones. None are being shut down. If you're already using them, nothing changes. If you're starting fresh, the modern alternatives are linked below. ## Legacy APIs in this section [Scraper API](/docs/scraper-api) Standalone JSON-extraction endpoint. Closed to new sign-ups Oct 2024. Modern path: [Crawling API](/docs/crawling-api) + `&scraper=`. [Leads API](/docs/leads-api) Email lead extraction by domain. Closed to new sign-ups Oct 2024. [Screenshots API](/docs/screenshots-api) Standalone screenshot endpoint. Closed to new sign-ups Nov 2024. Modern path: [Crawling API](/docs/crawling-api) + screenshot params, or [`crawl_screenshot`](/docs/ai-mcp#crawl-tools) via MCP. [Proxy Backconnect API](/docs/proxy-api) Management API for the Backconnect proxy. Modern path: [Smart AI Proxy](/docs/smart-proxy). [Account API](/docs/account-api) Monitor monthly usage, credits, and per-domain success rates across products. Still actively supported. [User Agents API](/docs/user-agents-api) Random User-Agent strings optimized for web crawling. Free to use, rate-limited to 1 req/sec. ## When to use legacy vs modern | If you need… | Modern API | Why | | --- | --- | --- | | Structured JSON from a known site | [Crawling API](/docs/crawling-api) with `&scraper=name` | Same scrapers, simpler endpoint, more parameters | | A page screenshot | [Crawling API](/docs/crawling-api) + screenshot params, or MCP's `crawl_screenshot` | Combined with the same JS-rendering pipeline you already use | | Rotating residential proxy | [Smart AI Proxy](/docs/smart-proxy) | Better routing, AI ban avoidance, fewer config knobs | | Account-level metadata | [Account API](/docs/account-api) | This is still the canonical endpoint - it lives in this section because it predates the modern split, not because it's deprecated | | Random User-Agent strings | [User Agents API](/docs/user-agents-api) | Same - free, supported, no replacement planned | [← PreviousSupport](/docs/support)[Next →Scraper API](/docs/scraper-api) --- Source: https://crawlbase.com/docs/proxy-api # Proxy API A residential proxy network with sticky sessions and a flat single-port endpoint. Use it when Smart AI Proxy isn't flexible enough - direct access to the IP pool with rotation under your control. Migrate to [Smart AI Proxy](/docs/smart-proxy) Faster routing, AI-driven ban avoidance, fewer config knobs. Proxy Backconnect is deprecated but existing integrations continue to work - no shutdown is scheduled. ## Endpoint PROXYproxy.crawlbase.com:9000 ``` # Username = your token + optional session/country qualifiers # Password = blank ``` ## Basic usage ``` curl -x 'http://YOUR_TOKEN:@proxy.crawlbase.com:9000' \ 'https://httpbin.org/ip' ``` ``` import requests proxies = { 'http': 'http://YOUR_TOKEN:@proxy.crawlbase.com:9000', 'https': 'http://YOUR_TOKEN:@proxy.crawlbase.com:9000', } res = requests.get('https://httpbin.org/ip', proxies=proxies) ``` Each request goes through a different residential IP by default - pure rotation. ## Sticky sessions To pin requests to a single IP, append a session ID to your username. The same session ID returns the same IP for ~30 minutes. ``` # Format: TOKEN-session-SESSION_ID curl -x 'http://YOUR_TOKEN-session-checkout42:@proxy.crawlbase.com:9000' \ 'https://shop.example.com/cart' ``` ## Country targeting ``` # Format: TOKEN-country-XX curl -x 'http://YOUR_TOKEN-country-DE:@proxy.crawlbase.com:9000' \ 'https://www.amazon.de' # Combine: country + session curl -x 'http://YOUR_TOKEN-country-DE-session-cart-1:@proxy.crawlbase.com:9000' \ 'https://www.amazon.de/cart' ``` ## Username qualifiers Reference of all available username modifiers, combined with hyphens. | Format | Effect | | --- | --- | | `TOKEN` | Default - random IP per request | | `TOKEN-country-XX` | IPs from country `XX` (ISO 3166) | | `TOKEN-session-NAME` | Sticky to one IP for ~30 min | | `TOKEN-country-XX-session-NAME` | Sticky session within a country | ## Static IPs and geolocalization When you need to keep the same exit IP across multiple requests - multi-step checkout flows, account login + scrape, anything that depends on the target site recognizing the same client - Backconnect can lock a static IP and hand back the port + session lifetime. The proxy then keeps that IP bound to your token as long as you keep using it inside the session window. The static-IP endpoint is rate-limited to **1 request per 5 minutes per country** (or 1 per 5 minutes overall if no country is specified). Cache the port your client receives - don't re-request on every crawl. Plan-gated feature Static IPs and country targeting may not be available on every Backconnect plan, and country availability varies by tier. If you get an unauthorized response, check your plan or contact support before retrying. ### Get a static IP Returns a port to use with `proxy.crawlbase.com`, plus the seconds the binding stays alive. The session timer auto-extends as long as you keep sending traffic through that port within the window. GEThttps://api.crawlbase.com/proxy/static ``` curl 'https://api.crawlbase.com/proxy/static?token=YOUR_TOKEN' # Response # { "port": 1234, "host": "proxy.crawlbase.com", "session_time": 10 } ``` ### Static IP from a specific country Pass a 2-letter ISO 3166 country code (`US`, `GB`, `DE`, `IT`, `RU`, …) to pin the static IP to that geography. If no port is currently free in the requested country, the response carries an error - back off and retry against the country-bucket rate limit. ``` curl 'https://api.crawlbase.com/proxy/static?token=YOUR_TOKEN&country=FR' # Response # { "port": 4551, "host": "proxy.crawlbase.com", "session_time": 10 } ``` ## IP whitelisting Backconnect supports IP whitelisting - let your server's outbound IPs authenticate against the proxy without sending the token on every request. Useful for fixed-IP pipelines that don't want credentials in code, and the only way to use Backconnect from environments that can't pass a Proxy-Authorization header. Three endpoints, all under `/proxy/whitelist_ips`, differentiated by HTTP method. Changes take **up to 1 minute** to propagate. Plan-gated feature Whitelist management may not be available on every plan, and a per-plan cap limits how many IPs can be whitelisted at once. An unauthorized response means your tier doesn't include the feature; contact support to upgrade. ### Add a whitelisted IP POSThttps://api.crawlbase.com/proxy/whitelist\_ips ``` curl -X POST 'https://api.crawlbase.com/proxy/whitelist_ips?token=YOUR_TOKEN&ip=123.123.123.123' ``` ### Remove a whitelisted IP DELETEhttps://api.crawlbase.com/proxy/whitelist\_ips ``` curl -X DELETE 'https://api.crawlbase.com/proxy/whitelist_ips?token=YOUR_TOKEN&ip=123.123.123.123' ``` ### View whitelisted IPs Returns the current list of whitelisted IPs for your token, in JSON. ``` curl 'https://api.crawlbase.com/proxy/whitelist_ips?token=YOUR_TOKEN' ``` ## Backconnect vs Smart AI Proxy | | Smart AI Proxy | Backconnect | | --- | --- | --- | | Use case | Easy mode - auto-routing, anti-bot bypass | Direct IP pool access, custom rotation | | JS rendering | Available | No (raw proxy) | | Auto-retry | Yes | No - handle yourself | | Sticky sessions | Via header | Via username | | Best for | Most users | Custom scrapers, network research | Pick Smart AI Proxy first Backconnect is the lower-level tool. If you don't have a specific reason to use it, Smart AI Proxy gives better results with less work. [← PreviousScreenshots API](/docs/screenshots-api) --- Source: https://crawlbase.com/docs/quick-start # Quick start Sign up, grab a token, send your first request. From zero to crawling in less time than it takes to brew coffee. ## Prerequisites You need exactly two things: - A free [Crawlbase account](https://crawlbase.com/signup) - gets you up to 20,000 free requests, no credit card. - Either `curl` in your shell, or one of our [official SDKs](/docs/sdk-python) in your project. Two tokens, one account Each account has a **Normal token** (TCP, fastest) and a **JavaScript token** (full Chrome rendering). Pick based on the site - most APIs and static pages work with the Normal token. ## Your first request The Crawling API takes a single required parameter - `url`: fully URL-encoded. Drop in your token and you're crawling. GEThttps://api.crawlbase.com/?token=YOUR\_TOKEN&url=ENCODED\_URL ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fhttpbin.org%2Fheaders' ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get('https://httpbin.org/headers') print(res['status_code']) print(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get('https://httpbin.org/headers'); console.log(res.statusCode, res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://httpbin.org/headers') puts res.status_code puts res.body ``` ``` 'YOUR_TOKEN']); $res = $api->get('https://httpbin.org/headers'); echo $res->statusCode . PHP_EOL; echo $res->body; ``` ``` package main import ( "fmt" "github.com/crawlbase/crawlbase-go" ) func main() { api := crawlbase.NewCrawlingAPI("YOUR_TOKEN") res, _ := api.Get("https://httpbin.org/headers") fmt.Println(res.StatusCode) fmt.Println(res.Body) } ``` Windows Command Prompt On Windows Command Prompt (`cmd.exe`), replace the single quotes around the URL with double quotes: `curl "https://api.crawlbase.com/?token=YOUR_TOKEN&url=ENCODED_URL"`. Single quotes are a Unix-shell convention; `cmd.exe` passes them through literally and the request fails. PowerShell, macOS, and Linux shells accept the single-quoted form above. You'll get back the page HTML, plus a few headers describing what happened upstream. The most important ones: original\_status int The HTTP status the target site returned to us. Useful for distinguishing "site says 404" from "we couldn't reach the site". cb\_status int The Crawlbase status code. `200` means success. See [status codes](/docs/status-codes) for the full list. url string The final URL after any redirects. Useful when you want to know where you actually landed. rid stringoptional A request identifier returned when you use `&async=true` or `&store=true`. Use it to look up the page in [Cloud Storage](/docs/cloud-storage). ## Need JavaScript rendering? Sites built with React, Vue, Angular, or anything that ships an empty HTML shell need a real browser. Switch to your **JavaScript token** : same endpoint, different token. ``` curl 'https://api.crawlbase.com/?token=YOUR_JS_TOKEN&url=https%3A%2F%2Freact-app.example.com&page_wait=2000' ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_JS_TOKEN'}) res = api.get('https://react-app.example.com', { 'page_wait': 2000, 'ajax_wait': True, }) print(res['body']) ``` Useful JS-rendering parameters: - `page_wait`: wait N milliseconds after load (defaults to 0). - `ajax_wait`: wait until network is idle. - `css_click_selector`: click an element before capturing. See the full list in [Crawling API parameters](/docs/crawling-api). ## Next steps You're crawling. Now pick a path: [Master the Crawling API](/docs/crawling-api) Every parameter, every header, every status code. [Use a ready-made scraper](/docs/scraper-api) Skip the parsing. Scrapers return clean JSON. [Scale to millions](/docs/crawler) Push URLs to the Enterprise Crawler queue. [Plug into your AI agent](/docs/ai-mcp) MCP server, Claude integration, prompt patterns. [← PreviousOverview](/docs/get-started)[Next →Authentication](/docs/authentication) --- Source: https://crawlbase.com/docs/rate-limits # Rate Limits Crawlbase enforces per-token concurrency limits, not requests-per-minute. Send as fast as you want, just keep parallel in-flight requests under your ceiling. ## Default limits Every account starts with the same generous defaults. You don't need to ask for these - they're already on every token the moment you sign up. | Limit | Default | Scope | | --- | --- | --- | | Concurrent requests | `20` | per token | | Requests per second | ~ `20` (derived) | per token | | Total monthly requests | up to `51,000,000` | per token | | Single-request timeout | `90` seconds | per request | | Crawler queue size | `100,000` URLs | per crawler | "Concurrent" means in-flight at the same moment. If each crawl takes 2 seconds and you keep 20 parallel, you'll do roughly 10 req/sec sustained - that math works out to ~864K requests per day per token. Concurrency, not RPS Don't try to throttle to "X requests per second" - just cap your worker pool at 20 and let request latency dictate throughput. It's simpler and matches how the limit actually works server-side. ## What happens when you hit the limit Cross the concurrency ceiling and Crawlbase responds with HTTP `429 Too Many Requests`. The request is rejected - it's not queued - so you should retry with backoff. ``` // HTTP/1.1 429 Too Many Requests // Retry-After: 1 // X-Crawlbase-Concurrency: 20 { "error": "Concurrency limit reached", "limit": 20 } ``` The `Retry-After` header tells you the minimum seconds before retrying. Always honor it. ## Handling rate limits gracefully The right pattern is **exponential backoff with jitter**. Most HTTP clients have this built in; here's the bare minimum if you need to roll your own. ``` import time, random from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) def crawl_with_retry(url, attempts=5): for i in range(attempts): res = api.get(url) if res['status_code'] != 429: return res wait = (2 ** i) + random.random() time.sleep(wait) raise RuntimeError('Rate limit exhausted') ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); async function crawlWithRetry(url, attempts = 5) { for (let i = 0; i < attempts; i++) { const res = await api.get(url); if (res.statusCode !== 429) return res; const wait = (2 ** i) * 1000 + Math.random() * 1000; await new Promise(r => setTimeout(r, wait)); } throw new Error('Rate limit exhausted'); } ``` For workloads that consistently push the ceiling, use a **worker pool** instead of fire-and-forget. Cap the pool at your concurrency limit and you'll never see a 429. ## Scaling beyond the defaults Three options when 20 concurrent isn't enough: Higher concurrency contact us We routinely set tokens to 100, 500, or 1,000+ concurrent for customers with the volume to justify it. [Email support](/docs/support) with your target throughput and use case. Multiple tokens free Each Crawlbase account can have multiple sub-accounts, each with its own token and concurrency budget. Use this to isolate workloads. Enterprise Crawler async Push URLs to a managed queue and let Crawlbase handle concurrency, retries, and delivery to your webhook. No client-side scheduling required. ## Best practices - **Cap worker pools at your token limit.** Don't oversubscribe and rely on 429s - the rejected request still takes a round-trip. - **Use `async=true` for slow targets.** Long-running JS-rendered crawls block a concurrency slot for the entire request. Async mode releases the slot immediately and delivers the result via webhook. - **Always honor `Retry-After`.** Ignoring it just creates more 429s and burns network round-trips. - **Add jitter to retries.** If 50 workers all retry at exactly 1s after a transient spike, you'll just spike again. - **Alert on sustained 429 rate.** An occasional 429 is fine. A sustained 5%+ rate means you need higher concurrency or smarter scheduling. [← PreviousAuthentication](/docs/authentication)[Next →Status Codes](/docs/status-codes) --- Source: https://crawlbase.com/docs/scraper-api # Scraper API Skip the parsing. Pick a scraper, point it at a URL, get back clean structured JSON. Scrapers cover Amazon, Google, LinkedIn, Instagram, eBay, and many more. Migrate to the [Crawling API](/docs/crawling-api) with `&scraper=name` Same scrapers, simpler endpoint, more parameters. The standalone Scraper API has been closed to new sign-ups since Oct 1, 2024 - existing integrations continue to work, no shutdown is scheduled, and migrating is a one-line URL change. ## Endpoint GEThttps://api.crawlbase.com/scraper?token=YOUR\_TOKEN&url=ENCODED\_URL&scraper=NAME ``` # Identical to the Crawling API, plus a required `scraper` parameter. # Returns parsed JSON instead of raw HTML. ``` ## Quickstart - Amazon product ``` curl 'https://api.crawlbase.com/scraper?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/dp/1098145356' \ --data-urlencode 'scraper=amazon-product-details' -G ``` ``` from crawlbase import ScraperAPI api = ScraperAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.amazon.com/dp/1098145356', {'scraper': 'amazon-product-details'} ) import json data = json.loads(res['body']) print(data['name'], data['price']) ``` ``` const { ScraperAPI } = require('crawlbase'); const api = new ScraperAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.amazon.com/dp/1098145356', { scraper: 'amazon-product-details' } ); const data = JSON.parse(res.body); console.log(data.name, data.price); ``` Sample response: ``` { "name": "Web Scraping with Python: Data Extraction from the Modern Web", "asin": "1098145356", "brand": "O'Reilly Media", "price": "$59.99", "availability": "In Stock", "rating": 4.6, "reviews_count": 214, "main_image": "https://m.media-amazon.com/images/I/…", "images": ["…"], "features": ["Paperback, 3rd Edition by Ryan Mitchell…"], "description": "A hands-on guide to extracting data from the modern web with Python…" } ``` ## Scraper catalog A representative slice of the scrapers available. Pass the scraper name as the `scraper` parameter. ### Amazon | Scraper | Returns | | --- | --- | | `amazon-product-details` | Product page: name, price, ratings, images, features | | `amazon-search-results` | Search listings page: products, pagination, filters | | `amazon-reviews` | Review page with rating, author, date, body, helpful counts | | `amazon-bestsellers` | Best Sellers ranked listings by category | | `amazon-questions` | Customer Q&A section | ### Google | Scraper | Returns | | --- | --- | | `google-serp` | Search results: organic, ads, knowledge panel, related searches | | `google-shopping` | Shopping tab listings with merchant, price, rating | | `google-news` | News tab results with source, snippet, time | | `google-maps` | Place page: name, address, hours, ratings, reviews | | `google-scholar` | Academic search results with citations | ### Social networks | Scraper | Returns | | --- | --- | | `linkedin-profile` | Public profile data: experience, education, skills | | `linkedin-company` | Company page: size, industry, headquarters | | `instagram-profile` | Profile metadata, recent posts, follower counts | | `tiktok-profile` | TikTok user profile and recent videos | | `youtube-channel` | Channel metadata, subscriber count, recent uploads | ### Other marketplaces | Scraper | Returns | | --- | --- | | `ebay-product-details` | eBay listing data | | `walmart-product` | Walmart product page | | `yelp-business` | Yelp business listing with reviews summary | | `booking-hotel` | Booking.com hotel page with rates and amenities | | `tripadvisor-attraction` | TripAdvisor attraction page | Don't see what you need? The full catalog is in your dashboard. New scrapers are added monthly. [Email us](/docs/support) if you need a custom scraper for a site we don't cover yet. ## Auto-detect with autoparse If you know the URL but don't want to look up the right scraper name, use `autoparse=true` on the standard [Crawling API](/docs/crawling-api) endpoint. We'll detect the page type and apply the matching scraper automatically. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/dp/1098145356' \ --data-urlencode 'autoparse=true' -G # Crawlbase recognizes the Amazon URL and auto-applies amazon-product-details ``` ## Parameters The Scraper API endpoint accepts the same shape as the [Crawling API](/docs/crawling-api#parameters), narrowed to the five params below plus the `scraper` name. For deeper notes on each shared param, the Crawling API reference is canonical - this list is the self-contained reference for the legacy `/scraper` endpoint. token stringrequired Your private Crawlbase token. The Normal token is the default; use the JavaScript token when combined with `javascript=true`. url stringrequired Target URL to scrape. Must start with `http` or `https` and be fully URL-encoded. scraper stringrequired Name of the scraper to apply. See the catalog above for the supported set. country ISO 3166optional Geolocate the request from a specific country (e.g. `US`, `GB`, `DE`). Country availability is plan-gated; full country list lives on the [Crawling API parameters](/docs/crawling-api#parameters) reference. javascript booleanfalse Render the page in a real Chrome browser before scraping. Set `javascript=true` for SPAs and JS-rendered pages. **Costs 2 credits per request** ; requires the JavaScript token, not the Normal token. premium booleanfalse Route the request through Crawlbase's premium residential network for tougher anti-bot targets. **Costs 10 credits** per request, or **20 credits** when combined with `javascript=true`. Plan-gated. ## Scraper-specific errors | Code | Meaning | | --- | --- | | `422` | Unknown scraper name. Check spelling against the catalog. | | `423` | URL doesn't match the scraper's expected pattern (e.g. `amazon-product-details` on a non-product URL). | | `425` | Page structure changed and the scraper couldn't extract data. Reported automatically; usually fixed within hours. | [← PreviousLegacy APIs](/docs/legacy)[Next →Leads API](/docs/leads-api) --- Source: https://crawlbase.com/docs/scrapers # All Scrapers Named data extractors that turn supported web pages into clean structured JSON. No HTML parsing on your end. Pick a scraper, point it at a URL, get back the fields you actually need. How scrapers work Scrapers ride on the [Crawling API](/docs/crawling-api). Add `&scraper=NAME` to any Crawling API call and you get parsed JSON back instead of raw HTML. Same token, same rate limits, same retries - different shape of response. ## Quickstart ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/dp/1098145356' \ --data-urlencode 'scraper=amazon-product-details' -G ``` ## Browse the catalog Each scraper has its own page with input format, output schema, and code samples in multiple languages. Click any name below to view full details. ### Amazon Product details, search results, reviews, best sellers, new releases. - [Amazon Product Details](/docs/scrapers/amazon-product-details) - [Amazon SERP](/docs/scrapers/amazon-serp) - [Amazon Offer Listing](/docs/scrapers/amazon-offer-listing) - [Amazon Product Reviews](/docs/scrapers/amazon-product-reviews) - [Amazon Best Sellers](/docs/scrapers/amazon-best-sellers) - [Amazon New Releases](/docs/scrapers/amazon-new-releases) ### Google Search engine results pages with all SERP features, plus Shopping product offers. - [Google SERP](/docs/scrapers/google-serp) - [Google Product Offers](/docs/scrapers/google-product-offers) - [Google Trends](/docs/scrapers/google-trends) - [Google Trends Explore](/docs/scrapers/google-trends-explore) ### Facebook Public groups, pages, profiles, hashtags, events. JS token recommended. - [Facebook Group](/docs/scrapers/facebook-group) - [Facebook Page](/docs/scrapers/facebook-page) - [Facebook Profile](/docs/scrapers/facebook-profile) - [Facebook Hashtag](/docs/scrapers/facebook-hashtag) - [Facebook Event](/docs/scrapers/facebook-event) ### Instagram Reels, posts, profiles, hashtags, audio aggregations. JS token recommended. - [Instagram Reel](/docs/scrapers/instagram-reel) - [Instagram Post](/docs/scrapers/instagram-post) - [Instagram Profile](/docs/scrapers/instagram-profile) - [Instagram Hashtag](/docs/scrapers/instagram-hashtag) - [Instagram Reels Audio](/docs/scrapers/instagram-reels-audio) ### TikTok Shop, products, profiles. Reviews, pricing, seller metrics, related videos. - [TikTok Product](/docs/scrapers/tiktok-product) - [TikTok Shop](/docs/scrapers/tiktok-shop) - [TikTok Profile](/docs/scrapers/tiktok-profile) ### LinkedIn Profiles, companies, public feeds. Get experience, education, skills, employee counts, and more. - [LinkedIn Profile](/docs/scrapers/linkedin-profile) - [LinkedIn Company](/docs/scrapers/linkedin-company) - [LinkedIn Feed](/docs/scrapers/linkedin-feed) ### Quora - [Quora SERP](/docs/scrapers/quora-serp) - [Quora Question](/docs/scrapers/quora-question) ### Airbnb - [Airbnb SERP](/docs/scrapers/airbnb-serp) ### eBay - [eBay SERP](/docs/scrapers/ebay-serp) - [eBay Product](/docs/scrapers/ebay-product) - [eBay Seller Shop](/docs/scrapers/ebay-seller-shop) ### AliExpress - [AliExpress Product](/docs/scrapers/aliexpress-product) - [AliExpress SERP](/docs/scrapers/aliexpress-serp) ### Galaxus - [Galaxus Product](/docs/scrapers/galaxus-product) - [Galaxus SERP](/docs/scrapers/galaxus-serp) - [Galaxus Product Reviews](/docs/scrapers/galaxus-product-reviews) ### Bing - [Bing SERP](/docs/scrapers/bing-serp) ### Immobilienscout24 - [ImmobilienScout24 Property](/docs/scrapers/immobilienscout24-property) ### Walmart - [Walmart SERP](/docs/scrapers/walmart-serp) - [Walmart Product Details](/docs/scrapers/walmart-product-details) - [Walmart Category](/docs/scrapers/walmart-category) ### Best Buy - [Best Buy SERP](/docs/scrapers/bestbuy-serp) - [Best Buy Product Details](/docs/scrapers/bestbuy-product-details) ### G2 - [G2 Product Reviews](/docs/scrapers/g2-product-reviews) ### Eventbrite - [Eventbrite Events List](/docs/scrapers/eventbrite-events-list) - [Eventbrite Event Details](/docs/scrapers/eventbrite-event-details) ### GitHub Developer-platform scrapers for GitHub - repository pages, repository search results, and user or organization profiles. See the [Developer category](/docs/scrapers/developer) for an overview. - [GitHub Repository](/docs/scrapers/github-repository) - [GitHub SERP](/docs/scrapers/github-serp) - [GitHub Profile](/docs/scrapers/github-profile) ### Reddit Community-platform scrapers for Reddit - subreddit listings, search results, and single posts with their comment trees. See the [Social Media category](/docs/scrapers/social-media) for an overview. - [Reddit Subreddit](/docs/scrapers/reddit-subreddit) - [Reddit Search](/docs/scrapers/reddit-serp) - [Reddit Post](/docs/scrapers/reddit-post) ### Booking.com Travel and hospitality scrapers for Booking.com - search-results listings and single hotel pages with pricing, review scores, and facilities. See the [Travel, Events & Real Estate category](/docs/scrapers/travel-events) for an overview. - [Booking SERP](/docs/scrapers/booking-serp) - [Booking Hotel](/docs/scrapers/booking-hotel) ### Product Hunt Product Hunt scrapers - daily and weekly leaderboards, and individual product pages with upvotes, makers, topics, and reviews. See the [Reviews & Q&A category](/docs/scrapers/reviews-qa) for an overview. - [Product Hunt Leaderboard](/docs/scrapers/producthunt-leaderboard) - [Product Hunt Product](/docs/scrapers/producthunt-product) ### Stack Exchange Stack Exchange scrapers - question, tag, and search listings, and single question threads with every answer and comment, across the whole Stack Exchange network. - [Stack Exchange Questions](/docs/scrapers/stackexchange-serp) - [Stack Exchange Thread](/docs/scrapers/stackexchange-thread) ### Exercism Developer-platform scrapers for Exercism - track exercise listings, single exercise instructions, community-solution listings, and single published solutions. See the [Developer category](/docs/scrapers/developer) for an overview. - [Exercism Exercises](/docs/scrapers/exercism-serp) - [Exercism Exercise](/docs/scrapers/exercism-exercise) - [Exercism Solutions](/docs/scrapers/exercism-solutions) - [Exercism Solution](/docs/scrapers/exercism-solution) ### Kaggle Data-science-platform scrapers for Kaggle - dataset search and single dataset pages, notebook search and single notebook pages. See the [Developer category](/docs/scrapers/developer) for an overview. - [Kaggle Dataset Search](/docs/scrapers/kaggle-dataset-serp) - [Kaggle Dataset](/docs/scrapers/kaggle-dataset) - [Kaggle Notebook Search](/docs/scrapers/kaggle-notebook-serp) - [Kaggle Notebook](/docs/scrapers/kaggle-notebook) ### LeetCode Coding-practice scrapers for LeetCode - the problem set listing, single problem pages, the community solutions tab and single solution posts. See the [Developer category](/docs/scrapers/developer) for an overview. - [LeetCode Problem Set](/docs/scrapers/leetcode-serp) - [LeetCode Problem](/docs/scrapers/leetcode-problem) - [LeetCode Solutions](/docs/scrapers/leetcode-solutions) - [LeetCode Solution](/docs/scrapers/leetcode-solution) ### OLX Classifieds-marketplace scrapers for OLX - search/category results pages and single ad pages, across the shared frontend (olx.pl, olx.ua, olx.pt, olx.ro, olx.bg, olx.kz, olx.uz). See the [E-commerce category](/docs/scrapers/ecommerce) for an overview. - [OLX SERP](/docs/scrapers/olx-serp) - [OLX Item](/docs/scrapers/olx-item) ### Generic Site-agnostic extractors that work on any URL. - [Generic Extractor](/docs/scrapers/generic-extractor) - [Email Extractor](/docs/scrapers/email-extractor) ## Don't see what you need? For unsupported sites, you can always use the [Crawling API](/docs/crawling-api) without a scraper to get the full HTML and parse it yourself. New scrapers ship monthly - [contact support](/docs/support) if you have a specific request. [← PreviousPrompt patterns](/docs/ai-prompts)[Next →E-Commerce](/docs/scrapers/ecommerce) --- Source: https://crawlbase.com/docs/scrapers/airbnb-serp # Airbnb SERP Extract Airbnb search results - array of listings with prices, ratings, and locations. ## API usage Add `&scraper=airbnb-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.airbnb.com/s/Beirut/homes' \ --data-urlencode 'scraper=airbnb-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.airbnb.com/s/Beirut/homes', {'scraper': 'airbnb-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.airbnb.com/s/Beirut/homes', { scraper: 'airbnb-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.airbnb.com/s/Beirut/homes', scraper: 'airbnb-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.airbnb.com/s/Beirut/homes ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. search\_location string Location query. total\_results integer | null Matching listings. listings array Listing summary objects. listings[].id string Listing ID. listings[].title string Title. listings[].url string URL. listings[].location string Neighborhood / city. listings[].price\_per\_night string Nightly price with currency. listings[].rating number | null Average rating. listings[].reviews\_count integer | null Reviews. listings[].images array Image URLs. listings[].guests integer Max guests. listings[].bedrooms integer Bedrooms. ## Sample response ``` { "search_location": "Beirut", "listings": [ { "id": "54281209", "title": "Sunny apartment in Hamra", "location": "Beirut, Lebanon", "price_per_night": "$45", "rating": 4.92, "reviews_count": 142, "guests": 2, "bedrooms": 1 } ] } ``` [← PreviousQuora Question](/docs/scrapers/quora-question)[Next →eBay SERP](/docs/scrapers/ebay-serp) --- Source: https://crawlbase.com/docs/scrapers/aliexpress-product # AliExpress Product Extract an AliExpress product page - title, price, variants, reviews, shipping, and seller information. Use the JS token AliExpress scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=aliexpress-product` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.aliexpress.com/item/1005008227636051.html' \ --data-urlencode 'scraper=aliexpress-product' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.aliexpress.com/item/1005008227636051.html', {'scraper': 'aliexpress-product'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.aliexpress.com/item/1005008227636051.html', { scraper: 'aliexpress-product' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.aliexpress.com/item/1005008227636051.html', scraper: 'aliexpress-product') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.aliexpress.com/item/1005008227636051.html ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. product\_id string Product ID. title string Title. price string Price. original\_price string | null Pre-discount price. discount\_percentage string | null Discount. rating number Rating. reviews\_count integer Reviews. orders\_count integer Total orders. images array Image URLs. variants array Variant options. shipping object Shipping methods, costs, deliveries. store object Store metadata. ## Sample response ``` { "product_id": "1005008227636051", "title": "Wireless Bluetooth Headphones", "price": "$18.99", "original_price": "$45.00", "rating": 4.8, "orders_count": 5240 } ``` [← PreviouseBay Seller Shop](/docs/scrapers/ebay-seller-shop)[Next →AliExpress SERP](/docs/scrapers/aliexpress-serp) --- Source: https://crawlbase.com/docs/scrapers/aliexpress-serp # AliExpress SERP Extract AliExpress search results - array of products with prices and seller summaries. Use the JS token AliExpress scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=aliexpress-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.aliexpress.com/wholesale?SearchText=water+bottle' \ --data-urlencode 'scraper=aliexpress-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.aliexpress.com/wholesale?SearchText=water+bottle', {'scraper': 'aliexpress-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.aliexpress.com/wholesale?SearchText=water+bottle', { scraper: 'aliexpress-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.aliexpress.com/wholesale?SearchText=water+bottle', scraper: 'aliexpress-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.aliexpress.com/wholesale?SearchText=water+bottle ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string Search query. products array Product summaries. products[].product\_id string Product ID. products[].title string Title. products[].price string Price. products[].url string URL. products[].rating number Rating. products[].orders\_count integer Orders. products[].image\_url string Thumbnail. ## Sample response ``` { "query": "water bottle", "products": [ { "product_id": "1005008227636051", "title": "Insulated Stainless Steel Water Bottle", "price": "$8.99", "rating": 4.7, "orders_count": 12430 } ] } ``` [← PreviousAliExpress Product](/docs/scrapers/aliexpress-product)[Next →Bing SERP](/docs/scrapers/bing-serp) --- Source: https://crawlbase.com/docs/scrapers/amazon-best-sellers # Amazon Best Sellers Get Amazon's Best Sellers ranking for a category as a structured product list. ## API usage Add `&scraper=amazon-best-sellers` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics' \ --data-urlencode 'scraper=amazon-best-sellers' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics', {'scraper': 'amazon-best-sellers'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics', { scraper: 'amazon-best-sellers' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics', scraper: 'amazon-best-sellers') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. category string Best Sellers category name. category\_url string Absolute URL of the category page. products array Ranked products. products[].rank integer Best-seller rank within the category. products[].asin string Product ASIN. products[].title string Product title. products[].price string | null Displayed price. products[].rating number | null Star rating. products[].reviews\_count integer | null Review count. products[].image string Thumbnail URL. products[].url string Absolute product URL. ## Sample response ``` { "category": "Electronics", "category_url": "https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics", "products": [ { "rank": 1, "asin": "B0BDHB9Y8H", "title": "Apple AirPods Pro (2nd Generation)", "price": "$189.00", "rating": 4.7, "reviews_count": 98421, "image": "https://m.media-amazon.com/images/I/...jpg", "url": "https://www.amazon.com/dp/B0BDHB9Y8H" } ] } ``` --- Source: https://crawlbase.com/docs/scrapers/amazon-new-releases # Amazon New Releases Get Amazon's New Releases page for a category - newly added products ranked by recency and momentum. ## API usage Add `&scraper=amazon-new-releases` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/new-releases/handmade' \ --data-urlencode 'scraper=amazon-new-releases' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.amazon.com/new-releases/handmade', {'scraper': 'amazon-new-releases'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.amazon.com/new-releases/handmade', { scraper: 'amazon-new-releases' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.amazon.com/new-releases/handmade', scraper: 'amazon-new-releases') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.amazon.com/new-releases/handmade ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. category string New-release category. category\_url string Absolute URL of the category page. products array Newly listed products with the same shape as `amazon-best-sellers`. ## Sample response ``` { "category": "Handmade", "products": [ { "rank": 1, "asin": "B0F3YC9NKH", "title": "Handmade Ceramic Vase", "price": "$39.00", "rating": null, "reviews_count": null, "image": "https://m.media-amazon.com/images/I/...jpg", "url": "https://www.amazon.com/dp/B0F3YC9NKH" } ] } ``` [← PreviousAmazon Best Sellers](/docs/scrapers/amazon-best-sellers)[Next →Google SERP](/docs/scrapers/google-serp) --- Source: https://crawlbase.com/docs/scrapers/amazon-offer-listing # Amazon Offer Listing Get every offer on an Amazon listing page - competing sellers, prices, condition, and shipping options for the same product. ## API usage Add `&scraper=amazon-offer-listing` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/gp/offer-listing/B01KJEOCDW' \ --data-urlencode 'scraper=amazon-offer-listing' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.amazon.com/gp/offer-listing/B01KJEOCDW', {'scraper': 'amazon-offer-listing'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.amazon.com/gp/offer-listing/B01KJEOCDW', { scraper: 'amazon-offer-listing' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.amazon.com/gp/offer-listing/B01KJEOCDW', scraper: 'amazon-offer-listing') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.amazon.com/gp/offer-listing/B01KJEOCDW ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. asin string Product ASIN being listed. offers array All competing offers. offers[].price string Offer price. offers[].shipping string Shipping cost or free-shipping note. offers[].condition string New / Used / Refurbished tier. offers[].seller string Seller name or marketplace identity. offers[].seller\_rating number | null Seller star rating. offers[].seller\_url string Absolute seller profile URL. offers[].fulfilled\_by\_amazon boolean True if FBA-fulfilled. ## Sample response ``` { "asin": "B01KJEOCDW", "offers": [ { "price": "$22.49", "shipping": "FREE Shipping", "condition": "New", "seller": "Amazon.com", "seller_rating": null, "seller_url": "https://www.amazon.com/gp/help/customer/display.html", "fulfilled_by_amazon": true } ] } ``` --- Source: https://crawlbase.com/docs/scrapers/amazon-product-details # Amazon Product Details Extract a complete Amazon product page - name, ASIN, brand, price, availability, ratings, images, features, and description - as structured JSON. ## API usage Add `&scraper=amazon-product-details` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/Apple-iPhone-Silicone-Case-MagSafe/dp/B0CHX2XFLN' \ --data-urlencode 'scraper=amazon-product-details' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.amazon.com/Apple-iPhone-Silicone-Case-MagSafe/dp/B0CHX2XFLN', {'scraper': 'amazon-product-details'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.amazon.com/Apple-iPhone-Silicone-Case-MagSafe/dp/B0CHX2XFLN', { scraper: 'amazon-product-details' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.amazon.com/Apple-iPhone-Silicone-Case-MagSafe/dp/B0CHX2XFLN', scraper: 'amazon-product-details') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.amazon.com/Apple-iPhone-Silicone-Case-MagSafe/dp/B0CHX2XFLN ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. name string Product title as displayed on the page. asin string Amazon Standard Identification Number. brand string | null Brand name when shown on the page. price string Currency-formatted price string (e.g. "$49.99"). availability string In-stock status text (e.g. "In Stock"). rating number Average customer rating, 1–5. reviews\_count integer Total number of reviews. main\_image string URL of the primary product image. images array\ All product image URLs. features array\ Bullet-list product features. description string Long-form product description. categories array\ Browse-node breadcrumb path. ## Sample response ``` { "name": "Apple iPhone Silicone Case with MagSafe", "asin": "B0CHX2XFLN", "brand": "Apple", "price": "$49.00", "availability": "In Stock", "rating": 4.7, "reviews_count": 12483, "main_image": "https://m.media-amazon.com/images/I/61MZi+B-OBL.jpg", "images": ["…"], "features": [ "Designed by Apple to complement iPhone", "MagSafe-compatible attachment and alignment" ], "description": "Silicone exterior with a soft microfiber lining…", "categories": ["Cell Phones & Accessories", "Cases"] } ``` --- Source: https://crawlbase.com/docs/scrapers/amazon-product-reviews # Amazon Product Reviews Pull the customer reviews for an Amazon product - text, rating, helpful counts, and verified-purchase status. Currently unavailable The `amazon-product-reviews` scraper is currently not available due to changes from Amazon. We are working on a fix, but we do not have an ETA at this time. ## API usage Add `&scraper=amazon-product-reviews` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/hz/reviews-render/ajax/medley-filtered-reviews/get/ref=cm_cr_dp_d_fltrs_srt?scope=reviewsAjax0&asin=B08PN7R2MZ&pageNumber=10' \ --data-urlencode 'scraper=amazon-product-reviews' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.amazon.com/hz/reviews-render/ajax/medley-filtered-reviews/get/ref=cm_cr_dp_d_fltrs_srt?scope=reviewsAjax0&asin=B08PN7R2MZ&pageNumber=10', {'scraper': 'amazon-product-reviews'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.amazon.com/hz/reviews-render/ajax/medley-filtered-reviews/get/ref=cm_cr_dp_d_fltrs_srt?scope=reviewsAjax0&asin=B08PN7R2MZ&pageNumber=10', { scraper: 'amazon-product-reviews' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.amazon.com/hz/reviews-render/ajax/medley-filtered-reviews/get/ref=cm_cr_dp_d_fltrs_srt?scope=reviewsAjax0&asin=B08PN7R2MZ&pageNumber=10', scraper: 'amazon-product-reviews') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.amazon.com/hz/reviews-render/ajax/medley-filtered-reviews/get/ref=cm_cr_dp_d_fltrs_srt?scope=reviewsAjax0&asin=B08PN7R2MZ&pageNumber=10 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. asin string Product ASIN being reviewed. reviews array Review records. reviews[].title string Review headline. reviews[].rating integer 1–5 star rating. reviews[].author string Reviewer display name. reviews[].date string Review date (ISO or human-readable). reviews[].body string Review text body. reviews[].verified\_purchase boolean True if Amazon marked the review as verified purchase. reviews[].helpful\_count integer Helpful votes. ## Sample response ``` { "asin": "B08PN7R2MZ", "reviews": [ { "title": "Better than expected", "rating": 5, "author": "R. Patel", "date": "2026-03-14", "body": "Battery life is incredible…", "verified_purchase": true, "helpful_count": 87 } ] } ``` --- Source: https://crawlbase.com/docs/scrapers/amazon-serp # Amazon SERP Parse an Amazon search results page into a structured array of products with prices, ratings, and pagination info. ## API usage Add `&scraper=amazon-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/s?k=games' \ --data-urlencode 'scraper=amazon-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.amazon.com/s?k=games', {'scraper': 'amazon-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.amazon.com/s?k=games', { scraper: 'amazon-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.amazon.com/s?k=games', scraper: 'amazon-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.amazon.com/s?k=games ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string Search query that produced these results. total\_results integer | null Reported total result count. page integer Current page number. products array Per-product results (see fields below). products[].asin string Product ASIN. products[].title string Product title. products[].price string | null Displayed price. products[].rating number | null Star rating. products[].reviews\_count integer | null Review count. products[].image string Thumbnail URL. products[].url string Absolute product URL. products[].sponsored boolean True if the listing is a Sponsored ad. ## Sample response ``` { "query": "games", "total_results": 60000, "page": 1, "products": [ { "asin": "B0CRJYSL5G", "title": "Catan: 5th Edition", "price": "$44.97", "rating": 4.8, "reviews_count": 5732, "image": "https://m.media-amazon.com/images/I/...jpg", "url": "https://www.amazon.com/dp/B0CRJYSL5G", "sponsored": false } ] } ``` --- Source: https://crawlbase.com/docs/scrapers/bestbuy-product-details # Best Buy Product Details Extract a Best Buy product page - title, price, full description, specifications, ratings, and reviews summary. ## API usage Add `&scraper=bestbuy-product-details` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.bestbuy.com/site/meta-quest-3-512gb-the-most-powerful-quest-ultimate-mixed-reality-experiences-get-batman-arkham-shadow-white/6596938.p' \ --data-urlencode 'scraper=bestbuy-product-details' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.bestbuy.com/site/meta-quest-3-512gb-the-most-powerful-quest-ultimate-mixed-reality-experiences-get-batman-arkham-shadow-white/6596938.p', {'scraper': 'bestbuy-product-details'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.bestbuy.com/site/meta-quest-3-512gb-the-most-powerful-quest-ultimate-mixed-reality-experiences-get-batman-arkham-shadow-white/6596938.p', { scraper: 'bestbuy-product-details' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.bestbuy.com/site/meta-quest-3-512gb-the-most-powerful-quest-ultimate-mixed-reality-experiences-get-batman-arkham-shadow-white/6596938.p', scraper: 'bestbuy-product-details') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.bestbuy.com/site/meta-quest-3-512gb-the-most-powerful-quest-ultimate-mixed-reality-experiences-get-batman-arkham-shadow-white/6596938.p ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. sku string SKU. title string Title. brand string Brand. price string Current price. original\_price string | null Pre-discount price. availability string In-store and online availability. rating number Rating. reviews\_count integer Review count. description string Description. features array Feature bullets. specifications object Spec pairs. images array Images. videos array Video URLs. ## Sample response ``` { "sku": "6596938", "title": "Meta Quest 3 512GB", "brand": "Meta", "price": "$649.99", "availability": "In stock", "rating": 4.7 } ``` [← PreviousBest Buy SERP](/docs/scrapers/bestbuy-serp)[Next →G2 Product Reviews](/docs/scrapers/g2-product-reviews) --- Source: https://crawlbase.com/docs/scrapers/bestbuy-serp # Best Buy SERP Extract Best Buy search results - array of products with prices, ratings, and availability. ## API usage Add `&scraper=bestbuy-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.bestbuy.com/site/searchpage.jsp?st=gaming+chair' \ --data-urlencode 'scraper=bestbuy-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.bestbuy.com/site/searchpage.jsp?st=gaming+chair', {'scraper': 'bestbuy-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.bestbuy.com/site/searchpage.jsp?st=gaming+chair', { scraper: 'bestbuy-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.bestbuy.com/site/searchpage.jsp?st=gaming+chair', scraper: 'bestbuy-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.bestbuy.com/site/searchpage.jsp?st=gaming+chair ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string Search query. products array Product summaries. products[].sku string Best Buy SKU. products[].title string Title. products[].price string Price. products[].rating number Rating. products[].reviews\_count integer Reviews. products[].availability string Stock status. products[].image\_url string Thumbnail. ## Sample response ``` { "query": "gaming chair", "products": [ { "sku": "6503480", "title": "Gaming Chair Pro RGB", "price": "$299.99", "rating": 4.5, "reviews_count": 412 } ] } ``` [← PreviousWalmart Category](/docs/scrapers/walmart-category)[Next →Best Buy Product Details](/docs/scrapers/bestbuy-product-details) --- Source: https://crawlbase.com/docs/scrapers/bing-serp # Bing SERP Extract Bing search results - organic links, video results, news, and related search suggestions. ## API usage Add `&scraper=bing-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.bing.com/search?q=iphone' \ --data-urlencode 'scraper=bing-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.bing.com/search?q=iphone', {'scraper': 'bing-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.bing.com/search?q=iphone', { scraper: 'bing-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.bing.com/search?q=iphone', scraper: 'bing-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.bing.com/search?q=iphone ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string Search query. total\_results integer | null Total result count. organic array Organic result objects. organic[].title string Result title. organic[].url string Destination URL. organic[].snippet string Snippet. organic[].position integer Position (1-indexed). videos array Video results. related\_searches array Related queries. ## Sample response ``` { "query": "iphone", "organic": [ { "title": "iPhone - Apple", "url": "https://www.apple.com/iphone/", "snippet": "Discover the new iPhone...", "position": 1 } ], "related_searches": ["iphone 15", "iphone pro"] } ``` [← PreviousAliExpress SERP](/docs/scrapers/aliexpress-serp)[Next →ImmobilienScout24 Property](/docs/scrapers/immobilienscout24-property) --- Source: https://crawlbase.com/docs/scrapers/booking-hotel # Booking Hotel Parse a Booking.com hotel page into structured JSON with the name, description, address, coordinates, star rating, review score, review count, review label, image, and facilities. ## API usage Add `&scraper=booking-hotel` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.booking.com/hotel/nl/example-amsterdam.html' \ --data-urlencode 'scraper=booking-hotel' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.booking.com/hotel/nl/example-amsterdam.html', {'scraper': 'booking-hotel'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.booking.com/hotel/nl/example-amsterdam.html', { scraper: 'booking-hotel' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.booking.com/hotel/nl/example-amsterdam.html', scraper: 'booking-hotel') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.booking.com/hotel/nl/example-amsterdam.html ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. name string Hotel name. url string Canonical Booking.com URL of the hotel. description string Full hotel description. address string Hotel address. coordinates object Geographic coordinates of the hotel. coordinates.latitude number | null Latitude of the hotel, or null when unavailable. coordinates.longitude number | null Longitude of the hotel, or null when unavailable. rating integer | null Star rating of the hotel, or null when unrated. reviewScore number | null Average review score, or null when there are no reviews. reviewCount integer | null Number of reviews, or null when there are no reviews. reviewLabel string | null Textual review label (for example `Superb`), when present. image string | null Main image URL, when present. facilities array Facility names offered by the hotel. ## Sample response ``` { "name": "Canal House Amsterdam", "url": "https://www.booking.com/hotel/nl/example-amsterdam.html", "description": "A restored 17th-century canal house in the heart of Amsterdam, steps from the Jordaan district. Rooms overlook the Keizersgracht with original beamed ceilings and a private garden terrace.", "address": "Keizersgracht 148, Amsterdam City Center, 1015 CX Amsterdam, Netherlands", "coordinates": { "latitude": 52.3738, "longitude": 4.8846 }, "rating": 4, "reviewScore": 9.1, "reviewCount": 1284, "reviewLabel": "Superb", "image": "https://cf.bstatic.com/xdata/images/hotel/canal-house.jpg", "facilities": [ "Free WiFi", "Non-smoking rooms", "Garden", "Bar", "Family rooms", "24-hour front desk" ] } ``` [← PreviousBooking SERP](/docs/scrapers/booking-serp)[Next →Product Hunt Leaderboard](/docs/scrapers/producthunt-leaderboard) --- Source: https://crawlbase.com/docs/scrapers/booking-serp # Booking SERP Parse a Booking.com search-results page into structured JSON with the matched properties, each property name, URL, address, distance, price, review score, review count, star rating, and image. ## API usage Add `&scraper=booking-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.booking.com/searchresults.html?ss=Amsterdam' \ --data-urlencode 'scraper=booking-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.booking.com/searchresults.html?ss=Amsterdam', {'scraper': 'booking-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.booking.com/searchresults.html?ss=Amsterdam', { scraper: 'booking-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.booking.com/searchresults.html?ss=Amsterdam', scraper: 'booking-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.booking.com/searchresults.html?ss=Amsterdam ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. searchUrl string Canonical search-results URL. destination string Destination the search was run for. propertyCount integer Number of properties returned in `properties`. properties array Properties matching the search, in the order Booking.com returned them. properties[].name string Property name. properties[].url string Canonical Booking.com URL of the property. properties[].address string Property address. properties[].distance string | null Distance from the destination centre, when present. properties[].price string Formatted price text as shown on the listing. properties[].priceAmount number | null Numeric price value, or null when it cannot be parsed. properties[].currency string | null Currency code of the price, when present. properties[].reviewScore number | null Average review score, or null when there are no reviews. properties[].reviewCount integer | null Number of reviews, or null when there are no reviews. properties[].rating integer | null Star rating of the property, or null when unrated. properties[].image string | null Thumbnail image URL, when present. ## Sample response ``` { "searchUrl": "https://www.booking.com/searchresults.html?ss=Amsterdam", "destination": "Amsterdam", "propertyCount": 2, "properties": [ { "name": "Canal House Amsterdam", "url": "https://www.booking.com/hotel/nl/canal-house-amsterdam.html", "address": "Keizersgracht 148, Amsterdam City Center, 1015 CX Amsterdam", "distance": "0.8 km from centre", "price": "€ 245", "priceAmount": 245, "currency": "EUR", "reviewScore": 9.1, "reviewCount": 1284, "rating": 4, "image": "https://cf.bstatic.com/xdata/images/hotel/canal-house.jpg" }, { "name": "Jordaan Boutique Stay", "url": "https://www.booking.com/hotel/nl/jordaan-boutique-stay.html", "address": "Prinsengracht 302, Jordaan, 1016 HX Amsterdam", "distance": "1.2 km from centre", "price": "€ 189", "priceAmount": 189, "currency": "EUR", "reviewScore": 8.7, "reviewCount": 642, "rating": 3, "image": null } ] } ``` [← PreviousReddit Post](/docs/scrapers/reddit-post)[Next →Booking Hotel](/docs/scrapers/booking-hotel) --- Source: https://crawlbase.com/docs/scrapers/developer # Developer Scrapers for developer platforms. Point at a GitHub, Stack Overflow, or Exercism URL and get clean structured JSON - repository metadata, search results, profiles, question-and-answer threads, or coding-exercise solutions - instead of HTML. ## Overview The Developer category covers the platforms engineering teams pull from for open-source intelligence, dependency and ecosystem analysis, developer-relations research, hiring/sourcing signals, and technical question-and-answer mining. Every scraper accepts a target URL, returns parsed JSON in milliseconds, and rides the same residential proxies and anti-bot bypass that powers the [Crawling API](/docs/crawling-api) - same uptime SLA, same one-token authentication, no per-target setup. The core set targets **GitHub** , the largest host of public source code. Pick a scraper by the surface you need: a single **repository** page, a repository **search** results page (SERP), or a user/organization **profile**. Each scraper targets a single page-type so the JSON shape stays stable as the underlying HTML changes - and you only pay for successful responses, so retries against a flaky upstream don't show up on your bill. **Stack Overflow** is covered too. It is part of the [Stack Exchange](/docs/scrapers/reviews-qa#stackexchange) network, so the same two host-agnostic scrapers handle it - there is no separate Stack Overflow scraper to learn. Point `stackexchange-serp` at a question list, tag, or search page (e.g. `https://stackoverflow.com/questions/tagged/python`) for a structured array of questions with scores, answer and view counts, and tags; point `stackexchange-thread` at a single question (e.g. `https://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-processing-an-unsorted-array`) for the full body plus every answer and comment. The network parser handles the markup whether the URL is on `stackoverflow.com` or any other `*.stackexchange.com` site. **Exercism** is covered by four scrapers for the coding-practice platform. Point `exercism-serp` at a track exercise list (e.g. `https://exercism.org/tracks/ruby/exercises`) for every exercise with its slug, difficulty, and blurb; `exercism-exercise` at a single exercise (e.g. `/tracks/ruby/exercises/two-fer`) for its full instructions as text and HTML; `exercism-solutions` at an exercise's community solutions for a paginated array of published solutions with author, language, and star counts; and `exercism-solution` at a single published solution (e.g. `/tracks/ruby/exercises/two-fer/solutions/handle`) for its iterations, metadata, and source code. Common pipelines: - **Ecosystem monitoring** : poll `github-repository` on the projects you depend on, snapshot `stars`, `forks`, `openIssuesCount`, `latestRelease`, and `archived`, and alert on Δ. - **Discovery** : feed `github-serp` queries (e.g. a topic or keyword search) into a list of `github-repository` calls to enrich each hit with full metadata. - **Developer sourcing / DevRel** : resolve a `github-profile` to read `followers`, `publicRepos`, `pinnedRepos`, and `organizations`. - **Dependency intelligence** : track `primaryLanguage`, `languages`, `license`, and `topics` across a portfolio of repositories to flag license or maintenance risk. - **Q&A mining** : run `stackexchange-serp` on a tag or search (e.g. `stackoverflow.com/questions/tagged/python`), then fan out to `stackexchange-thread` per question to capture accepted answers, code blocks, and vote scores - a clean corpus for support automation or model training. - **Solution mining** : run `exercism-serp` on a track, fan out to `exercism-solutions` per exercise, then `exercism-solution` per author to build a labelled corpus of working code across languages for the same problem. Every field maps directly to what the page renders, and nullable fields come back as `null` when the source page omits the value rather than silently disappearing - so your schema stays predictable across calls. No GitHub or Stack Exchange API token, rate-limit juggling, or pagination bookkeeping on your end: the scraper handles the fetch and the parse, and you get back the fields you actually need. ## GitHub Three scrapers covering the GitHub surfaces teams query most - a single repository page, repository search results, and user or organization profiles. - [GitHub Repository](/docs/scrapers/github-repository) - repository page (description, language, stars, forks, issues, license, latest release). - [GitHub SERP](/docs/scrapers/github-serp) - repository search-results page on GitHub. - [GitHub Profile](/docs/scrapers/github-profile) - user or organization profile (bio, followers, pinned repos, organizations). ## Stack Overflow Stack Overflow is served by the [Stack Exchange](/docs/scrapers/reviews-qa#stackexchange) scrapers - point them at `stackoverflow.com` URLs. Use `stackexchange-serp` for question lists, tag pages, and search results, and `stackexchange-thread` for a single question with its full answer thread. Both are host-agnostic across the Stack Exchange network, so the same call shape works for every `*.stackexchange.com` site. - [Stack Overflow Questions](/docs/scrapers/stackexchange-serp) - a Stack Overflow questions, tagged (e.g. `/questions/tagged/python`), or search-results page as a structured array with scores, answer and view counts, tags, and pagination. - [Stack Overflow Thread](/docs/scrapers/stackexchange-thread) - a single Stack Overflow question with its full body plus every answer and comment, with scores, accepted state, and authors. ## Exercism Four scrapers covering the Exercism coding-practice platform - a track exercise listing, a single exercise with its instructions, an exercise community-solutions listing, and a single published solution. Point them at `exercism.org` URLs; track and exercise slugs are read from the URL path. - [Exercism Exercises](/docs/scrapers/exercism-serp) - a track exercise-list page (e.g. `/tracks/ruby/exercises`) as a structured array of exercises with slug, title, difficulty, and blurb. - [Exercism Exercise](/docs/scrapers/exercism-exercise) - a single exercise overview page with its title, difficulty, and full instructions as text and HTML. - [Exercism Solutions](/docs/scrapers/exercism-solutions) - an exercise community-solutions page as a paginated array of published solutions with author, language, stars, and iteration counts. - [Exercism Solution](/docs/scrapers/exercism-solution) - a single published community solution with its iterations, language, star count, and source code. ## Kaggle Four scrapers covering the Kaggle data-science platform - dataset search, a single dataset with its files and license, notebook search, and a single notebook with its metadata and inputs. Point them at `kaggle.com` URLs; owner and dataset or notebook slugs are read from the URL path. - [Kaggle Dataset Search](/docs/scrapers/kaggle-dataset-serp) - a dataset search or listing page (e.g. `/datasets?search=heart+disease`) as a ranked array of datasets with owner, size, usability rating, downloads, and notebook count. - [Kaggle Dataset](/docs/scrapers/kaggle-dataset) - a single dataset page with its description, keywords, owner, license, file list, and engagement counts. - [Kaggle Notebook Search](/docs/scrapers/kaggle-notebook-serp) - a notebook search or listing page (e.g. `/code?searchQuery=titanic`) as a ranked array of notebooks with author, co-authors, competition context, votes, and comments. - [Kaggle Notebook](/docs/scrapers/kaggle-notebook) - a single notebook page with its author, language, runtime, version history, engagement counts, and attached inputs. ## LeetCode Four scrapers covering the LeetCode coding-practice platform - the problem set listing, a single problem with its statement and starter code, the community solutions tab, and a single solution post. Point them at `leetcode.com` URLs; problem and post slugs are read from the URL path. - [LeetCode Problem Set](/docs/scrapers/leetcode-serp) - a problem set listing (e.g. `/problemset/?difficulty=EASY`) as an array of problems with id, difficulty, acceptance rate, and premium flag. - [LeetCode Problem](/docs/scrapers/leetcode-problem) - a single problem page with its statement, difficulty, topic tags, hints, starter code snippets, and engagement counts. - [LeetCode Solutions](/docs/scrapers/leetcode-solutions) - the community solutions tab of a problem as an array of posts with author, tags, and engagement counts. - [LeetCode Solution](/docs/scrapers/leetcode-solution) - a single community solution post with its write-up, extracted code blocks, and engagement counts. ## Example call Below: a single `github-repository` call. Replace `YOUR_TOKEN` with your [Crawling API token](/docs/authentication); the only required parameters are the target URL and the scraper name. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://github.com/rails/rails' \ --data-urlencode 'scraper=github-repository' -G ``` ### Sample response ``` { "name": "rails", "owner": "rails", "fullName": "rails/rails", "url": "https://github.com/rails/rails", "description": "Ruby on Rails", "primaryLanguage": "Ruby", "languages": ["Ruby", "JavaScript", "HTML", "SCSS", "CSS", "Dockerfile"], "stars": 58789, "forks": 22414, "watchers": 2400, "hasIssues": true, "openIssuesCount": 478, "openPrsCount": 1081, "topics": ["ruby", "rails", "html", "activerecord", "framework", "mvc", "activejob"], "license": "MIT license", "defaultBranch": "main", "latestRelease": "v8.1.3", "readmePresent": true, "archived": false } ``` Full reference (parameters, all 4 SDK languages, edge cases): [GitHub Repository - full reference](/docs/scrapers/github-repository) [← PreviousTravel, Events & Real Estate](/docs/scrapers/travel-events)[Next →Generic Extractors](/docs/scrapers/generic) --- Source: https://crawlbase.com/docs/scrapers/ebay-product # eBay Product Extract a single eBay listing - title, price, images, condition, shipping, and seller details. JS token recommended Some eBay pages load content via JavaScript or iframes. For complete data extraction, use your **JavaScript token**. ## API usage Add `&scraper=ebay-product` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.ebay.com/itm/156078647276' \ --data-urlencode 'scraper=ebay-product' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.ebay.com/itm/156078647276', {'scraper': 'ebay-product'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.ebay.com/itm/156078647276', { scraper: 'ebay-product' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.ebay.com/itm/156078647276', scraper: 'ebay-product') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.ebay.com/itm/156078647276 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. item\_id string Item ID. title string Title. price string Current price. original\_price string | null Original price. condition string Item condition. availability string Stock or quantity. description string Listing description. images array Product images. shipping object Shipping cost and methods. seller object Seller name, feedback score, percentage positive. returns string Returns policy. ## Sample response ``` { "item_id": "156078647276", "title": "Apple iPhone X 64GB Unlocked", "price": "$129.99", "condition": "Used - Excellent", "availability": "3 available", "seller": { "name": "phonecollector_us", "feedback_score": 12420, "positive_pct": "99.6%" } } ``` [← PreviouseBay SERP](/docs/scrapers/ebay-serp)[Next →eBay Seller Shop](/docs/scrapers/ebay-seller-shop) --- Source: https://crawlbase.com/docs/scrapers/ebay-seller-shop # eBay Seller Shop Extract an eBay seller storefront - shop info, ratings, total feedback, and a sample of items currently for sale. JS token recommended Some eBay pages load content via JavaScript or iframes. For complete data extraction, use your **JavaScript token**. ## API usage Add `&scraper=ebay-seller-shop` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.ebay.com/str/watcheshalfprice' \ --data-urlencode 'scraper=ebay-seller-shop' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.ebay.com/str/watcheshalfprice', {'scraper': 'ebay-seller-shop'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.ebay.com/str/watcheshalfprice', { scraper: 'ebay-seller-shop' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.ebay.com/str/watcheshalfprice', scraper: 'ebay-seller-shop') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.ebay.com/str/watcheshalfprice ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. shop\_name string Shop display name. seller\_username string Seller username. feedback\_score integer Total feedback score. positive\_feedback\_pct string Positive feedback percentage. member\_since string Date the seller joined eBay. location string Seller location. items\_for\_sale integer Total items listed. items array Sample of items. ## Sample response ``` { "shop_name": "Watches Half Price", "seller_username": "watcheshalfprice", "feedback_score": 8420, "positive_feedback_pct": "99.8%", "items_for_sale": 120 } ``` [← PreviouseBay Product](/docs/scrapers/ebay-product)[Next →AliExpress Product](/docs/scrapers/aliexpress-product) --- Source: https://crawlbase.com/docs/scrapers/ebay-serp # eBay SERP Extract eBay search results - array of products with prices, conditions, and seller info. JS token recommended Some eBay pages load content via JavaScript or iframes. For complete data extraction, use your **JavaScript token**. ## API usage Add `&scraper=ebay-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.ebay.com/sch/i.html?_nkw=iphone+x' \ --data-urlencode 'scraper=ebay-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.ebay.com/sch/i.html?_nkw=iphone+x', {'scraper': 'ebay-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.ebay.com/sch/i.html?_nkw=iphone+x', { scraper: 'ebay-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.ebay.com/sch/i.html?_nkw=iphone+x', scraper: 'ebay-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.ebay.com/sch/i.html?_nkw=iphone+x ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string Search query. total\_results integer | null Total results. products array Product summaries. products[].item\_id string eBay item ID. products[].title string Listing title. products[].url string Listing URL. products[].price string Price. products[].shipping string Shipping cost. products[].condition string New / Used / Refurbished. products[].seller string Seller username. products[].image\_url string Thumbnail. ## Sample response ``` { "query": "iphone x", "products": [ { "item_id": "156078647276", "title": "Apple iPhone X 64GB Unlocked", "price": "$129.99", "shipping": "Free shipping", "condition": "Used", "seller": "phonecollector_us" } ] } ``` [← PreviousAirbnb SERP](/docs/scrapers/airbnb-serp)[Next →eBay Product](/docs/scrapers/ebay-product) --- Source: https://crawlbase.com/docs/scrapers/ecommerce # E-Commerce Nineteen scrapers for the largest online retailers and marketplaces. Pick a site, point at a URL, get clean structured JSON instead of HTML. ## Overview The E-Commerce category covers the marketplaces most teams pull from for price intelligence, catalog enrichment, competitor monitoring, and ad-attribution analytics. Every scraper accepts a target URL, returns parsed JSON in milliseconds, and rides the same residential proxies and anti-bot bypass that powers the [Crawling API](/docs/crawling-api) - same uptime SLA, same one-token authentication, no per-target setup. Pick a scraper by the surface you need: **product detail** pages, **search results** (SERP), **category/department** feeds, **seller/merchant** shop pages, **offer listings** , or **review** threads. Each scraper targets a single page-type so the JSON shape stays stable as the underlying HTML changes - and you only pay for successful responses, so retries against a flaky upstream don't show up on your bill. Common pipelines: - **Price monitoring** : poll `amazon-product-details` / `walmart-product-details` daily, snapshot `price` and `availability`, alert on Δ. - **Competitive intelligence** : feed `amazon-serp` queries into a list of `amazon-product-details` calls. - **Catalog enrichment** : enrich your own SKUs with brand, image, feature data via the `*-product-details` scrapers. - **Ad measurement** : monitor `amazon-best-sellers` / `amazon-new-releases` rankings before/after campaign launches. Geo-routing is automatic - pass `country=DE` in the request and you'll see what a German shopper sees (currency, locale, regional availability). For login-walled or session-bound pages, pair the scraper with `cookies_session` to reuse the same residential session across calls. ## Amazon Six scrapers covering the full Amazon retail surface - product detail, search, offer listings, reviews, best-seller charts, and new-release feeds. Country-aware: pass `country=US` / `UK` / `DE` / etc. to hit the regional storefront. - [Amazon Product Details](/docs/scrapers/amazon-product-details) - product page (title, price, ratings, description). - [Amazon SERP](/docs/scrapers/amazon-serp) - search-results page on Amazon. - [Amazon Offer Listing](/docs/scrapers/amazon-offer-listing) - all sellers offering a given product. - [Amazon Product Reviews](/docs/scrapers/amazon-product-reviews) - reviews for a product. - [Amazon Best Sellers](/docs/scrapers/amazon-best-sellers) - best-seller rankings by category. - [Amazon New Releases](/docs/scrapers/amazon-new-releases) - newly released products by category. ## Walmart Walmart.com product pages, search results, and department/category feeds. Useful when you're tracking pricing parity between Amazon and Walmart for the same SKU. - [Walmart Product Details](/docs/scrapers/walmart-product-details) - product page on Walmart. - [Walmart SERP](/docs/scrapers/walmart-serp) - search-results page on Walmart. - [Walmart Category](/docs/scrapers/walmart-category) - products under a department/sub-department. ## eBay eBay listings, search results, and seller-shop pages. The seller-shop scraper is the right fit when you're tracking a specific reseller's catalog over time rather than a fixed product. - [eBay Product](/docs/scrapers/ebay-product) - product listing (price, seller, condition, shipping). - [eBay SERP](/docs/scrapers/ebay-serp) - search-results page on eBay. - [eBay Seller Shop](/docs/scrapers/ebay-seller-shop) - all listings from a single seller's shop. ## AliExpress Cross-border AliExpress scrapers - product pages and search. Pair with `country` to see localised price/shipping for the buyer market you care about. - [AliExpress Product](/docs/scrapers/aliexpress-product) - product page on AliExpress. - [AliExpress SERP](/docs/scrapers/aliexpress-serp) - search-results page on AliExpress. ## Galaxus Galaxus and Digitec product, search, and reviews scrapers for the Swiss market. Pair with `country` set to CH for accurate pricing and availability. - [Galaxus Product](/docs/scrapers/galaxus-product) - product page on Galaxus. - [Galaxus SERP](/docs/scrapers/galaxus-serp) - search-results page on Galaxus. - [Galaxus Product Reviews](/docs/scrapers/galaxus-product-reviews) - customer reviews for a product on Galaxus. ## Best Buy Best Buy product detail and search. Strong choice for consumer-electronics price tracking in North America. - [Best Buy Product Details](/docs/scrapers/bestbuy-product-details) - product page on Best Buy. - [Best Buy SERP](/docs/scrapers/bestbuy-serp) - search-results page on Best Buy. ## Google Shopping Merchant offers for a Google Shopping product - useful when you need a cross-retailer price comparison without scraping each retailer separately. - [Google Product Offers](/docs/scrapers/google-product-offers) - merchant offers for a Google Shopping product. ## TikTok Shop TikTok Shop product listings - a growing channel for impulse-purchase commerce, especially in APAC and increasingly in the US. - [TikTok Shop](/docs/scrapers/tiktok-shop) - TikTok Shop product listing. ## OLX Classifieds-marketplace scrapers for OLX. Both work on OLX's shared frontend across `olx.pl`, `olx.ua`, `olx.pt`, `olx.ro`, `olx.bg`, `olx.kz`, and `olx.uz` - only the domain in the request URL changes. OLX rate-limits datacenter IPs, but you don't pick a proxy pool yourself - the Crawling API routes through its residential network by default and auto-selects the best exit per request. Pass `country=` (e.g. `country=PL` for olx.pl) only when you need a specific market's geo. - [OLX SERP](/docs/scrapers/olx-serp) - search or category results page - array of ads with pagination. - [OLX Item](/docs/scrapers/olx-item) - single ad page with parameters, photos, location, and seller. ## Example call Below: a single `amazon-product-details` call. Replace `YOUR_TOKEN` with your [Crawling API token](/docs/authentication); the only required parameters are the target URL and the scraper name. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.amazon.com/Apple-iPhone-Silicone-Case-MagSafe/dp/B0CHX2XFLN' \ --data-urlencode 'scraper=amazon-product-details' -G ``` ### Sample response ``` { "name": "Apple iPhone Silicone Case with MagSafe", "asin": "B0CHX2XFLN", "brand": "Apple", "price": "$49.00", "availability": "In Stock", "rating": 4.7, "reviews_count": 12483, "main_image": "https://m.media-amazon.com/images/I/61MZi+B-OBL.jpg", "images": ["…"], "features": [ "Designed by Apple to complement iPhone", "MagSafe-compatible attachment and alignment" ], "description": "Silicone exterior with a soft microfiber lining…", "categories": ["Cell Phones & Accessories", "Cases"] } ``` Full reference (parameters, all 4 SDK languages, edge cases): [Amazon Product Details - full reference](/docs/scrapers/amazon-product-details) [← PreviousOverview](/docs/scrapers)[Next →Search Engines](/docs/scrapers/search-engines) --- Source: https://crawlbase.com/docs/scrapers/email-extractor # Email Extractor Extract every email address visible on a web page. Useful for contact pages, team directories, and public listings. ## API usage Add `&scraper=email-extractor` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://letsencrypt.org/contact/' \ --data-urlencode 'scraper=email-extractor' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://letsencrypt.org/contact/', {'scraper': 'email-extractor'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://letsencrypt.org/contact/', { scraper: 'email-extractor' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://letsencrypt.org/contact/', scraper: 'email-extractor') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://letsencrypt.org/contact/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string Final URL. title string Page title. emails array Email address strings (deduplicated). emails\_with\_context array Email + surrounding text. emails\_with\_context[].email string Email address. emails\_with\_context[].context string Surrounding text. ## Sample response ``` { "url": "https://letsencrypt.org/contact/", "title": "Contact - Let's Encrypt", "emails": [ "press@letsencrypt.org", "webmaster@letsencrypt.org" ] } ``` [← PreviousGeneric Extractor](/docs/scrapers/generic-extractor)[Next →Python SDK](/docs/sdk-python) --- Source: https://crawlbase.com/docs/scrapers/eventbrite-event-details # Eventbrite Event Details Extract a single Eventbrite event page - full description, organizer, agenda, ticket tiers, and venue details. ## API usage Add `&scraper=eventbrite-event-details` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.eventbrite.com/e/fordham-ai-quantitative-hedge-fund-conference-tickets-1981959936523' \ --data-urlencode 'scraper=eventbrite-event-details' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.eventbrite.com/e/fordham-ai-quantitative-hedge-fund-conference-tickets-1981959936523', {'scraper': 'eventbrite-event-details'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.eventbrite.com/e/fordham-ai-quantitative-hedge-fund-conference-tickets-1981959936523', { scraper: 'eventbrite-event-details' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.eventbrite.com/e/fordham-ai-quantitative-hedge-fund-conference-tickets-1981959936523', scraper: 'eventbrite-event-details') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.eventbrite.com/e/fordham-ai-quantitative-hedge-fund-conference-tickets-1981959936523 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. event\_id string Event ID. title string Title. description string Full description. start\_datetime string ISO start datetime. end\_datetime string ISO end datetime. timezone string Event timezone. venue object Venue name, address, lat/lng, online flag. organizer object Organizer name, bio, follower count. ticket\_tiers array Ticket tiers with name, price, availability. agenda array Schedule items. tags array Event tags. images array Event images. ## Sample response ``` { "event_id": "1981959936523", "title": "Fordham AI Quantitative Hedge Fund Conference", "start_datetime": "2026-05-15T09:00:00-04:00", "timezone": "America/New_York", "ticket_tiers": [ { "name": "General Admission", "price": "$25" }, { "name": "VIP", "price": "$120" } ] } ``` [← PreviousEventbrite Events List](/docs/scrapers/eventbrite-events-list)[Next →GitHub Repository](/docs/scrapers/github-repository) --- Source: https://crawlbase.com/docs/scrapers/eventbrite-events-list # Eventbrite Events List Extract an Eventbrite search/browse page - array of upcoming events with date, location, and price. ## API usage Add `&scraper=eventbrite-events-list` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.eventbrite.com/d/ny--new-york/ai/' \ --data-urlencode 'scraper=eventbrite-events-list' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.eventbrite.com/d/ny--new-york/ai/', {'scraper': 'eventbrite-events-list'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.eventbrite.com/d/ny--new-york/ai/', { scraper: 'eventbrite-events-list' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.eventbrite.com/d/ny--new-york/ai/', scraper: 'eventbrite-events-list') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.eventbrite.com/d/ny--new-york/ai/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. search\_query string Search query / category. location string | null Location filter. total\_events integer | null Total events. events array Event summary objects. events[].event\_id string Event ID. events[].title string Title. events[].url string URL. events[].date string Start date. events[].time string Start time. events[].venue string Venue or "Online". events[].price\_from string Min price (or "Free"). events[].image\_url string Image. ## Sample response ``` { "search_query": "ai", "location": "New York", "events": [ { "event_id": "1981959936523", "title": "Fordham AI Quantitative Hedge Fund Conference", "date": "2026-05-15", "time": "9: 00 AM", "venue": "Fordham University", "price_from": "$25" } ] } ``` [← PreviousG2 Product Reviews](/docs/scrapers/g2-product-reviews)[Next →Eventbrite Event Details](/docs/scrapers/eventbrite-event-details) --- Source: https://crawlbase.com/docs/scrapers/exercism-exercise # Exercism Exercise Parse a single Exercism exercise overview page into structured JSON with its title, difficulty, and full instructions as text and HTML. ## API usage Add `&scraper=exercism-exercise` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://exercism.org/tracks/ruby/exercises/two-fer' \ --data-urlencode 'scraper=exercism-exercise' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://exercism.org/tracks/ruby/exercises/two-fer', {'scraper': 'exercism-exercise'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://exercism.org/tracks/ruby/exercises/two-fer', { scraper: 'exercism-exercise' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://exercism.org/tracks/ruby/exercises/two-fer', scraper: 'exercism-exercise') data = JSON.parse(res.body) ``` ## Example input URL Any Exercism exercise overview page works in the `url` parameter - the page at `/tracks//exercises/`. For example: ``` https://exercism.org/tracks/ruby/exercises/two-fer https://exercism.org/tracks/python/exercises/leap https://exercism.org/tracks/go/exercises/hello-world https://exercism.org/tracks/rust/exercises/grains ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the exercise page that was scraped. track string | null Track slug read from the URL path (for example `ruby`), or null when it cannot be determined. slug string | null Exercise slug read from the URL path (for example `two-fer`), or null when it cannot be determined. title string | null Exercise display title. difficulty string | null Difficulty label (for example `easy`, `medium`, `hard`). instructions string | null Full exercise instructions as plain text, with whitespace collapsed. instructionsHtml string | null Full exercise instructions as HTML, preserving headings, code blocks, and lists. ## Sample response ``` { "url": "https://exercism.org/tracks/ruby/exercises/two-fer", "track": "ruby", "slug": "two-fer", "title": "Two Fer", "difficulty": "easy", "instructions": "Two Fer Two-fer is short for two for one. One for you and one for me. Given a name, return a string with the message: One for name, one for me. Where \"name\" is the given name. However, if the name is missing, return the string: One for you, one for me.", "instructionsHtml": "Instructions\nTwo-fer is short for two for one. One for you and one for me.\nGiven a name, return a string with the message:\nOne for name, one for me." } ``` [← PreviousExercism Exercises](/docs/scrapers/exercism-serp)[Next →Exercism Solutions](/docs/scrapers/exercism-solutions) --- Source: https://crawlbase.com/docs/scrapers/exercism-serp # Exercism Exercises Parse an Exercism track exercise-list page into structured JSON with each exercise slug, title, difficulty, blurb, and unlock state. ## API usage Add `&scraper=exercism-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://exercism.org/tracks/ruby/exercises' \ --data-urlencode 'scraper=exercism-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://exercism.org/tracks/ruby/exercises', {'scraper': 'exercism-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://exercism.org/tracks/ruby/exercises', { scraper: 'exercism-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://exercism.org/tracks/ruby/exercises', scraper: 'exercism-serp') data = JSON.parse(res.body) ``` ## Example input URL Any Exercism track exercise-list page works in the `url` parameter - the page at `/tracks//exercises` for any language track. For example: ``` https://exercism.org/tracks/ruby/exercises https://exercism.org/tracks/python/exercises https://exercism.org/tracks/go/exercises https://exercism.org/tracks/javascript/exercises https://exercism.org/tracks/rust/exercises ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. trackUrl string URL of the track exercise-list page that was scraped. track string | null Track slug read from the URL path (for example `ruby`), or null when it cannot be determined. exerciseCount integer Number of exercises returned in `exercises`. exercises array Exercises on the track, in listing order. exercises[].slug string Exercise slug (URL fragment, for example `two-fer`). exercises[].type string Exercise type (for example `practice` or `tutorial`). exercises[].title string Exercise display title. exercises[].iconUrl string URL of the exercise icon. exercises[].difficulty string Difficulty label (for example `easy`, `medium`, `hard`). exercises[].blurb string Short one-line description of the exercise. exercises[].isExternal boolean True when the exercise is hosted outside the track. exercises[].isUnlocked boolean True when the exercise is unlocked for the viewer. exercises[].isRecommended boolean True when Exercism recommends the exercise next. exercises[].url string | null Canonical URL of the exercise overview page, or null when absent. ## Sample response ``` { "trackUrl": "https://exercism.org/tracks/ruby/exercises", "track": "ruby", "exerciseCount": 3, "exercises": [ { "slug": "two-fer", "type": "practice", "title": "Two Fer", "iconUrl": "https://assets.exercism.org/exercises/two-fer.svg", "difficulty": "easy", "blurb": "Create a sentence of the form \"One for X, one for me.\"", "isExternal": false, "isUnlocked": true, "isRecommended": false, "url": "https://exercism.org/tracks/ruby/exercises/two-fer" }, { "slug": "hello-world", "type": "practice", "title": "Hello World", "iconUrl": "https://assets.exercism.org/exercises/hello-world.svg", "difficulty": "easy", "blurb": "The classical introductory exercise. Just say \"Hello, World!\"", "isExternal": false, "isUnlocked": true, "isRecommended": true, "url": "https://exercism.org/tracks/ruby/exercises/hello-world" }, { "slug": "grains", "type": "practice", "title": "Grains", "iconUrl": "https://assets.exercism.org/exercises/grains.svg", "difficulty": "medium", "blurb": "Calculate the number of grains of wheat on a chessboard.", "isExternal": false, "isUnlocked": true, "isRecommended": false, "url": "https://exercism.org/tracks/ruby/exercises/grains" } ] } ``` [← PreviousStack Exchange Thread](/docs/scrapers/stackexchange-thread)[Next →Exercism Exercise](/docs/scrapers/exercism-exercise) --- Source: https://crawlbase.com/docs/scrapers/exercism-solution # Exercism Solution Parse a single published Exercism solution page into structured JSON with its track, exercise, author, language, iteration history, and full source code. ## API usage Add `&scraper=exercism-solution` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://exercism.org/tracks/ruby/exercises/two-fer/solutions/neilnorthrop' \ --data-urlencode 'scraper=exercism-solution' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://exercism.org/tracks/ruby/exercises/two-fer/solutions/neilnorthrop', {'scraper': 'exercism-solution'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://exercism.org/tracks/ruby/exercises/two-fer/solutions/neilnorthrop', { scraper: 'exercism-solution' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://exercism.org/tracks/ruby/exercises/two-fer/solutions/neilnorthrop', scraper: 'exercism-solution') data = JSON.parse(res.body) ``` ## Example input URL Any single published Exercism solution page works in the `url` parameter - the page at `/tracks//exercises//solutions/`. For example: ``` https://exercism.org/tracks/ruby/exercises/two-fer/solutions/neilnorthrop https://exercism.org/tracks/go/exercises/hello-world/solutions/erikschierboom ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the solution page that was scraped. track object Track the solution belongs to. track.slug string | null Track slug (for example `ruby`). track.title string | null Track display title (for example `Ruby`). track.iconUrl string | null URL of the track icon. exercise object Exercise the solution belongs to. exercise.slug string | null Exercise slug (for example `two-fer`). exercise.type string | null Exercise type (for example `practice` or `concept`). exercise.title string | null Exercise display title. exercise.iconUrl string | null URL of the exercise icon. exercise.difficulty string | null Difficulty label (for example `easy`, `medium`, `hard`). exercise.blurb string | null Short exercise description. author object Author of the solution. author.handle string | null Author handle (Exercism username). language string | null Programming language of the solution. indentSize integer | null Editor indent size the author used, or null when not reported. numStars integer | null Number of stars on the solution. numIterations integer Number of iterations in the `iterations` array. isOutOfDate boolean True when the solution is out of date with the current exercise. publishedIterationIdx integer | null Index (`idx`) of the iteration that is published, or null when none. publishedAt string | null Publish time of the solution (ISO 8601), or null when absent. iterations array Iteration history for the solution, oldest first. iterations[].idx integer | null Iteration index (1-based). iterations[].status string | null Analysis status of the iteration (for example `no_automated_feedback`). iterations[].testsStatus string | null Test run status of the iteration (for example `passed`). iterations[].submissionMethod string | null How the iteration was submitted (for example `cli` or `api`). iterations[].createdAt string | null Creation time of the iteration (ISO 8601), or null when absent. iterations[].isPublished boolean True when this iteration is the published one. iterations[].isLatest boolean True when this iteration is the most recent. code string | null Full source code of the published solution, or null when absent. ## Sample response ``` { "url": "https://exercism.org/tracks/ruby/exercises/two-fer/solutions/neilnorthrop", "track": { "slug": "ruby", "title": "Ruby", "iconUrl": "https://assets.exercism.org/tracks/ruby.svg" }, "exercise": { "slug": "two-fer", "type": "practice", "title": "Two Fer", "iconUrl": "https://assets.exercism.org/exercises/two-fer.svg", "difficulty": "easy", "blurb": "Create a sentence of the form \"One for X, one for me.\"" }, "author": { "handle": "neilnorthrop" }, "language": "ruby", "indentSize": 2, "numStars": 42, "numIterations": 2, "isOutOfDate": false, "publishedIterationIdx": 2, "publishedAt": "2026-05-18T14:03:27Z", "iterations": [ { "idx": 1, "status": "no_automated_feedback", "testsStatus": "passed", "submissionMethod": "cli", "createdAt": "2026-05-17T22:41:03Z", "isPublished": false, "isLatest": false }, { "idx": 2, "status": "no_automated_feedback", "testsStatus": "passed", "submissionMethod": "cli", "createdAt": "2026-05-18T14:03:27Z", "isPublished": true, "isLatest": true } ], "code": "def two_fer(name = \"you\")\n \"One for #{name}, one for me.\"\nend" } ``` [← PreviousExercism Solutions](/docs/scrapers/exercism-solutions)[Next →Kaggle Dataset Search](/docs/scrapers/kaggle-dataset-serp) --- Source: https://crawlbase.com/docs/scrapers/exercism-solutions # Exercism Solutions Parse an Exercism exercise community-solutions page into structured JSON with each published solution author, language, stars, iteration counts, and pagination. ## API usage Add `&scraper=exercism-solutions` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. The listing is paginated (24 per page); pass a `?page=N` query on the target URL for later pages. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://exercism.org/tracks/ruby/exercises/two-fer/solutions' \ --data-urlencode 'scraper=exercism-solutions' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://exercism.org/tracks/ruby/exercises/two-fer/solutions', {'scraper': 'exercism-solutions'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://exercism.org/tracks/ruby/exercises/two-fer/solutions', { scraper: 'exercism-solutions' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://exercism.org/tracks/ruby/exercises/two-fer/solutions', scraper: 'exercism-solutions') data = JSON.parse(res.body) ``` ## Example input URL Any Exercism exercise community-solutions page works in the `url` parameter - the page at `/tracks//exercises//solutions`, optionally with a `?page=N` query. For example: ``` https://exercism.org/tracks/ruby/exercises/two-fer/solutions https://exercism.org/tracks/go/exercises/hello-world/solutions https://exercism.org/tracks/python/exercises/leap/solutions?page=2 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the solutions listing page that was scraped. track object Track context lifted from the first solution row. track.title string | null Track display title (for example `Ruby`). track.iconUrl string | null URL of the track icon. exercise object Exercise context lifted from the first solution row. exercise.title string | null Exercise display title. exercise.iconUrl string | null URL of the exercise icon. page integer | null Page number of the listing currently returned (1-based). totalPages integer | null Total number of pages of solutions for the exercise. totalCount integer | null Total number of published solutions for the exercise. solutionCount integer Number of solutions returned in `solutions` on this page. solutions array Published solutions on this page, in listing order. solutions[].uuid string Exercism solution UUID. solutions[].author object Author of the solution. solutions[].author.handle string Author handle (Exercism username). solutions[].author.avatarUrl string | null URL of the author avatar. solutions[].author.flair string | null Author flair badge (for example `insider`), or null when none. solutions[].language string Programming language of the solution. solutions[].numStars integer Number of stars on the solution. solutions[].numComments integer Number of comments on the solution. solutions[].numViews integer Number of views on the solution. solutions[].numIterations integer Number of iterations the author submitted. solutions[].numLoc integer | null Lines of code in the solution, or null when not reported. solutions[].isOutOfDate boolean True when the solution is out of date with the current exercise. solutions[].publishedAt string | null Publish time of the solution (ISO 8601), or null when absent. solutions[].snippet string | null Short code snippet preview of the solution, or null when absent. solutions[].url string | null Public URL of the single-solution page, or null when absent. ## Sample response ``` { "url": "https://exercism.org/tracks/ruby/exercises/two-fer/solutions", "track": { "title": "Ruby", "iconUrl": "https://assets.exercism.org/tracks/ruby.svg" }, "exercise": { "title": "Two Fer", "iconUrl": "https://assets.exercism.org/exercises/two-fer.svg" }, "page": 1, "totalPages": 412, "totalCount": 9876, "solutionCount": 2, "solutions": [ { "uuid": "b3f2c1a0-9d8e-4f7a-8c6b-1a2b3c4d5e6f", "author": { "handle": "neilnorthrop", "avatarUrl": "https://avatars.exercism.org/u/12345", "flair": "insider" }, "language": "ruby", "numStars": 42, "numComments": 3, "numViews": 1280, "numIterations": 2, "numLoc": 3, "isOutOfDate": false, "publishedAt": "2026-05-18T14:03:27Z", "snippet": "def two_fer(name = \"you\")\n \"One for #{name}, one for me.\"\nend", "url": "https://exercism.org/tracks/ruby/exercises/two-fer/solutions/neilnorthrop" }, { "uuid": "c4e3d2b1-0a9f-4e8b-9d7c-2b3c4d5e6f70", "author": { "handle": "erikschierboom", "avatarUrl": "https://avatars.exercism.org/u/67890", "flair": null }, "language": "ruby", "numStars": 17, "numComments": 0, "numViews": 640, "numIterations": 1, "numLoc": 4, "isOutOfDate": false, "publishedAt": "2026-04-02T09:11:50Z", "snippet": "def two_fer(name = \"you\")\n format(\"One for %s, one for me.\", name)\nend", "url": "https://exercism.org/tracks/ruby/exercises/two-fer/solutions/erikschierboom" } ] } ``` [← PreviousExercism Exercise](/docs/scrapers/exercism-exercise)[Next →Exercism Solution](/docs/scrapers/exercism-solution) --- Source: https://crawlbase.com/docs/scrapers/facebook-event # Facebook Event Get the metadata for a public Facebook event: title, host, location, time, and attendee counts. Use the JS token Facebook scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=facebook-event` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.facebook.com/events/1543404119289643' \ --data-urlencode 'scraper=facebook-event' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.facebook.com/events/1543404119289643', {'scraper': 'facebook-event'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.facebook.com/events/1543404119289643', { scraper: 'facebook-event' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.facebook.com/events/1543404119289643', scraper: 'facebook-event') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.facebook.com/events/1543404119289643 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. name string Event title. host string Hosting page or person. description string Event description body. start\_time string ISO 8601 start time. end\_time string | null ISO 8601 end time when set. location\_name string Venue name. location\_address string | null Full address text. interested\_count integer Users marked Interested. going\_count integer Users marked Going. cover\_image string Event cover image URL. ## Sample response ``` { "name": "Web Scraping Meetup", "host": "Crawlers and Scraping Enthusiasts", "description": "Quarterly meetup for the community…", "start_time": "2026-05-20T18:00:00-04:00", "end_time": "2026-05-20T21:00:00-04:00", "location_name": "WeWork Bryant Park", "interested_count": 237, "going_count": 82 } ``` [← PreviousFacebook Hashtag](/docs/scrapers/facebook-hashtag)[Next →Instagram Reel](/docs/scrapers/instagram-reel) --- Source: https://crawlbase.com/docs/scrapers/facebook-group # Facebook Group Extract a public Facebook group's metadata and recent feed: name, description, member count, post bodies, and reactions. Use the JS token Facebook scrapers work best with your **JavaScript token**. With the Normal token, content rendered after page load may not be captured. ## API usage Add `&scraper=facebook-group` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.facebook.com/groups/198722650913932' \ --data-urlencode 'scraper=facebook-group' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.facebook.com/groups/198722650913932', {'scraper': 'facebook-group'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.facebook.com/groups/198722650913932', { scraper: 'facebook-group' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.facebook.com/groups/198722650913932', scraper: 'facebook-group') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.facebook.com/groups/198722650913932 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. name string Group name. description string Group description text. members\_count integer Number of members. privacy string "Public" or "Private". cover\_image string Cover image URL. posts array Recent posts. posts[].author string Author display name. posts[].body string Post text. posts[].date string Posted-at. posts[].reactions integer Total reactions. posts[].comments\_count integer Number of comments. ## Sample response ``` { "name": "Crawlers and Scraping Enthusiasts", "description": "A community for web data professionals…", "members_count": 8472, "privacy": "Public", "posts": [ { "author": "M. Chen", "body": "Anyone tried the new MCP-based agents?", "date": "2026-04-26", "reactions": 42, "comments_count": 12 } ] } ``` [← PreviousGoogle Trends Explore](/docs/scrapers/google-trends-explore)[Next →Facebook Page](/docs/scrapers/facebook-page) --- Source: https://crawlbase.com/docs/scrapers/facebook-hashtag # Facebook Hashtag Extract recent posts surfaced by a Facebook hashtag page. Use the JS token Facebook scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=facebook-hashtag` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.facebook.com/hashtag/robots' \ --data-urlencode 'scraper=facebook-hashtag' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.facebook.com/hashtag/robots', {'scraper': 'facebook-hashtag'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.facebook.com/hashtag/robots', { scraper: 'facebook-hashtag' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.facebook.com/hashtag/robots', scraper: 'facebook-hashtag') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.facebook.com/hashtag/robots ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. hashtag string Hashtag (without the #). posts array Posts appearing on the hashtag page. posts[].author string Author name. posts[].body string Post text. posts[].date string Posted-at. posts[].image string | null Attached image URL when present. posts[].reactions integer Reactions count. posts[].url string Permalink to the post. ## Sample response ``` { "hashtag": "robots", "posts": [ { "author": "Boston Dynamics", "body": "New Atlas demo at the lab today…", "date": "2026-04-22", "reactions": 8412, "url": "https://www.facebook.com/BostonDynamics/posts/..." } ] } ``` [← PreviousFacebook Profile](/docs/scrapers/facebook-profile)[Next →Facebook Event](/docs/scrapers/facebook-event) --- Source: https://crawlbase.com/docs/scrapers/facebook-page # Facebook Page Get structured data for a Facebook page: name, category, follower count, contact info, posts and reactions. Use the JS token Facebook scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=facebook-page` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.facebook.com/Amazon/' \ --data-urlencode 'scraper=facebook-page' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.facebook.com/Amazon/', {'scraper': 'facebook-page'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.facebook.com/Amazon/', { scraper: 'facebook-page' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.facebook.com/Amazon/', scraper: 'facebook-page') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.facebook.com/Amazon/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. name string Page name. category string Page category (e.g. "Retail company"). about string About-section text. followers\_count integer Followers. likes\_count integer Page likes. phone string | null Contact phone if listed. website string | null External website URL. address string | null Listed address. profile\_image string Profile image URL. cover\_image string Cover image URL. ## Sample response ``` { "name": "Amazon", "category": "Retail company", "about": "Amazon's mission is to be Earth's most customer-centric company.", "followers_count": 29000000, "likes_count": 29380000, "website": "https://www.amazon.com", "address": "410 Terry Ave N, Seattle, WA" } ``` [← PreviousFacebook Group](/docs/scrapers/facebook-group)[Next →Facebook Profile](/docs/scrapers/facebook-profile) --- Source: https://crawlbase.com/docs/scrapers/facebook-profile # Facebook Profile Extract a public Facebook profile: name, profile and cover images, work and education history, and similar-name suggestions. Use the JS token Facebook scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=facebook-profile` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.facebook.com/zuck' \ --data-urlencode 'scraper=facebook-profile' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.facebook.com/zuck', {'scraper': 'facebook-profile'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.facebook.com/zuck', { scraper: 'facebook-profile' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.facebook.com/zuck', scraper: 'facebook-profile') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.facebook.com/zuck ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. name string Profile owner display name. profile\_image string Profile image URL. cover\_image string | null Cover image URL. intro string | null Bio / intro text from the profile. work array Listed work entries. work[].employer string Employer name. work[].title string Job title. work[].period string | null Date range (e.g. "2004 - present"). education array Education entries with `school`, `degree`, `period`. similar\_profiles array\ Names of profiles Facebook surfaces as similar. ## Sample response ``` { "name": "Mark Zuckerberg", "profile_image": "https://scontent.fbcd…/profile.jpg", "intro": "Founder and CEO of Meta.", "work": [ { "employer": "Meta", "title": "Founder & CEO", "period": "Feb 2004 – present" } ] } ``` [← PreviousFacebook Page](/docs/scrapers/facebook-page)[Next →Facebook Hashtag](/docs/scrapers/facebook-hashtag) --- Source: https://crawlbase.com/docs/scrapers/g2-product-reviews # G2 Product Reviews Extract G2 product reviews - name, ratings distribution, individual reviews with reviewer metadata. ## API usage Add `&scraper=g2-product-reviews` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.g2.com/products/zoom/reviews' \ --data-urlencode 'scraper=g2-product-reviews' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.g2.com/products/zoom/reviews', {'scraper': 'g2-product-reviews'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.g2.com/products/zoom/reviews', { scraper: 'g2-product-reviews' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.g2.com/products/zoom/reviews', scraper: 'g2-product-reviews') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.g2.com/products/zoom/reviews ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. product\_name string Product name. product\_logo\_url string Product logo. overall\_rating number Average rating. total\_reviews integer Review count. rating\_distribution object Counts by star (5\_star, 4\_star, etc). reviews array Individual reviews. reviews[].title string Review title. reviews[].rating number Star rating. reviews[].body string Review body. reviews[].pros string | null Pros section. reviews[].cons string | null Cons section. reviews[].reviewer\_name string Reviewer name. reviews[].reviewer\_role string | null Job title. reviews[].reviewer\_company\_size string | null Company size. reviews[].review\_date string Review date. ## Sample response ``` { "product_name": "Zoom", "overall_rating": 4.5, "total_reviews": 52840, "reviews": [ { "title": "Reliable for daily standups", "rating": 4.5, "reviewer_role": "Engineering Manager", "reviewer_company_size": "51-200 employees" } ] } ``` [← PreviousBest Buy Product Details](/docs/scrapers/bestbuy-product-details)[Next →Eventbrite Events List](/docs/scrapers/eventbrite-events-list) --- Source: https://crawlbase.com/docs/scrapers/galaxus-product # Galaxus Product Extract a Galaxus product page - title, price, images, ratings, delivery, and specifications. ## API usage Add `&scraper=galaxus-product` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.galaxus.ch/en/s1/product/sony-playstation-5-slim-disc-edition-game-consoles-39726201' \ --data-urlencode 'scraper=galaxus-product' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.galaxus.ch/en/s1/product/sony-playstation-5-slim-disc-edition-game-consoles-39726201', {'scraper': 'galaxus-product'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.galaxus.ch/en/s1/product/sony-playstation-5-slim-disc-edition-game-consoles-39726201', { scraper: 'galaxus-product' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.galaxus.ch/en/s1/product/sony-playstation-5-slim-disc-edition-game-consoles-39726201', scraper: 'galaxus-product') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.galaxus.ch/en/s1/product/sony-playstation-5-slim-disc-edition-game-consoles-39726201 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. title string | null Product title. url string Canonical product URL. price object Price object with `amount` and `currency`. images array Product image URLs. reviewsCount string Number of reviews. ratings string Average rating. deliveryTime string Estimated delivery time. deliveryStatusIcon string | null Availability label. deliveryNote string Stock and shipping note. supplier string | null Supplier or brand. description string | null Product description. specifications array Specification key/value pairs. returnPolicy string | null Return policy. warranty string | null Warranty terms. ## Sample response ``` { "title": "Sony Playstation 5 Slim Disc Edition", "url": "https://www.galaxus.ch/en/s1/product/sony-playstation-5-slim-disc-edition-game-consoles-39726201?tagIds=1", "price": { "amount": "475.00", "currency": "CHF" }, "images": [ "https://static01.galaxus.com/productimages/sample1.jpeg", "https://static01.galaxus.com/productimages/sample2.jpeg" ], "reviewsCount": "837", "ratings": "4.74", "deliveryTime": "Delivered Sat, 16.5.", "deliveryStatusIcon": "available in a few days", "deliveryNote": "More than 10 pieces in stock | free shipping", "supplier": "Sony", "description": "PlayStation 5 console - 1 TB", "specifications": [ { "Console": "PS5" }, { "Game world": "Playstation" } ], "returnPolicy": "30-day right of return", "warranty": "24 Months Warranty (Bring-in)" } ``` --- Source: https://crawlbase.com/docs/scrapers/galaxus-product-reviews # Galaxus Product Reviews Extract Galaxus product reviews - rating distribution, review text, reviewer metadata, comments, and pros/cons. ## API usage Add `&scraper=galaxus-product-reviews` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.galaxus.ch/en/s6/product/ratings/lattafa-khamrah-eau-de-parfum-100-ml-fragrances-36152090' \ --data-urlencode 'scraper=galaxus-product-reviews' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.galaxus.ch/en/s6/product/ratings/lattafa-khamrah-eau-de-parfum-100-ml-fragrances-36152090', {'scraper': 'galaxus-product-reviews'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.galaxus.ch/en/s6/product/ratings/lattafa-khamrah-eau-de-parfum-100-ml-fragrances-36152090', { scraper: 'galaxus-product-reviews' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.galaxus.ch/en/s6/product/ratings/lattafa-khamrah-eau-de-parfum-100-ml-fragrances-36152090', scraper: 'galaxus-product-reviews') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.galaxus.ch/en/s6/product/ratings/lattafa-khamrah-eau-de-parfum-100-ml-fragrances-36152090 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. productName string Product name. productBrand string Product brand. productId string Product ID. averageRating string Average rating. totalRatings string Total ratings. reviewsCount integer Reviews returned in this response. ratingDistribution array Star buckets with counts. reviews array Review entries. reviews[].title string Review title. reviews[].text string Review body text. reviews[].rating integer Rating from 1-5 stars. reviews[].insertDate string ISO 8601 timestamp. reviews[].isVerifiedBuyer boolean True when the reviewer is a verified buyer. reviews[].reviewer object Reviewer profile (userName, avatar, profile URL). reviews[].comments array Replies to the review. reviews[].proContra array Pros and cons entries. ## Sample response ``` { "productName": "Khamrah", "productBrand": "Lattafa", "productId": "36152090", "averageRating": "4.57", "totalRatings": "690", "reviewsWithTextCount": "157", "reviewsCount": 30, "ratingDistribution": [ { "stars": 5, "count": 522 }, { "stars": 4, "count": 96 } ], "reviews": [ { "position": 1, "id": "7541657", "title": "Super", "text": "Wirklich sehr gut, hält lange und riecht angenehm.", "rating": 5, "insertDate": "2024-07-01T19:58:44.418Z", "upVoteCount": 11, "downVoteCount": 3, "isVerifiedBuyer": true, "language": "de", "reviewer": { "userName": "DavidM88", "userAvatarLink": "https://static.digitecgalaxus.ch/Files/gax_avatar_13.png", "profileUrl": "https://www.galaxus.ch/en/user/ovy4mx8xwc" }, "comments": [ { "id": "120430", "text": "Wo kann man das kaufen?", "insertDate": "2024-08-31T17:16:40.757Z", "upVoteCount": 1, "downVoteCount": 0, "language": "de" } ], "proContra": [ { "id": "7541657-p-0", "isPro": true, "comment": "Perfekt" } ] } ] } ``` --- Source: https://crawlbase.com/docs/scrapers/galaxus-serp # Galaxus SERP Extract Galaxus search results - array of products with prices, ratings, and availability. ## API usage Add `&scraper=galaxus-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.galaxus.ch/en/search?q=iphone 17 pro' \ --data-urlencode 'scraper=galaxus-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.galaxus.ch/en/search?q=iphone 17 pro', {'scraper': 'galaxus-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.galaxus.ch/en/search?q=iphone 17 pro', { scraper: 'galaxus-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.galaxus.ch/en/search?q=iphone 17 pro', scraper: 'galaxus-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.galaxus.ch/en/search?q=iphone 17 pro ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. resultsFor string Normalized search query. resultInfo string Human-readable result count text. productsCount integer Number of products returned. products array Product listings. products[].position integer Position in results. products[].title string Product title. products[].brand string | null Brand. products[].description string Short description. products[].category string Category. products[].price object Price with `amount`, `currency`, and `text`. products[].image string Thumbnail URL. products[].url string Product URL. products[].ratings string Average rating. products[].reviewsCount string Number of reviews. products[].availability string Availability. ## Sample response ``` { "products": [ { "position": 1, "title": "Apple iPhone 17 Pro", "brand": "Apple", "description": "256 GB, Deep Blue, 6.30\", Dual SIM, 5G", "category": "Smartphones", "price": { "amount": "1079", "currency": "CHF", "text": "CHF1079.–" }, "image": "https://static01.galaxus.com/productimages/sample.avif", "url": "https://www.galaxus.ch/en/s1/product/apple-iphone-17-pro-256-gb-smartphones-61962936", "ratings": "4.6", "reviewsCount": "604", "availability": "available" } ], "productsCount": 48, "resultInfo": "9434 products", "resultsFor": "iphone 17 pro" } ``` --- Source: https://crawlbase.com/docs/scrapers/generic # Generic Extractors Two universal extractors for sites without a named scraper. Define your own fields and selectors - we'll handle the request, anti-bot, and parsing. ## Overview Generic extractors fill the gaps between named scrapers. When the site you need isn't in the catalog yet - niche marketplaces, regional retailers, internal portals - these two scrapers let you describe the page yourself and we run the extraction. `generic-extractor` takes a CSS-selector schema (or our auto-detection) and returns the parsed values. `email-extractor` is purpose-built for one common task: pulling every email address visible on a page, regardless of how the page hides them (mailto links, plain text, slightly-obfuscated patterns like `name [at] domain.com`). Common use cases: - **Long-tail catalog ingestion** : drop a schema for a regional retailer, run nightly imports without us shipping a dedicated scraper for it. - **Lead generation** : walk a list of company websites, run `email-extractor`, build a contactable prospect list (subject to your jurisdiction's outbound-email rules). - **Research pipelines** : extract structured fields (titles, headings, meta) from any page for downstream NLP - useful when you need normalised input from heterogeneous sources. - **Site monitoring** : define a schema once, monitor a competitor's pricing or copy changes by diffing the parsed JSON over time. Both scrapers ride the same anti-bot, residential-routing, and JS-rendering stack as the named scrapers - so the auto-detection works on JS-heavy SPAs without you wiring up a separate browser. If a target needs a dedicated parser eventually, the schema you wrote is a good handoff document for our scraper team. ## Generic extractors Two universal building blocks - one for arbitrary structured extraction, one for the always-needed task of pulling emails. Use these when there's no named scraper for the site you care about. - [Generic Extractor](/docs/scrapers/generic-extractor) - schema-driven HTML extractor. Pass selectors, get back structured JSON. - [Email Extractor](/docs/scrapers/email-extractor) - pulls every email address visible on a page. ## Example call Below: a `generic-extractor` call against Stack Overflow's homepage. With no schema specified, the scraper returns auto-detected metadata - page title, language, and headings grouped by level. Pass a custom `selectors` object (see the full reference) to extract specific fields. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://stackoverflow.com/' \ --data-urlencode 'scraper=generic-extractor' -G ``` ### Sample response ``` { "url": "https://stackoverflow.com/", "title": "Stack Overflow - Where Developers Learn...", "language": "en", "headings": { "h1": ["Where developers grow together"], "h2": ["Hot Network Questions"] } } ``` Full reference (parameters, all 4 SDK languages, edge cases): [Generic Extractor - full reference](/docs/scrapers/generic-extractor) [← PreviousDeveloper](/docs/scrapers/developer)[Next →Overview](/docs/sdks) --- Source: https://crawlbase.com/docs/scrapers/generic-extractor # Generic Extractor A site-agnostic extractor - pulls links, images, headings, and main content from any web page. ## API usage Add `&scraper=generic-extractor` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://stackoverflow.com/' \ --data-urlencode 'scraper=generic-extractor' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://stackoverflow.com/', {'scraper': 'generic-extractor'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://stackoverflow.com/', { scraper: 'generic-extractor' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://stackoverflow.com/', scraper: 'generic-extractor') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://stackoverflow.com/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string Final URL. title string Page title tag. meta\_description string | null Meta description. canonical\_url string | null Canonical link. language string | null Detected language. headings object h1/h2/h3 arrays of heading text. links array Outbound links with href, text, rel. images array Image URLs with alt text. main\_content string Extracted readable body text. ## Sample response ``` { "url": "https://stackoverflow.com/", "title": "Stack Overflow - Where Developers Learn...", "language": "en", "headings": { "h1": ["Where developers grow together"], "h2": ["Hot Network Questions"] } } ``` [← PreviousOLX Item](/docs/scrapers/olx-item)[Next →Email Extractor](/docs/scrapers/email-extractor) --- Source: https://crawlbase.com/docs/scrapers/github-profile # GitHub Profile Parse a GitHub user or organization profile page into structured JSON with name, bio, followers, following, public repos, pinned repos, and organizations. ## API usage Add `&scraper=github-profile` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://github.com/karpathy' \ --data-urlencode 'scraper=github-profile' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://github.com/karpathy', {'scraper': 'github-profile'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://github.com/karpathy', { scraper: 'github-profile' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://github.com/karpathy', scraper: 'github-profile') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://github.com/karpathy ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. name string | null Display name. username string | null Account login (the profile slug). bio string | null Profile bio. url string Canonical profile URL. followers integer | null Follower count. following integer | null Following count. publicRepos integer | null Public repository count shown on the Repositories tab. pinnedRepos array Names of the pinned repositories. organizations array Logins of organizations shown on the profile. ## Sample response ``` { "name": "Andrej", "username": "karpathy", "bio": "I like to train Deep Neural Nets on large datasets.", "url": "https://github.com/karpathy", "followers": 210000, "following": 8, "publicRepos": 63, "pinnedRepos": ["nanoGPT", "nanochat", "llm.c", "llama2.c", "micrograd", "microgpt"], "organizations": [] } ``` [← PreviousGitHub SERP](/docs/scrapers/github-serp)[Next →Reddit Subreddit](/docs/scrapers/reddit-subreddit) --- Source: https://crawlbase.com/docs/scrapers/github-repository # GitHub Repository Parse a single GitHub repository page into structured JSON with stars, forks, watchers, languages, topics, license, open issues and PRs, default branch, and latest release. ## API usage Add `&scraper=github-repository` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://github.com/rails/rails' \ --data-urlencode 'scraper=github-repository' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://github.com/rails/rails', {'scraper': 'github-repository'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://github.com/rails/rails', { scraper: 'github-repository' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://github.com/rails/rails', scraper: 'github-repository') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://github.com/rails/rails ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. name string | null Repository name. owner string | null Owner (user or organization) login. fullName string Full `owner/name` identifier. url string Canonical repository URL. description string | null Repository description (the tagline under the header). primaryLanguage string | null Primary programming language (first entry of `languages`). languages array Languages listed in the repository language bar. stars integer | null Star count. forks integer | null Fork count. watchers integer | null Watcher count. hasIssues boolean True if the Issues tab is enabled. openIssuesCount integer | null Open issue count. openPrsCount integer | null Open pull request count. topics array Repository topic tags. license string | null License name shown in the sidebar. defaultBranch string | null Default branch name. latestRelease string | null Latest release tag, when the repository has releases. readmePresent boolean True if a rendered README is present. archived boolean True if the repository is archived (read-only). ## Sample response ``` { "name": "rails", "owner": "rails", "fullName": "rails/rails", "url": "https://github.com/rails/rails", "description": "Ruby on Rails", "primaryLanguage": "Ruby", "languages": ["Ruby", "JavaScript", "HTML", "SCSS", "CSS", "Dockerfile"], "stars": 58789, "forks": 22414, "watchers": 2400, "hasIssues": true, "openIssuesCount": 478, "openPrsCount": 1081, "topics": ["ruby", "rails", "html", "activerecord", "framework", "mvc", "activejob"], "license": "MIT license", "defaultBranch": "main", "latestRelease": "v8.1.3", "readmePresent": true, "archived": false } ``` [← PreviousEventbrite Event Details](/docs/scrapers/eventbrite-event-details)[Next →GitHub SERP](/docs/scrapers/github-serp) --- Source: https://crawlbase.com/docs/scrapers/github-serp # GitHub SERP Parse a GitHub repository search results page into a structured array of repositories with stars, language, topics, and pagination info. ## API usage Add `&scraper=github-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. One request returns one page; follow `pagination.next_page_url` to walk further, up to GitHub's 1000-result cap. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://github.com/search?q=web+scraping&type=repositories' \ --data-urlencode 'scraper=github-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://github.com/search?q=web+scraping&type=repositories', {'scraper': 'github-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://github.com/search?q=web+scraping&type=repositories', { scraper: 'github-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://github.com/search?q=web+scraping&type=repositories', scraper: 'github-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://github.com/search?q=web+scraping&type=repositories ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string | null Search query that produced these results. searchType string | null Search type, e.g. `repositories`. sort string | null Sort field, e.g. `stars`. Absent when sorting by best match. order string | null Sort direction, `desc` or `asc`. totalResults integer | null Reported total result count across all pages. currentPage integer Current page number. pagination object Pagination info for walking the result set. pagination.current\_page integer Current page number. pagination.total\_pages integer | null Highest page number GitHub exposes for this query. pagination.next\_page\_url string | null Absolute URL of the next page, or `null` at the last page / result cap. pagination.previous\_page\_url string | null Absolute URL of the previous page, or `null` on the first page. pagination.has\_next boolean True while a next page exists below the 1000-result cap. results array Per-repository results (see fields below). results[].position integer Rank of the result on this page (1-based). results[].name string | null Repository name. results[].owner string | null Owner (user or organization) login. results[].fullName string Full `owner/name` identifier. results[].url string Absolute repository URL. results[].description string | null Repository description. results[].primaryLanguage string | null Primary programming language. results[].stars integer | null Star count. results[].topics array Topic labels shown on the result card. results[].updatedAt string | null Last-updated timestamp or relative label, when present. results[].license string | null License shown on the result card, when present. results[].archived boolean True if the repository is a public archive. results[].sponsorable boolean True if the owner can be sponsored. ## Sample response ``` { "query": "web scraping", "searchType": "repositories", "sort": "stars", "order": "desc", "totalResults": 133816, "currentPage": 1, "pagination": { "current_page": 1, "total_pages": 100, "next_page_url": "https://github.com/search?q=web+scraping&type=repositories&s=stars&o=desc&p=2", "previous_page_url": null, "has_next": true }, "results": [ { "position": 1, "name": "scrapy", "owner": "scrapy", "fullName": "scrapy/scrapy", "url": "https://github.com/scrapy/scrapy", "description": "Scrapy, a fast high-level web crawling & scraping framework for Python.", "primaryLanguage": "Python", "stars": 63119, "topics": ["python", "crawler", "framework", "scraping", "crawling"], "updatedAt": null, "license": null, "archived": false, "sponsorable": false } ] } ``` [← PreviousGitHub Repository](/docs/scrapers/github-repository)[Next →GitHub Profile](/docs/scrapers/github-profile) --- Source: https://crawlbase.com/docs/scrapers/google-product-offers # Google Product Offers Get all merchant offers from a Google Shopping product page with prices, ratings, and seller info. ## API usage Add `&scraper=google-product-offers` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.google.com/shopping/product/7015446445080090940/offers' \ --data-urlencode 'scraper=google-product-offers' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.google.com/shopping/product/7015446445080090940/offers', {'scraper': 'google-product-offers'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.google.com/shopping/product/7015446445080090940/offers', { scraper: 'google-product-offers' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.google.com/shopping/product/7015446445080090940/offers', scraper: 'google-product-offers') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.google.com/shopping/product/7015446445080090940/offers ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. product\_title string Product name as displayed. offers array Merchant offers. offers[].seller string Merchant name. offers[].price string Listed price. offers[].shipping string | null Shipping cost or note. offers[].condition string New / Used / Refurbished. offers[].seller\_rating number | null Aggregated seller rating. offers[].url string Click-out URL to the merchant. ## Sample response ``` { "product_title": "Sony WH-1000XM5 Wireless Headphones", "offers": [ { "seller": "Best Buy", "price": "$348.00", "shipping": "Free shipping", "condition": "New", "seller_rating": 4.8, "url": "https://www.bestbuy.com/site/..." } ] } ``` [← PreviousGoogle SERP](/docs/scrapers/google-serp)[Next →Google Trends](/docs/scrapers/google-trends) --- Source: https://crawlbase.com/docs/scrapers/google-serp # Google SERP Parse a Google search results page into structured organic results, ads, related searches, knowledge panels, and "People also ask" boxes. ## API usage Add `&scraper=google-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.google.com/search?q=samsung+social+accounts' \ --data-urlencode 'scraper=google-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.google.com/search?q=samsung+social+accounts', {'scraper': 'google-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.google.com/search?q=samsung+social+accounts', { scraper: 'google-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.google.com/search?q=samsung+social+accounts', scraper: 'google-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.google.com/search?q=samsung+social+accounts ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string Search query. results array Organic search results. results[].title string Result title. results[].url string Result destination URL. results[].snippet string Description text shown under the title. results[].sitelinks array Optional sitelinks under the main result. ads array Sponsored ads with the same shape as `results[]`. related\_searches array\ "Searches related to" suggestions. people\_also\_ask array PAA section with question/answer pairs. knowledge\_panel object | null Knowledge panel block when present. ## Sample response ``` { "query": "samsung social accounts", "results": [ { "title": "Samsung Mobile (@SamsungMobile) / X", "url": "https://x.com/SamsungMobile", "snippet": "The official Samsung Mobile X account…" } ], "related_searches": ["samsung instagram", "samsung official tiktok"], "people_also_ask": [ { "question": "Does Samsung have a TikTok?", "answer": "Yes, Samsung is on TikTok at @samsung." } ] } ``` [← PreviousAmazon New Releases](/docs/scrapers/amazon-new-releases)[Next →Google Product Offers](/docs/scrapers/google-product-offers) --- Source: https://crawlbase.com/docs/scrapers/google-trends # Google Trends Parse the Google Trends "Trending now" page into a structured array of trending searches with search-volume metrics, trend breakdowns, filters, and pagination. ## API usage Add `&scraper=google-trends` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. Filters such as `geo`, time range, and category are passed as query-string parameters on the Google Trends URL itself. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://trends.google.com/trending?geo=AE-DU' \ --data-urlencode 'scraper=google-trends' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://trends.google.com/trending?geo=AE-DU', {'scraper': 'google-trends'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://trends.google.com/trending?geo=AE-DU', { scraper: 'google-trends' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://trends.google.com/trending?geo=AE-DU', scraper: 'google-trends') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://trends.google.com/trending?geo=AE-DU ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. trends array Trending searches on the page, ordered by position. trends[].position integer Rank of the trend within the page (1-based). trends[].title string Trend query as it appears on the Google Trends page. trends[].searchVolume string Abbreviated search volume for the trend (for example `20K+`, `1M+`). trends[].searchVolumeText string Localized human-readable search-volume label (for example `20K+ searches`). trends[].increasePercentage string Recent increase in search interest, as shown on the page. trends[].started string When the trend started rising, as a relative time string (for example `6 hours ago`). trends[].status string Trend status as shown by Google (typically `Active` or `Lasted`). trends[].trendBreakdown array\ Related sub-queries that Google groups under this trend. trends[].trendBreakdownMoreCount integer Number of additional sub-queries beyond those listed in `trendBreakdown`. trendsCount integer Total number of trends returned in `trends`. filters object Filter state reflected by the page, decoded from the Google Trends URL. filters.geo string Resolved geography label (country or sub-region) the trends are scoped to. filters.timeRange string Time-range label the page is filtered by (for example `Past 7 days`). filters.category string Category filter label (for example `All categories`). filters.trendType string Trend-type filter label (for example `All trends`). filters.sortBy string Sort-order label (for example `By relevance`, `By search volume`). filters.updatedAtText string Localized label showing when the page was last updated by Google. pagination object Pagination state for the current request. pagination.currentPage integer Page index of the current response (1-based). pagination.itemsPerPage integer Number of trends Google returns per page. pagination.totalItems integer Total trends available across all pages for the current filters. pagination.hasNextPage boolean `true` when more trends are available on a subsequent page. pagination.hasPreviousPage boolean `true` when a previous page is available. ## Sample response ``` { "trends": [ { "position": 1, "title": "al maktoum international airport (dwc)", "searchVolume": "20K+", "searchVolumeText": "20K+ searches", "increasePercentage": "600%", "started": "6 hours ago", "status": "Active", "trendBreakdown": ["dubai international airport"], "trendBreakdownMoreCount": 0 } ], "trendsCount": 5, "filters": { "geo": "Dubai", "timeRange": "Past 7 days", "category": "All categories", "trendType": "All trends", "sortBy": "By relevance", "updatedAtText": "Updated Jun 2, 2:56 PM" }, "pagination": { "currentPage": 1, "itemsPerPage": 25, "totalItems": 25, "hasNextPage": false, "hasPreviousPage": false } } ``` [← PreviousGoogle Product Offers](/docs/scrapers/google-product-offers)[Next →Google Trends Explore](/docs/scrapers/google-trends-explore) --- Source: https://crawlbase.com/docs/scrapers/google-trends-explore # Google Trends Explore Parse a Google Trends Explore page into structured topic interest over time, interest by sub-region, related topics, and related queries. ## API usage Add `&scraper=google-trends-explore` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. Filters such as `geo`, time range, category, and search type are passed as query-string parameters on the Google Trends URL itself. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://trends.google.com/trends/explore?q=/g/11cs9m5kkd' \ --data-urlencode 'scraper=google-trends-explore' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://trends.google.com/trends/explore?q=/g/11cs9m5kkd', {'scraper': 'google-trends-explore'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://trends.google.com/trends/explore?q=/g/11cs9m5kkd', { scraper: 'google-trends-explore' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://trends.google.com/trends/explore?q=/g/11cs9m5kkd', scraper: 'google-trends-explore') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://trends.google.com/trends/explore?q=/g/11cs9m5kkd ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. topic string Resolved topic label for the Explore query (a Knowledge-Graph topic name, or the literal search term when no topic is matched). filters object Filter state reflected by the page, decoded from the Google Trends URL. filters.geo string Resolved geography label (country or sub-region) the comparison is scoped to. Defaults to `Worldwide` when no `geo` is set. filters.timeRange string Time-range label the comparison is filtered by (for example `Past 12 months`, `Past 7 days`). filters.category string Category filter label (for example `All categories`). filters.searchType string Search-type filter label (for example `Web Search`, `YouTube Search`). filters.description string Human-readable summary of the active filters (typically `geo` joined with `timeRange`). interestOverTime object Interest-over-time chart block. interestOverTime.title string Section heading as rendered by Google (typically `Interest over time`). interestOverTime.timeline array Time-series points for the topic, ordered chronologically. interestOverTime.timeline[].date string Localized date label for the data point. interestOverTime.timeline[].value number Normalized interest score for the data point, on a 0-100 scale where 100 is peak popularity for the period. interestOverTime.timelineCount integer Total number of points in `timeline`. interestBySubregion object Interest-by-sub-region block (country, sub-region, metro, or city, depending on the active `geo`). interestBySubregion.title string Section heading as rendered by Google (typically `Interest by region`, `Interest by sub-region`, etc.). interestBySubregion.view string Active view tab (typically `Top` or `Rising`). interestBySubregion.totalItems integer Total number of regions available across all pages. interestBySubregion.paginationText string Localized pagination summary (for example `Showing 1-5 of 87 regions`). interestBySubregion.items array Regions on the current page, ordered by position. interestBySubregion.items[].position integer Rank of the region within the response (1-based). interestBySubregion.items[].name string Region name as shown by Google. interestBySubregion.items[].value number Normalized interest score for the region on a 0-100 scale. relatedTopics object Related-topics block. relatedTopics.title string Section heading as rendered by Google (typically `Related topics`). relatedTopics.view string Active view tab (typically `Top` or `Rising`). relatedTopics.items array Related topics on the current page, ordered by position. relatedTopics.items[].position integer Rank of the topic within the response (1-based). relatedTopics.items[].name string Topic name as shown by Google. relatedTopics.items[].value string Score label as shown by Google: a percentage growth string (for example `+250%`) on the Rising view, or `Breakout` for new entrants; a relative score on the Top view. relatedTopics.items[].link string Google Trends Explore URL for the related topic. relatedQueries object Related-queries block. relatedQueries.title string Section heading as rendered by Google (typically `Related queries`). relatedQueries.view string Active view tab (typically `Top` or `Rising`). relatedQueries.items array Related queries on the current page, ordered by position. relatedQueries.items[].position integer Rank of the query within the response (1-based). relatedQueries.items[].name string Search query as shown by Google. relatedQueries.items[].value string Score label as shown by Google: a percentage growth string (for example `+250%`) on the Rising view, or `Breakout` for new entrants; a relative score on the Top view. relatedQueries.items[].link string Google Trends Explore URL for the related query. ## Sample response ``` { "topic": "YouTube Music", "filters": { "geo": "Worldwide", "timeRange": "Past 12 months", "category": "All categories", "searchType": "Web Search", "description": "Worldwide, Past 12 months" }, "interestOverTime": { "title": "Interest over time", "timeline": [ { "date": "Jun 15, 2025", "value": 100 } ], "timelineCount": 53 }, "interestBySubregion": { "title": "Interest by region", "view": "Top", "totalItems": 87, "paginationText": "Showing 1-5 of 87 regions", "items": [ { "position": 1, "name": "South Africa", "value": 100 } ] }, "relatedTopics": { "title": "Related topics", "view": "Rising", "items": [ { "position": 1, "name": "Spotify - Topic", "value": "Breakout", "link": "https://trends.google.com/trends/explore?q=/g/11hy9l1k8h&date=today+12-m" } ] }, "relatedQueries": { "title": "Related queries", "view": "Rising", "items": [ { "position": 1, "name": "youtube muzica", "value": "+250%", "link": "https://trends.google.com/trends/explore?q=youtube+muzica&date=today+12-m" } ] } } ``` [← PreviousGoogle Trends](/docs/scrapers/google-trends)[Next →Facebook Group](/docs/scrapers/facebook-group) --- Source: https://crawlbase.com/docs/scrapers/immobilienscout24-property # ImmobilienScout24 Property Extract a German real-estate listing from Immobilienscout24 - address, price, size, features, and agent contact. ## API usage Add `&scraper=immobilienscout24-property` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.immobilienscout24.de/expose/167174293' \ --data-urlencode 'scraper=immobilienscout24-property' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.immobilienscout24.de/expose/167174293', {'scraper': 'immobilienscout24-property'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.immobilienscout24.de/expose/167174293', { scraper: 'immobilienscout24-property' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.immobilienscout24.de/expose/167174293', scraper: 'immobilienscout24-property') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.immobilienscout24.de/expose/167174293 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. property\_id string Listing ID. title string Title. property\_type string "Wohnung", "Haus", etc. listing\_type string "Kauf" or "Miete". address object Street, postal code, city, region. price object Total price plus monthly cost breakdown. size\_sqm number Living area in m². rooms number Number of rooms. year\_built integer | null Year built. features array Feature tags. description string Full description in German. images array Image URLs. agent object Agent contact. ## Sample response ``` { "property_id": "167174293", "property_type": "Wohnung", "listing_type": "Kauf", "price": { "purchase_price": "€549,000" }, "size_sqm": 85, "rooms": 3, "year_built": 1972 } ``` [← PreviousBing SERP](/docs/scrapers/bing-serp)[Next →Walmart SERP](/docs/scrapers/walmart-serp) --- Source: https://crawlbase.com/docs/scrapers/instagram-hashtag # Instagram Hashtag Extract recent and top posts from an Instagram hashtag page. Currently unavailable The `instagram-hashtag` scraper is currently not available due to changes from Instagram. We are working on a fix, but we do not have an ETA at this time. Use the JS token Instagram scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=instagram-hashtag` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.instagram.com/explore/tags/love/' \ --data-urlencode 'scraper=instagram-hashtag' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.instagram.com/explore/tags/love/', {'scraper': 'instagram-hashtag'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.instagram.com/explore/tags/love/', { scraper: 'instagram-hashtag' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.instagram.com/explore/tags/love/', scraper: 'instagram-hashtag') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.instagram.com/explore/tags/love/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. hashtag string Hashtag (without the #). post\_count integer | null Total posts using this hashtag, when shown. top\_posts array Top-ranked posts under the tag. recent\_posts array Recent posts under the tag. posts[].id string Post shortcode. posts[].thumbnail\_url string Thumbnail URL. posts[].like\_count integer Likes. posts[].comment\_count integer Comments. posts[].url string Permalink. ## Sample response ``` { "hashtag": "love", "post_count": 2100000000, "top_posts": [ { "id": "C7xY...", "thumbnail_url": "https://scontent.cdninstagram.com/...jpg", "like_count": 847000, "comment_count": 2102, "url": "https://www.instagram.com/p/C7xY..." } ] } ``` [← PreviousInstagram Profile](/docs/scrapers/instagram-profile)[Next →Instagram Reels Audio](/docs/scrapers/instagram-reels-audio) --- Source: https://crawlbase.com/docs/scrapers/instagram-post # Instagram Post Extract Instagram post metadata: caption, owner, media URLs, hashtags, mentions, and engagement counts. Use the JS token Instagram scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=instagram-post` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.instagram.com/p/B5LQhLiFFCX' \ --data-urlencode 'scraper=instagram-post' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.instagram.com/p/B5LQhLiFFCX', {'scraper': 'instagram-post'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.instagram.com/p/B5LQhLiFFCX', { scraper: 'instagram-post' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.instagram.com/p/B5LQhLiFFCX', scraper: 'instagram-post') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.instagram.com/p/B5LQhLiFFCX ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. id string Post shortcode. caption string Caption text. owner\_username string Author username. media\_type string `image`, `video`, or `carousel`. media\_urls array\ Direct URLs to image(s) or video(s). hashtags array\ Hashtags from the caption. mentions array\ @-mentioned usernames. location string | null Tagged location. like\_count integer Likes. comment\_count integer Comments. posted\_at string ISO 8601 timestamp. ## Sample response ``` { "id": "B5LQhLiFFCX", "caption": "New product launch! #design #craft", "owner_username": "apple", "media_type": "image", "media_urls": ["https://scontent.cdninstagram.com/...jpg"], "hashtags": ["design", "craft"], "mentions": [], "like_count": 142000, "comment_count": 3210, "posted_at": "2026-04-15T14:22:00Z" } ``` [← PreviousInstagram Reel](/docs/scrapers/instagram-reel)[Next →Instagram Profile](/docs/scrapers/instagram-profile) --- Source: https://crawlbase.com/docs/scrapers/instagram-profile # Instagram Profile Get Instagram profile data: bio, follower/following counts, recent posts, profile pic, and verification status. Use the JS token Instagram scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=instagram-profile` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.instagram.com/apple/' \ --data-urlencode 'scraper=instagram-profile' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.instagram.com/apple/', {'scraper': 'instagram-profile'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.instagram.com/apple/', { scraper: 'instagram-profile' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.instagram.com/apple/', scraper: 'instagram-profile') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.instagram.com/apple/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. username string Profile username (without @). full\_name string Profile display name. biography string Bio text. followers\_count integer Followers. following\_count integer Following. posts\_count integer Total posts on the profile. is\_verified boolean Verified-account flag. is\_private boolean Private-account flag. profile\_pic\_url string Profile picture URL. recent\_posts array Up to 12 recent posts (same shape as `instagram-post`). ## Sample response ``` { "username": "apple", "full_name": "Apple", "biography": "Welcome to @apple. The latest creativity going on around us.", "followers_count": 33800000, "following_count": 9, "posts_count": 1284, "is_verified": true, "is_private": false, "profile_pic_url": "https://scontent.cdninstagram.com/...jpg" } ``` [← PreviousInstagram Post](/docs/scrapers/instagram-post)[Next →Instagram Hashtag](/docs/scrapers/instagram-hashtag) --- Source: https://crawlbase.com/docs/scrapers/instagram-reel # Instagram Reel Extract Instagram Reel metadata: caption, owner, video URL, audio track, view counts, and engagement metrics. Use the JS token Instagram scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=instagram-reel` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.instagram.com/reels/DHq4bFpID1_/' \ --data-urlencode 'scraper=instagram-reel' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.instagram.com/reels/DHq4bFpID1_/', {'scraper': 'instagram-reel'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.instagram.com/reels/DHq4bFpID1_/', { scraper: 'instagram-reel' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.instagram.com/reels/DHq4bFpID1_/', scraper: 'instagram-reel') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.instagram.com/reels/DHq4bFpID1_/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. id string Reel shortcode (last segment of the URL). caption string Reel caption text. owner\_username string Username of the Reel's author. video\_url string Direct URL to the video file. audio object Linked audio details: `title`, `artist`, `url`. view\_count integer Reel view count. like\_count integer Likes. comment\_count integer Comments. posted\_at string ISO 8601 timestamp. ## Sample response ``` { "id": "DHq4bFpID1_", "caption": "Behind the scenes 🎬", "owner_username": "craft_studio", "video_url": "https://scontent.cdninstagram.com/...mp4", "audio": { "title": "Original audio", "artist": "craft_studio" }, "view_count": 142000, "like_count": 8432, "comment_count": 203 } ``` [← PreviousFacebook Event](/docs/scrapers/facebook-event)[Next →Instagram Post](/docs/scrapers/instagram-post) --- Source: https://crawlbase.com/docs/scrapers/instagram-reels-audio # Instagram Reels Audio Find every Reel that uses a specific audio track. Useful for trend monitoring and creator discovery. Use the JS token Instagram scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=instagram-reels-audio` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.instagram.com/reels/audio/430642407673774' \ --data-urlencode 'scraper=instagram-reels-audio' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.instagram.com/reels/audio/430642407673774', {'scraper': 'instagram-reels-audio'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.instagram.com/reels/audio/430642407673774', { scraper: 'instagram-reels-audio' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.instagram.com/reels/audio/430642407673774', scraper: 'instagram-reels-audio') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.instagram.com/reels/audio/430642407673774 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. audio\_id string Audio identifier from the URL. audio\_title string Audio track title. audio\_artist string Audio artist or original creator. reels\_using\_audio array Reels using this audio with shape similar to `instagram-reel`. reels\_using\_audio[].id string Reel shortcode. reels\_using\_audio[].owner\_username string Reel author username. reels\_using\_audio[].view\_count integer Reel view count. reels\_using\_audio[].thumbnail\_url string Thumbnail URL. ## Sample response ``` { "audio_id": "430642407673774", "audio_title": "Original audio", "audio_artist": "craft_studio", "reels_using_audio": [ { "id": "DHq4bFpID1_", "owner_username": "craft_studio", "view_count": 142000, "thumbnail_url": "https://scontent.cdninstagram.com/...jpg" } ] } ``` [← PreviousInstagram Hashtag](/docs/scrapers/instagram-hashtag)[Next →TikTok Product](/docs/scrapers/tiktok-product) --- Source: https://crawlbase.com/docs/scrapers/kaggle-dataset # Kaggle Dataset Parse a single Kaggle dataset page into structured JSON with its description, owner, license, file list, keywords, and engagement counts. ## API usage Add `&scraper=kaggle-dataset` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.kaggle.com/datasets/uciml/iris' \ --data-urlencode 'scraper=kaggle-dataset' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.kaggle.com/datasets/uciml/iris', {'scraper': 'kaggle-dataset'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.kaggle.com/datasets/uciml/iris', { scraper: 'kaggle-dataset' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.kaggle.com/datasets/uciml/iris', scraper: 'kaggle-dataset') data = JSON.parse(res.body) ``` ## Example input URL Any Kaggle dataset page works in the `url` parameter - the page at `/datasets//`. For example: ``` https://www.kaggle.com/datasets/uciml/iris https://www.kaggle.com/datasets/johnsmith88/heart-disease-dataset https://www.kaggle.com/datasets/zynicide/wine-reviews ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the dataset page that was scraped. id string | null Numeric Kaggle dataset identifier, returned as a string. title string | null Dataset display title. subtitle string | null One-line summary shown under the title. description string | null Full dataset description in Markdown, as the owner wrote it - links, images, and lists are preserved rather than flattened. version integer | null Current dataset version number. keywords array Topic tags applied to the dataset, as an array of strings. owner object Account that publishes the dataset. owner.name string | null Owner display name. owner.url string | null Absolute URL of the owner profile or organization page. owner.image string | null Owner avatar URL. license object License the dataset is published under. license.name string | null License display name (for example `CC0: Public Domain`). license.url string | null Canonical URL of the license text. files array Downloadable archives attached to the dataset. files[].format string | null Archive format (for example `zip`). files[].sizeBytes integer | null Archive size in bytes. files[].downloadUrl string | null Absolute download URL, pinned to the current dataset version. files[].requiresSubscription boolean Whether Kaggle requires a signed-in account to start the download. downloads integer | null Total download count for the dataset. views integer | null Total page view count. votes integer | null Upvote count. commentCount integer | null Number of comments in the dataset discussion. lastUpdated string | null ISO 8601 timestamp of the most recent dataset update. discussionUrl string | null Absolute URL of the dataset discussion tab. thumbnail string | null Dataset card image URL. isAccessibleForFree boolean Whether Kaggle marks the dataset as freely accessible. ## Sample response ``` { "url": "https://www.kaggle.com/datasets/uciml/iris", "id": "19", "title": "Iris Species", "subtitle": "Classify iris plants into three species in this classic dataset", "description": "The Iris dataset was used in R.A. Fisher's classic 1936 paper, [The Use of Multiple Measurements in Taxonomic Problems](http://rcs.chemometrics.ru/Tutorials/classification/Fisher.pdf), and can also be found on the [UCI Machine Learning Repository][1].\n\nIt includes three iris species with 50 samples each as well as some properties about each flower.", "version": 2, "keywords": [ "subject", "earth and nature", "biology" ], "owner": { "name": "UCI Machine Learning", "url": "https://www.kaggle.com/organizations/uciml", "image": "https://storage.googleapis.com/kaggle-organizations/7/thumbnail.png" }, "license": { "name": "CC0: Public Domain", "url": "https://creativecommons.org/publicdomain/zero/1.0/" }, "files": [ { "format": "zip", "sizeBytes": 3687, "downloadUrl": "https://www.kaggle.com/datasets/uciml/iris/download?datasetVersionNumber=2", "requiresSubscription": true } ], "downloads": 907784, "views": 3131083, "votes": 4861, "commentCount": 33, "lastUpdated": "2016-09-27T07:38:05.44Z", "discussionUrl": "https://www.kaggle.com/uciml/iris/discussion", "thumbnail": "https://storage.googleapis.com/kaggle-datasets-images/19/19/default-backgrounds/dataset-card.jpg", "isAccessibleForFree": true } ``` [← PreviousKaggle Dataset Search](/docs/scrapers/kaggle-dataset-serp)[Next →Kaggle Notebook Search](/docs/scrapers/kaggle-notebook-serp) --- Source: https://crawlbase.com/docs/scrapers/kaggle-dataset-serp # Kaggle Dataset Search Parse a Kaggle dataset search or listing page into structured JSON with one row per dataset, including owner, size, usability rating, downloads, and notebook count. ## API usage Add `&scraper=kaggle-dataset-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.kaggle.com/datasets?search=heart+disease' \ --data-urlencode 'scraper=kaggle-dataset-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.kaggle.com/datasets?search=heart+disease', {'scraper': 'kaggle-dataset-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.kaggle.com/datasets?search=heart+disease', { scraper: 'kaggle-dataset-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.kaggle.com/datasets?search=heart+disease', scraper: 'kaggle-dataset-serp') data = JSON.parse(res.body) ``` ## Example input URL Any Kaggle dataset search or listing page works in the `url` parameter - the page at `/datasets`, with or without a query string. For example: ``` https://www.kaggle.com/datasets?search=heart+disease https://www.kaggle.com/datasets?search=nlp&fileType=csv https://www.kaggle.com/datasets https://www.kaggle.com/datasets?search=weather&page=2 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the search page that was scraped. query string | null Search term Kaggle actually ran, read from the rendered search box and falling back to the request URL. totalCount integer | null Total number of datasets matching the search across all pages, or null when the page does not state a total. page integer | null Current page number, or null when it cannot be determined. hasNextPage boolean Whether a further page of results is available. resultCount integer Number of rows returned on this page. results array Dataset rows in the order they appear on the page. results[].position integer One-based rank of the row within this page. results[].title string | null Dataset display title. results[].url string | null Absolute URL of the dataset page. results[].slug string | null Dataset identifier as `owner/dataset`, read from the URL path. results[].owner object Account that publishes the dataset. results[].owner.name string | null Owner display name. results[].owner.url string | null Absolute URL of the owner profile. results[].thumbnail string | null Dataset card image URL. Kaggle serves it as a CSS background rather than an image tag, so it is read from the inline style. results[].lastUpdated string | null Absolute timestamp of the last dataset update. The card shows only a relative age, so the value is taken from the tooltip behind it and keeps Kaggle formatting rather than ISO 8601. results[].usabilityRating number | null Kaggle usability score for the dataset, from 0 to 10. results[].fileCount integer | null Number of files in the dataset. results[].fileFormat string | null File format label shown on the card (for example `CSV`). results[].size string | null Total dataset size as displayed, units included (for example `6 kB`). Kaggle does not expose the exact byte count in the list view. results[].downloads integer | null Download count. Kaggle abbreviates figures over a thousand on cards (for example `362K`); the value is expanded back to an exact integer, and is null when the card omits the figure. results[].notebookCount integer | null Number of public notebooks built on the dataset, expanded from the same abbreviated form as `downloads`. results[].votes integer | null Upvote count for the dataset. results[].medal string | null Kaggle medal on the dataset (for example `bronze`), or null when the row carries none. ## Sample response ``` { "url": "https://www.kaggle.com/datasets?search=heart+disease", "query": "heart disease", "totalCount": 1628, "page": 1, "hasNextPage": true, "resultCount": 20, "results": [ { "position": 1, "title": "Heart Disease Dataset", "url": "https://www.kaggle.com/datasets/johnsmith88/heart-disease-dataset", "slug": "johnsmith88/heart-disease-dataset", "owner": { "name": "David Lapp", "url": "https://www.kaggle.com/johnsmith88" }, "thumbnail": "https://storage.googleapis.com/kaggle-datasets-images/216167/469115/23af5a37ef4e938b2c9a1b97662c3efc/dataset-thumbnail.jpg?t=2019-06-04-03-29-56", "lastUpdated": "Thu Jun 06 2019 17:33:55 GMT+0200 (Central European Summer Time)", "usabilityRating": 8.8, "fileCount": 1, "fileFormat": "CSV", "size": "6 kB", "downloads": 362000, "notebookCount": 984, "votes": 1859, "medal": "gold" }, { "position": 2, "title": "Heart Disease", "url": "https://www.kaggle.com/datasets/oktayrdeki/heart-disease", "slug": "oktayrdeki/heart-disease", "owner": { "name": "Oktay Ördekçi", "url": "https://www.kaggle.com/oktayrdeki" }, "thumbnail": "https://storage.googleapis.com/kaggle-datasets-images/6393782/10326308/95bfd024a1dc7d059e7b6f3632ed37c9/dataset-thumbnail.png?t=2024-12-29-13-32-57", "lastUpdated": "Sun Dec 29 2024 14:26:49 GMT+0100 (Central European Standard Time)", "usabilityRating": 10, "fileCount": 1, "fileFormat": "CSV", "size": "582 kB", "downloads": 22700, "notebookCount": 48, "votes": 179, "medal": "bronze" } ] } ``` [← PreviousExercism Solution](/docs/scrapers/exercism-solution)[Next →Kaggle Dataset](/docs/scrapers/kaggle-dataset) --- Source: https://crawlbase.com/docs/scrapers/kaggle-notebook # Kaggle Notebook Parse a single Kaggle notebook page into structured JSON with its author, language, runtime, version history, engagement counts, and attached inputs. The notebook body is served in an iframe Kaggle renders the executed notebook - cells, output, and charts - inside an iframe rather than in the page itself, so it is not part of this response. `contentUrl` returns the address of that frame; fetch it as a second request when you need the rendered notebook. The URL carries a short-lived signed token, so request it soon after scraping rather than storing it. ## API usage Add `&scraper=kaggle-notebook` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.kaggle.com/code/alexisbcook/titanic-tutorial' \ --data-urlencode 'scraper=kaggle-notebook' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.kaggle.com/code/alexisbcook/titanic-tutorial', {'scraper': 'kaggle-notebook'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.kaggle.com/code/alexisbcook/titanic-tutorial', { scraper: 'kaggle-notebook' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.kaggle.com/code/alexisbcook/titanic-tutorial', scraper: 'kaggle-notebook') data = JSON.parse(res.body) ``` ## Example input URL Any Kaggle notebook page works in the `url` parameter - the page at `/code//`. For example: ``` https://www.kaggle.com/code/alexisbcook/titanic-tutorial https://www.kaggle.com/code/startupsci/titanic-data-science-solutions https://www.kaggle.com/code/ldfreeman3/a-data-science-framework-to-achieve-99-accuracy ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the notebook page that was scraped. slug string | null Notebook identifier as `author/notebook`, read from the URL path. title string | null Notebook display title. author object Account that owns the notebook. author.username string | null Owner handle. author.name string | null Owner display name. author.url string | null Absolute URL of the owner profile. publishedAt string | null ISO 8601 timestamp of first publication. lastUpdated string | null ISO 8601 timestamp of the most recent version. thumbnail string | null Notebook card image URL. language string | null Notebook language (for example `Python` or `R`). runtime string | null Execution time of the last run as displayed, units included (for example `16s`). version integer | null Version number currently on display. versionCount integer | null Total number of published versions. views integer | null Total page view count. votes integer | null Upvote count. copies integer | null Number of times the notebook has been copied or forked. commentCount integer | null Number of comments on the notebook. medal string | null Kaggle medal on the notebook (for example `gold`), or null when it carries none. license object License the notebook is published under. license.name string | null License display name (for example `Apache 2.0`). license.url string | null Canonical URL of the license text. inputs array Competitions, datasets, and other notebooks attached as inputs. inputs[].category string | null Input group heading as Kaggle labels it (for example `COMPETITIONS`). inputs[].title string | null Display name of the attached input. contentUrl string | null Address of the iframe that holds the rendered notebook body. Carries a short-lived signed token. ## Sample response ``` { "url": "https://www.kaggle.com/code/alexisbcook/titanic-tutorial", "slug": "alexisbcook/titanic-tutorial", "title": "Titanic Tutorial", "author": { "username": "alexisbcook", "name": "Alexis Cook", "url": "https://www.kaggle.com/alexisbcook" }, "publishedAt": "2022-06-24T00:25:16.1066667Z", "lastUpdated": "2022-06-24T00:25:16.1066667Z", "thumbnail": "https://storage.googleapis.com/kaggle-avatars/thumbnails/2603295-kg.jpg", "language": "Python", "runtime": "16s", "version": 22, "versionCount": 22, "views": 3265744, "votes": 60135, "copies": 51320, "commentCount": 30626, "medal": "gold", "license": { "name": "Apache 2.0", "url": "http://www.apache.org/licenses/LICENSE-2.0" }, "inputs": [ { "category": "COMPETITIONS", "title": "Titanic - Machine Learning from Disaster" } ], "contentUrl": "https://www.kaggleusercontent.com/kf/99170538// __results__.html" } ``` [← PreviousKaggle Notebook Search](/docs/scrapers/kaggle-notebook-serp)[Next →LeetCode Problem Set](/docs/scrapers/leetcode-serp) --- Source: https://crawlbase.com/docs/scrapers/kaggle-notebook-serp # Kaggle Notebook Search Parse a Kaggle notebook search or listing page into structured JSON with one row per notebook, including author, co-authors, competition context, votes, and comment count. What the listing page does not expose Kaggle labels the result total only as `Results (100+)`, so `totalCountLabel` is returned verbatim as a string rather than converted to a number. Rows with more than one author render stacked avatars with no image URL in the document, so `author.image` is empty on those rows. A notebook attached to several competitions shows the first one plus a `+2` badge - the extra names are not in the page, so only `additionalContextCount` can be reported. ## API usage Add `&scraper=kaggle-notebook-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.kaggle.com/code?searchQuery=titanic' \ --data-urlencode 'scraper=kaggle-notebook-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.kaggle.com/code?searchQuery=titanic', {'scraper': 'kaggle-notebook-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.kaggle.com/code?searchQuery=titanic', { scraper: 'kaggle-notebook-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.kaggle.com/code?searchQuery=titanic', scraper: 'kaggle-notebook-serp') data = JSON.parse(res.body) ``` ## Example input URL Any Kaggle notebook search or listing page works in the `url` parameter - the page at `/code`, with or without a query string. For example: ``` https://www.kaggle.com/code?searchQuery=titanic https://www.kaggle.com/code?searchQuery=xgboost&language=Python https://www.kaggle.com/code https://www.kaggle.com/code?searchQuery=nlp&page=2 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the search page that was scraped. query string | null Search term read from the request URL. Kaggle leaves the rendered search box empty on this page, so the URL is the only source. totalCountLabel string | null Result total exactly as Kaggle labels it, for example `100+`. Returned as a string because the page does not state an exact number. page integer | null Current page number, or null when it cannot be determined. hasNextPage boolean Whether a further page of results is available. resultCount integer Number of rows returned on this page. results array Notebook rows in the order they appear on the page. results[].position integer One-based rank of the row within this page. results[].title string | null Notebook display title. results[].url string | null Absolute URL of the notebook. results[].slug string | null Notebook identifier as `author/notebook`, read from the URL path. results[].author object Account that owns the notebook. results[].author.username string | null Owner handle, taken from the first path segment of the notebook URL so it resolves on co-authored rows too. results[].author.name string | null Owner display name. results[].author.url string | null Absolute URL of the owner profile. results[].author.image string | null Owner avatar URL. Empty on co-authored rows, where Kaggle renders stacked avatars with no URL in the document. results[].coauthors array Display names of the additional authors, as an array of strings. Empty on single-author notebooks. results[].lastUpdated string | null Absolute timestamp of the last notebook update. The card shows only a relative age, so the value is taken from the tooltip behind it and keeps Kaggle formatting rather than ISO 8601. results[].commentCount integer | null Number of comments on the notebook. results[].context string | null Name of the competition or dataset the notebook is attached to. results[].additionalContextCount integer How many further competitions or datasets the notebook is attached to beyond `context`. Their names are not in the page, so only the count is available. results[].votes integer | null Upvote count for the notebook. results[].medal string | null Kaggle medal on the notebook (for example `gold`), or null when the row carries none. ## Sample response ``` { "url": "https://www.kaggle.com/code?searchQuery=titanic", "query": "titanic", "totalCountLabel": "100+", "page": 1, "hasNextPage": true, "resultCount": 20, "results": [ { "position": 1, "title": "Titanic Tutorial", "url": "https://www.kaggle.com/code/alexisbcook/titanic-tutorial", "slug": "alexisbcook/titanic-tutorial", "author": { "username": "alexisbcook", "name": "Alexis Cook", "url": "https://www.kaggle.com/alexisbcook", "image": "https://storage.googleapis.com/kaggle-avatars/thumbnails/2603295-kg.jpg" }, "coauthors": [], "lastUpdated": "Fri Jun 24 2022 00:25:16 GMT+0000 (Coordinated Universal Time)", "commentCount": 30626, "context": "Titanic - Machine Learning from Disaster", "additionalContextCount": 0, "votes": 60135, "medal": "gold" }, { "position": 3, "title": "Spaceship Titanic with TFDF", "url": "https://www.kaggle.com/code/gusthema/spaceship-titanic-with-tfdf", "slug": "gusthema/spaceship-titanic-with-tfdf", "author": { "username": "gusthema", "name": "Gusthema", "url": "https://www.kaggle.com/gusthema", "image": "" }, "coauthors": [ "Nidhin PD" ], "lastUpdated": "Mon Apr 17 2023 09:36:29 GMT+0000 (Coordinated Universal Time)", "commentCount": 378, "context": "Spaceship Titanic", "additionalContextCount": 0, "votes": 7863, "medal": "gold" }, { "position": 10, "title": "Titanic Top Solution", "url": "https://www.kaggle.com/code/brendan45774/titanic-top-solution", "slug": "brendan45774/titanic-top-solution", "author": { "username": "brendan45774", "name": "Brenda N", "url": "https://www.kaggle.com/brendan45774", "image": "https://storage.googleapis.com/kaggle-avatars/thumbnails/2681031-kg.jpg" }, "coauthors": [], "lastUpdated": "Wed Sep 01 2021 16:03:11 GMT+0000 (Coordinated Universal Time)", "commentCount": 249, "context": "Titanic - Machine Learning from Disaster", "additionalContextCount": 2, "votes": 2294, "medal": "gold" } ] } ``` [← PreviousKaggle Dataset](/docs/scrapers/kaggle-dataset)[Next →Kaggle Notebook](/docs/scrapers/kaggle-notebook) --- Source: https://crawlbase.com/docs/scrapers/leetcode-problem # LeetCode Problem Parse a single LeetCode problem page into structured JSON with its statement, difficulty, topic tags, hints, starter code snippets, and engagement counts. Premium problems return metadata only A problem with `isPremium` set to `true` keeps its identity fields - id, title, difficulty, topic tags - but the statement and the engagement counters are not part of the page, so `description`, `descriptionHtml`, and the counts come back empty. ## API usage Add `&scraper=leetcode-problem` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://leetcode.com/problems/two-sum/' \ --data-urlencode 'scraper=leetcode-problem' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://leetcode.com/problems/two-sum/', {'scraper': 'leetcode-problem'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://leetcode.com/problems/two-sum/', { scraper: 'leetcode-problem' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://leetcode.com/problems/two-sum/', scraper: 'leetcode-problem') data = JSON.parse(res.body) ``` ## Example input URL Any LeetCode problem page works in the `url` parameter - the page at `/problems/`. For example: ``` https://leetcode.com/problems/two-sum/ https://leetcode.com/problems/add-two-numbers/ https://leetcode.com/problems/longest-substring-without-repeating-characters/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the problem page that was scraped. id string | null Problem number as shown on the page, returned as a string. internalId string | null LeetCode's internal question identifier. Usually equal to `id`, but the two diverge on older problems. title string | null Problem display title. titleSlug string | null URL slug of the problem. difficulty string | null Difficulty label - `Easy`, `Medium`, or `Hard`. category string | null Problem category, for example `Algorithms` or `Database`. isPremium boolean Whether the problem is locked behind a LeetCode subscription. descriptionHtml string | null Problem statement as HTML, with the original markup preserved so examples, constraints, and inline code survive intact. description string | null Problem statement flattened to plain text. likes integer | null Upvote count on the problem. dislikes integer | null Downvote count on the problem. acceptedCount integer | null Number of accepted submissions. submissionCount integer | null Total number of submissions. acceptanceRate string | null Acceptance rate as displayed, including the percent sign. commentCount integer | null Number of comments in the problem discussion. topicTags array Algorithm and data structure tags applied to the problem. topicTags[].name string | null Tag display name. topicTags[].slug string | null Tag URL slug. hints array Hints published with the problem, as an array of plain-text strings. exampleTestcases array Default test case inputs, as an array of strings. Each string keeps the newlines that separate its arguments. codeSnippets array Starter code stubs LeetCode provides for the problem. codeSnippets[].language string | null Language display name, for example `Python3`. codeSnippets[].languageSlug string | null Language slug, for example `python3`. codeSnippets[].code string | null The stub itself, with its original indentation. similarQuestions array Problems LeetCode links as related. similarQuestions[].title string | null Related problem title. similarQuestions[].titleSlug string | null Related problem URL slug. similarQuestions[].url string | null Absolute URL of the related problem. similarQuestions[].difficulty string | null Related problem difficulty label. similarQuestions[].isPremium boolean Whether the related problem is subscription-locked. officialSolution object State of LeetCode's own write-up for the problem. officialSolution.available boolean Whether an official solution exists. officialSolution.isFree boolean Whether the official solution is readable without a subscription. officialSolution.hasVideo boolean Whether the official solution includes a video walkthrough. solutionsUrl string | null Absolute URL of the community solutions tab for this problem. ## Sample response ``` { "url": "https://leetcode.com/problems/two-sum/", "id": "1", "internalId": "1", "title": "Two Sum", "titleSlug": "two-sum", "difficulty": "Easy", "category": "Algorithms", "isPremium": false, "descriptionHtml": "You are given an array of integers nums and an integer target, return indices of t...", "description": "You are given an array of integers nums and an integer target, return indices of the two numbers such that they add up t...", "likes": 69478, "dislikes": 2592, "acceptedCount": 22929319, "submissionCount": 39569455, "acceptanceRate": "57.9%", "commentCount": 3490, "topicTags": [ { "name": "Array", "slug": "array" }, { "name": "Hash Table", "slug": "hash-table" } ], "hints": [ "A really brute force way would be to search for all possible pairs of numbers but that would be too slow. Again, it's best to try out brute force solutions just for completeness. It is from these brute force solutions that you can come up with optimizations." ], "exampleTestcases": [ "[2,7,11,15]\n9", "[3,2,4]\n6", "[3,3]\n6" ], "codeSnippets": [ { "language": "Python3", "languageSlug": "python3", "code": "class Solution:\n def twoSum(self, nums: List[int], target: int) -> List[int]:\n " }, { "language": "JavaScript", "languageSlug": "javascript", "code": "/**\n * @param {number[]} nums\n * @param {number} target\n * @return {number[]}\n */\nvar twoSum = function(nums, target) {\n \n};" } ], "similarQuestions": [ { "title": "3Sum", "titleSlug": "3sum", "url": "https://leetcode.com/problems/3sum/", "difficulty": "Medium", "isPremium": false }, { "title": "4Sum", "titleSlug": "4sum", "url": "https://leetcode.com/problems/4sum/", "difficulty": "Medium", "isPremium": false } ], "officialSolution": { "available": true, "isFree": true, "hasVideo": true }, "solutionsUrl": "https://leetcode.com/problems/two-sum/solutions/" } ``` [← PreviousLeetCode Problem Set](/docs/scrapers/leetcode-serp)[Next →LeetCode Solutions](/docs/scrapers/leetcode-solutions) --- Source: https://crawlbase.com/docs/scrapers/leetcode-serp # LeetCode Problem Set Parse a LeetCode problem set listing into structured JSON with each problem id, title, difficulty, acceptance rate, premium flag, and URL. The daily challenge is returned as an extra row LeetCode pins the daily challenge above the paginated table, so a full page returns one entry more than the page size - the pinned problem first, then the table rows. It is the only entry with `isDailyQuestion` set to `true`, which makes it straightforward to drop if you only want the table. ## API usage Add `&scraper=leetcode-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://leetcode.com/problemset/' \ --data-urlencode 'scraper=leetcode-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://leetcode.com/problemset/', {'scraper': 'leetcode-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://leetcode.com/problemset/', { scraper: 'leetcode-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://leetcode.com/problemset/', scraper: 'leetcode-serp') data = JSON.parse(res.body) ``` ## Example input URL Any LeetCode problem set listing works in the `url` parameter - the unfiltered set at `/problemset/` or a filtered view. For example: ``` https://leetcode.com/problemset/ https://leetcode.com/problemset/?difficulty=EASY https://leetcode.com/problemset/?topicSlugs=dynamic-programming ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the problem set listing that was scraped. totalProblems integer | null Total number of problems LeetCode reports for the current filter, across every page. problemCount integer Number of entries returned in `problems` for this page. problems array Problem rows found on the page, in the order they are listed. problems[].id string | null Problem number as shown in the listing, returned as a string. problems[].title string | null Problem display title. problems[].titleSlug string | null URL slug of the problem, usable to build any other LeetCode problem URL. problems[].url string | null Absolute URL of the problem page. problems[].difficulty string | null Difficulty label - `Easy`, `Medium`, or `Hard`. problems[].acceptanceRate string | null Acceptance rate as displayed, including the percent sign. problems[].isPremium boolean Whether the problem is locked behind a LeetCode subscription. problems[].isDailyQuestion boolean Whether this row is the pinned daily challenge. ## Sample response ``` { "url": "https://leetcode.com/problemset/", "totalProblems": 4013, "problemCount": 101, "problems": [ { "id": "3626", "title": "Smallest Divisible Digit Product I", "titleSlug": "smallest-divisible-digit-product-i", "url": "https://leetcode.com/problems/smallest-divisible-digit-product-i", "difficulty": "Easy", "acceptanceRate": "71.0%", "isPremium": false, "isDailyQuestion": true }, { "id": "1", "title": "Two Sum", "titleSlug": "two-sum", "url": "https://leetcode.com/problems/two-sum", "difficulty": "Easy", "acceptanceRate": "57.9%", "isPremium": false, "isDailyQuestion": false }, { "id": "2", "title": "Add Two Numbers", "titleSlug": "add-two-numbers", "url": "https://leetcode.com/problems/add-two-numbers", "difficulty": "Medium", "acceptanceRate": "49.1%", "isPremium": false, "isDailyQuestion": false } ] } ``` [← PreviousKaggle Notebook](/docs/scrapers/kaggle-notebook)[Next →LeetCode Problem](/docs/scrapers/leetcode-problem) --- Source: https://crawlbase.com/docs/scrapers/leetcode-solution # LeetCode Solution Parse a single LeetCode community solution post into structured JSON with its author, tags, write-up, extracted code blocks, and engagement counts. Only the rendered code blocks are returned Many posts advertise several languages but put the alternatives behind tabs that render one at a time. `codeBlocks` holds the blocks present in the write-up as served, so a post titled for three languages can come back with blocks in one. ## API usage Add `&scraper=leetcode-solution` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://leetcode.com/problems/two-sum/solutions/3619262/3-methods-c-java-python-beginner-friendl-x595/' \ --data-urlencode 'scraper=leetcode-solution' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://leetcode.com/problems/two-sum/solutions/3619262/3-methods-c-java-python-beginner-friendl-x595/', {'scraper': 'leetcode-solution'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://leetcode.com/problems/two-sum/solutions/3619262/3-methods-c-java-python-beginner-friendl-x595/', { scraper: 'leetcode-solution' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://leetcode.com/problems/two-sum/solutions/3619262/3-methods-c-java-python-beginner-friendl-x595/', scraper: 'leetcode-solution') data = JSON.parse(res.body) ``` ## Example input URL Any LeetCode community solution post works in the `url` parameter - the page at `/problems//solutions///`. For example: ``` https://leetcode.com/problems/two-sum/solutions/3619262/3-methods-c-java-python-beginner-friendl-x595/ https://leetcode.com/problems/two-sum/solutions/127810/two-sum-by-leetcode-kwuq/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the solution post that was scraped. id string | null Post identifier taken from its URL, returned as a string. problem string | null Slug of the problem the post belongs to. problemUrl string | null Absolute URL of the problem page. title string | null Post title. author object Account that published the post. author.name string | null Author display name. author.username string | null Author profile handle. author.url string | null Absolute URL of the author profile. postedAt string | null Publication date as displayed on the post. tags array Tag chips on the post - topics and languages - as an array of strings. views integer | null View count on the post. upvotes integer | null Upvote count on the post. comments integer | null Comment count on the post. content string | null The write-up flattened to plain text. contentHtml string | null The write-up as HTML, with headings, lists, and code markup preserved. codeBlocks array Code blocks extracted from the write-up, in document order. codeBlocks[].language string | null Language of the block as LeetCode labels it, for example `cpp`. codeBlocks[].code string | null The block source, with its original indentation. ## Sample response ``` { "url": "https://leetcode.com/problems/two-sum/solutions/3619262/3-methods-c-java-python-beginner-friendl-x595/", "id": "3619262", "problem": "two-sum", "problemUrl": "https://leetcode.com/problems/two-sum/", "title": "✅3 Method's || C++ || JAVA || PYTHON || Beginner Friendly🔥🔥🔥", "author": { "name": "Rahul Varma", "username": "rahulvarma5297", "url": "https://leetcode.com/u/rahulvarma5297/" }, "postedAt": "Jun 09, 2023", "tags": [ "Array", "Hash Table", "C++", "Java" ], "views": 2368905, "upvotes": 12000, "comments": 285, "content": "Intuition\n\nThe Two Sum problem asks us to find two numbers in an array that sum up to a given target value. We need to r...", "contentHtml": "Intuition\n\nThe Two Sum problem asks us to find two numbers in an array that sum up to a given...", "codeBlocks": [ { "language": "cpp", "code": "class Solution {\npublic:\n vector twoSum(vector& nums, int target) {\n int n = nums.size();\n..." } ] } ``` [← PreviousLeetCode Solutions](/docs/scrapers/leetcode-solutions)[Next →OLX SERP](/docs/scrapers/olx-serp) --- Source: https://crawlbase.com/docs/scrapers/leetcode-solutions # LeetCode Solutions Parse the community solutions tab of a LeetCode problem into structured JSON with each post title, author, tags, and engagement counts. Counters are rounded above a thousand LeetCode abbreviates upvote, view, and comment counts once they pass a thousand - `11.9K`, `2.3M`. Those are expanded back to integers, so a value like `11900` carries the precision the page showed rather than the exact figure. ## API usage Add `&scraper=leetcode-solutions` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://leetcode.com/problems/two-sum/solutions/' \ --data-urlencode 'scraper=leetcode-solutions' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://leetcode.com/problems/two-sum/solutions/', {'scraper': 'leetcode-solutions'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://leetcode.com/problems/two-sum/solutions/', { scraper: 'leetcode-solutions' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://leetcode.com/problems/two-sum/solutions/', scraper: 'leetcode-solutions') data = JSON.parse(res.body) ``` ## Example input URL The solutions tab of any LeetCode problem works in the `url` parameter - the page at `/problems//solutions/`. For example: ``` https://leetcode.com/problems/two-sum/solutions/ https://leetcode.com/problems/add-two-numbers/solutions/ https://leetcode.com/problems/valid-parentheses/solutions/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. url string URL of the solutions listing that was scraped. problem string | null Slug of the problem the solutions belong to. problemUrl string | null Absolute URL of the problem page. solutionCount integer Number of entries returned in `solutions` for this page. solutions array Solution posts found on the page, in the order they are listed. solutions[].title string | null Post title. solutions[].url string | null Absolute URL of the post. solutions[].id string | null Post identifier taken from its URL, returned as a string. solutions[].author object Account that published the post. solutions[].author.name string | null Author display name. solutions[].author.username string | null Author profile handle. solutions[].author.url string | null Absolute URL of the author profile. solutions[].postedAt string | null Publication date as displayed on the card. solutions[].tags array Tag chips on the card - topics and languages - as an array of strings. solutions[].upvotes integer | null Upvote count on the post. solutions[].views integer | null View count on the post. solutions[].comments integer | null Comment count on the post. ## Sample response ``` { "url": "https://leetcode.com/problems/two-sum/solutions/", "problem": "two-sum", "problemUrl": "https://leetcode.com/problems/two-sum/", "solutionCount": 15, "solutions": [ { "title": "Two Sum", "url": "https://leetcode.com/problems/two-sum/solutions/127810/two-sum-by-leetcode-kwuq/", "id": "127810", "author": { "name": "LeetCode", "username": "leetcode", "url": "https://leetcode.com/u/leetcode/" }, "postedAt": "Jun 25, 2021", "tags": [ "Editorial" ], "upvotes": 5000, "views": 13800000, "comments": 2700 }, { "title": "✅3 Method's || C++ || JAVA || PYTHON || Beginner Friendly🔥🔥🔥", "url": "https://leetcode.com/problems/two-sum/solutions/3619262/3-methods-c-java-python-beginner-friendl-x595/", "id": "3619262", "author": { "name": "Rahul Varma", "username": "rahulvarma5297", "url": "https://leetcode.com/u/rahulvarma5297/" }, "postedAt": "Jun 09, 2023", "tags": [ "Array", "Hash Table", "C++", "Java" ], "upvotes": 11900, "views": 2300000, "comments": 285 } ] } ``` [← PreviousLeetCode Problem](/docs/scrapers/leetcode-problem)[Next →LeetCode Solution](/docs/scrapers/leetcode-solution) --- Source: https://crawlbase.com/docs/scrapers/linkedin-company # LinkedIn Company Extract a LinkedIn company page - description, industry, headcount, headquarters, employee samples, and locations. ## API usage Add `&scraper=linkedin-company` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.linkedin.com/company/amazon' \ --data-urlencode 'scraper=linkedin-company' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.linkedin.com/company/amazon', {'scraper': 'linkedin-company'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.linkedin.com/company/amazon', { scraper: 'linkedin-company' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.linkedin.com/company/amazon', scraper: 'linkedin-company') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.linkedin.com/company/amazon ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. name string Company name. description string About section. industry string Primary industry. company\_size string Employee size bracket. headquarters string HQ city. founded string | null Founding year. specialties array Specialty tags. website string Website URL. logo\_url string Logo image URL. employee\_count\_on\_linkedin integer LinkedIn member count. locations array Office locations. ## Sample response ``` { "name": "Amazon", "industry": "Software Development", "company_size": "10,001+ employees", "headquarters": "Seattle, Washington", "founded": "1994", "website": "https://www.amazon.com", "employee_count_on_linkedin": 2840000 } ``` [← PreviousLinkedIn Profile](/docs/scrapers/linkedin-profile)[Next →LinkedIn Feed](/docs/scrapers/linkedin-feed) --- Source: https://crawlbase.com/docs/scrapers/linkedin-feed # LinkedIn Feed Extract a LinkedIn feed post - author, body, attachments, reactions, and comment counts. ## API usage Add `&scraper=linkedin-feed` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.linkedin.com/feed/update/urn:li:activity:7022155503770251267' \ --data-urlencode 'scraper=linkedin-feed' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.linkedin.com/feed/update/urn:li:activity:7022155503770251267', {'scraper': 'linkedin-feed'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.linkedin.com/feed/update/urn:li:activity:7022155503770251267', { scraper: 'linkedin-feed' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.linkedin.com/feed/update/urn:li:activity:7022155503770251267', scraper: 'linkedin-feed') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.linkedin.com/feed/update/urn:li:activity:7022155503770251267 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. post\_id string Activity URN. author\_name string Author display name. author\_headline string Author headline. author\_profile\_url string Author profile URL. posted\_at string Relative timestamp. body string Post text. media array Attached images, videos, document links. reactions\_count integer Total reactions. comments\_count integer Comments count. reposts\_count integer Reposts count. ## Sample response ``` { "post_id": "urn:li:activity:7022155503770251267", "author_name": "Jane Doe", "author_headline": "VP Engineering", "posted_at": "2 weeks ago", "reactions_count": 1842, "comments_count": 87, "reposts_count": 42 } ``` [← PreviousLinkedIn Company](/docs/scrapers/linkedin-company)[Next →Quora SERP](/docs/scrapers/quora-serp) --- Source: https://crawlbase.com/docs/scrapers/linkedin-profile # LinkedIn Profile Extract a public LinkedIn profile - experience, education, skills, certifications, publications, and volunteering. ## API usage Add `&scraper=linkedin-profile` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.linkedin.com/in/kaitlyn-owen' \ --data-urlencode 'scraper=linkedin-profile' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.linkedin.com/in/kaitlyn-owen', {'scraper': 'linkedin-profile'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.linkedin.com/in/kaitlyn-owen', { scraper: 'linkedin-profile' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.linkedin.com/in/kaitlyn-owen', scraper: 'linkedin-profile') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.linkedin.com/in/kaitlyn-owen ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. full\_name string Full display name. headline string Profile headline. location string City, region, country. about string | null About / summary section. avatar\_url string Profile photo URL. connections\_count string Connection count (often "500+"). experience array Work experiences with title, company, dates, description. education array Degrees with school, field, dates. skills array Skill name strings. certifications array Certifications with name, issuer, date. languages array Languages with proficiency level. ## Sample response ``` { "full_name": "Kaitlyn Owen", "headline": "Senior Software Engineer at Acme Corp", "location": "San Francisco Bay Area", "connections_count": "500+", "experience": [ { "title": "Senior Software Engineer", "company": "Acme Corp", "start_date": "2022", "end_date": "Present" } ] } ``` [← PreviousTikTok Profile](/docs/scrapers/tiktok-profile)[Next →LinkedIn Company](/docs/scrapers/linkedin-company) --- Source: https://crawlbase.com/docs/scrapers/olx-item # OLX Item Turn a single OLX ad page into structured JSON with the full description, ad parameters, photos, location, and seller. Pass `scraper=olx-item` to the Crawling API with any OLX ad URL. ## Usage Send the individual OLX ad URL to the Crawling API with the `scraper=olx-item` parameter. Ad pages live under the `/d/` path. The scraper covers OLX's shared frontend, so the same scraper name works across markets - only the domain in the request URL changes. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.olx.pl/d/oferta/bmw-320d-f30-2016-xdrive-IDABC12.html' \ --data-urlencode 'scraper=olx-item' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.olx.pl/d/oferta/bmw-320d-f30-2016-xdrive-IDABC12.html', {'scraper': 'olx-item'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.olx.pl/d/oferta/bmw-320d-f30-2016-xdrive-IDABC12.html', { scraper: 'olx-item' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.olx.pl/d/oferta/bmw-320d-f30-2016-xdrive-IDABC12.html', scraper: 'olx-item') data = JSON.parse(res.body) ``` ## Example input URL Any individual OLX ad URL (the `/d/` detail path). A few examples across OLX markets: ``` https://www.olx.pl/d/oferta/bmw-320d-f30-2016-xdrive-IDABC12.html https://www.olx.ua/d/uk/obyavlenie/planka-IDVDALa.html https://www.olx.pt/d/anuncio/iphone-15-pro-IDXYZ99.html ``` The `olx-item` scraper works on the shared OLX frontend across `olx.pl`, `olx.ua`, `olx.pt`, `olx.ro`, `olx.bg`, `olx.kz`, and `olx.uz` - only the TLD in the request URL changes. OLX rate-limits datacenter IPs, but you don't pick a proxy pool yourself - the Crawling API routes through its residential network by default and auto-selects the best exit per request. Pass `country=` (e.g. `country=PL` for olx.pl) only when you need a specific market's geo. ## Response The scraper returns a JSON object with the full ad, including its parameters, photos, location, and seller. url string The ad page URL that was scraped. id integer | null OLX ad id. title string | null Ad title. description string | null Full ad description text as written by the seller. category object Category object for the ad. category.id integer | null OLX category id. category.type string | null Category type slug (e.g. `automotive`). price object Price object for the ad. price.value number | null Numeric price value. price.currency string | null ISO currency code (e.g. `PLN`, `UAH`, `EUR`). price.currencySymbol string | null Display currency symbol (e.g. `zł`). price.displayValue string | null Human-readable formatted price as shown on OLX. price.negotiable boolean Whether the seller marked the price as negotiable. price.free boolean Whether the ad is listed as free. price.exchange boolean Whether the seller is open to an exchange instead of cash. itemCondition string | null Item condition when provided (e.g. `new`, `used`). params array Array of ad-specific parameters (make, year, mileage, size, and so on). params[].key string | null Machine-readable parameter key. params[].name string | null Localized parameter label as shown on OLX. params[].value string | null Localized parameter value. params[].normalizedValue string | null Normalized value suitable for filtering, when available. location object Location object for the ad. location.city string | null City name. location.region string | null Region, voivodeship, or oblast name. location.district string | null District or neighborhood, when present. location.path string | null Human-readable location path (region, city, district). location.latitude number | null Approximate latitude. location.longitude number | null Approximate longitude. photos array Array of full-size photo URLs for the ad. seller object Seller object for the ad. seller.id integer | null OLX seller id. seller.name string | null Seller display name. seller.type string | null Seller type (e.g. `private`, `business`). seller.companyName string | null Registered company name for business sellers. seller.registeredAt string | null ISO 8601 timestamp of when the seller registered on OLX. seller.lastSeenAt string | null ISO 8601 timestamp of the seller's last activity. contact object Contact options object for the ad. contact.name string | null Contact name shown on the ad. contact.phoneAvailable boolean Whether a phone number is available for this ad. contact.chatAvailable boolean Whether OLX chat is enabled for this ad. contact.courier boolean Whether OLX courier/delivery is offered. contact.negotiation boolean Whether the seller accepts price negotiation. createdTime string | null ISO 8601 timestamp of when the ad was posted. lastRefreshTime string | null ISO 8601 timestamp of when the ad was last refreshed by the seller. validToTime string | null ISO 8601 timestamp of when the ad expires. status string | null Ad status (e.g. `active`, `removed`). isActive boolean Whether the ad is currently active. isBusiness boolean Whether the ad was posted by a business account. isPromoted boolean Whether the ad is a paid/promoted listing. isHighlighted boolean Whether the ad is visually highlighted. ## Sample response ``` { "url": "https://www.olx.pl/d/oferta/bmw-320d-f30-2016-xdrive-IDABC12.html", "id": 915581849, "title": "BMW 320d F30 2016 xDrive - pełen serwis", "description": "Sprzedam BMW 320d F30, rok 2016, xDrive. Pełna historia serwisowa, bezwypadkowy, jeden właściciel.", "category": { "id": 4, "type": "automotive" }, "price": { "value": 45900, "currency": "PLN", "currencySymbol": "zł", "displayValue": "45 900 zł", "negotiable": true, "free": false, "exchange": false }, "itemCondition": "used", "params": [ { "key": "make", "name": "Marka", "value": "BMW", "normalizedValue": "bmw" }, { "key": "year", "name": "Rok produkcji", "value": "2016", "normalizedValue": "2016" }, { "key": "mileage", "name": "Przebieg", "value": "142 000 km", "normalizedValue": "142000" } ], "location": { "city": "Warszawa", "region": "Mazowieckie", "district": "Mokotów", "path": "Mazowieckie, Warszawa, Mokotów", "latitude": 52.2297, "longitude": 21.0122 }, "photos": [ "https://ireland.apollo.olxcdn.com:443/v1/files/765inat2qu8k1-PL/image;s=1024x768", "https://ireland.apollo.olxcdn.com:443/v1/files/a1b2c3d4e5f6-PL/image;s=1024x768" ], "seller": { "id": 8842013, "name": "Auto Komis Premium", "type": "business", "companyName": "Auto Komis Premium Sp. z o.o.", "registeredAt": "2019-03-11T00:00:00+01:00", "lastSeenAt": "2026-07-21T18:42:00+02:00" }, "contact": { "name": "Auto Komis Premium", "phoneAvailable": true, "chatAvailable": true, "courier": false, "negotiation": true }, "createdTime": "2026-07-18T10:24:55+02:00", "lastRefreshTime": "2026-07-21T08:05:12+02:00", "validToTime": "2026-08-18T10:24:55+02:00", "status": "active", "isActive": true, "isBusiness": true, "isPromoted": true, "isHighlighted": false } ``` [← PreviousOLX SERP](/docs/scrapers/olx-serp)[Next →Generic Extractor](/docs/scrapers/generic-extractor) --- Source: https://crawlbase.com/docs/scrapers/olx-serp # OLX SERP Turn an OLX search or category results page into a structured array of ads with pagination metadata. Pass `scraper=olx-serp` to the Crawling API with any OLX listing URL. ## Usage Send the OLX search or category URL to the Crawling API with the `scraper=olx-serp` parameter. The scraper covers OLX's shared frontend, so the same scraper name works across markets - only the domain in the request URL changes. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.olx.pl/motoryzacja/samochody/' \ --data-urlencode 'scraper=olx-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.olx.pl/motoryzacja/samochody/', {'scraper': 'olx-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.olx.pl/motoryzacja/samochody/', { scraper: 'olx-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.olx.pl/motoryzacja/samochody/', scraper: 'olx-serp') data = JSON.parse(res.body) ``` ## Example input URL Any OLX search, category, or query results URL. A few examples across OLX markets: ``` https://www.olx.pl/motoryzacja/samochody/ https://www.olx.pl/nieruchomosci/mieszkania/wynajem/ https://www.olx.pl/oferty/q-iphone-15/ https://www.olx.ua/uk/list/ https://www.olx.pt/imoveis/ https://www.olx.ro/auto-masini-moto-ambarcatiuni/ ``` The `olx-serp` scraper works on the shared OLX frontend across `olx.pl`, `olx.ua`, `olx.pt`, `olx.ro`, `olx.bg`, `olx.kz`, and `olx.uz` - only the TLD in the request URL changes. OLX rate-limits datacenter IPs, but you don't pick a proxy pool yourself - the Crawling API routes through its residential network by default and auto-selects the best exit per request. Pass `country=` (e.g. `country=PL` for olx.pl) only when you need a specific market's geo. ## Response The scraper returns a JSON object describing the results page and an `items` array of ads. url string The results page URL that was scraped. categoryId integer | null OLX category id for the results page, when present. page integer | null Zero-based index of the current results page. totalPages integer | null Total number of results pages available for this query. totalCount integer | null Total number of ads matching the query across all pages. itemCount integer Number of ads returned on this page (length of `items`). items array Array of ads on the current results page. items[].id integer | null OLX ad id. items[].title string | null Ad title. items[].url string | null Canonical URL of the individual ad page. items[].price object Price object for the ad. items[].price.value number | null Numeric price value. items[].price.currency string | null ISO currency code (e.g. `PLN`, `UAH`, `EUR`). items[].price.currencySymbol string | null Display currency symbol (e.g. `zł`). items[].price.displayValue string | null Human-readable formatted price as shown on OLX. items[].price.negotiable boolean Whether the seller marked the price as negotiable. items[].price.free boolean Whether the ad is listed as free. items[].price.exchange boolean Whether the seller is open to an exchange instead of cash. items[].location object Location object for the ad. items[].location.city string | null City name. items[].location.region string | null Region, voivodeship, or oblast name. items[].location.district string | null District or neighborhood, when present. items[].location.path string | null Human-readable location path (region, city, district). items[].location.latitude number | null Approximate latitude. items[].location.longitude number | null Approximate longitude. items[].category object Category object for the ad. items[].category.id integer | null OLX category id for the ad. items[].category.type string | null Category type slug (e.g. `automotive`). items[].createdTime string | null ISO 8601 timestamp of when the ad was posted. items[].isPromoted boolean Whether the ad is a paid/promoted listing. items[].isHighlighted boolean Whether the ad is visually highlighted in results. items[].isBusiness boolean Whether the ad was posted by a business account. items[].thumbnailUrl string | null URL of the ad's thumbnail image. ## Sample response ``` { "url": "https://www.olx.pl/motoryzacja/samochody/", "categoryId": 4, "page": 0, "totalPages": 25, "totalCount": 1000, "itemCount": 2, "items": [ { "id": 915581849, "title": "BMW 320d F30 2016 xDrive - pełen serwis", "url": "https://www.olx.pl/d/oferta/bmw-320d-f30-2016-xdrive-IDABC12.html", "price": { "value": 45900, "currency": "PLN", "currencySymbol": "zł", "displayValue": "45 900 zł", "negotiable": true, "free": false, "exchange": false }, "location": { "city": "Warszawa", "region": "Mazowieckie", "district": "Mokotów", "path": "Mazowieckie, Warszawa, Mokotów", "latitude": 52.2297, "longitude": 21.0122 }, "category": { "id": 4, "type": "automotive" }, "createdTime": "2026-07-18T10:24:55+02:00", "isPromoted": true, "isHighlighted": false, "isBusiness": true, "thumbnailUrl": "https://ireland.apollo.olxcdn.com:443/v1/files/765inat2qu8k1-PL/image;s=1024x768" }, { "id": 927067590, "title": "Audi A4 B9 2.0 TDI 2017 - bezwypadkowy", "url": "https://www.olx.pl/d/oferta/audi-a4-b9-20-tdi-2017-IDDEF34.html", "price": { "value": 62000, "currency": "PLN", "currencySymbol": "zł", "displayValue": "62 000 zł", "negotiable": false, "free": false, "exchange": false }, "location": { "city": "Kraków", "region": "Małopolskie", "district": null, "path": "Małopolskie, Kraków", "latitude": 50.0647, "longitude": 19.945 }, "category": { "id": 4, "type": "automotive" }, "createdTime": "2026-07-16T09:12:41+02:00", "isPromoted": false, "isHighlighted": false, "isBusiness": false, "thumbnailUrl": "https://ireland.apollo.olxcdn.com:443/v1/files/c0riwdwc88q11-PL/image;s=1024x768" } ] } ``` [← PreviousLeetCode Solution](/docs/scrapers/leetcode-solution)[Next →OLX Item](/docs/scrapers/olx-item) --- Source: https://crawlbase.com/docs/scrapers/producthunt-leaderboard # Product Hunt Leaderboard Parse a Product Hunt leaderboard page into structured JSON with the ranked products, each product name, tagline, upvotes, comment count, topics, thumbnail, maker count, and promotion flag. ## API usage Add `&scraper=producthunt-leaderboard` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.producthunt.com/leaderboard/daily/2026/7/14/' \ --data-urlencode 'scraper=producthunt-leaderboard' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.producthunt.com/leaderboard/daily/2026/7/14/', {'scraper': 'producthunt-leaderboard'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.producthunt.com/leaderboard/daily/2026/7/14/', { scraper: 'producthunt-leaderboard' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.producthunt.com/leaderboard/daily/2026/7/14/', scraper: 'producthunt-leaderboard') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.producthunt.com/leaderboard/daily/2026/7/14/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. period string Leaderboard period (for example `daily`, `weekly`). date string Date of the leaderboard. url string Canonical leaderboard URL. productCount integer Number of products returned in `products`. products array Products on the leaderboard, in ranked order. products[].position integer Position of the product in the returned list (1-based). products[].rank integer Rank of the product on the leaderboard. products[].id string Product Hunt identifier of the product. products[].name string Product name. products[].tagline string Product tagline. products[].slug string URL slug of the product. products[].url string Canonical Product Hunt URL of the product. products[].upvotes integer Number of upvotes on the product. products[].commentsCount integer Number of comments on the product. products[].topics array Topics the product is tagged with. products[].thumbnail string | null Thumbnail image URL, when present. products[].makerCount integer Number of makers credited on the product. products[].isPromoted boolean True when the product is a promoted (sponsored) listing. ## Sample response ``` { "period": "daily", "date": "2026-7-14", "url": "https://www.producthunt.com/leaderboard/daily/2026/7/14/", "productCount": 2, "products": [ { "position": 1, "rank": 1, "id": "612345", "name": "Acme Analytics", "tagline": "Product analytics that ships itself", "slug": "acme-analytics", "url": "https://www.producthunt.com/products/acme-analytics", "upvotes": 842, "commentsCount": 74, "topics": ["Analytics", "SaaS", "Developer Tools"], "thumbnail": "https://ph-files.imgix.net/acme-thumb.png", "makerCount": 3, "isPromoted": false } ] } ``` [← PreviousBooking Hotel](/docs/scrapers/booking-hotel)[Next →Product Hunt Product](/docs/scrapers/producthunt-product) --- Source: https://crawlbase.com/docs/scrapers/producthunt-product # Product Hunt Product Parse a Product Hunt product page into structured JSON with the name, tagline, description, upvotes, comment count, reviews count and rating, launch dates, website, topics, and makers. ## API usage Add `&scraper=producthunt-product` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.producthunt.com/products/acme-analytics' \ --data-urlencode 'scraper=producthunt-product' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.producthunt.com/products/acme-analytics', {'scraper': 'producthunt-product'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.producthunt.com/products/acme-analytics', { scraper: 'producthunt-product' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.producthunt.com/products/acme-analytics', scraper: 'producthunt-product') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.producthunt.com/products/acme-analytics ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. id string Product Hunt identifier of the product. name string Product name. tagline string Product tagline. slug string URL slug of the product. url string Canonical Product Hunt URL of the product. description string Full product description. upvotes integer Number of upvotes on the product. commentsCount integer Number of comments on the product. reviewsCount integer Number of reviews on the product. reviewsRating number | null Average review rating, or null when there are no reviews. featuredAt string | null Time the product was featured (ISO 8601). createdAt string Time the product was created (ISO 8601). website string | null External website of the product. thumbnail string | null Thumbnail image URL, when present. topics array Topics the product is tagged with. makers array Names of the makers credited on the product. makerCount integer Number of makers credited on the product. ## Sample response ``` { "id": "612345", "name": "Acme Analytics", "tagline": "Product analytics that ships itself", "slug": "acme-analytics", "url": "https://www.producthunt.com/products/acme-analytics", "description": "Acme Analytics is a self-serve product analytics platform that auto-instruments your app and surfaces the metrics that matter without a data team.", "upvotes": 842, "commentsCount": 74, "reviewsCount": 128, "reviewsRating": 4.8, "featuredAt": "2026-07-14T07:01:00Z", "createdAt": "2026-07-14T07:01:00Z", "website": "https://acme-analytics.com", "thumbnail": "https://ph-files.imgix.net/acme-thumb.png", "topics": ["Analytics", "SaaS", "Developer Tools"], "makers": ["Jane Doe", "Sam Rivera", "Priya Patel"], "makerCount": 3 } ``` [← PreviousProduct Hunt Leaderboard](/docs/scrapers/producthunt-leaderboard)[Next →Stack Exchange Questions](/docs/scrapers/stackexchange-serp) --- Source: https://crawlbase.com/docs/scrapers/quora-question # Quora Question Extract a Quora question page - full question, all answers with author metadata, and related questions. ## API usage Add `&scraper=quora-question` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.quora.com/Which-is-the-best-tool-for-scraping-customer-reviews' \ --data-urlencode 'scraper=quora-question' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.quora.com/Which-is-the-best-tool-for-scraping-customer-reviews', {'scraper': 'quora-question'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.quora.com/Which-is-the-best-tool-for-scraping-customer-reviews', { scraper: 'quora-question' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.quora.com/Which-is-the-best-tool-for-scraping-customer-reviews', scraper: 'quora-question') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.quora.com/Which-is-the-best-tool-for-scraping-customer-reviews ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. question\_title string Question title. question\_details string | null Additional question detail. tags array Topic tags. answer\_count integer Total answers. follower\_count integer Follower count. view\_count integer | null Total views. answers array Answer objects. answers[].author\_name string Answer author. answers[].author\_credentials string | null Author credentials line. answers[].body string Answer text. answers[].upvotes integer Upvote count. related\_questions array Related question URLs and titles. ## Sample response ``` { "question_title": "Which is the best tool for scraping customer reviews?", "answer_count": 14, "follower_count": 42, "answers": [ { "author_name": "John Smith", "author_credentials": "Data Engineer", "upvotes": 218 } ] } ``` [← PreviousQuora SERP](/docs/scrapers/quora-serp)[Next →Airbnb SERP](/docs/scrapers/airbnb-serp) --- Source: https://crawlbase.com/docs/scrapers/quora-serp # Quora SERP Extract Quora search results - array of matching questions with metadata. Currently unavailable The `quora-serp` scraper is currently not available due to changes from Quora. We are working on a fix, but we do not have an ETA at this time. Tip Append `&css_click_selector=.q-text.qu-cursor--pointer` when calling this scraper to maximize data extraction. ## API usage Add `&scraper=quora-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.quora.com/search?q=websitevoice' \ --data-urlencode 'scraper=quora-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.quora.com/search?q=websitevoice', {'scraper': 'quora-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.quora.com/search?q=websitevoice', { scraper: 'quora-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.quora.com/search?q=websitevoice', scraper: 'quora-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.quora.com/search?q=websitevoice ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string Original search query. total\_results integer | null Total result count if visible. questions array Question summary objects. questions[].id string Question slug. questions[].title string Question title. questions[].url string Question URL. questions[].answers\_count integer Answer count. questions[].followers\_count integer Follower count. ## Sample response ``` { "query": "websitevoice", "questions": [ { "id": "What-is-WebsiteVoice", "title": "What is WebsiteVoice?", "url": "https://www.quora.com/What-is-WebsiteVoice", "answers_count": 3, "followers_count": 12 } ] } ``` [← PreviousLinkedIn Feed](/docs/scrapers/linkedin-feed)[Next →Quora Question](/docs/scrapers/quora-question) --- Source: https://crawlbase.com/docs/scrapers/reddit-post # Reddit Post Parse a single Reddit post into structured JSON with the post body, score, upvote ratio, comment count, and the full nested comment tree with each comment author, score, and text. ## API usage Add `&scraper=reddit-post` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.reddit.com/r/programming/comments/1c8a1bb/' \ --data-urlencode 'scraper=reddit-post' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.reddit.com/r/programming/comments/1c8a1bb/', {'scraper': 'reddit-post'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.reddit.com/r/programming/comments/1c8a1bb/', { scraper: 'reddit-post' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.reddit.com/r/programming/comments/1c8a1bb/', scraper: 'reddit-post') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.reddit.com/r/programming/comments/1c8a1bb/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. post object The post itself. post.id string Reddit fullname of the post (for example `t3_...`). post.title string Post title. post.author string Username of the post author. post.subreddit string Subreddit the post belongs to (without the `r/` prefix). post.score integer Post score (upvotes minus downvotes). post.upvoteRatio number | null Fraction of votes that are upvotes (0 to 1). post.commentsCount integer Total number of comments on the post. post.createdAt string Post creation time (ISO 8601). post.url string | null Outbound link the post points to (null for text posts). post.domain string | null Domain of the outbound link. post.permalink string Permalink to the post on Reddit. post.flair string | null Post flair text, when set. post.isNsfw boolean True when the post is marked NSFW. post.selfText string Body text of a self (text) post; empty for link posts. commentCount integer Number of top-level comments returned in `comments`. comments array Top-level comments, each with a nested `replies` tree. comments[].id string Reddit fullname of the comment (for example `t1_...`). comments[].author string Username of the comment author. comments[].score integer Comment score (upvotes minus downvotes). comments[].createdAt string Comment creation time (ISO 8601). comments[].body string Comment text. comments[].permalink string Permalink to the comment on Reddit. comments[].isSubmitter boolean True when the comment author is the post author (OP). comments[].replies array Nested replies, each the same shape as a comment. ## Sample response ``` { "post": { "id": "t3_1c8a1bb", "title": "The hidden cost of deep dependency trees", "author": "buildmaster", "subreddit": "programming", "score": 2417, "upvoteRatio": 0.96, "commentsCount": 312, "createdAt": "2026-07-14T09:12:44+00:00", "url": "https://example.com/blog/dependency-trees", "domain": "example.com", "permalink": "https://old.reddit.com/r/programming/comments/1c8a1bb/the_hidden_cost_of_deep_dependency_trees/", "flair": null, "isNsfw": false, "selfText": "" }, "commentCount": 3, "comments": [ { "id": "t1_l0aa111", "author": "perftuner", "score": 184, "createdAt": "2026-07-14T09:31:07+00:00", "body": "This matches what we saw. Flattening the tree cut our cold install time in half.", "permalink": "https://old.reddit.com/r/programming/comments/1c8a1bb/the_hidden_cost_of_deep_dependency_trees/l0aa111/", "isSubmitter": false, "replies": [ { "id": "t1_l0aa222", "author": "buildmaster", "score": 71, "createdAt": "2026-07-14T09:44:52+00:00", "body": "Yep. The transitive fan-out is where most of the time goes.", "permalink": "https://old.reddit.com/r/programming/comments/1c8a1bb/the_hidden_cost_of_deep_dependency_trees/l0aa222/", "isSubmitter": true, "replies": [] } ] } ] } ``` [← PreviousReddit Search](/docs/scrapers/reddit-serp)[Next →Booking SERP](/docs/scrapers/booking-serp) --- Source: https://crawlbase.com/docs/scrapers/reddit-serp # Reddit Search Parse a Reddit search results page into structured JSON with the matching posts, each title, author, subreddit, score, comment count, timestamp, link domain, related subreddits, and pagination. ## API usage Add `&scraper=reddit-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.reddit.com/search/?q=web+scraping' \ --data-urlencode 'scraper=reddit-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.reddit.com/search/?q=web+scraping', {'scraper': 'reddit-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.reddit.com/search/?q=web+scraping', { scraper: 'reddit-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.reddit.com/search/?q=web+scraping', scraper: 'reddit-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.reddit.com/search/?q=web+scraping ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string Search query that produced these results. sort string Sort order applied to the search (for example `relevance`, `new`, `top`). time string Time window applied to the search (for example `all`, `day`, `week`). restrictSubreddit boolean True when the search was scoped to a single subreddit. resultCount integer Number of results returned in `results`. pagination object Pagination cursor for the next and previous pages. pagination.next\_page\_url string | null URL for the next page of results, or null on the last page. pagination.previous\_page\_url string | null URL for the previous page of results, or null on the first page. pagination.has\_next boolean True when a next page is available. results array Search results, in ranked order. results[].position integer Position of the result in the returned list (1-based). results[].id string Reddit fullname of the post (for example `t3_...`). results[].title string Post title. results[].url string URL of the result on Reddit. results[].author string Username of the post author. results[].subreddit string Subreddit the post belongs to (without the `r/` prefix). results[].score integer Post score (upvotes minus downvotes). results[].commentsCount integer Number of comments on the post. results[].createdAt string Post creation time (ISO 8601). results[].linkUrl string Outbound link the post points to. results[].domain string Domain of the outbound link. subreddits array Related subreddits surfaced alongside the results. subreddits[].name string Subreddit name (without the `r/` prefix). subreddits[].url string Canonical subreddit URL. subreddits[].subscribers integer | null Subscriber count of the subreddit. subreddits[].description string | null Short subreddit description. ## Sample response ``` { "query": "web scraping", "sort": "relevance", "time": "all", "restrictSubreddit": false, "resultCount": 2, "pagination": { "next_page_url": "https://old.reddit.com/search?q=web+scraping&count=25&after=t3_1c7aaaa", "previous_page_url": null, "has_next": true }, "results": [ { "position": 1, "id": "t3_1c7aaaa", "title": "What is the most reliable way to do web scraping at scale in 2026?", "url": "https://old.reddit.com/r/webdev/comments/1c7aaaa/what_is_the_most_reliable_way_to_do_web_scraping/", "author": "datawrangler", "subreddit": "webdev", "score": 486, "commentsCount": 133, "createdAt": "2026-06-30T14:22:10+00:00", "linkUrl": "https://old.reddit.com/r/webdev/comments/1c7aaaa/what_is_the_most_reliable_way_to_do_web_scraping/", "domain": "old.reddit.com" } ], "subreddits": [ { "name": "webscraping", "url": "https://old.reddit.com/r/webscraping/", "subscribers": 84210, "description": "A community for web scraping, crawling and data extraction." } ] } ``` [← PreviousReddit Subreddit](/docs/scrapers/reddit-subreddit)[Next →Reddit Post](/docs/scrapers/reddit-post) --- Source: https://crawlbase.com/docs/scrapers/reddit-subreddit # Reddit Subreddit Parse a subreddit listing page into structured JSON with the ranked posts, each post title, author, score, comment count, timestamp, permalink, link domain, flair, and pagination. ## API usage Add `&scraper=reddit-subreddit` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.reddit.com/r/programming/' \ --data-urlencode 'scraper=reddit-subreddit' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.reddit.com/r/programming/', {'scraper': 'reddit-subreddit'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.reddit.com/r/programming/', { scraper: 'reddit-subreddit' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.reddit.com/r/programming/', scraper: 'reddit-subreddit') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.reddit.com/r/programming/ ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. subreddit string Subreddit name (without the `r/` prefix). url string Canonical subreddit URL. sort string Sort order of the listing (for example `hot`, `new`, `top`). postCount integer Number of posts returned in `posts`. pagination object Pagination cursor for the next and previous pages. pagination.next\_page\_url string | null URL for the next page of posts, or null on the last page. pagination.previous\_page\_url string | null URL for the previous page of posts, or null on the first page. pagination.has\_next boolean True when a next page is available. posts array Posts on the listing, in display order. posts[].position integer Position of the post in the returned list (1-based). posts[].id string Reddit fullname of the post (for example `t3_...`). posts[].rank integer Rank of the post on the subreddit listing. posts[].title string Post title. posts[].author string Username of the post author. posts[].score integer Post score (upvotes minus downvotes). posts[].commentsCount integer Number of comments on the post. posts[].createdAt string Post creation time (ISO 8601). posts[].permalink string Permalink to the post on Reddit. posts[].url string | null Outbound link the post points to (null for text posts). posts[].domain string | null Domain of the outbound link. posts[].flair string | null Post flair text, when set. posts[].isNsfw boolean True when the post is marked NSFW. posts[].isStickied boolean True when the post is pinned to the top of the subreddit. posts[].thumbnail string | null Thumbnail image URL, when present. ## Sample response ``` { "subreddit": "programming", "url": "https://old.reddit.com/r/programming/", "sort": "hot", "postCount": 3, "pagination": { "next_page_url": "https://old.reddit.com/r/programming/?count=25&after=t3_1c8x9aa", "previous_page_url": null, "has_next": true }, "posts": [ { "position": 1, "id": "t3_1c8a1bb", "rank": 1, "title": "The hidden cost of deep dependency trees", "author": "buildmaster", "score": 2417, "commentsCount": 312, "createdAt": "2026-07-14T09:12:44+00:00", "permalink": "https://old.reddit.com/r/programming/comments/1c8a1bb/the_hidden_cost_of_deep_dependency_trees/", "url": "https://example.com/blog/dependency-trees", "domain": "example.com", "flair": null, "isNsfw": false, "isStickied": false, "thumbnail": "https://b.thumbs.redditmedia.com/abc.jpg" } ] } ``` [← PreviousGitHub Profile](/docs/scrapers/github-profile)[Next →Reddit Search](/docs/scrapers/reddit-serp) --- Source: https://crawlbase.com/docs/scrapers/reviews-qa # Reviews & Q&A Seven scrapers for review platforms and question-and-answer communities. Pull structured reviews and answer threads without fighting markup changes. ## Overview Reviews and Q&A pages carry some of the highest-signal user-generated content on the web, verified product feedback, lived-experience answers, and the language real customers use to describe problems. These scrapers turn that content into JSON you can pipe into LLMs, sentiment models, or product-research dashboards without scraping each domain by hand. Common use cases: - **Product research** : surface real pros/cons from `g2-product-reviews` for a category you're entering or a competitor you're sizing up. - **VoC analytics** (voice of customer): pipe verified-buyer reviews into a sentiment model and segment by reviewer-role / company-size to find which customer profiles love or hate a feature. - **Content discovery** : walk `quora-question` threads to find the exact phrasing buyers use when describing the problem your product solves, input for landing-page copy and SEO. - **AI training** : build a retrieval index of high-quality Q&A pairs for an answer engine or domain-specific copilot. The G2 and Quora scrapers handle pagination internally, a single G2 product call returns the rating distribution and the visible review batch; pass `page` for older pages. Quora threads return the question + all loaded answers in one shot. The Product Hunt scrapers return a leaderboard's ranked launches or a single product's upvotes, makers, topics, and reviews in one call. ## G2 G2 software-product reviews. Returns the overall rating, total review count, rating distribution, and the parsed reviews, including reviewer role and company size where G2 surfaces them. - [G2 Product Reviews](/docs/scrapers/g2-product-reviews) - software-product reviews on G2, rating distribution, individual reviews, pros/cons. ## Quora Quora question pages, the question, all answers, author info, and engagement counts. - [Quora Question](/docs/scrapers/quora-question) - a Quora question page with all answers. - [Quora SERP](/docs/scrapers/quora-serp) - Quora search results. ## Product Hunt Product Hunt launch pages, daily and weekly leaderboards plus individual product pages with upvotes, makers, topics, and reviews. - [Product Hunt Leaderboard](/docs/scrapers/producthunt-leaderboard) - daily and weekly Product Hunt leaderboards with ranks, upvotes, and makers. - [Product Hunt Product](/docs/scrapers/producthunt-product) - a single Product Hunt product page with upvotes, makers, topics, and reviews. ## Stack Exchange Stack Exchange network Q&A, question lists and full question threads across Stack Overflow, Super User, Ask Ubuntu, Server Fault, MathOverflow, and every \*.stackexchange.com site. One parser per page type, host-agnostic across the network. - [Stack Exchange Questions](/docs/scrapers/stackexchange-serp) - a questions, tagged, or search-results page as a structured array with scores, answer and view counts, tags, and pagination. - [Stack Exchange Thread](/docs/scrapers/stackexchange-thread) - a single question with its full body plus every answer and comment, with scores, accepted state, and authors. ## Example call Below: a single `g2-product-reviews` call. The response carries the product-level aggregates plus a parsed review batch, title, rating, reviewer role, and company size. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.g2.com/products/zoom/reviews' \ --data-urlencode 'scraper=g2-product-reviews' -G ``` ### Sample response ``` { "product_name": "Zoom", "overall_rating": 4.5, "total_reviews": 52840, "reviews": [ { "title": "Reliable for daily standups", "rating": 4.5, "reviewer_role": "Engineering Manager", "reviewer_company_size": "51-200 employees" } ] } ``` Full reference (parameters, all 4 SDK languages, edge cases): [G2 Product Reviews, full reference](/docs/scrapers/g2-product-reviews) [← PreviousSocial Media](/docs/scrapers/social-media)[Next →Travel, Events & Real Estate](/docs/scrapers/travel-events) --- Source: https://crawlbase.com/docs/scrapers/search-engines # Search Engines Two scrapers that turn Google and Bing search-results pages into clean structured JSON, including organic listings, ads, and side-panels. Useful for SEO monitoring, rank tracking, and competitive research. ## Overview Search-engine scrapers turn a Google or Bing results page into structured JSON: organic listings, ads, knowledge panel, related searches, and "people also ask" - all surfaces, one parser, one bill. The browser dance (consent walls, JS rendering, anti-bot, geo-routing) happens server-side; you receive the parsed shape directly. Common use cases: - **Rank tracking** : monitor a list of keywords daily, store the position of your domain (and your competitors') in `organic[].url`, alert on movement. - **SERP-feature monitoring** : detect when Google promotes a People-Also-Ask box, knowledge panel, or AI overview that pushes organic results below the fold. - **SEO research** : harvest `related_searches` and `people_also_ask` for content-gap analysis without subscribing to a third-party SaaS. - **AI-training datasets** : build a corpus of "what Google shows for query X" snapshots for retrieval-augmented generation and answer-engine evaluations. Geo and language are explicit: pass `country` + `language` on the request and the scraper hits the right Google domain (e.g. `google.de` for `country=DE`) so the SERP you see matches what a user in that market sees. Combine with the [Smart Proxy](/docs/smart-proxy) when you need sticky-session pagination across deep result pages. ## Google Full Google SERP - organic, ads, snippets, related searches, and people-also-ask. The most-used scraper in the catalog; pairs naturally with the e-commerce `*-product-details` scrapers when you need a discovery → enrichment pipeline. - [Google SERP](/docs/scrapers/google-serp) - Google search-results page - organic, ads, knowledge panel, related searches. ## Bing Bing search-results page. Useful when you need a second data source for SEO research, or when your audience skews enterprise/Windows where Bing share is meaningfully higher than its overall market share suggests. - [Bing SERP](/docs/scrapers/bing-serp) - Bing search-results page. ## Example call Below: a single `google-serp` call. Pass `country` and `language` alongside the URL when you need the regional storefront - Google routes its SERP off both your IP location and your `hl`/`gl` URL params, so we set them explicitly. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.google.com/search?q=samsung+social+accounts' \ --data-urlencode 'scraper=google-serp' \ --data-urlencode 'country=US' -G ``` ### Sample response ``` { "query": "samsung social accounts", "results": [ { "title": "Samsung Mobile (@SamsungMobile) / X", "url": "https://x.com/SamsungMobile", "snippet": "The official Samsung Mobile X account…" } ], "related_searches": ["samsung instagram", "samsung official tiktok"], "people_also_ask": [ { "question": "Does Samsung have a TikTok?", "answer": "Yes, Samsung is on TikTok at @samsung." } ] } ``` Full reference (parameters, all 4 SDK languages, edge cases): [Google SERP - full reference](/docs/scrapers/google-serp) [← PreviousE-Commerce](/docs/scrapers/ecommerce)[Next →Social Media](/docs/scrapers/social-media) --- Source: https://crawlbase.com/docs/scrapers/social-media # Social Media Eighteen scrapers for the biggest social platforms - public profiles, posts, hashtags, events, feeds, and comment threads. We handle the anti-bot, you get clean JSON. ## Overview Social-media scrapers cover the public surfaces of the five platforms most teams pull from: Facebook, Instagram, TikTok, LinkedIn, and Reddit. Each scraper targets one page-type - profile, post, hashtag, event, feed, subreddit listing, comment thread - so the JSON shape stays predictable even when the underlying app refactors its UI. We handle the anti-bot, the rate limiting, and the residential routing; you receive the parsed structure. Common use cases: - **Brand monitoring** : poll `instagram-profile` / `tiktok-profile` for follower count, posting cadence, and engagement trends across competitors and influencers. - **Influencer discovery** : walk `instagram-hashtag` or `instagram-reels-audio` to find creators using a given hashtag or trending sound. - **Recruiting / sales prospecting** : enrich a CRM with `linkedin-profile` and `linkedin-company` data - title, headcount, headline, current role. - **Event marketing** : pull `facebook-event` attendance and timing for tracking event saturation in a city or industry vertical. - **Trend research** : feed `tiktok-product` and `instagram-post` into trend-detection pipelines for marketing or product research. - **Community & sentiment research** : pull `reddit-subreddit` and `reddit-post` for ranked posts and full comment trees to track sentiment, product feedback, and emerging discussion. Only public surfaces are supported - login-walled or private content is out of scope and not something the scrapers attempt. Where a platform has multiple post types (Instagram has profile + post + reel + hashtag, for example), each gets its own scraper so you don't pay for fields you don't need. ## Facebook Public Facebook surfaces - pages, profiles, groups, hashtags, events. Useful for brand monitoring, event marketing, and discovery in regions where Facebook has higher share than Instagram (most of EMEA, parts of LATAM). - [Facebook Page](/docs/scrapers/facebook-page) - public Facebook page (name, category, description, posts). - [Facebook Profile](/docs/scrapers/facebook-profile) - public Facebook profile. - [Facebook Group](/docs/scrapers/facebook-group) - public Facebook group. - [Facebook Hashtag](/docs/scrapers/facebook-hashtag) - posts under a Facebook hashtag. - [Facebook Event](/docs/scrapers/facebook-event) - public Facebook event. ## Instagram Five Instagram scrapers covering the surfaces most relevant to influencer marketing and trend research - profiles, individual posts and reels, hashtag feeds, and reels-audio for music/sound-driven trend tracking. - [Instagram Profile](/docs/scrapers/instagram-profile) - public Instagram profile. - [Instagram Post](/docs/scrapers/instagram-post) - single Instagram post. - [Instagram Reel](/docs/scrapers/instagram-reel) - single Instagram Reel. - [Instagram Hashtag](/docs/scrapers/instagram-hashtag) - posts under an Instagram hashtag. - [Instagram Reels Audio](/docs/scrapers/instagram-reels-audio) - reels using a given audio track. ## TikTok TikTok profiles and individual video/product posts. Pair with `tiktok-shop` in the E-Commerce category if you're tracking creator commerce. - [TikTok Profile](/docs/scrapers/tiktok-profile) - public TikTok profile. - [TikTok Product](/docs/scrapers/tiktok-product) - single TikTok video / product post. ## LinkedIn Public LinkedIn - profiles, company pages, and feed posts. The standard fit for B2B prospecting, recruiting, and competitive headcount tracking. - [LinkedIn Profile](/docs/scrapers/linkedin-profile) - public LinkedIn profile. - [LinkedIn Company](/docs/scrapers/linkedin-company) - public LinkedIn company page. - [LinkedIn Feed](/docs/scrapers/linkedin-feed) - LinkedIn feed posts. ## Reddit Public Reddit surfaces - subreddit listings, search results, and single posts with their full comment trees. The fit for community monitoring, sentiment analysis, and trend detection across topic-specific communities. - [Reddit Subreddit](/docs/scrapers/reddit-subreddit) - ranked posts from a subreddit listing (title, author, score, comment count, flair, pagination). - [Reddit Search](/docs/scrapers/reddit-serp) - Reddit search results - matching posts plus related subreddits. - [Reddit Post](/docs/scrapers/reddit-post) - single post with body, score, upvote ratio, and the full nested comment tree. ## Example call Below: a single `instagram-profile` call. The scraper returns the public profile snapshot - username, bio, follower/following counts, post count, verification flag, and avatar URL. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.instagram.com/apple/' \ --data-urlencode 'scraper=instagram-profile' -G ``` ### Sample response ``` { "username": "apple", "full_name": "Apple", "biography": "Welcome to @apple. The latest creativity going on around us.", "followers_count": 33800000, "following_count": 9, "posts_count": 1284, "is_verified": true, "is_private": false, "profile_pic_url": "https://scontent.cdninstagram.com/...jpg" } ``` Full reference (parameters, all 4 SDK languages, edge cases): [Instagram Profile - full reference](/docs/scrapers/instagram-profile) [← PreviousSearch Engines](/docs/scrapers/search-engines)[Next →Reviews & Q&A](/docs/scrapers/reviews-qa) --- Source: https://crawlbase.com/docs/scrapers/stackexchange-serp # Stack Exchange Questions Parse a Stack Exchange questions, tagged, or search results page into structured JSON with each question title, score, answer and view counts, tags, excerpt, author, and pagination. ## API usage Add `&scraper=stackexchange-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://stackoverflow.com/questions/tagged/web-scraping' \ --data-urlencode 'scraper=stackexchange-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://stackoverflow.com/questions/tagged/web-scraping', {'scraper': 'stackexchange-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://stackoverflow.com/questions/tagged/web-scraping', { scraper: 'stackexchange-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://stackoverflow.com/questions/tagged/web-scraping', scraper: 'stackexchange-serp') data = JSON.parse(res.body) ``` ## Example input URL Any Stack Exchange listing works in the `url` parameter: a tag, search, or questions page on any site in the network. For example: ``` https://stackoverflow.com/questions/tagged/web-scraping https://superuser.com/search?q=ssh+tunnel https://askubuntu.com/questions?tab=Votes https://unix.stackexchange.com/questions/tagged/bash https://mathoverflow.net/questions ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. site string Stack Exchange site the results came from (for example `stackoverflow`). query string Search query or tag that produced these results. tag string | null Tag the listing was scoped to, or null when not a tag page. sort string Sort order applied to the listing (for example `newest`, `votes`, `active`). currentPage integer Page number of the listing currently returned (1-based). totalQuestions integer | null Total number of questions matching the listing, or null when the page omits the count. results array Questions on this page, in listing order. results[].position integer Position of the question in the returned list (1-based). results[].id string Stack Exchange question id. results[].title string Question title. results[].questionUrl string Canonical URL of the question. results[].score integer Question score (upvotes minus downvotes). results[].answerCount integer Number of answers on the question. results[].viewCount integer Number of views on the question. results[].hasAcceptedAnswer boolean True when the question has an accepted answer. results[].excerpt string Short excerpt of the question body. results[].tags array Tags applied to the question. results[].author string Display name of the question author. results[].authorUrl string Profile URL of the question author. results[].askedAt string Question creation time (ISO 8601). resultCount integer Number of results returned in `results`. pagination object Pagination cursor for the next page. pagination.currentPage integer Page number this response covers (1-based). pagination.nextPageUrl string | null URL for the next page of results, or null on the last page. pagination.hasNext boolean True when a next page is available. ## Sample response ``` { "site": "stackoverflow.com", "query": "web scraping infinite scroll pagination", "tag": null, "sort": "Relevance", "currentPage": 1, "totalQuestions": null, "results": [ { "position": 1, "id": "78912345", "title": "How to handle pagination when scraping an infinite scroll page?", "questionUrl": "https://stackoverflow.com/questions/78912345/how-to-handle-pagination-when-scraping-an-infinite-scroll-page", "score": 4, "answerCount": 2, "viewCount": 137, "hasAcceptedAnswer": true, "excerpt": "I'm trying to scrape a product listing that loads more items as you scroll. The network tab shows a POST request returning JSON, but the cursor token keeps changing...", "tags": ["python", "web-scraping", "pagination", "requests"], "author": "dev_ana", "authorUrl": "https://stackoverflow.com/users/1234567/dev-ana", "askedAt": "2026-07-15T09:42:11Z" }, { "position": 2, "id": "78911002", "title": "Rotating proxies with residential IPs to avoid rate limiting", "questionUrl": "https://stackoverflow.com/questions/78911002/rotating-proxies-with-residential-ips-to-avoid-rate-limiting", "score": 1, "answerCount": 0, "viewCount": 42, "hasAcceptedAnswer": false, "excerpt": "My crawler gets blocked after roughly 500 requests from a datacenter IP. I want to rotate through a residential proxy pool but I'm unsure how to detect a soft ban...", "tags": ["web-scraping", "proxy", "http"], "author": "crawler_joe", "authorUrl": "https://stackoverflow.com/users/9988776/crawler-joe", "askedAt": "2026-07-15T08:05:47Z" } ], "resultCount": 2, "pagination": { "currentPage": 1, "nextPageUrl": "https://stackoverflow.com/search?q=web+scraping+infinite+scroll+pagination&tab=Relevance&page=2", "hasNext": true } } ``` [← PreviousProduct Hunt Product](/docs/scrapers/producthunt-product)[Next →Stack Exchange Thread](/docs/scrapers/stackexchange-thread) --- Source: https://crawlbase.com/docs/scrapers/stackexchange-thread # Stack Exchange Thread Parse a single Stack Exchange question page into structured JSON with the question body, every answer and comment, score, accepted state, author, reputation, and timestamps. ## API usage Add `&scraper=stackexchange-thread` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file' \ --data-urlencode 'scraper=stackexchange-thread' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file', {'scraper': 'stackexchange-thread'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file', { scraper: 'stackexchange-thread' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file', scraper: 'stackexchange-thread') data = JSON.parse(res.body) ``` ## Example input URL Any Stack Exchange question URL works in the `url` parameter, from any site in the network. For example: ``` https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file https://superuser.com/questions/441895/automate-opening-html-and-printing-to-pdf https://askubuntu.com/questions/1206658/why-is-apt-held-back https://serverfault.com/questions/439471/which-openvpn-cipher-should-i-use https://mathoverflow.net/questions/44265/is-the-riemann-hypothesis ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. question object The question itself. question.id string Stack Exchange question id. question.title string Question title. question.url string Canonical URL of the question. question.body string Question body text. question.score integer Question score (upvotes minus downvotes). question.viewCount integer Number of views on the question. question.tags array Tags applied to the question. question.askedAt string Question creation time (ISO 8601). question.author object Author of the question. question.author.name string Display name of the question author. question.author.url string Profile URL of the question author. question.author.reputation integer Reputation score of the question author. question.comments array Comments on the question. question.comments[].id string Comment id. question.comments[].score integer Comment score. question.comments[].body string Comment text. question.comments[].author string Display name of the comment author. question.comments[].authorUrl string Profile URL of the comment author. question.comments[].createdAt string Comment creation time (ISO 8601). answerCount integer Number of answers returned in `answers`. answers array Answers to the question, in listing order. answers[].id string Answer id. answers[].score integer Answer score (upvotes minus downvotes). answers[].isAccepted boolean True when this is the accepted answer. answers[].body string Answer body text. answers[].author object Author of the answer. answers[].author.name string Display name of the answer author. answers[].author.url string Profile URL of the answer author. answers[].author.reputation integer Reputation score of the answer author. answers[].createdAt string Answer creation time (ISO 8601). answers[].comments array Comments on the answer. answers[].comments[].id string Comment id. answers[].comments[].score integer Comment score. answers[].comments[].body string Comment text. answers[].comments[].author string Display name of the comment author. answers[].comments[].authorUrl string Profile URL of the comment author. answers[].comments[].createdAt string Comment creation time (ISO 8601). ## Sample response ``` { "question": { "id": "2861071", "title": "How to modify a text file?", "url": "https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file", "body": "I'm using Python and I need to insert a line at the start of a text file without loading the whole file into memory. Is there a way to do this in place, or do I have to rewrite the file?", "score": 312, "viewCount": 486201, "tags": ["python", "file", "text-files"], "askedAt": "2010-05-18T20:11:33Z", "author": { "name": "Nathan Fellman", "url": "https://stackoverflow.com/users/8460/nathan-fellman", "reputation": 127843 }, "comments": [ { "id": "2954120", "score": 3, "body": "Do you need to preserve the original file, or is rewriting acceptable?", "author": "Greg Hewgill", "authorUrl": "https://stackoverflow.com/users/893/greg-hewgill", "createdAt": "2010-05-18T20:19:04Z" } ] }, "answerCount": 2, "answers": [ { "id": "2861108", "score": 401, "isAccepted": true, "body": "You cannot insert into the middle of a file without rewriting it. Read the file into a list of lines, insert your new line, then write it all back:\n\n with open('file.txt') as f:\n lines = f.readlines()\n lines.insert(0, 'new first line\\n')\n with open('file.txt', 'w') as f:\n f.writelines(lines)\n", "author": { "name": "Roberto Bonvallet", "url": "https://stackoverflow.com/users/193568/roberto-bonvallet", "reputation": 30215 }, "createdAt": "2010-05-18T20:15:52Z", "comments": [ { "id": "2954260", "score": 12, "body": "For very large files, stream through a temporary file instead of holding everything in memory.", "author": "John Machin", "authorUrl": "https://stackoverflow.com/users/253537/john-machin", "createdAt": "2010-05-18T21:02:11Z" } ] }, { "id": "2861180", "score": 47, "isAccepted": false, "body": "If the file is large, use fileinput with inplace=True to edit it line by line without loading it all at once.", "author": { "name": "codeape", "url": "https://stackoverflow.com/users/18770/codeape", "reputation": 98412 }, "createdAt": "2010-05-18T20:24:39Z", "comments": [] } ] } ``` [← PreviousStack Exchange Questions](/docs/scrapers/stackexchange-serp)[Next →Exercism Exercises](/docs/scrapers/exercism-serp) --- Source: https://crawlbase.com/docs/scrapers/tiktok-product # TikTok Product Extract a TikTok Shop product page - title, price, variants, seller info, reviews, and related videos. Use the JS token TikTok scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=tiktok-product` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.tiktok.com/view/product/1729493620818874839' \ --data-urlencode 'scraper=tiktok-product' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.tiktok.com/view/product/1729493620818874839', {'scraper': 'tiktok-product'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.tiktok.com/view/product/1729493620818874839', { scraper: 'tiktok-product' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.tiktok.com/view/product/1729493620818874839', scraper: 'tiktok-product') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.tiktok.com/view/product/1729493620818874839 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. product\_id string Product identifier from the URL. title string Product title. price string Current selling price. original\_price string | null Pre-discount price when on sale. discount\_percentage string | null Discount as percentage string e.g. "20%". rating number Average product rating. reviews\_count integer Total number of reviews. sold\_count string Total units sold (formatted as "10K+", "1.2M+"). description string Product description. images array Array of product image URLs. variants array Array of variant objects (color, size, etc). seller object Seller metadata: name, rating, follower\_count. related\_videos array TikTok videos featuring this product. ## Sample response ``` { "product_id": "1729493620818874839", "title": "Wireless Bluetooth Earbuds Pro", "price": "$24.99", "original_price": "$49.99", "discount_percentage": "50%", "rating": 4.7, "reviews_count": 12453, "sold_count": "50K+", "seller": { "name": "AudioPro Store", "rating": 4.8, "follower_count": 28000 } } ``` [← PreviousInstagram Reels Audio](/docs/scrapers/instagram-reels-audio)[Next →TikTok Shop](/docs/scrapers/tiktok-shop) --- Source: https://crawlbase.com/docs/scrapers/tiktok-profile # TikTok Profile Extract a TikTok creator profile - bio, follower counts, total likes, and recent video previews. Use the JS token TikTok scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=tiktok-profile` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.tiktok.com/@loveyourboka' \ --data-urlencode 'scraper=tiktok-profile' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.tiktok.com/@loveyourboka', {'scraper': 'tiktok-profile'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.tiktok.com/@loveyourboka', { scraper: 'tiktok-profile' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.tiktok.com/@loveyourboka', scraper: 'tiktok-profile') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.tiktok.com/@loveyourboka ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. username string Profile @username. display\_name string Display name. bio string | null Profile bio. avatar\_url string Profile picture URL. verified boolean Verification badge present. follower\_count integer Total followers. following\_count integer Number of accounts followed. likes\_count integer Total likes received. videos\_count integer Total videos posted. recent\_videos array Recent video objects. ## Sample response ``` { "username": "loveyourboka", "display_name": "Love Your Boka", "verified": false, "follower_count": 82400, "following_count": 213, "likes_count": 1820000, "videos_count": 412 } ``` [← PreviousTikTok Shop](/docs/scrapers/tiktok-shop)[Next →LinkedIn Profile](/docs/scrapers/linkedin-profile) --- Source: https://crawlbase.com/docs/scrapers/tiktok-shop # TikTok Shop Extract a TikTok Shop storefront - store metadata, products, ratings, and follower counts. Use the JS token TikTok scrapers work best with your **JavaScript token**. ## API usage Add `&scraper=tiktok-shop` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.tiktok.com/shop/store/medcursor/7495198098029185725' \ --data-urlencode 'scraper=tiktok-shop' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.tiktok.com/shop/store/medcursor/7495198098029185725', {'scraper': 'tiktok-shop'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.tiktok.com/shop/store/medcursor/7495198098029185725', { scraper: 'tiktok-shop' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.tiktok.com/shop/store/medcursor/7495198098029185725', scraper: 'tiktok-shop') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.tiktok.com/shop/store/medcursor/7495198098029185725 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. shop\_id string Shop identifier. shop\_name string Display name of the shop. logo\_url string Shop logo URL. rating number Average shop rating. reviews\_count integer Total reviews across the shop. follower\_count integer Number of shop followers. products\_count integer Total products listed. products array Sample products on the landing page. ## Sample response ``` { "shop_id": "7495198098029185725", "shop_name": "Medcursor", "rating": 4.6, "reviews_count": 8420, "follower_count": 125000, "products_count": 342 } ``` [← PreviousTikTok Product](/docs/scrapers/tiktok-product)[Next →TikTok Profile](/docs/scrapers/tiktok-profile) --- Source: https://crawlbase.com/docs/scrapers/travel-events # Travel, Events & Real Estate Six scrapers for travel listings, event marketplaces, and real-estate platforms. ## Overview Travel, events, and real-estate platforms publish constantly-changing inventory at high frequency - prices, availability, calendar - that's only useful as a structured time series. These scrapers turn the listing pages into JSON so you can run dynamic-pricing analyses, market saturation studies, or property-comp queries against fresh data. Common use cases: - **Hotel rate tracking** : poll `booking-serp` for a destination and snapshot each property's price and review score, then drill into `booking-hotel` for room-level detail - competitive rate intelligence for revenue teams. - **Short-term rental pricing** : poll `airbnb-serp` for a destination across check-in dates, snapshot `price_per_night`, build a yield curve for hosts or revenue managers. - **Event-density analytics** : walk `eventbrite-events-list` by city + category to track event-marketing saturation week-over-week. - **Conference / tradeshow research** : enrich `eventbrite-event-details` with venue, organizer, and ticket-tier info for a sales prospecting workflow. - **Real-estate comps** : pull `immobilienscout24-property` snapshots for German residential listings and feed them into valuation models or buyer-search alerts. Inventory data ages fast - design pipelines to refresh frequently (hourly or daily) for the price-sensitive use cases, less often for static metadata. Geo-routing matters more in this category than most: pass `country=DE` for Immobilienscout24 to ensure you hit the German storefront from a German residential IP. ## Booking.com Booking.com travel and hospitality pages - search-results listings and single hotel pages. Returns pricing, review scores, star ratings, and facilities, the fit for dynamic-pricing analysis, competitive rate tracking, and hotel-metadata enrichment. - [Booking SERP](/docs/scrapers/booking-serp) - a Booking.com search-results page - matched properties with name, price, address, distance, review score, star rating, and image. - [Booking Hotel](/docs/scrapers/booking-hotel) - a single Booking.com hotel page - name, description, address, coordinates, star rating, review score, and facilities. ## Airbnb Airbnb search-results pages - the destination listing feed. Use the search URL with your filters (dates, guests, amenities) and the scraper returns the resulting listings with price, rating, capacity, and ID. For per-listing detail, point at the listing page itself; we'll follow up with a dedicated scraper if there's enough demand. - [Airbnb SERP](/docs/scrapers/airbnb-serp) - Airbnb search-results page. ## Eventbrite Two Eventbrite scrapers - search/listing pages and individual event details. Pair them in a discovery → detail pipeline: walk a city/category listing, then enrich each event ID with the full details. - [Eventbrite Events List](/docs/scrapers/eventbrite-events-list) - Eventbrite events listing / search page. - [Eventbrite Event Details](/docs/scrapers/eventbrite-event-details) - full Eventbrite event page (date, venue, organizer, tickets). ## Immobilienscout24 German real-estate listings on Immobilienscout24. Returns price, location, area (sqm), and agent contact. The largest residential property platform in DACH - a strong fit for property-tech and proptech analytics. - [ImmobilienScout24 Property](/docs/scrapers/immobilienscout24-property) - real-estate listing on Immobilienscout24 (price, location, area, agent). ## Example call Below: a single `airbnb-serp` call for a destination. Build the search URL with whatever filter combination you need (check-in/out dates, guest count, amenities) - Airbnb encodes them in the URL and the scraper preserves them in the response shape. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.airbnb.com/s/Beirut/homes' \ --data-urlencode 'scraper=airbnb-serp' -G ``` ### Sample response ``` { "search_location": "Beirut", "listings": [ { "id": "54281209", "title": "Sunny apartment in Hamra", "location": "Beirut, Lebanon", "price_per_night": "$45", "rating": 4.92, "reviews_count": 142, "guests": 2, "bedrooms": 1 } ] } ``` Full reference (parameters, all 4 SDK languages, edge cases): [Airbnb SERP - full reference](/docs/scrapers/airbnb-serp) [← PreviousReviews & Q&A](/docs/scrapers/reviews-qa)[Next →Developer](/docs/scrapers/developer) --- Source: https://crawlbase.com/docs/scrapers/walmart-category # Walmart Category Extract a Walmart category browse page - title, breadcrumbs, filters, and array of products. ## API usage Add `&scraper=walmart-category` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.walmart.com/browse/home/dorm-decor/4044_1225301_1225229_7471338' \ --data-urlencode 'scraper=walmart-category' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.walmart.com/browse/home/dorm-decor/4044_1225301_1225229_7471338', {'scraper': 'walmart-category'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.walmart.com/browse/home/dorm-decor/4044_1225301_1225229_7471338', { scraper: 'walmart-category' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.walmart.com/browse/home/dorm-decor/4044_1225301_1225229_7471338', scraper: 'walmart-category') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.walmart.com/browse/home/dorm-decor/4044_1225301_1225229_7471338 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. category\_path array Breadcrumb from root to current. category\_title string Category title. total\_products integer | null Total products. filters array Filter options. products array Product summaries (same shape as walmart-serp). ## Sample response ``` { "category_path": ["Home", "Dorm Decor"], "category_title": "Dorm Decor", "total_products": 1240, "products": [ { "product_id": "3551794083", "title": "LED String Lights", "price": "$12.99" } ] } ``` --- Source: https://crawlbase.com/docs/scrapers/walmart-product-details # Walmart Product Details Extract a Walmart product page - title, price, description, images, ratings, and reviews. ## API usage Add `&scraper=walmart-product-details` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.walmart.com/ip/Meta-Quest-3-512GB-Breakthrough-Mixed-Reality-Powerful-Performance-Asgard-s-Wrath-2/3551794083' \ --data-urlencode 'scraper=walmart-product-details' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.walmart.com/ip/Meta-Quest-3-512GB-Breakthrough-Mixed-Reality-Powerful-Performance-Asgard-s-Wrath-2/3551794083', {'scraper': 'walmart-product-details'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.walmart.com/ip/Meta-Quest-3-512GB-Breakthrough-Mixed-Reality-Powerful-Performance-Asgard-s-Wrath-2/3551794083', { scraper: 'walmart-product-details' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.walmart.com/ip/Meta-Quest-3-512GB-Breakthrough-Mixed-Reality-Powerful-Performance-Asgard-s-Wrath-2/3551794083', scraper: 'walmart-product-details') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.walmart.com/ip/Meta-Quest-3-512GB-Breakthrough-Mixed-Reality-Powerful-Performance-Asgard-s-Wrath-2/3551794083 ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. product\_id string Product ID. title string Title. brand string | null Brand. price string Current price. original\_price string | null Pre-discount price. availability string Stock status. rating number Rating. reviews\_count integer Reviews count. description string Description. features array Feature bullets. images array Image URLs. specifications object Spec key/value pairs. seller string Sold by. ## Sample response ``` { "product_id": "3551794083", "title": "Meta Quest 3 512GB", "brand": "Meta", "price": "$649.99", "availability": "In stock", "rating": 4.7, "reviews_count": 8420 } ``` [← PreviousWalmart SERP](/docs/scrapers/walmart-serp)[Next →Walmart Category](/docs/scrapers/walmart-category) --- Source: https://crawlbase.com/docs/scrapers/walmart-serp # Walmart SERP Extract Walmart search results - array of products with prices, ratings, and shipping info. ## API usage Add `&scraper=walmart-serp` to a [Crawling API](/docs/crawling-api) request. URL-encode the target URL in the `url` parameter. ``` curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \ --data-urlencode 'url=https://www.walmart.com/search?q=samsung+galaxy' \ --data-urlencode 'scraper=walmart-serp' -G ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.walmart.com/search?q=samsung+galaxy', {'scraper': 'walmart-serp'} ) import json data = json.loads(res['body']) ``` ``` const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.walmart.com/search?q=samsung+galaxy', { scraper: 'walmart-serp' } ); const data = JSON.parse(res.body); ``` ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://www.walmart.com/search?q=samsung+galaxy', scraper: 'walmart-serp') data = JSON.parse(res.body) ``` ## Example input URL The URL passed in the `url` parameter (URL-decoded for readability): ``` https://www.walmart.com/search?q=samsung+galaxy ``` ## Response shape JSON response body. Field types may be `null` when the source page omits the value. query string Search query. total\_results integer | null Total products. products array Product summaries. products[].product\_id string Product ID. products[].title string Title. products[].price string Current price. products[].rating number Rating. products[].reviews\_count integer Reviews. products[].image\_url string Thumbnail. products[].sponsored boolean Sponsored result flag. ## Sample response ``` { "query": "samsung galaxy", "products": [ { "product_id": "5101854842", "title": "Samsung Galaxy S24 128GB", "price": "$799.99", "rating": 4.6, "reviews_count": 1240, "sponsored": false } ] } ``` [← PreviousImmobilienScout24 Property](/docs/scrapers/immobilienscout24-property)[Next →Walmart Product Details](/docs/scrapers/walmart-product-details) --- Source: https://crawlbase.com/docs/screenshots-api # Screenshots API Render any URL as a PNG or JPEG. Viewport-only or full-page, desktop or mobile, with all the JS rendering controls of the main API. Perfect for previews, monitoring, and snapshots. Migrate to the [Crawling API](/docs/crawling-api) (or `crawl_screenshot` via [MCP](/docs/ai-mcp)) Same JS-rendering pipeline, screenshot parameters added on the standard endpoint. The standalone Screenshots API has been closed to new sign-ups since Nov 1, 2024 - existing integrations continue to work, no shutdown is scheduled. ## Endpoint GEThttps://api.crawlbase.com/screenshots?token=YOUR\_JS\_TOKEN&url=ENCODED\_URL ``` # Requires a JavaScript token (rendering happens in headless Chrome). # Returns the image bytes directly. Content-Type: image/png (default). ``` ## Quickstart ``` # Save the screenshot to disk curl 'https://api.crawlbase.com/screenshots?token=YOUR_JS_TOKEN' \ --data-urlencode 'url=https://github.com/anthropic' \ -o screenshot.png -G ``` ``` from crawlbase import ScreenshotsAPI api = ScreenshotsAPI({'token': 'YOUR_JS_TOKEN'}) res = api.get('https://github.com/anthropic') with open('screenshot.png', 'wb') as f: f.write(res['body']) ``` ``` const { ScreenshotsAPI } = require('crawlbase'); const fs = require('node:fs/promises'); const api = new ScreenshotsAPI({ token: 'YOUR_JS_TOKEN' }); const res = await api.get('https://github.com/anthropic'); await fs.writeFile('screenshot.png', res.body); ``` ## Parameters ### Required token stringrequired Your private Crawlbase token. url stringrequired Target page URL. Must start with `http` or `https` and be fully URL-encoded. ### Screenshot-specific mode viewport | fullpageviewport Capture only the visible area, or the entire scrollable page. format png | jpegpng PNG for crisp text and UI; JPEG for smaller payloads on photo-heavy pages. width int (px)1280 Viewport width. height int (px)800 Viewport height. Ignored when `mode=fullpage`. device desktop | mobiledesktop Use a preset device profile. Mobile presets force `width=375`, `height=812`, and a phone User-Agent. store booleanfalse Persist the screenshot in [Cloud Storage](/docs/cloud-storage). When `true`, the response includes a `screenshot_url` header pointing at the stored copy - useful when you want a stable URL to embed in dashboards or share with downstream systems. ### Rendering control Inherited from the [Crawling API parameter set](/docs/crawling-api#parameters). The rendering controls clients use most often with screenshots: user\_agent stringoptional Custom User-Agent forwarded to the target site verbatim. URL-encode it. If omitted, Crawlbase rotates a realistic UA per request. css\_click\_selector stringoptional CSS selector for an element to click before the screenshot is captured (`#some-button`, `.some-other-button`). URL-encode the value. scroll booleanfalse Auto-scroll the page before capture. Defaults to a 10-second scroll. Pair with `scroll_interval` (10–60 s) to extend it. Useful for lazy-loaded content above the fold of a `mode=fullpage` shot. page\_wait integer (ms)optional Wait this many milliseconds after the page loads before capturing - gives time for animations or JS-heavy renders to settle. ajax\_wait booleanfalse Wait until in-flight AJAX requests finish before capturing. country ISO 3166optional Geolocate the screenshot from a specific country (e.g. `US`, `GB`, `DE`). Country availability is plan-gated; full country list lives on the [Crawling API parameters](/docs/crawling-api#parameters) reference. ## Common patterns ### Full-page mobile screenshot ``` curl 'https://api.crawlbase.com/screenshots?token=YOUR_JS_TOKEN' \ --data-urlencode 'url=https://news.ycombinator.com' \ --data-urlencode 'mode=fullpage' \ --data-urlencode 'device=mobile' \ --data-urlencode 'format=jpeg' \ -o hn-mobile.jpg -G ``` ### Screenshot after a click ``` # Open a "Show details" panel before capturing curl 'https://api.crawlbase.com/screenshots?token=YOUR_JS_TOKEN' \ --data-urlencode 'url=https://example.com/product/123' \ --data-urlencode 'css_click_selector=button.show-details' \ --data-urlencode 'page_wait=1500' \ -o detail.png -G ``` ## Common use cases - **Link previews** : generate Open Graph fallbacks for sites without proper meta tags. - **Visual monitoring** : capture a site weekly to detect layout regressions. - **Compliance archives** : pair with [Cloud Storage](/docs/cloud-storage) to archive what a page looked like on a specific date. - **Email reports** : embed live screenshots in scheduled reports. [← PreviousLeads API](/docs/leads-api)[Next →Proxy API](/docs/proxy-api) --- Source: https://crawlbase.com/docs/sdk-csharp # C# / .NET Official .NET client for the Crawlbase platform. Every method has a sync version (e.g. `Get`) and an async version (`GetAsync`) - same artifact, every API, sensible defaults. ## How the SDK is shaped The .NET SDK is a thin wrapper around the same HTTP API documented in [API Reference](/docs/api-reference). Every Crawling API parameter you'd append as a query string in a raw HTTP call is reachable as a `Dictionary` option - names, defaults, and behavior all map one-to-one. One quirk worth knowing up front: the .NET SDK exposes the response state _on the API instance itself_, not on a returned value object. Calls like `api.Get(url)` return void; you read the result via `api.StatusCode`, `api.Body`, and so on. This is different from the Python / Node / Ruby / PHP SDKs (which return a response object). The Storage API is the exception - its methods return a response object you read directly. What you get for using it instead of `HttpClient` directly: - URL encoding, parameter validation, and response parsing handled out of the box. - Sync + async pair on every verb - pick whichever fits your call site. - A single client class per Crawlbase API, all sharing the same constructor / call shape. - Sensible defaults (90-second timeout, automatic JSON parsing of `format=json` responses). Source on [github.com/crawlbase/crawlbase-net](https://github.com/crawlbase/crawlbase-net). ## Install Latest version on NuGet. Targets .NET 6+; tested through .NET 9. ``` # .NET CLI dotnet add package CrawlbaseAPI # Package Manager Console Install-Package CrawlbaseAPI # Or in csproj: # ``` ## Authentication Every Crawlbase API authenticates with the same token model. Two token types live on a single account: - **Normal Token (TCP)** - for static HTML, JSON endpoints, anything that doesn't need a browser. Faster + cheaper. - **JavaScript Token** - for SPAs, lazy-loaded feeds, anything that hides content behind client-side rendering. Required to use `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Use environment variables or your DI container's configuration in production. Pattern: ``` // Pick the right token at instantiation; the SDK doesn't switch // tokens per-call, so keep two clients if you alternate. var api = new Crawlbase.API(Environment.GetEnvironmentVariable("CRAWLBASE_TOKEN")); var js = new Crawlbase.API(Environment.GetEnvironmentVariable("CRAWLBASE_JS_TOKEN")); await api.GetAsync("https://github.com/anthropic"); var opts = new Dictionary { ["page_wait"] = 2000 }; await js.GetAsync("https://feed.example.com", opts); ``` Full token model + dashboard locations on the [Authentication](/docs/authentication) page. ## Quickstart Three lines from the namespace to a crawled response. Note that response state lives on the api instance: ``` var api = new Crawlbase.API("YOUR_TOKEN"); await api.GetAsync("https://github.com/anthropic"); if (api.StatusCode == 200) { Console.WriteLine(api.Body); } ``` Branch on `api.StatusCode` (the SDK's HTTP status to Crawlbase) and `api.CrawlbaseStatus` (the Crawlbase verdict - see [Errors](#errors) below) when deciding whether to retry. Pass `new Dictionary { ["format"] = "json" }` to receive a JSON envelope instead of raw page content. ## All APIs in one package Each Crawlbase product has a matching client class. Same constructor (single token string), same `Get` / `GetAsync` / `Post` / `PostAsync` shape. ``` string token = "YOUR_TOKEN"; var crawl = new Crawlbase.API(token); // Crawling API var scraper = new Crawlbase.ScraperAPI(token); // parsed JSON for supported sites var leads = new Crawlbase.LeadsAPI(token); // domain-scoped email extraction (legacy) var shots = new Crawlbase.ScreenshotsAPI(token); // body is base64-encoded image var storage = new Crawlbase.StorageAPI(token); // Cloud Storage CRUD // Push high-volume async jobs to the Enterprise Crawler via the Crawling API: // api.Get(url, options) where options carries `callback=true` + `crawler=YourCrawler`. // See /docs/crawler for the queue-management workflow. ``` ## Common patterns ### JavaScript rendering For SPAs, lazy-loaded feeds, and pages where the initial HTML is empty, instantiate with the JavaScript token and pass any combination of `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Order to think about: a fixed wait, then network-idle, then scroll for lazy-load, then click for any gating UI element. ``` var api = new Crawlbase.API("YOUR_JS_TOKEN"); await api.GetAsync("https://spa.example.com", new Dictionary { ["page_wait"] = 2000, ["ajax_wait"] = true, ["scroll"] = true, }); ``` ### Use a built-in scraper Skip the parser entirely on supported sites. Pass `["scraper"] = "NAME"` and the body becomes a JSON string with the structured fields documented on the per-scraper page. ``` using System.Text.Json; var api = new Crawlbase.ScraperAPI("YOUR_TOKEN"); await api.GetAsync( "https://www.amazon.com/dp/1098145356", new Dictionary { ["scraper"] = "amazon-product-details" } ); var data = JsonSerializer.Deserialize(api.Body); Console.WriteLine($"{data.GetProperty("name")} - {data.GetProperty("price")}"); ``` ### Geo-routing Pass `["country"] = "ISO"` to route the crawl through that country's exit nodes. Use it any time the target serves localized content based on IP. ``` var api = new Crawlbase.API("YOUR_TOKEN"); // Hit the German Amazon catalog from a German residential IP await api.GetAsync( "https://www.amazon.com/dp/1098145356", new Dictionary { ["country"] = "DE" } ); ``` ### Retry with backoff The recommended retry shape: exponential backoff capped at 3-5 attempts, retry on transient errors only (5xx or empty body), don't retry on 4xx. ``` public async Task CrawlAsync(Crawlbase.API api, string url, int attempts = 5) { var rand = new Random(); for (int i = 0; i < attempts; i++) { try { await api.GetAsync(url); } catch (Exception) { // SDK throws on transport failures - fall through to retry } if (api.StatusCode == 200 && api.CrawlbaseStatus == 200) { return true; } if (api.StatusCode is >= 400 and < 500) { throw new InvalidOperationException($"client error {api.StatusCode}: {url}"); } // Exponential backoff with jitter var ms = (int) (rand.NextDouble() * Math.Pow(2, i) * 1000); await Task.Delay(ms); } return false; } ``` ### Async crawls + webhooks Fire-and-forget mode. Pass `["async"] = true` with a `["callback"]` URL; the call returns immediately and Crawlbase POSTs the result to your webhook when the page is ready. Useful for batch jobs and slow targets. ``` var api = new Crawlbase.API("YOUR_TOKEN"); await api.GetAsync("https://example.com", new Dictionary { ["async"] = true, ["callback"] = "https://your-app.com/webhook", }); // api.Body is a JSON envelope { rid: ... } - use that to correlate // the eventual webhook delivery. // // Your ASP.NET / Minimal API endpoint receives a POST with: // { rid, url, original_status, cb_status, body } ``` For very high volumes (millions of URLs), use the [Enterprise Crawler](/docs/crawler) which sits in front of this same async pipeline. ### Sticky sessions Some flows need the same residential IP across multiple calls. Pass `cookies_session` with a stable identifier and Crawlbase reuses the same exit node for ~30 minutes. ``` var api = new Crawlbase.API("YOUR_JS_TOKEN"); var session = $"checkout-{userId}"; var opts = new Dictionary { ["cookies_session"] = session }; await api.GetAsync("https://shop.example.com/cart", opts); await api.GetAsync("https://shop.example.com/checkout", opts); await api.GetAsync("https://shop.example.com/confirm", opts); ``` ### Cloud Storage CRUD The Storage API is the exception to the "response on api instance" pattern - its methods return a response object you read directly. Useful when reading back results stored from a previous Crawling API call (`store=true`). ``` var storage = new Crawlbase.StorageAPI("YOUR_TOKEN"); // Fetch by URL var response = storage.GetByUrl("https://www.apple.com"); Console.WriteLine(response.OriginalStatus); Console.WriteLine(response.CrawlbaseStatus); Console.WriteLine(response.URL); Console.WriteLine(response.RID); Console.WriteLine(response.StoredAt); // Or fetch by RID, delete, bulk-fetch, list RIDs, total count var item = storage.GetByRID(rid); bool deleted = storage.Delete(rid); var items = storage.Bulk(new List { rid1, rid2 }); var rids = storage.RIDs(100); // optional limit var total = storage.TotalCount(); ``` ## Errors & retries The platform surfaces two status codes on every response: the SDK's own `api.StatusCode` (HTTP status of the request to Crawlbase itself) and `api.CrawlbaseStatus` (Crawlbase's verdict on the target - see the [Crawling API errors table](/docs/crawling-api#errors) for the full list). Always branch on `api.CrawlbaseStatus` when deciding whether to retry - a target can return `200` with empty body, in which case `StatusCode` is `200` but `CrawlbaseStatus` is `520`. ``` try { await api.GetAsync(url); } catch (Exception ex) { log.LogError(ex, "transport error"); return; } int pc = api.CrawlbaseStatus; switch (pc) { case 200: UseBody(api.Body); break; case 520 or 525: // 520 = empty body, 525 = anti-bot couldn't be solved. // Switch to JS token and retry. await RetryWithJsTokenAsync(url); break; case 521 or 522 or 523: // Target unreachable or timed out. Retry with backoff. ScheduleRetry(url); break; default: log.LogError("crawl failed url={Url} crawlbase_status={CrawlbaseStatus}", url, pc); break; } ``` All retries against the platform are free - only successful responses (`CrawlbaseStatus: 200`) count against your quota. ## Performance & best practices - **Reuse a single client per token.** Register it as a singleton in your DI container - each instance opens its own underlying `HttpClient`. Don't construct one per request. - **Use the cheapest token that works.** Don't default to the JavaScript token "just in case" - Normal-token requests are faster and use less concurrency. - **Prefer `ajax_wait` over `page_wait`.** Fixed delays burn concurrency on every request, even fast ones. - **Mind shared state on the API instance.** Because Crawling/Scraper/Leads/Screenshots APIs write response state onto the api object (not a return value), do not share one instance across concurrent `Task`s - a second await's `GetAsync()` will overwrite the first task's response state mid-read. Pool one instance per worker, or use the StorageAPI's return-object methods which are safe to interleave. - **For batch jobs: async + webhook, or push to the Enterprise Crawler.** Awaitable Tasks blocking on synchronous calls saturate concurrency caps quickly; async + webhook releases the slot the moment a request is queued. ## Method reference All non-Storage client classes share the same surface. Constructors take a token string; verbs come in sync + async pairs and write response state onto the api instance. new Crawlbase.API(string token) constructor Initialize a Crawling API client. Same shape for `Crawlbase.ScraperAPI`, `Crawlbase.LeadsAPI`, `Crawlbase.ScreenshotsAPI`, `Crawlbase.StorageAPI`. api.Get(string url, Dictionary options = null) method Send a GET (synchronous). Returns void; read response via properties on api. api.GetAsync(string url, Dictionary options = null) method Send a GET (async). Returns `Task`. Same response model. api.Post(...) / api.PostAsync(...) method Send a POST. `data` is the body - pass a Dictionary for form-encoded, a string for raw. Response state - properties on the api instance after a call: api.StatusCode int HTTP status of the SDK's request to Crawlbase. api.CrawlbaseStatus int Crawlbase verdict on the target. Branch on this for retry decisions. api.OriginalStatus int HTTP status the target returned to Crawlbase. api.Body string Page content (or JSON string when `format=json` / `scraper=` was used). For `ScreenshotsAPI`, this is base64-encoded - convert with `Convert.FromBase64String(api.Body)`. api.StorageURL / api.StorageRID string Set when the call carried `store=true`. Use these to fetch the stored response back via `StorageAPI`. [← PreviousJava](/docs/sdk-java)[Next →Overview](/docs/integrations) --- Source: https://crawlbase.com/docs/sdk-go # Go Official Go client for the Crawlbase platform. Idiomatic Go - error returns instead of exceptions, `context.Context` support on every verb, zero external dependencies (only `net/http` + stdlib). ## How the SDK is shaped The Go SDK is intentionally lean. One client - `CrawlingAPI`: covers every Crawlbase product through the unified Crawling API endpoint: | Use case | Pass in `options` | | --- | --- | | Plain crawl | _(nothing - the default)_ | | Built-in scraper | `"scraper": "amazon-product-details"` (and the rest of the [catalog](/docs/scrapers)) | | Screenshot | `"screenshot": "true"` | | Email extraction | `"scraper": "email-extractor"` | | Async + webhook | `"async": "true"` + `"callback": "https://..."` | | Push to Enterprise Crawler | `"async": "true"` + `"callback"` + `"crawler": "YourCrawler"` | The standalone `/scraper`, `/leads`, and `/screenshots` endpoints (which the older Crawlbase SDKs wrap with separate client classes) have been closed to new sign-ups since 2024. The Go SDK ships only the modern path - one client, every product, no vestigial classes. What you get for using it instead of `net/http` directly: - URL encoding, parameter validation, and response parsing handled out of the box. - Idiomatic Go surface - `(result, error)` returns, named struct fields, no panics on transport failures. - `context.Context` support on every verb via `*WithContext` variants for cancellation / deadlines / trace propagation. - Sensible defaults (90-second timeout, transparent gzip decompression, automatic JSON parsing of `format=json` / `scraper=` responses). Source on [github.com/crawlbase/crawlbase-go](https://github.com/crawlbase/crawlbase-go). Reference on [pkg.go.dev](https://pkg.go.dev/github.com/crawlbase/crawlbase-go). Issues + PRs welcome. ## Install Latest version on pkg.go.dev. Requires Go 1.21+. ``` go get github.com/crawlbase/crawlbase-go@latest # Or pin a specific version go get github.com/crawlbase/crawlbase-go@v0.1.0 ``` ## Authentication Every Crawlbase API authenticates with the same token model. Two token types live on a single account: - **Normal Token (TCP)** - for static HTML, JSON endpoints, anything that doesn't need a browser. Faster + cheaper. - **JavaScript Token** - for SPAs, lazy-loaded feeds, anything that hides content behind client-side rendering. Required to use `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Use environment variables in production. The SDK doesn't read env vars itself - that's deliberate so you stay in control of where credentials come from. Pattern: ``` package main import ( "log" "os" "github.com/crawlbase/crawlbase-go" ) func main() { // Pick the right token at instantiation; the SDK doesn't switch // tokens per-call, so keep two clients if you alternate. api, err := crawlbase.NewCrawlingAPI(os.Getenv("CRAWLBASE_TOKEN")) if err != nil { log.Fatal(err) } js, err := crawlbase.NewCrawlingAPI(os.Getenv("CRAWLBASE_JS_TOKEN")) if err != nil { log.Fatal(err) } api.Get("https://github.com/anthropic", nil) js.Get("https://feed.example.com", map[string]string{"page_wait": "2000"}) } ``` The constructor returns `crawlbase.ErrTokenRequired` if the token string is empty. Full token model + dashboard locations on the [Authentication](/docs/authentication) page. ## Quickstart Three lines from import to crawled HTML: ``` package main import ( "fmt" "log" "github.com/crawlbase/crawlbase-go" ) func main() { api, err := crawlbase.NewCrawlingAPI("YOUR_TOKEN") if err != nil { log.Fatal(err) } res, err := api.Get("https://github.com/anthropic", nil) if err != nil { log.Fatal(err) } if res.StatusCode == 200 { fmt.Println(res.Body) } } ``` Branch on `res.StatusCode` (the SDK's HTTP status to Crawlbase) and `res.CBStatus` (the Crawlbase verdict - see [Errors](#errors) below) when deciding whether to retry. Pass `map[string]string{"format": "json"}` to receive a JSON envelope instead of raw page content (auto-parsed into `res.JSON`). ## Common patterns ### JavaScript rendering For SPAs, lazy-loaded feeds, and pages where the initial HTML is empty, instantiate with the JavaScript token and pass any combination of `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Order to think about: a fixed wait, then network-idle, then scroll for lazy-load, then click for any gating UI element. ``` api, _ := crawlbase.NewCrawlingAPI("YOUR_JS_TOKEN") res, err := api.Get("https://spa.example.com", map[string]string{ "page_wait": "2000", "ajax_wait": "true", "scroll": "true", }) ``` ### Use a built-in scraper Skip the parser entirely on supported sites. Pass `"scraper": "NAME"` and the response Body becomes a JSON string with the structured fields documented on the per-scraper page. The body is also pre-decoded into `res.JSON` so you can read fields directly. ``` api, _ := crawlbase.NewCrawlingAPI("YOUR_TOKEN") res, err := api.Get( "https://www.amazon.com/dp/1098145356", map[string]string{"scraper": "amazon-product-details"}, ) if err != nil { log.Fatal(err) } if name, ok := res.JSON["name"].(string); ok { fmt.Println(name) } ``` ### Geo-routing Pass `"country": "ISO"` to route the crawl through that country's exit nodes. Use it any time the target serves localized content based on IP. ``` api, _ := crawlbase.NewCrawlingAPI("YOUR_TOKEN") // Hit the German Amazon catalog from a German residential IP res, _ := api.Get( "https://www.amazon.com/dp/1098145356", map[string]string{"country": "DE"}, ) ``` ### Retry with backoff The recommended retry shape: exponential backoff capped at 3-5 attempts, retry on transient errors only (5xx or empty body), don't retry on 4xx. ``` import ( "fmt" "math" "math/rand" "time" "github.com/crawlbase/crawlbase-go" ) func Crawl(api *crawlbase.CrawlingAPI, url string, attempts int) (*crawlbase.Response, error) { for i := 0; i < attempts; i++ { res, err := api.Get(url, nil) if err != nil { return nil, err } if res.StatusCode == 200 && res.CBStatus == 200 { return res, nil } if res.StatusCode >= 400 && res.StatusCode < 500 { return nil, fmt.Errorf("client error %d: %s", res.StatusCode, url) } // Exponential backoff with jitter d := time.Duration(rand.Float64() * math.Pow(2, float64(i)) * float64(time.Second)) time.Sleep(d) } return nil, fmt.Errorf("failed: %s", url) } ``` ### Async crawls + webhooks Fire-and-forget mode. The SDK call returns immediately with an `RID`; Crawlbase POSTs the result to your callback URL when the page is ready. Useful for batch jobs and slow targets. ``` api, _ := crawlbase.NewCrawlingAPI("YOUR_TOKEN") res, _ := api.Get("https://example.com", map[string]string{ "async": "true", "callback": "https://your-app.com/webhook", }) rid := res.RID // correlate the eventual webhook delivery // Your net/http handler receives a POST with: // { rid, url, original_status, cb_status, body } ``` For very high volumes (millions of URLs), push to the [Enterprise Crawler](/docs/crawler) by adding `"crawler": "YourCrawlerName"` alongside the async + callback options. ### Sticky sessions Some flows need the same residential IP across multiple calls. Pass `cookies_session` with a stable identifier and Crawlbase reuses the same exit node for ~30 minutes. ``` api, _ := crawlbase.NewCrawlingAPI("YOUR_JS_TOKEN") session := fmt.Sprintf("checkout-%d", userID) opts := map[string]string{"cookies_session": session} api.Get("https://shop.example.com/cart", opts) api.Get("https://shop.example.com/checkout", opts) api.Get("https://shop.example.com/confirm", opts) ``` ### Screenshots Pass `"screenshot": "true"` to capture a full-page screenshot. The body comes back as a base64-encoded image; use `crawlbase.ImageBytes(res)` to decode into raw bytes for `os.WriteFile` / `image.Decode`. ``` api, _ := crawlbase.NewCrawlingAPI("YOUR_JS_TOKEN") res, _ := api.Get("https://www.apple.com", map[string]string{ "screenshot": "true", }) img, err := crawlbase.ImageBytes(res) if err != nil { log.Fatal(err) } os.WriteFile("apple.png", img, 0o644) ``` ### Context for cancellation Every verb has a `*WithContext` variant for use with `context.Context`: useful any time the call should respect upstream cancellation, deadlines, or trace propagation (HTTP handlers, gRPC servers, anything in a request loop). ``` import ( "context" "time" ) api, _ := crawlbase.NewCrawlingAPI("YOUR_TOKEN") ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() res, err := api.GetWithContext(ctx, "https://example.com", nil) ``` ## Errors & retries The platform surfaces two status codes on every response: the SDK's own `res.StatusCode` (HTTP status of the request to Crawlbase itself) and `res.CBStatus` (Crawlbase's verdict on the target - lifted out of the `cb_status` response header for typed access; see the [Crawling API errors table](/docs/crawling-api#errors) for the full list). Always branch on `CBStatus` when deciding whether to retry - a target can return `200` with empty body, in which case `StatusCode` is `200` but `CBStatus` is `520`. ``` res, err := api.Get(url, nil) if err != nil { return err } switch res.CBStatus { case 200: use(res.Body) case 520, 525: // 520 = empty body, 525 = anti-bot couldn't be solved. // Switch to JS token and retry. retryWithJSToken(url) case 521, 522, 523: // Target unreachable or timed out. Retry with backoff. scheduleRetry(url) default: log.Printf("crawl failed: url=%s cb_status=%d", url, res.CBStatus) } ``` All retries against the platform are free - only successful responses (`CBStatus: 200`) count against your quota. ## Performance & best practices - **Reuse a single client per token.** The constructor is cheap, but each `*CrawlingAPI` instance has its own underlying `http.Client` with its own connection pool. Build it once at service init, share it across goroutines (the SDK is goroutine-safe). - **Use the cheapest token that works.** Don't default to the JavaScript token "just in case" - Normal-token requests are faster and use less concurrency. Promote on a `CBStatus == 520` or `525`. - **Prefer `ajax_wait` over `page_wait`.** Fixed delays burn concurrency on every request, even fast ones. - **For batch jobs: async + webhook, or push to the Enterprise Crawler.** Goroutine pools blocking on synchronous calls saturate concurrency caps quickly; async + webhook releases the slot the moment a request is queued. - **Use `GetWithContext` / `PostWithContext` in server code.** A request-scoped context propagates cancellation when the caller goes away - without it, a hung crawl will continue past the caller's deadline. ## Response fields Full method signatures, godoc, and per-method examples live on [pkg.go.dev](https://pkg.go.dev/github.com/crawlbase/crawlbase-go). The fields below are the bit Crawlbase users reach for most - the typed verdict on the target, returned on every `*crawlbase.Response`: StatusCode int HTTP status of the SDK's request to Crawlbase. CBStatus int Crawlbase verdict on the target. Lifted from the `cb_status` (or legacy `pc_status`) response header for typed access. Branch on this for retry decisions. `PCStatus` is a deprecated alias of `CBStatus`. OriginalStatus int HTTP status the target site returned to Crawlbase. URL string Final URL after target-side redirects. Body string Page content (or JSON string when `format=json` / `scraper=` was used; or base64-encoded image when `screenshot=true`). Headers map[string]string Lower-cased response headers. RID string Request ID - set when the call carried `"async": "true"` or `"store": "true"`. JSON map[string]any Pre-parsed JSON when the response Content-Type is JSON. Saves a `json.Unmarshal` step on scraper / format=json calls. [← PreviousPHP](/docs/sdk-php)[Next →Java](/docs/sdk-java) --- Source: https://crawlbase.com/docs/sdk-java # Java Official Java client for the Crawlbase platform. JDK 8+, dependency-light - same artifact, every API, sensible defaults. ## How the SDK is shaped The Java SDK is a thin wrapper around the same HTTP API documented in [API Reference](/docs/api-reference). Every Crawling API parameter you'd append as a query string in a raw HTTP call is reachable as a `HashMap` option - names, defaults, and behavior all map one-to-one. One quirk worth knowing up front: the Java SDK exposes the response state _on the API instance itself_, not on a returned value object. Calls like `api.get(url)` return void; you read the result via `api.getStatusCode()`, `api.getBody()`, and so on. This is different from the Python / Node / Ruby / PHP SDKs (which return a response object) - kept in mind, the rest of the surface is straightforward. What you get for using it instead of `HttpClient` / OkHttp directly: - URL encoding, parameter validation, and response parsing handled out of the box. - A single client class per Crawlbase API, all sharing the same constructor / call shape. - Idiomatic Java - runtime exceptions for transport failures (no checked exceptions to declare). - Sensible defaults (90-second timeout, automatic decoding of JSON / gzip responses). Source on [github.com/crawlbase/crawlbase-java](https://github.com/crawlbase/crawlbase-java). ## Install Latest version on Maven Central. Requires JDK 8+; tested through JDK 21. ``` com.crawlbase crawlbase-java-sdk-pom 1.1 implementation 'com.crawlbase:crawlbase-java-sdk-pom:1.1' ``` ## Authentication Every Crawlbase API authenticates with the same token model. Two token types live on a single account: - **Normal Token (TCP)** - for static HTML, JSON endpoints, anything that doesn't need a browser. Faster + cheaper. - **JavaScript Token** - for SPAs, lazy-loaded feeds, anything that hides content behind client-side rendering. Required to use `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Use environment variables or your Spring config in production. The SDK doesn't read either itself. Pattern: ``` import java.util.*; import com.crawlbase.*; // Pick the right token at instantiation; the SDK doesn't switch // tokens per-call, so keep two clients if you alternate. API api = new API(System.getenv("CRAWLBASE_TOKEN")); API js = new API(System.getenv("CRAWLBASE_JS_TOKEN")); api.get("https://github.com/anthropic"); HashMap opts = new HashMap<>(); opts.put("page_wait", 2000); js.get("https://feed.example.com", opts); ``` Full token model + dashboard locations on the [Authentication](/docs/authentication) page. ## Quickstart Three lines from import to crawled HTML. Note that response state lives on the API instance: ``` import com.crawlbase.*; API api = new API("YOUR_TOKEN"); api.get("https://github.com/anthropic"); if (api.getStatusCode() == 200) { System.out.println(api.getBody()); } ``` Branch on `api.getStatusCode()` (the SDK's HTTP status to Crawlbase) and `api.getCrawlbaseStatus()` (the Crawlbase verdict - see [Errors](#errors) below) when deciding whether to retry. Pass a `HashMap` with `"format" → "json"` to receive a JSON envelope instead of raw page content. ## All APIs in one artifact Each Crawlbase product has a matching client class. Same constructor (single token string), same method shape. ``` import com.crawlbase.*; String token = "YOUR_TOKEN"; API crawl = new API(token); // Crawling API: general-purpose page fetch ScraperAPI scraper = new ScraperAPI(token); // parsed JSON for supported sites LeadsAPI leads = new LeadsAPI(token); // domain-scoped email extraction (legacy) ScreenshotsAPI shots = new ScreenshotsAPI(token); // screenshots; body is base64-encoded image bytes // Push high-volume async jobs to the Enterprise Crawler via the Crawling API: // api.get(url, options) where options carries `callback=true` + `crawler=YourCrawler`. // See /docs/crawler for the queue-management workflow. ``` ## Common patterns ### JavaScript rendering For SPAs, lazy-loaded feeds, and pages where the initial HTML is empty, instantiate with the JavaScript token and pass any combination of `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Order to think about: a fixed wait, then network-idle, then scroll for lazy-load, then click for any gating UI element. ``` API api = new API("YOUR_JS_TOKEN"); HashMap opts = new HashMap<>(); opts.put("page_wait", 2000); opts.put("ajax_wait", true); opts.put("scroll", true); api.get("https://spa.example.com", opts); ``` ### Use a built-in scraper Skip the parser entirely on supported sites. Pass `"scraper" → "NAME"` and the body becomes a JSON string with the structured fields documented on the per-scraper page. ``` import com.crawlbase.*; import com.fasterxml.jackson.databind.*; import java.util.*; API api = new API("YOUR_TOKEN"); HashMap opts = new HashMap<>(); opts.put("scraper", "amazon-product-details"); api.get("https://www.amazon.com/dp/1098145356", opts); ObjectMapper mapper = new ObjectMapper(); Map data = mapper.readValue(api.getBody(), Map.class); System.out.println(data.get("name") + " - " + data.get("price")); ``` ### Geo-routing Pass `"country" → "ISO"` to route the crawl through that country's exit nodes. Use it any time the target serves localized content based on IP. ``` API api = new API("YOUR_TOKEN"); // Hit the German Amazon catalog from a German residential IP HashMap opts = new HashMap<>(); opts.put("country", "DE"); api.get("https://www.amazon.com/dp/1098145356", opts); ``` ### Retry with backoff The recommended retry shape: exponential backoff capped at 3-5 attempts, retry on transient errors only (5xx or empty body), don't retry on 4xx. ``` import com.crawlbase.*; import java.util.concurrent.ThreadLocalRandom; public boolean crawl(API api, String url, int attempts) throws InterruptedException { for (int i = 0; i < attempts; i++) { api.get(url); if (api.getStatusCode() == 200 && api.getCrawlbaseStatus() == 200) { return true; } if (api.getStatusCode() >= 400 && api.getStatusCode() < 500) { throw new RuntimeException("client error " + api.getStatusCode() + ": " + url); } // Exponential backoff with jitter long ms = (long) (ThreadLocalRandom.current().nextDouble() * Math.pow(2, i) * 1000); Thread.sleep(ms); } return false; } ``` ### Async crawls + webhooks Fire-and-forget mode. Pass `"async" → true` with a `"callback"` URL; the call returns immediately and Crawlbase POSTs the result to your webhook when the page is ready. Useful for batch jobs and slow targets. ``` API api = new API("YOUR_TOKEN"); HashMap opts = new HashMap<>(); opts.put("async", true); opts.put("callback", "https://your-app.com/webhook"); api.get("https://example.com", opts); // api.getBody() now contains a JSON envelope with { rid: ... }. // use that to correlate the eventual webhook delivery. // // Your Spring / Jakarta servlet receives a POST with: // { rid, url, original_status, cb_status, body } ``` For very high volumes (millions of URLs), use the [Enterprise Crawler](/docs/crawler) which sits in front of this same async pipeline. ### Sticky sessions Some flows need the same residential IP across multiple calls. Pass `cookies_session` with a stable identifier and Crawlbase reuses the same exit node for ~30 minutes. ``` API api = new API("YOUR_JS_TOKEN"); String session = "checkout-" + userId; HashMap opts = new HashMap<>(); opts.put("cookies_session", session); api.get("https://shop.example.com/cart", opts); api.get("https://shop.example.com/checkout", opts); api.get("https://shop.example.com/confirm", opts); ``` ## Errors & retries The platform surfaces two status codes on every response: the SDK's own `api.getStatusCode()` (HTTP status of the request to Crawlbase itself) and `api.getCrawlbaseStatus()` (Crawlbase's verdict on the target - see the [Crawling API errors table](/docs/crawling-api#errors) for the full list). Always branch on `getCrawlbaseStatus()` when deciding whether to retry - a target can return `200` with empty body, in which case `getStatusCode()` is `200` but `getCrawlbaseStatus()` is `520`. ``` api.get(url); int pc = api.getCrawlbaseStatus(); switch (pc) { case 200: useBody(api.getBody()); break; case 520: case 525: // 520 = empty body, 525 = anti-bot couldn't be solved. // Switch to JS token and retry. retryWithJsToken(url); break; case 521: case 522: case 523: // Target unreachable or timed out. Retry with backoff. scheduleRetry(url); break; default: log.error("crawl failed url={} crawlbase_status={}", url, pc); } ``` Note that all SDK methods throw `RuntimeException` (not checked exceptions) on transport failures. Wrap your retry loop accordingly. All retries against the platform are free - only successful responses (`crawlbaseStatus: 200`) count against your quota. ## Performance & best practices - **Reuse a single client per token.** Define it as a Spring bean / CDI singleton - each instance opens its own underlying HTTP client. Don't construct one per request. - **Use the cheapest token that works.** Don't default to the JavaScript token "just in case" - Normal-token requests are faster and use less concurrency. - **Prefer `ajax_wait` over `page_wait`.** Fixed delays burn concurrency on every request, even fast ones. - **Mind the shared-state on the API instance.** Because response data lives on the api object (not a return value), do not share one `API` instance across multiple threads making concurrent calls - a second thread's `api.get()` will overwrite the first thread's response state mid-read. Pool one instance per worker thread, or guard with a mutex. - **For batch jobs: async + webhook, or push to the Enterprise Crawler.** Thread pools blocking on synchronous calls saturate concurrency caps quickly; async + webhook releases the slot the moment a request is queued. ## Method reference All client classes share the same surface. Constructors take a token string; verbs mirror the underlying HTTP methods and write response state onto the api instance. new API(String token) constructor Initialize a Crawling API client. Optional second-argument constructors set timeout / proxy. Same shape for `ScraperAPI`, `LeadsAPI`, `ScreenshotsAPI`. api.get(String url) method Send a GET. Returns void; read response via getters. api.get(String url, HashMap\ options) method Send a GET with options. `options` maps any [Crawling API parameter](/docs/crawling-api) name to its value. api.post(String url, HashMap\ data) method Send a POST. `data` is the form-encoded body. Optional third-argument options. Response state - getters on the api instance after a call: api.getStatusCode() int HTTP status of the SDK's request to Crawlbase. api.getCrawlbaseStatus() int Crawlbase verdict on the target. Branch on this for retry decisions. api.getOriginalStatus() int HTTP status the target site returned to Crawlbase. api.getBody() String Page content (or JSON string when `format=json` / `scraper=` was used). For `ScreenshotsAPI`, this is a base64-encoded image - use `Base64.getDecoder().decode(...)` to convert. [← PreviousGolang](/docs/sdk-go)[Next →C# / .NET](/docs/sdk-csharp) --- Source: https://crawlbase.com/docs/sdk-node # Node.js Official Node.js client for the Crawlbase platform. One package, every API, full async/await with proper Promise rejection on transport errors and structured response objects on success. ## How the SDK is shaped The Node SDK is a thin wrapper around the same HTTP API documented in [API Reference](/docs/api-reference). Every Crawling API parameter you'd append as a query string in a raw HTTP call is reachable from the SDK as a key in the options object - names, defaults, and behavior all map one-to-one. There is no parameter the SDK adds; there is no parameter it hides. What you get for using it instead of `fetch` / `axios` directly: - URL encoding, parameter validation, and response parsing handled out of the box - code reads like product code, not HTTP plumbing. - Both ESM (`import { CrawlingAPI } from 'crawlbase'`) and CommonJS (`const { CrawlingAPI } = require('crawlbase')`) supported. - A single client class per Crawlbase API, all sharing the same constructor / call shape. - Sensible defaults (90-second timeout, automatic JSON parsing of `format=json` responses, UTF-8 decoding) that match what most teams configure by hand on their first integration. The SDK is open source, MIT-licensed, and accepts community PRs at [github.com/crawlbase/crawlbase-node](https://github.com/crawlbase/crawlbase-node). ## Install Latest version on npm. Works with Node.js 16+ on all major package managers. ``` npm install crawlbase # Or via pnpm / yarn / bun pnpm add crawlbase yarn add crawlbase bun add crawlbase ``` Source on [GitHub](https://github.com/crawlbase/crawlbase-node). Issues + PRs welcome. ## Authentication Every Crawlbase API authenticates with the same token model. Two token types live on a single account: - **Normal Token (TCP)** - for static HTML, JSON endpoints, anything that doesn't need a browser. Faster + cheaper. - **JavaScript Token** - for SPAs, lazy-loaded feeds, anything that hides content behind client-side rendering. Required to use `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Use environment variables in production. The SDK doesn't read env vars itself - that's deliberate so you stay in control of where credentials come from - but the idiomatic pattern is: ``` import { CrawlingAPI } from 'crawlbase'; // Pick the right token at instantiation; the SDK doesn't switch // tokens per-call, so keep two clients if you alternate. const api = new CrawlingAPI({ token: process.env.CRAWLBASE_TOKEN }); const js = new CrawlingAPI({ token: process.env.CRAWLBASE_JS_TOKEN }); await api.get('https://github.com/anthropic'); await js.get('https://feed.example.com', { page_wait: 2000 }); ``` Full token model + dashboard locations on the [Authentication](/docs/authentication) page. ## Quickstart Three lines from import to crawled HTML. Both ESM and CommonJS work: ``` // ESM import { CrawlingAPI } from 'crawlbase'; const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get('https://github.com/anthropic'); if (res.statusCode === 200) { console.log(res.body); } // CommonJS - same shape // const { CrawlingAPI } = require('crawlbase'); ``` Branch on `response.statusCode` (the SDK's HTTP status to Crawlbase) and `response.headers.cb_status` (the Crawlbase verdict - see [Errors](#errors) below) when deciding whether to retry. Pass `{ format: 'json' }` to receive a JSON envelope instead of raw page content. ## All APIs in one package Every Crawlbase API has a matching client class. Same constructor, same `get` / `post` verbs. ``` import { CrawlingAPI, // general-purpose page fetch ScraperAPI, // parsed JSON for supported sites LeadsAPI, // domain-scoped email extraction (legacy) ScreenshotsAPI, // screenshots of any URL } from 'crawlbase'; const token = { token: 'YOUR_TOKEN' }; const crawl = new CrawlingAPI(token); const scraper = new ScraperAPI(token); const leads = new LeadsAPI(token); const shots = new ScreenshotsAPI(token); // Push high-volume async jobs to the Enterprise Crawler via the // Crawling API: api.get(url, { async: true, callback: '...', // crawler: 'YourCrawler' }). See /docs/crawler for the queue // workflow. ``` ## Common patterns ### JavaScript rendering For SPAs, lazy-loaded feeds, and pages where the initial HTML is empty, instantiate with the JavaScript token and pass any combination of `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Order to think about: a fixed wait, then network-idle, then scroll for lazy-load, then click for any gating UI element. ``` const api = new CrawlingAPI({ token: 'YOUR_JS_TOKEN' }); const res = await api.get('https://spa.example.com', { page_wait: 2000, ajax_wait: true, scroll: true, }); ``` ### Use a built-in scraper Skip the parser entirely on supported sites. Pass `scraper: 'NAME'` and the response body becomes a JSON string with the structured fields documented on the per-scraper page. ``` import { ScraperAPI } from 'crawlbase'; const api = new ScraperAPI({ token: 'YOUR_TOKEN' }); const res = await api.get( 'https://www.amazon.com/dp/1098145356', { scraper: 'amazon-product-details' } ); const data = JSON.parse(res.body); console.log(data.name, data.price); ``` ### Geo-routing Pass `country: 'ISO'` to route the crawl through that country's exit nodes. Use it any time the target serves localized content based on IP. ``` const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); // Hit the German Amazon catalog from a German residential IP const res = await api.get( 'https://www.amazon.com/dp/1098145356', { country: 'DE' } ); ``` ### Retry with backoff The recommended retry shape: exponential backoff capped at 3-5 attempts, retry on transient errors only (5xx or empty body), don't retry on 4xx. ``` const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const sleep = ms => new Promise(r => setTimeout(r, ms)); async function crawl(url, attempts = 5) { for (let i = 0; i < attempts; i++) { const res = await api.get(url); if (res.statusCode === 200 && Number(res.headers.cb_status) === 200) { return res; } if (res.statusCode >= 400 && res.statusCode < 500) { throw new Error(`client error ${res.statusCode}: ${url}`); } await sleep(Math.random() * (2 ** i) * 1000); } throw new Error(`Failed: ${url}`); } ``` ### Async crawls + webhooks Fire-and-forget mode. The SDK call resolves immediately with an `rid`; Crawlbase POSTs the result to your callback URL when the page is ready. Useful for batch jobs and slow targets. ``` const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const res = await api.get('https://example.com', { async: true, callback: 'https://your-app.com/webhook', }); const rid = res.rid; // correlate the eventual webhook delivery // Your Express / Fastify / Hono webhook receives a POST with: // { rid, url, original_status, cb_status, body } ``` For very high volumes (millions of URLs), use the [Enterprise Crawler](/docs/crawler) which sits in front of this same async pipeline with retries, rate management, and result delivery. ### Sticky sessions Some flows need the same residential IP across multiple calls - a checkout, a paginated search, a logged-in session. Pass `cookies_session` with a stable identifier and Crawlbase reuses the same exit node for ~30 minutes. ``` const api = new CrawlingAPI({ token: 'YOUR_JS_TOKEN' }); const session = `checkout-${userId}`; await api.get('https://shop.example.com/cart', { cookies_session: session }); await api.get('https://shop.example.com/checkout', { cookies_session: session }); await api.get('https://shop.example.com/confirm', { cookies_session: session }); ``` ## Errors & retries The Crawlbase platform surfaces two status codes on every response: the SDK's own `response.statusCode` (HTTP status of the request to Crawlbase itself) and the `cb_status` response header (Crawlbase's verdict on the target - see the [Crawling API errors table](/docs/crawling-api#errors) for the full list). The Node SDK exposes response headers as a plain object on `response.headers`, so the verdict reads as `response.headers.cb_status`. Always branch on that when deciding whether to retry - a target can return `200` with empty body, in which case `response.statusCode` is `200` but `response.headers.cb_status` is `520`. ``` const res = await api.get(url); const cb = Number(res.headers.cb_status); if (cb === 200) { use(res.body); } else if (cb === 520 || cb === 525) { // 520 = empty body, 525 = anti-bot couldn't be solved. // Switch to JS token and retry. await retryWithJsToken(url); } else if ([521, 522, 523].includes(cb)) { // Target unreachable or timed out. Retry with backoff. schedule_retry(url); } else { logger.error('crawl failed', { url, cb_status: cb }); } ``` All retries against the platform are free - only successful responses (`cb_status: 200`) count against your quota. That makes aggressive backoff cheap; the only real cost of retrying is added latency. ## Performance & best practices - **Reuse a single client per token.** The constructor is cheap but each instance opens its own underlying connection pool. Build it once at module scope, share it across calls. - **Use the cheapest token that works.** Don't default to the JavaScript token "just in case" - Normal-token requests are faster and use less concurrency. Promote to JS only when the Normal response is empty or anti-bot-blocked. - **Prefer `ajax_wait` over `page_wait`.** Fixed delays burn concurrency on every request, even fast ones. - **For batch jobs: async + webhook, or push to the Enterprise Crawler.** Synchronous mode is the right default for ad-hoc and interactive use; for sustained high-volume submission switch to async so your concurrency slot frees up the moment a request is queued. - **Watch the `remaining` response header.** It carries the number of concurrency slots you have left - a healthy client backs off proactively before hitting the cap. ## Method reference All client classes share the same surface. Constructor takes a single options object; verbs mirror the underlying HTTP methods. Every method returns a Promise. new CrawlingAPI({ token, timeout }) constructor Initialize a client with your token. Optional: `timeout` in milliseconds (default `90000`). .get(url, options?) method Send a GET. `options` maps any [Crawling API parameter](/docs/crawling-api) to its value. .post(url, data, options?) method Send a POST. `data` is the body - pass an object for form-encoded, a string for raw. Response shape (object, all properties present even when empty): response.statusCode number HTTP status of the SDK's request to Crawlbase. response.body string Page content (or JSON string when `format=json` / `scraper=` was used). UTF-8 decoded by default. response.url string Final URL after target-side redirects. response.headers object Lower-cased response headers. Crawlbase-specific status fields are exposed here: - `response.headers.cb_status` - Crawlbase verdict on the target. Lifted from the `cb_status` (or legacy `pc_status`) response header. Branch on this for retry decisions. `pc_status` is retained as a deprecated alias key. - `response.headers.original_status` - HTTP status the target site returned to Crawlbase. - `response.headers.rid` - Request ID (when the call carried `async: true` or `store: true`). response.json object | undefined Pre-parsed JSON when the response Content-Type is JSON. Parsed once by the SDK so you don't have to. [← PreviousPython](/docs/sdk-python)[Next →Ruby](/docs/sdk-ruby) --- Source: https://crawlbase.com/docs/sdk-php # PHP Official PHP client for the Crawlbase platform. PSR-compatible, Composer-installable, works with PHP 7.4+ - same package, every API, sensible defaults. ## How the SDK is shaped The PHP SDK is a thin wrapper around the same HTTP API documented in [API Reference](/docs/api-reference). Every Crawling API parameter you'd append as a query string in a raw HTTP call is reachable from the SDK as a key in the options array - names, defaults, and behavior all map one-to-one. There is no parameter the SDK adds; there is no parameter it hides. What you get for using it instead of cURL or Guzzle directly: - URL encoding, parameter validation, and response parsing handled out of the box. - PSR-4 autoloading - drop into any modern PHP framework (Laravel, Symfony, Slim) without ceremony. - A single client class per Crawlbase API, all sharing the same constructor / call shape. - Sensible defaults (90-second timeout, automatic JSON parsing of `format=json` responses, UTF-8-encoded bodies). Source on [github.com/crawlbase/crawlbase-php](https://github.com/crawlbase/crawlbase-php). Issues + PRs welcome. ## Install Latest version on Packagist. Requires PHP 7.4+; tested through PHP 8.3. ``` composer require crawlbase/crawlbase # Or add to composer.json directly: # "crawlbase/crawlbase": "^1.0" ``` ## Authentication Every Crawlbase API authenticates with the same token model. Two token types live on a single account: - **Normal Token (TCP)** - for static HTML, JSON endpoints, anything that doesn't need a browser. Faster + cheaper. - **JavaScript Token** - for SPAs, lazy-loaded feeds, anything that hides content behind client-side rendering. Required to use `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Use environment variables (or your framework's config - Laravel `config()`, Symfony parameters) in production. The SDK doesn't read env vars itself - that's deliberate so you stay in control of where credentials come from. Pattern: ``` getenv('CRAWLBASE_TOKEN')]); $js = new CrawlingAPI(['token' => getenv('CRAWLBASE_JS_TOKEN')]); $api->get('https://github.com/anthropic'); $js->get('https://feed.example.com', ['page_wait' => 2000]); ``` Full token model + dashboard locations on the [Authentication](/docs/authentication) page. ## Quickstart Three lines from autoload to crawled HTML: ``` 'YOUR_TOKEN']); $res = $api->get('https://github.com/anthropic'); if ($res->statusCode == 200) { echo $res->body; } ``` Branch on `->statusCode` (the SDK's HTTP status to Crawlbase) and `->headers->cb_status` (the Crawlbase verdict - see [Errors](#errors) below) when deciding whether to retry. Pass `['format' => 'json']` to receive a JSON envelope instead of raw page content. ## All APIs in one package Every Crawlbase API has a matching client class. Same constructor, same `get` / `post` verbs. ``` 'YOUR_TOKEN']; $crawl = new CrawlingAPI($token); // general-purpose page fetch $scraper = new ScraperAPI($token); // parsed JSON for supported sites $leads = new LeadsAPI($token); // domain-scoped email extraction (legacy) $shots = new ScreenshotsAPI($token); // screenshots of any URL $storage = new StorageAPI($token); // Cloud Storage CRUD // Push high-volume async jobs to the Enterprise Crawler via the Crawling API: // $api->get($url, ['async' => true, 'callback' => '...', 'crawler' => 'YourCrawler']). // See /docs/crawler for the queue workflow. ``` ## Common patterns ### JavaScript rendering For SPAs, lazy-loaded feeds, and pages where the initial HTML is empty, instantiate with the JavaScript token and pass any combination of `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Order to think about: a fixed wait, then network-idle, then scroll for lazy-load, then click for any gating UI element. ``` $api = new \\Crawlbase\\CrawlingAPI(['token' => 'YOUR_JS_TOKEN']); $res = $api->get('https://spa.example.com', [ 'page_wait' => 2000, 'ajax_wait' => true, 'scroll' => true, ]); ``` ### Use a built-in scraper Skip the parser entirely on supported sites. Pass `'scraper' => 'NAME'` and the response body becomes a JSON string with the structured fields documented on the per-scraper page. ``` 'YOUR_TOKEN']); $res = $api->get('https://www.amazon.com/dp/1098145356', ['scraper' => 'amazon-product-details']); $data = json_decode($res->body, true); echo $data['name'] . ' - ' . $data['price']; ``` ### Geo-routing Pass `'country' => 'ISO'` to route the crawl through that country's exit nodes. Use it any time the target serves localized content based on IP. ``` $api = new \\Crawlbase\\CrawlingAPI(['token' => 'YOUR_TOKEN']); // Hit the German Amazon catalog from a German residential IP $res = $api->get('https://www.amazon.com/dp/1098145356', ['country' => 'DE']); ``` ### Retry with backoff The recommended retry shape: exponential backoff capped at 3-5 attempts, retry on transient errors only (5xx or empty body), don't retry on 4xx. ``` get($url); if ($res->statusCode === 200 && (int) $res->headers->cb_status === 200) { return $res; } if ($res->statusCode >= 400 && $res->statusCode < 500) { throw new RuntimeException("client error {$res->statusCode}: $url"); } usleep((int) (mt_rand() / mt_getrandmax() * pow(2, $i) * 1_000_000)); } throw new RuntimeException("Failed: $url"); } ``` ### Async crawls + webhooks Fire-and-forget mode. The SDK call returns immediately with an `rid`; Crawlbase POSTs the result to your callback URL when the page is ready. Useful for batch jobs and slow targets. ``` $api = new \\Crawlbase\\CrawlingAPI(['token' => 'YOUR_TOKEN']); $res = $api->get('https://example.com', [ 'async' => true, 'callback' => 'https://your-app.com/webhook', ]); $rid = $res->rid; // correlate the eventual webhook delivery // Your Laravel / Symfony / Slim webhook receives a POST with: // { rid, url, original_status, cb_status, body } ``` For very high volumes (millions of URLs), use the [Enterprise Crawler](/docs/crawler) which sits in front of this same async pipeline. ### Sticky sessions Some flows need the same residential IP across multiple calls. Pass `cookies_session` with a stable identifier and Crawlbase reuses the same exit node for ~30 minutes. ``` $api = new \Crawlbase\CrawlingAPI(['token' => 'YOUR_JS_TOKEN']); $session = "checkout-{$userId}"; $api->get('https://shop.example.com/cart', ['cookies_session' => $session]); $api->get('https://shop.example.com/checkout', ['cookies_session' => $session]); $api->get('https://shop.example.com/confirm', ['cookies_session' => $session]); ``` ## Errors & retries The platform surfaces two status codes on every response: the SDK's own `->statusCode` (HTTP status of the request to Crawlbase itself) and `->headers->cb_status` (Crawlbase's verdict on the target - see the [Crawling API errors table](/docs/crawling-api#errors) for the full list). Always branch on `->headers->cb_status` when deciding whether to retry - a target can return `200` with empty body, in which case `->statusCode` is `200` but `->headers->cb_status` is `520`. ``` $res = $api->get($url); $cb = (int) $res->headers->cb_status; switch (true) { case $cb === 200: use_body($res->body); break; case in_array($cb, [520, 525], true): // 520 = empty body, 525 = anti-bot couldn't be solved. // Switch to JS token and retry. retry_with_js_token($url); break; case in_array($cb, [521, 522, 523], true): // Target unreachable or timed out. Retry with backoff. schedule_retry($url); break; default: $logger->error('crawl failed', ['url' => $url, 'cb_status' => $cb]); } ``` All retries against the platform are free - only successful responses (`cb_status: 200`) count against your quota. ## Performance & best practices - **Reuse a single client per token.** Build it once at app boot (Laravel service provider, Symfony service container) and inject everywhere - each instance opens its own connection. - **Use the cheapest token that works.** Don't default to the JavaScript token "just in case" - Normal-token requests are faster and use less concurrency. Promote to JS only when the Normal response is empty or anti-bot-blocked. - **Prefer `ajax_wait` over `page_wait`.** Fixed delays burn concurrency on every request, even fast ones. - **For batch jobs: async + webhook, or push to the Enterprise Crawler.** Queue workers calling the SDK synchronously will saturate your concurrency cap; async + webhook releases the slot the moment a request is queued. - **Watch the `remaining` response header.** It carries the number of concurrency slots you have left. ## Method reference All client classes share the same surface. Constructor takes an options array; verbs mirror the underlying HTTP methods. new CrawlingAPI(['token' =\> T, 'timeout' =\> N]) constructor Initialize a client with your token. Optional: `'timeout'` in seconds (default `90`). -\>get($url, $options = []) method Send a GET. `$options` maps any [Crawling API parameter](/docs/crawling-api) to its value. -\>post($url, $data, $options = []) method Send a POST. `$data` is the body - pass an array for form-encoded, a string for raw. Response shape - public properties on the response object returned from each verb: -\>statusCode int HTTP status of the SDK's request to Crawlbase. -\>body string Page content (or JSON string when `format=json` / `scraper=` was used). -\>headers object Response headers as an object. Crawlbase-specific status fields are exposed here: - `->headers->cb_status`: Crawlbase verdict on the target. Lifted from the `cb_status` (or legacy `pc_status`) response header. Branch on this for retry decisions. `pc_status` is retained as a deprecated alias key. - `->headers->original_status`: HTTP status the target site returned to Crawlbase. - `->headers->storage_url` / `->headers->rid`: set when the call carried `'store' => true`. [← PreviousRuby](/docs/sdk-ruby)[Next →Golang](/docs/sdk-go) --- Source: https://crawlbase.com/docs/sdk-python # Python Official Python client for the Crawlbase platform. One package wraps every API - Crawling, Scraper, Smart AI Proxy, Storage, Crawler, Screenshots - with idiomatic Python ergonomics for parameters, errors, and retries. ## How the SDK is shaped The Python SDK is a thin, dependency-light wrapper around the same HTTP API documented in [API Reference](/docs/api-reference). Every Crawling API parameter you'd append as a query string in a raw HTTP call is reachable from the SDK as a keyword in the options dict - names, defaults, and behavior all map one-to-one. There is no parameter the SDK adds; there is no parameter the SDK hides. What you get for using it instead of `requests` directly: - URL encoding, parameter validation, and response parsing handled out of the box - your application code reads like product code, not HTTP plumbing. - A single client class per Crawlbase API, all sharing the same constructor / call shape so once you've used one, you've used all of them. - Sensible defaults (90-second timeout, JSON parsing of `format=json` responses, automatic UTF-8 decoding) that match what most teams configure by hand on their first integration. - A small surface area to learn - five client classes, two verbs (`get` / `post`), one response shape. The SDK is open source, MIT-licensed, and accepts community PRs at [github.com/crawlbase/crawlbase-python](https://github.com/crawlbase/crawlbase-python). Most reported issues land in a release within a sprint. ## Install Latest version on PyPI. Requires Python 3.7+; tested through Python 3.13. ``` pip install crawlbase # Or via Poetry / uv / pip-tools poetry add crawlbase uv add crawlbase ``` Source on [GitHub](https://github.com/crawlbase/crawlbase-python). Issues + PRs welcome. ## Authentication Every Crawlbase API authenticates with the same token model - there's no separate API key per product. Two token types live on a single account: - **Normal Token (TCP)**- for static HTML, JSON endpoints, anything that doesn't need a browser. Faster + cheaper. - **JavaScript Token** - for SPAs, lazy-loaded feeds, and any target that hides content behind client-side rendering. Required to use `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Use environment variables in production rather than hard-coding tokens. The SDK doesn't read env vars itself - that's a deliberate choice so you stay in control of where credentials come from - but the idiomatic pattern is: ``` import os from crawlbase import CrawlingAPI # Pick the right token at instantiation; the SDK doesn't switch # tokens per-call, so keep two clients if you alternate. api = CrawlingAPI({'token': os.environ['CRAWLBASE_TOKEN']}) js = CrawlingAPI({'token': os.environ['CRAWLBASE_JS_TOKEN']}) res = api.get('https://github.com/anthropic') res = js.get('https://feed.example.com', {'page_wait': 2000}) ``` Full token model + dashboard locations on the [Authentication](/docs/authentication) page. ## Quickstart Three lines from import to crawled HTML: ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get('https://github.com/anthropic') if res['status_code'] == 200: print(res['body']) ``` Branch on `status_code` (the HTTP status of the SDK's request to Crawlbase) and `cb_status` (the Crawlbase verdict - see [Errors](#errors) below) when deciding whether to retry. The body is bytes by default; pass `'format': 'json'` to receive a JSON envelope instead of raw page content. ## All APIs in one package Every Crawlbase API has a matching client class. Same constructor, same `get` / `post` verbs. Pick the class by what you're doing; behind the scenes they all hit a different endpoint of the same platform. ``` from crawlbase import ( CrawlingAPI, # general-purpose page fetch (HTML / JSON / etc.) ScraperAPI, # parsed JSON for supported sites (Amazon, Google, etc.) LeadsAPI, # domain-scoped email extraction (legacy) ScreenshotsAPI, # screenshots of any URL StorageAPI, # Cloud Storage CRUD ) token = {'token': 'YOUR_TOKEN'} crawl = CrawlingAPI(token) scraper = ScraperAPI(token) leads = LeadsAPI(token) shots = ScreenshotsAPI(token) storage = StorageAPI(token) # Push high-volume async jobs to the Enterprise Crawler via the # Crawling API: api.get(url, {'async': True, 'callback': '...', # 'crawler': 'YourCrawler'}). See /docs/crawler for the queue # workflow. ``` ## Common patterns ### JavaScript rendering For SPAs, lazy-loaded feeds, and pages where the initial HTML is empty, instantiate with the JavaScript token and pass any combination of `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Order to think about: a fixed wait, then network-idle, then scroll for lazy-load, then click for any gating UI element. ``` api = CrawlingAPI({'token': 'YOUR_JS_TOKEN'}) res = api.get('https://spa.example.com', { 'page_wait': 2000, 'ajax_wait': True, 'scroll': True, }) ``` ### Use a built-in scraper Skip the parser entirely on supported sites. Pass `'scraper': 'NAME'` and the response `body` becomes a JSON string with the structured fields documented on the per-scraper page. ``` import json from crawlbase import ScraperAPI api = ScraperAPI({'token': 'YOUR_TOKEN'}) res = api.get( 'https://www.amazon.com/dp/1098145356', {'scraper': 'amazon-product-details'} ) data = json.loads(res['body']) print(data['name'], data['price']) ``` ### Geo-routing Pass `'country'='ISO'` to route the crawl through that country's exit nodes. Use it any time the target serves localized content based on IP - most retailers, all SERPs, geo-restricted streaming pages. ``` api = CrawlingAPI({'token': 'YOUR_TOKEN'}) # Hit the German Amazon catalog from a German residential IP res = api.get( 'https://www.amazon.com/dp/1098145356', {'country': 'DE'} ) ``` ### Async with retries The recommended retry shape: exponential backoff capped at 3-5 attempts, retry on transient errors only (5xx or empty body), don't retry on 4xx (the request shape is wrong and won't fix itself). ``` import time, random from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) def crawl(url, attempts=5): for i in range(attempts): res = api.get(url) # 200 from Crawlbase + non-empty body from the target if res['status_code'] == 200 and int(res.get('cb_status', 0)) == 200: return res # Don't bother retrying client errors (4xx) if 400 <= res['status_code'] < 500: raise ValueError(f"client error {res['status_code']}: {url}") # Exponential backoff with jitter time.sleep(random.uniform(0, 2 ** i)) raise RuntimeError(f'Failed: {url}') ``` ### Async crawls + webhooks Fire-and-forget mode. The SDK call returns immediately with an `rid`; Crawlbase POSTs the result to your callback URL when the page is ready. Useful for batch jobs and slow targets where you don't want a synchronous request to occupy a concurrency slot for 30+ seconds. ``` api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get('https://example.com', { 'async': True, 'callback': 'https://your-app.com/webhook', }) rid = res['rid'] # use this to correlate the eventual webhook delivery # Webhook handler (Flask / FastAPI / etc.) receives a POST with: # { rid, url, original_status, cb_status, body } ``` For very high volumes (millions of URLs), use the [Enterprise Crawler](/docs/crawler) which sits in front of this same async pipeline with retries, rate management, and result delivery. ### Sticky sessions Some flows need the same residential IP across multiple calls - a checkout, a paginated search, a logged-in session. Pass `'cookies_session'` with a stable identifier and Crawlbase reuses the same exit node for ~30 minutes. ``` api = CrawlingAPI({'token': 'YOUR_JS_TOKEN'}) session = f'checkout-{user_id}' api.get('https://shop.example.com/cart', {'cookies_session': session}) api.get('https://shop.example.com/checkout', {'cookies_session': session}) api.get('https://shop.example.com/confirm', {'cookies_session': session}) ``` ## Errors & retries The Crawlbase platform surfaces two status codes on every response: the SDK's own `status_code` (the HTTP status of the request to Crawlbase itself) and `cb_status` (Crawlbase's verdict on the target - see the [Crawling API errors table](/docs/crawling-api#errors) for the full list). Always branch on `cb_status` when deciding whether to retry - a target can return `200` with empty body, in which case `status_code` is `200` but `cb_status` is `520`. ``` res = api.get(url) cb = int(res.get('cb_status', 0)) if cb == 200: use(res['body']) elif cb in (520, 525): # 520 = empty body, 525 = anti-bot couldn't be solved. # Switch to JS token and retry. retry_with_js_token(url) elif cb in (521, 522, 523): # Target unreachable or timed out. Retry with backoff. schedule_retry(url) else: log.error('crawl failed', extra={'url': url, 'cb_status': cb}) ``` All retries against the platform are free - only successful responses (`cb_status: 200`) count against your quota. That makes aggressive backoff cheap; the only real cost of retrying is added latency. ## Performance & best practices - **Reuse a single client per token.** The constructor is cheap but each instance opens its own connection pool. Build it once at module scope, share it across calls. - **Use the cheapest token that works.** Don't default to the JavaScript token "just in case" - Normal-token requests are faster and use less concurrency. Promote to JS only when the Normal response is empty or anti-bot-blocked. - **Prefer `ajax_wait` over `page_wait`.** Fixed delays burn concurrency on every request, even fast ones. `ajax_wait` returns the moment the page goes network-idle. - **For batch jobs: async + webhook, or push to the Enterprise Crawler.** Synchronous mode is the right default for ad-hoc and interactive use; for sustained high-volume submission switch to async so your concurrency slot frees up the moment a request is queued rather than when it completes. - **Watch the `remaining` response header.** It carries the number of concurrency slots you have left - a healthy client backs off proactively before hitting the cap rather than reacting to 429s. ## Method reference All client classes share the same surface. Constructor takes a single options dict; verbs mirror the underlying HTTP methods. CrawlingAPI({'token': T, 'timeout': N}) constructor Initialize a client with your token. Optional: `'timeout'` in seconds (default `90`) - applies to the SDK's HTTP call to Crawlbase, not the upstream crawl. .get(url, options=None) method Send a GET. `options` is a dict mapping any [Crawling API parameter](/docs/crawling-api) to its value. Returns a response dict. .post(url, data, options=None) method Send a POST. `data` is the body - pass a dict for form-encoded, a string for raw. `options` works the same as `.get`. Response shape (dict, all keys present even when their value is empty): status\_code int HTTP status of the SDK's request to Crawlbase. `200` means the request was accepted; check `cb_status` for the target outcome. cb\_status int Crawlbase verdict on the target. Lifted from the `cb_status` (or legacy `pc_status`) response header. Branch on this for retry decisions. `pc_status` is retained as a deprecated alias key. original\_status int HTTP status the target site returned to Crawlbase. url str Final URL after target-side redirects. body bytes | str Page content (or JSON string when `format=json` / `scraper=` was used). headers dict Response headers from the target site. rid str Request ID (when `async=true` or `store=true`). [← PreviousOverview](/docs/sdks)[Next →Node.js](/docs/sdk-node) --- Source: https://crawlbase.com/docs/sdk-ruby # Ruby Official Ruby gem for the Crawlbase platform. Idiomatic Ruby across Ruby 2.7+ and JRuby - same gem, every API, sensible defaults that match what most Rails apps configure by hand. ## How the SDK is shaped The Ruby gem is a thin wrapper around the same HTTP API documented in [API Reference](/docs/api-reference). Every Crawling API parameter you'd append as a query string in a raw HTTP call is reachable from the gem as a keyword on the call - names, defaults, and behavior all map one-to-one. There is no parameter the gem adds; there is no parameter it hides. What you get for using it instead of `Net::HTTP` / `Faraday` directly: - URL encoding, parameter validation, and response parsing handled out of the box - application code stays focused on the business logic. - Idiomatic Ruby surface - keyword args, snake\_case parameter names, exception-raising for transport failures, plain-old-Ruby response objects. - A single client class per Crawlbase API, all sharing the same constructor / call shape. - Sensible defaults (90-second timeout, automatic JSON parsing of `format=json` responses, UTF-8-encoded bodies) that match what most teams configure by hand on their first integration. Source on [github.com/crawlbase/crawlbase-ruby](https://github.com/crawlbase/crawlbase-ruby). Issues + PRs welcome. ## Install Latest version on RubyGems. Tested on Ruby 2.7, 3.0, 3.1, 3.2, 3.3 + JRuby. ``` gem install crawlbase # Or in your Gemfile gem 'crawlbase' ``` ## Authentication Every Crawlbase API authenticates with the same token model. Two token types live on a single account: - **Normal Token (TCP)** - for static HTML, JSON endpoints, anything that doesn't need a browser. Faster + cheaper. - **JavaScript Token** - for SPAs, lazy-loaded feeds, anything that hides content behind client-side rendering. Required to use `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Use Rails credentials (`Rails.application.credentials.crawlbase_token`) or environment variables in production. The gem doesn't read either itself - that's deliberate so you stay in control of where credentials come from. Pattern: ``` require 'crawlbase' # Pick the right token at instantiation; the gem doesn't switch # tokens per-call, so keep two clients if you alternate. api = Crawlbase::API.new(token: ENV.fetch('CRAWLBASE_TOKEN')) js = Crawlbase::API.new(token: ENV.fetch('CRAWLBASE_JS_TOKEN')) api.get('https://github.com/anthropic') js.get('https://feed.example.com', page_wait: 2000) ``` Full token model + dashboard locations on the [Authentication](/docs/authentication) page. ## Quickstart Three lines from require to crawled HTML: ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://github.com/anthropic') puts res.body if res.status_code == 200 ``` Branch on `.status_code` (the gem's HTTP status to Crawlbase) and `.cb_status` (the Crawlbase verdict - see [Errors](#errors) below) when deciding whether to retry. Pass `format: 'json'` to receive a JSON envelope instead of raw page content. ## All APIs in one gem Every Crawlbase API has a matching class. Same constructor, same `get` / `post` verbs. ``` require 'crawlbase' token = { token: 'YOUR_TOKEN' } crawl = Crawlbase::API.new(**token) # general-purpose page fetch scraper = Crawlbase::ScraperAPI.new(**token) # parsed JSON for supported sites leads = Crawlbase::LeadsAPI.new(**token) # domain-scoped email extraction (legacy) shots = Crawlbase::ScreenshotsAPI.new(**token) # screenshots of any URL storage = Crawlbase::StorageAPI.new(**token) # Cloud Storage CRUD # Push high-volume async jobs to the Enterprise Crawler via the Crawling API: # api.get(url, async: true, callback: '...', crawler: 'YourCrawler'). # See /docs/crawler for the queue workflow. ``` ## Common patterns ### JavaScript rendering For SPAs, lazy-loaded feeds, and pages where the initial HTML is empty, instantiate with the JavaScript token and pass any combination of `page_wait`, `ajax_wait`, `scroll`, and `css_click_selector`. Order to think about: a fixed wait, then network-idle, then scroll for lazy-load, then click for any gating UI element. ``` api = Crawlbase::API.new(token: 'YOUR_JS_TOKEN') res = api.get('https://spa.example.com', page_wait: 2000, ajax_wait: true, scroll: true) ``` ### Use a built-in scraper Skip the parser entirely on supported sites. Pass `scraper: 'NAME'` and the response body becomes a JSON string with the structured fields documented on the per-scraper page. ``` require 'crawlbase' require 'json' api = Crawlbase::ScraperAPI.new(token: 'YOUR_TOKEN') res = api.get('https://www.amazon.com/dp/1098145356', scraper: 'amazon-product-details') data = JSON.parse(res.body) puts data['name'], data['price'] ``` ### Geo-routing Pass `country: 'ISO'` to route the crawl through that country's exit nodes. Use it any time the target serves localized content based on IP. ``` api = Crawlbase::API.new(token: 'YOUR_TOKEN') # Hit the German Amazon catalog from a German residential IP res = api.get('https://www.amazon.com/dp/1098145356', country: 'DE') ``` ### Retry with backoff The recommended retry shape: exponential backoff capped at 3-5 attempts, retry on transient errors only (5xx or empty body), don't retry on 4xx. ``` require 'crawlbase' api = Crawlbase::API.new(token: 'YOUR_TOKEN') def crawl(api, url, attempts: 5) attempts.times do |i| res = api.get(url) return res if res.status_code == 200 && res.cb_status.to_i == 200 raise "client error: %d" % res.status_code if (400..499).include?(res.status_code) sleep(rand * (2**i)) # exponential backoff with jitter end raise "Failed: %s" % url end ``` ### Async crawls + webhooks Fire-and-forget mode. The gem call returns immediately with an `rid`; Crawlbase POSTs the result to your callback URL when the page is ready. Useful for batch jobs and slow targets. ``` api = Crawlbase::API.new(token: 'YOUR_TOKEN') res = api.get('https://example.com', async: true, callback: 'https://your-app.com/webhook') rid = res.rid # correlate the eventual webhook delivery # Your Rails / Sinatra webhook receives a POST with: # { rid, url, original_status, cb_status, body } ``` For very high volumes (millions of URLs), use the [Enterprise Crawler](/docs/crawler) which sits in front of this same async pipeline. ### Sticky sessions Some flows need the same residential IP across multiple calls. Pass `cookies_session` with a stable identifier and Crawlbase reuses the same exit node for ~30 minutes. ``` api = Crawlbase::API.new(token: 'YOUR_JS_TOKEN') session = "checkout-#{user_id}" api.get('https://shop.example.com/cart', cookies_session: session) api.get('https://shop.example.com/checkout', cookies_session: session) api.get('https://shop.example.com/confirm', cookies_session: session) ``` ## Errors & retries The platform surfaces two status codes on every response: the gem's own `.status_code` (HTTP status of the request to Crawlbase itself) and `.cb_status` (Crawlbase's verdict on the target - see the [Crawling API errors table](/docs/crawling-api#errors) for the full list). Always branch on `.cb_status` when deciding whether to retry - a target can return `200` with empty body, in which case `.status_code` is `200` but `.cb_status` is `520`. ``` res = api.get(url) pc = res.cb_status.to_i case pc when 200 use(res.body) when 520, 525 # 520 = empty body, 525 = anti-bot couldn't be solved. # Switch to JS token and retry. retry_with_js_token(url) when 521, 522, 523 # Target unreachable or timed out. Retry with backoff. schedule_retry(url) else Rails.logger.error('crawl failed', url: url, cb_status: pc) end ``` All retries against the platform are free - only successful responses (`cb_status: 200`) count against your quota. ## Performance & best practices - **Reuse a single client per token.** The constructor is cheap but each instance opens its own connection. Build it once at app boot (Rails initializer is the natural spot), share it across requests. - **Use the cheapest token that works.** Don't default to the JavaScript token "just in case" - Normal-token requests are faster and use less concurrency. Promote to JS only when the Normal response is empty or anti-bot-blocked. - **Prefer `ajax_wait` over `page_wait`.** Fixed delays burn concurrency on every request, even fast ones. - **For batch jobs: async + webhook, or push to the Enterprise Crawler.** Sidekiq workers calling the gem synchronously will saturate your concurrency cap; async + webhook releases the slot the moment a request is queued. - **Watch the `remaining` response header.** It carries the number of concurrency slots you have left - back off proactively before hitting the cap rather than reacting to 429s. ## Method reference All client classes share the same surface. Constructor takes keyword arguments; verbs mirror the underlying HTTP methods. Crawlbase::API.new(token:, timeout:) constructor Initialize a client with your token. Optional: `timeout` in seconds (default `90`). #get(url, \*\*options) method Send a GET. `options` maps any [Crawling API parameter](/docs/crawling-api) to its value. Returns a response object. #post(url, data, \*\*options) method Send a POST. `data` is the body - pass a hash for form-encoded, a string for raw. Response shape - methods on the response object: .status\_code Integer HTTP status of the gem's request to Crawlbase. .cb\_status Integer Crawlbase verdict on the target. Branch on this for retry decisions. Lifted from the `cb_status` (or legacy `pc_status`) response header; `pc_status` is retained as a deprecated alias. .original\_status Integer HTTP status the target returned to Crawlbase. .url String Final URL after target-side redirects. .body String Page content (or JSON string when `format=json` / `scraper=` was used). .headers Hash Response headers from the target site. .rid String Request ID (when `async: true` or `store: true`). [← PreviousNode.js](/docs/sdk-node)[Next →PHP](/docs/sdk-php) --- Source: https://crawlbase.com/docs/sdks # Official SDKs Native client libraries for the seven languages developers actually ship in. Same interface across all of them - install, authenticate, call. Each SDK wraps the Crawling, Scraper, Leads, and Screenshots APIs (plus Cloud Storage where the host language supports it) so you get one dependency for the whole platform. What the SDKs give you The SDKs are thin wrappers that handle the request-shaping (URL encoding, parameter validation, response parsing, retry helpers) so your application code reads like product code instead of HTTP plumbing. Every SDK exposes the same set of clients - [Crawling API](/docs/crawling-api), Scraper API, Leads API, Screenshots API (plus Cloud Storage on Python / Ruby / PHP / .NET) - and the API surface mirrors the underlying parameters one-to-one. If a parameter is documented on the API page, it works in every SDK. The Enterprise Crawler is reached through the Crawling API itself by passing `async` + `callback` + `crawler` options; there's no separate Crawler client class. ## Pick your language Each language has its own page with install instructions, authentication, multi-API examples, and the method reference. ![Python logo](/assets/images/python-logo-round-6275a9b1c7.png) [Python](/docs/sdk-python) Most popular SDK. Install with `pip install crawlbase`. Works on Python 3.9+ and ships with async helpers. Learn more ![Node.js logo](/assets/images/node-js-logo-round-dcd8bf0db6.png) [Node.js](/docs/sdk-node) Install with `npm install crawlbase`. ESM and CommonJS supported. Promise-based across every API. Learn more ![Ruby logo](/assets/images/ruby-logo-round-42f0935ab8.png) [Ruby](/docs/sdk-ruby) Dependency-free gem for scraping and crawling with the Crawlbase APIs. Idiomatic Ruby; supports 2.7+ and JRuby. Learn more ![PHP logo](/assets/images/php-logo-round-8d5ebe870d.png) [PHP](/docs/sdk-php) Lightweight PSR-compatible class package. Install via `composer require crawlbase/crawlbase`. PHP 7.4+. Learn more Go [Golang](/docs/sdk-go) Idiomatic, context-aware client. `go get github.com/crawlbase/crawlbase-go` on Go 1.18+. Learn more ![Java logo](/assets/images/java-logo-round-67f69502de.png) [Java](/docs/sdk-java) Maven / Gradle artifact `com.crawlbase:crawlbase`. JDK 11+, Jakarta-compatible across every API. Learn more ![.NET logo](/assets/images/dot-net-logo-round-31464a3974.png) [C# / .NET](/docs/sdk-csharp) NuGet package `CrawlbaseAPI`. .NET 6+ supported, async/await throughout. Install with `dotnet add package CrawlbaseAPI`. Learn more ## More Other ways to integrate when one of the official SDKs isn't the right fit. [Other languages](/docs/crawling-api) No SDK for your stack? Hit the Crawling API directly over HTTP - every SDK is doing exactly that under the hood. Use the API [Missing one?](/contact) Built an SDK for a language we don't cover, or want to collaborate on one? Reach out and we'll get you set up. Contact us ## Which SDK should I use? Use the SDK that matches your project's primary language - that's almost always the right answer. The interfaces are the same shape across languages, so picking one over another is purely about ecosystem fit (your dependency manager, your runtime, your existing types). If your stack isn't listed, you can use the [Crawling API](/docs/crawling-api) directly over HTTP - every SDK is doing exactly that under the hood. The [API Playground](/docs/api-playground) generates raw curl/HTTP examples you can port to any client. ## Open source All SDKs are open source on GitHub at [github.com/crawlbase](https://github.com/crawlbase). Issues, PRs, and feature requests welcome - most user-reported gaps in the SDKs are fixed within a release cycle. [← PreviousGeneric Extractors](/docs/scrapers/generic)[Next →Python](/docs/sdk-python) --- Source: https://crawlbase.com/docs/smart-proxy # Smart AI Proxy A single rotating proxy endpoint. Configure it once in your HTTP client and every request you make gets routed through Crawlbase's network - no API rewrites, no special SDK. ## Endpoint HTTPSsmartproxy.crawlbase.com:8013 HTTPsmartproxy.crawlbase.com:8012 - Prefer the HTTPS proxy on port `8013` (recommended). The HTTP proxy on port `8012` is available for clients that only speak HTTP to upstream proxies. - Authenticate with your token as the username; leave the password blank. - Both ports work with any target URL, HTTP or HTTPS. ## Quickstart Set Smart AI Proxy as the proxy in your HTTP client. That's the entire setup. Disable TLS verification Smart AI Proxy intercepts TLS connections to add proxy headers. Your client will see Crawlbase's certificate instead of the target's, so set `verify=False` / `InsecureSkipVerify: true` / equivalent. The connection from Crawlbase to the target site is still verified. ``` curl -x 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013' \ -k 'https://httpbin.org/ip' ``` ``` import requests proxies = { 'http': 'http://YOUR_TOKEN:@smartproxy.crawlbase.com:8012', 'https': 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013', } res = requests.get('https://httpbin.org/ip', proxies=proxies, verify=False) print(res.text) ``` ``` const { HttpsProxyAgent } = require('https-proxy-agent'); const agent = new HttpsProxyAgent( 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013' ); const res = await fetch('https://httpbin.org/ip', { agent }); console.log(await res.text()); ``` ``` require 'net/http' uri = URI('https://httpbin.org/ip') proxy = Net::HTTP::Proxy('smartproxy.crawlbase.com', 8013, 'YOUR_TOKEN', '') http = proxy.new(uri.host, uri.port) http.use_ssl = true http.verify_mode = OpenSSL::SSL::VERIFY_NONE puts http.get(uri.request_uri).body ``` ``` package main import ( "crypto/tls" "fmt" "io" "net/http" "net/url" ) func main() { proxyURL, _ := url.Parse("https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013") client := &http.Client{Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, }} res, _ := client.Get("https://httpbin.org/ip") body, _ := io.ReadAll(res.Body) fmt.Println(string(body)) } ``` ## POST requests Smart AI Proxy forwards POST requests to the target like any other HTTP method. Set the proxy on your client and `POST` as you normally would - the proxy preserves your method, headers, and body. Examples below cover the two body shapes most clients use: form-encoded and JSON. ### Form-encoded body ``` # HTTPS proxy on :8013 (use http:// + :8012 for HTTP-only clients) curl -X POST \ -H 'Content-Type: application/x-www-form-urlencoded' \ -F 'param=value' \ -x 'https://YOUR_TOKEN@smartproxy.crawlbase.com:8013' \ -k 'https://httpbin.org/anything' ``` ``` import requests from urllib3.exceptions import InsecureRequestWarning requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning) proxies = { 'http': 'http://YOUR_TOKEN:@smartproxy.crawlbase.com:8012', 'https': 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013', } res = requests.post( 'https://httpbin.org/anything', data={'param': 'value'}, proxies=proxies, verify=False, ) print(res.status_code, res.text) ``` ``` const { HttpsProxyAgent } = require('https-proxy-agent'); const querystring = require('querystring'); const agent = new HttpsProxyAgent( 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013' ); const res = await fetch('https://httpbin.org/anything', { method: 'POST', headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, body: querystring.stringify({ param: 'value' }), agent, }); console.log(res.status, await res.text()); ``` ``` require 'net/http' require 'openssl' require 'uri' uri = URI('https://httpbin.org/anything') proxy = Net::HTTP::Proxy('smartproxy.crawlbase.com', 8013, 'YOUR_TOKEN', '') http = proxy.new(uri.host, uri.port) http.use_ssl = true http.verify_mode = OpenSSL::SSL::VERIFY_NONE req = Net::HTTP::Post.new(uri.request_uri) req.set_form_data('param' => 'value') res = http.request(req) puts res.code, res.body ``` ``` package main import ( "crypto/tls" "fmt" "io" "net/http" "net/url" "strings" ) func main() { proxyURL, _ := url.Parse("https://YOUR_TOKEN@smartproxy.crawlbase.com:8013") client := &http.Client{Transport: &http.Transport{ Proxy: http.ProxyURL(proxyURL), TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, }} data := url.Values{} data.Set("param", "value") req, _ := http.NewRequest("POST", "https://httpbin.org/anything", strings.NewReader(data.Encode())) req.Header.Set("Content-Type", "application/x-www-form-urlencoded") res, _ := client.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res.Status, string(body)) } ``` ### JSON body ``` curl -X POST \ -H 'Content-Type: application/json' \ --data '{"key1":"value1","key2":"value2"}' \ -x 'https://YOUR_TOKEN@smartproxy.crawlbase.com:8013' \ -k 'https://httpbin.org/anything' ``` ``` import requests proxies = { 'http': 'http://YOUR_TOKEN:@smartproxy.crawlbase.com:8012', 'https': 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013', } res = requests.post( 'https://httpbin.org/anything', json={'key1': 'value1', 'key2': 'value2'}, proxies=proxies, verify=False, ) print(res.status_code, res.text) ``` ``` const { HttpsProxyAgent } = require('https-proxy-agent'); const agent = new HttpsProxyAgent( 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013' ); const res = await fetch('https://httpbin.org/anything', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ key1: 'value1', key2: 'value2' }), agent, }); console.log(res.status, await res.text()); ``` ``` require 'net/http' require 'json' require 'openssl' require 'uri' uri = URI('https://httpbin.org/anything') proxy = Net::HTTP::Proxy('smartproxy.crawlbase.com', 8013, 'YOUR_TOKEN', '') http = proxy.new(uri.host, uri.port) http.use_ssl = true http.verify_mode = OpenSSL::SSL::VERIFY_NONE req = Net::HTTP::Post.new(uri.request_uri, 'Content-Type' => 'application/json') req.body = { key1: 'value1', key2: 'value2' }.to_json puts http.request(req).body ``` ## Forwarding headers and cookies Smart AI Proxy passes most of the headers and cookies on your outgoing request through to the target, so existing clients keep working without modification. Two notable behaviors: - Your `User-Agent` is forwarded as-is. Send a blank one and the proxy rotates a realistic UA for you. - Hop-by-hop and proxy-control headers (`Host`, `Proxy-Authorization`) are stripped - they describe the proxy itself, not the request being forwarded. ``` curl -H 'Accept-Language: en-US,en;q=0.9' \ -H 'X-Custom-Header: My-Custom-Value' \ -H 'User-Agent: MyCustomBrowser/1.0' \ --cookie 'sid=abc123; cart=xyz789' \ -x 'https://YOUR_TOKEN@smartproxy.crawlbase.com:8013' \ -k 'https://httpbin.org/anything' ``` The example above arrives at the target with all four custom headers and both cookies intact. To override proxy behavior (country, device, session, JS rendering, scrapers, etc.) use the [CrawlbaseAPI-\* headers](#control-headers) instead - those are interpreted by the proxy and never reach the target. ## Headless browser rendering Smart AI Proxy is backed by the same headless browser fleet as the [Crawling API](/docs/crawling-api). To execute JavaScript, capture client-rendered SPAs, or apply Crawling API features that require a real browser (screenshots, scroll, click-selectors, autoparse), pass `CrawlbaseAPI-Parameters: javascript=true` as a header on your outgoing request. ``` # Render with a headless browser, force a 2s wait, scroll to load lazy content curl -H 'CrawlbaseAPI-Parameters: javascript=true&page_wait=2000&scroll=true' \ -x 'https://YOUR_TOKEN@smartproxy.crawlbase.com:8013' \ -k 'https://spa.example.com/feed' ``` Authenticate with your **Normal token** (the Smart AI Proxy token from your dashboard), not the JavaScript token - Smart AI Proxy rejects the JavaScript token with `401 "Your private token is required!"`. JavaScript rendering through Smart AI Proxy is available on the **Premium** plan only. The full set of browser-tier parameters (`page_wait`, `scroll`, `css_click_selector`, `wait_for`, screenshots) is reachable through `CrawlbaseAPI-Parameters`; see the [JavaScript parameters reference](/docs/crawling-api#request-params-js) for the canonical list. ## When to use Smart AI Proxy vs the Crawling API Smart AI Proxy and the [Crawling API](/docs/crawling-api) run on the same network and expose the same feature surface - JS rendering, anti-bot bypass, country routing, device emulation, sessions, scrapers, async + storage, all of it. The choice between them isn't about capability; it's about **interface shape** , **which subscription you hold** , and **what concurrency tier** that subscription provides. | Pick Smart AI Proxy when… | Pick the Crawling API (REST) when… | | --- | --- | | You can't change client code (third-party tool, browser extension, Scrapy, an existing scraper) | You're building from scratch and want explicit per-request control | | You'd rather configure a proxy once than rewrite every request to a new endpoint | You'd rather see the URL and parameters in plain GET form for logging / debugging | | Your subscription is on the Smart AI Proxy plan, with its own thread / concurrency tier | Your subscription is on the Crawling API plan, with its own monthly quota and concurrency budget | | You want to drop Crawlbase in front of an existing pipeline with zero code changes | You want one of the SDKs to handle retries, async polling, and response parsing for you | All Crawling API parameters are reachable from Smart AI Proxy via the `CrawlbaseAPI-Parameters` header (see below). The capability surface is the same - pick the lane your subscription and integration shape favor. ## Control headers Pass custom headers prefixed with `CrawlbaseAPI-` on your outgoing request to control proxy behavior. The three single-purpose headers below are convenience shortcuts; the full Crawling API parameter set is reachable via `CrawlbaseAPI-Parameters` (documented after the table). CrawlbaseAPI-Country ISO 3166optional Force a specific country: `US`, `GB`, `DE`, etc. CrawlbaseAPI-Device desktop | mobiledesktop Emulate device class. CrawlbaseAPI-Session-Id stringoptional Pin a session to the same exit IP. Useful for multi-step flows that need a stable identity. Sessions live for ~30 minutes. CrawlbaseAPI-Parameters query stringoptional The full [Crawling API parameter set](/docs/crawling-api#parameters) passed as a single ampersand-joined string. Anything you'd append to a REST request - `javascript=true`, `page_wait=2000`, `scroll=true`, `store=true`, `&scraper=amazon-product-details`, `autoparse=true`: works here. Combine multiple with `&`: e.g. `"javascript=true&country=US&store=true"`. ### Using CrawlbaseAPI-Parameters The single-purpose headers above (Country, Device, Session-Id) are shortcuts for the most common controls. Anything else from the Crawling API parameter set - JS rendering, scroll, click selectors, scrapers, async + webhooks + storage, get\_cookies, get\_headers - is reachable via the `CrawlbaseAPI-Parameters` header. The format is the same query-string you'd append to a REST call: ``` # JS-rendered SPA, store the result, force US geo curl -x 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013' \ -H 'CrawlbaseAPI-Parameters: javascript=true&country=US&store=true&page_wait=2000' \ -k 'https://spa.example.com/feed' # Apply a scraper - same as &scraper=… on the REST endpoint curl -x 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013' \ -H 'CrawlbaseAPI-Parameters: scraper=amazon-product-details' \ -k 'https://www.amazon.com/dp/B0CHX2XFLN' ``` Conflict resolution: if you pass both a single-purpose header (e.g. `CrawlbaseAPI-Country: GB`) and the same field inside `CrawlbaseAPI-Parameters`, the single-purpose header wins. Pick one style per request to keep behavior predictable. ``` # Pin to a US session for a multi-step checkout flow curl -x 'https://YOUR_TOKEN:@smartproxy.crawlbase.com:8013' \ -H 'CrawlbaseAPI-Country: US' \ -H 'CrawlbaseAPI-Session-Id: checkout-user-42' \ -k 'https://shop.example.com/cart' ``` ## Errors Smart AI Proxy returns standard HTTP responses. Status codes follow the same model as the Crawling API. Auth errors (`401`, `402`) are returned by the proxy itself; site errors (`404`, `500`, etc.) come from the target. [← PreviousEnterprise Crawler](/docs/crawler)[Next →Cloud Storage](/docs/cloud-storage) --- Source: https://crawlbase.com/docs/status-codes # Status Codes Crawlbase returns two status signals on every response: the standard HTTP status, and a `cb_status` header describing what Crawlbase did. Here's what each combination means. ## Two statuses, two questions Most HTTP APIs give you a single status code. Crawlbase gives you two because crawling involves two layers - Crawlbase's infrastructure, and the target site behind it. HTTP status int The status of **your request to Crawlbase**. `200` means we processed it; `4xx/5xx` means we couldn't. cb\_status intheader The status of **Crawlbase's request to the target site**. `200` means we got a clean page; other codes describe what went wrong upstream. original\_status intheader The raw HTTP status the target site returned. Useful when the site itself returns a non-200 you need to handle (404, 403, etc.). The mental model Always check HTTP status first. If it's 200, then check `cb_status`. If _that's_ 200, then check `original_status` for site-side errors. ## HTTP status codes What Crawlbase itself returned to your client. | Code | Meaning | Action | | --- | --- | --- | | `200` | Request processed. Check `cb_status` for outcome. | Continue to `cb_status` | | `401` | Token missing or invalid. | Verify token; check it hasn't been reset | | `402` | Out of credits or trial expired. | Top up account | | `403` | Token doesn't have access to this product. | Use the right token type (Normal vs JS) | | `422` | Malformed request - usually missing or unencoded URL. | URL-encode the `url` parameter | | `429` | Concurrency limit reached. | Back off and retry; see [Rate Limits](/docs/rate-limits) | | `500` | Crawlbase internal error. Rare and transient. | Retry with backoff; [check status page](https://status.crawlbase.com) | | `503` | Service temporarily unavailable. | Retry with backoff | ## cb\_status codes What happened during the actual crawl. Returned as the `cb_status` response header on every `200`-OK request. Formerly named `pc_status`. ### Success | Code | Meaning | | --- | --- | | `200` | Page crawled successfully. Body is the target page's HTML or JSON. | | `201` | Async request accepted. Result will be delivered to your webhook or stored under the `rid`. | ### Target site responded with an error Crawlbase reached the site, but the site itself returned a non-2xx. The body contains whatever the site sent back. | Code | Meaning | | --- | --- | | `404` | Target page does not exist. | | `410` | Target page has been permanently removed. | | `451` | Page blocked for legal reasons in the target geography. | ### Blocked or filtered | Code | Meaning | What to try | | --- | --- | --- | | `520` | Target site returned an empty or invalid response. | Retry; switch to JS token if not already | | `521` | Target site refused the connection. | Check URL is correct; site may be down | | `522` | Crawlbase couldn't reach the target site (timeout). | Retry; consider `page_wait` tuning | | `523` | Target site sent a TLS handshake error. | Site may have certificate issues; report to support | | `525` | Bot challenge couldn't be solved automatically. | Switch to JS token; some sites may need custom handling | | `599` | Generic upstream failure. | Retry with backoff | ## Reading the response ``` curl -i 'https://api.crawlbase.com/?token=YOUR_TOKEN&url=https%3A%2F%2Fexample.com' # HTTP/1.1 200 OK # cb_status: 200 # original_status: 200 # url: https://example.com/ # content-type: text/html ``` ``` from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) res = api.get('https://example.com') # Layer 1: did Crawlbase accept the request? if res['status_code'] != 200: raise RuntimeError(f"Crawlbase: {res['status_code']}") # Layer 2: did Crawlbase succeed in fetching the page? if res['cb_status'] != 200: raise RuntimeError(f"Crawl failed: {res['cb_status']}") # Layer 3: did the target site return content? if res['original_status'] != 200: print(f"Site returned {res['original_status']}") print(res['body']) ``` ## Next steps [Error Handling](/docs/errors) Patterns for retries, dead-letter queues, and observability. [Rate Limits](/docs/rate-limits) Specific guidance on 429s and concurrency. [← PreviousRate Limits](/docs/rate-limits)[Next →Error Handling](/docs/errors) --- Source: https://crawlbase.com/docs/support # Support Stuck? We're a small team that responds fast. Pick the channel that fits - most issues get resolved within a few hours. ## Channels [Email](mailto:support@crawlbase.com) support@crawlbase.com - best for account-specific issues, billing, and bug reports. [GitHub](https://github.com/crawlbase) SDK bugs, MCP server issues, feature requests. Public and trackable. [Live chat](mailto:support@crawlbase.com) Opens the chat widget. Business hours response, fastest for time-sensitive issues. [Status page](https://status.crawlbase.com) Real-time uptime and incident history. Subscribe to alerts here. ## Before you write us The fastest support tickets are the ones that include enough context for us to reproduce the issue without asking. Please include: - **The full request URL** (with token redacted to `YOUR_TOKEN`) and any headers you set. - **The full response** : body, status code, and the `cb_status` / `original_status` headers. - **The `rid`** if one was returned. Lets us trace the request through our logs immediately. - **What you expected to happen** versus what actually happened. - **Whether it's reproducible** : every time, intermittent, or one-off. ## Support priority by product Priority follows the product and its billing model, not a single account plan. Email is available for every product; live chat is available on paid subscriptions. | Product | Billing | Support priority | | --- | --- | --- | | Crawling API | Pay as you go | High, prioritized | | Crawler | Pay as you go | High, prioritized | | Smart Proxy | Subscription | By plan tier | | Cloud Storage | Subscription | By plan tier | ## What we can't help with - **Custom scraper requests with no volume.** If you need a one-off page parsed, write your own selector. We build new [scrapers](/docs/scraper-api) when there's broader demand. - **Legal interpretation.** We can't advise on whether your specific use case is compliant with GDPR / CCPA / target site ToS - talk to your lawyer. [← PreviousChangelog](/docs/changelog)[Next →Legacy APIs](/docs/legacy) --- Source: https://crawlbase.com/docs/user-agents-api # User Agents API Free, rate-limited endpoint that returns a random User-Agent string optimized for web crawling. Use it when you're running your own crawler outside the Crawlbase pipeline. Free to use The User Agents API is completely free to use, regardless of your Crawlbase plan. Rate-limited to 1 request per second. ## Overview Returns a random User-Agent string optimized for web crawling. The selection logic favors UAs that look natural in production traffic - modern Chrome, Firefox, and Safari builds across desktop and mobile, weighted to match real-world usage. **Endpoint:** `https://api.crawlbase.com/user_agents` ## Quickstart ``` curl 'https://api.crawlbase.com/user_agents?token=YOUR_TOKEN' ``` Returns a single User-Agent string. Use it directly in the `User-Agent` header of your outbound HTTP request. ## Parameters token stringrequired Your Crawlbase token. The endpoint is free, but the token is needed for rate-limit accounting. device desktop | tablet | mobiledesktop Filter to a device class. Useful when your scraper needs realistic UAs for a specific platform mix. size integer1 Number of UAs to return per call. Maximum 10. Combine with `device` to pre-filter, then cache the batch client-side and rotate locally - that's both faster and avoids the 1 req/s rate limit. ## Rate limit 1 request per second. Cache UA strings client-side and rotate them yourself rather than calling on every request - that's both faster and avoids hitting the limit. ## When to use this You only need this if you're running your own crawler against websites that don't go through Crawlbase. The [Crawling API](/docs/crawling-api) and [Smart AI Proxy](/docs/smart-proxy) already handle UA randomization automatically - you don't need to set `User-Agent` on requests through them. [← PreviousAPI Playground](/docs/api-playground)[Next →Changelog](/docs/changelog)