Cloud Storage · Crawlbase Documentation

端点

GEThttps://api.crawlbase.com/storage

# Two operations: store (write) and retrieve (read).
# Storage is implicit - set store=true on a Crawling API call to write.
# Use this endpoint to read.

存储页面

您不需要调用此端点来存储。只需在任何 Crawling API 调用中添加 store=true，页面就会被自动存储，您将在响应中收到一个 rid。

curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \
  --data-urlencode 'url=https://example.com' \
  --data-urlencode 'store=true' -G

# Response includes rid: a1B2c3D4e5F6
import requests

res = requests.get(
    'https://api.crawlbase.com/',
    params={'token': 'YOUR_TOKEN', 'url': 'https://example.com', 'store': 'true'},
)
print(res.text)
const params = new URLSearchParams({ token: 'YOUR_TOKEN', url: 'https://example.com', store: 'true' });
const res = await fetch(`https://api.crawlbase.com/?${params}`);
console.log(await res.text());
require 'net/http'

uri = URI('https://api.crawlbase.com/')
uri.query = URI.encode_www_form(token: 'YOUR_TOKEN', url: 'https://example.com', store: 'true')
puts Net::HTTP.get(uri)

获取页面

通过 RID

curl 'https://api.crawlbase.com/storage?token=YOUR_TOKEN&rid=a1B2c3D4e5F6'
from crawlbase import StorageAPI

api = StorageAPI({'token': 'YOUR_TOKEN'})
res = api.get(rid='a1B2c3D4e5F6')
print(res['body'])
const { StorageAPI } = require('crawlbase');
const api = new StorageAPI({ token: 'YOUR_TOKEN' });

const res = await api.get({ rid: 'a1B2c3D4e5F6' });
console.log(res.body);

通过 URL

通过原始 URL 查找已存储的页面。返回最新存储的版本。

curl 'https://api.crawlbase.com/storage?token=YOUR_TOKEN&url=https%3A%2F%2Fexample.com'
import requests

res = requests.get(
    'https://api.crawlbase.com/storage',
    params={'token': 'YOUR_TOKEN', 'url': 'https://example.com'},
)
print(res.text)
const params = new URLSearchParams({ token: 'YOUR_TOKEN', url: 'https://example.com' });
const res = await fetch(`https://api.crawlbase.com/storage?${params}`);
console.log(await res.text());
require 'net/http'

uri = URI('https://api.crawlbase.com/storage')
uri.query = URI.encode_www_form(token: 'YOUR_TOKEN', url: 'https://example.com')
puts Net::HTTP.get(uri)

参数

token

string必填

您的 Crawlbase token。

rid

string可选值

页面存储时返回的请求标识符。

url

string可选值

原始 URL。返回最新存储的版本。需进行 URL 编码。

format

html | jsonhtml

响应封装格式。json 会将 body 和元数据一起包裹。

批量获取

通过 RID 在一次往返请求中拉取最多 100 个已存储的页面。POST 一个包含列表的 JSON body，并（可选地）让服务器在返回每个条目时将其删除 - 适用于无需保持存储的「清空队列」式流水线。

POSThttps://api.crawlbase.com/storage/bulk

curl -X POST 'https://api.crawlbase.com/storage/bulk?token=YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{ "rids": ["RID1","RID2","RID3"], "auto_delete": true }'
import requests

res = requests.post(
    'https://api.crawlbase.com/storage/bulk?token=YOUR_TOKEN',
    json={'rids': ['RID1', 'RID2', 'RID3'], 'auto_delete': True},
)
print(res.json())
const res = await fetch('https://api.crawlbase.com/storage/bulk?token=YOUR_TOKEN', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ rids: ['RID1', 'RID2', 'RID3'], auto_delete: true }),
});
console.log(await res.json());
require 'json'
require 'net/http'

res = Net::HTTP.post(URI('https://api.crawlbase.com/storage/bulk?token=YOUR_TOKEN'),
                     { rids: %w[RID1 RID2 RID3], auto_delete: true }.to_json,
                     'Content-Type' => 'application/json')
puts res.body

rids

string[]必填

要获取的 RID 数组。每次请求最多 100 个 - 超过 100 的部分会被静默丢弃。

auto_delete

booleanfalse

当为 true 时，每个成功返回的条目会在同一次调用中从存储中删除。当您正在清空一次性结果队列且无需保留时使用此选项。

响应是一个 JSON 数组，每个返回的 RID 对应一个对象。body 字段经过 base64 编码和 gzip 压缩 - 先 base64 解码再 gzip 解压即可得到原始页面。

[
  {
    "stored_at": "2021-03-01T14:22:58+02:00",
    "original_status": 200,
    "cb_status": 200,
    "rid": "RID1",
    "url": "https://example.com/a",
    "body": "H4sIAAAA…"  // base64(gzip(html))
  },
  {
    "stored_at": "2021-03-01T14:30:51+02:00",
    "original_status": 200,
    "cb_status": 200,
    "rid": "RID2",
    "url": "https://example.com/b",
    "body": "H4sIAAAA…"
  }
]

批量删除

在一次调用中删除最多一组 RID。返回每个 RID 的状态，便于发现那些已经不存在或删除失败的条目。

POSThttps://api.crawlbase.com/storage/bulk_delete

curl -X POST 'https://api.crawlbase.com/storage/bulk_delete?token=YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{ "rids": ["RID1","RID2","RID3"] }'
import requests

res = requests.post(
    'https://api.crawlbase.com/storage/bulk_delete?token=YOUR_TOKEN',
    json={'rids': ['RID1', 'RID2', 'RID3']},
)
print(res.json())
const res = await fetch('https://api.crawlbase.com/storage/bulk_delete?token=YOUR_TOKEN', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ rids: ['RID1', 'RID2', 'RID3'] }),
});
console.log(await res.json());
require 'json'
require 'net/http'

res = Net::HTTP.post(URI('https://api.crawlbase.com/storage/bulk_delete?token=YOUR_TOKEN'),
                     { rids: %w[RID1 RID2 RID3] }.to_json,
                     'Content-Type' => 'application/json')
puts res.body

响应是一个 JSON 数组，每个提交的 RID 对应一个条目。status: true 表示该条目已被删除；status: false 加上 result: "Not Found" 表示该 RID 不存在（已被清理、已过期或从未写入）。

[
  { "rid": "RID1", "result": "Deleted",   "status": true  },
  { "rid": "RID2", "result": "Not Found", "status": false },
  { "rid": "RID3", "result": "Failed",    "status": false }
]

删除不可逆

发送前请仔细核对 RID 列表。没有软删除或撤销机制 - 如果您需要可恢复的工作流，请先使用 auto_delete=false 获取，并在本地持久化保存 body 之后再调用 /bulk_delete。

删除单个页面

通过 RID 从存储中删除一个条目。使用 DELETE /storage，并将 RID 放在查询字符串中。

DELETEhttps://api.crawlbase.com/storage

curl -X DELETE 'https://api.crawlbase.com/storage?token=YOUR_TOKEN&rid=RID'
import requests

res = requests.delete('https://api.crawlbase.com/storage?token=YOUR_TOKEN&rid=RID')
print(res.text)
const res = await fetch('https://api.crawlbase.com/storage?token=YOUR_TOKEN&rid=RID', { method: 'DELETE' });
console.log(await res.text());
require 'net/http'

uri = URI('https://api.crawlbase.com/storage?token=YOUR_TOKEN&rid=RID')
res = Net::HTTP.start(uri.host, uri.port, use_ssl: true) { |http| http.delete(uri.request_uri) }
puts res.body

三种响应形态：

结果	Body
找到并已删除	`{"success": "The Storage item has been deleted successfully"}`
找到但删除失败	`{"error": "The Storage item could not be deleted"}`
不在存储中	`{"error": "Not Found"}`

列出 RID

对您存储区中的 RID 进行分页浏览 - 即清单调用。对于单次响应无法容纳的数据集，请使用基于滚动的分页（scroll=true 会启动一个滚动会话并返回一个 scroll_id，您可以在后续调用中复用它）。

GEThttps://api.crawlbase.com/storage/rids

# First page
curl 'https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&limit=100&scroll=true'

# Next page - replay the scroll_id from the previous response
curl 'https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&scroll_id=dXVlcnlUaGVuRmV0Y2g7…'
import requests

# First page
res = requests.get('https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&limit=100&scroll=true')
print(res.json())

# Next page - replay the scroll_id from the previous response
res = requests.get('https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&scroll_id=dXVlcnlUaGVuRmV0Y2g7…')
print(res.json())
// First page
const first = await fetch('https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&limit=100&scroll=true');
console.log(await first.json());

// Next page - replay the scroll_id from the previous response
const next = await fetch('https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&scroll_id=dXVlcnlUaGVuRmV0Y2g7…');
console.log(await next.json());
require 'net/http'

# First page
puts Net::HTTP.get(URI('https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&limit=100&scroll=true'))

# Next page - replay the scroll_id from the previous response
puts Net::HTTP.get(URI('https://api.crawlbase.com/storage/rids?token=YOUR_TOKEN&scroll_id=dXVlcnlUaGVuRmV0Y2g7…'))

limit

integer可选

每次调用返回的 RID 最大数量。上限为 10000。无默认值 - 请显式设置。

scroll

booleanfalse

当为 true 时，响应中会包含一个 scroll_id，您可用它来获取下一页。不设置则只能获取第一页。

scroll_id

string可选

来自上一次响应的 token。复用它以推进滚动。后续调用中不要传 scroll=true - 仅在首次调用时传入。

scroll_order

asc | descdesc

按存储时间戳对 RID 排序。默认按最新优先排序。

{
  "rids": ["RID1", "RID2", "RID3", "..."],
  "scroll_id": "dXVlcnlUaGVuRmV0Y2g7NTs1NDpDV…"
}

滚动会话会过期

一个 scroll_id 的非活动有效期约为 15 秒。如果您看到 "Scroll session has expired or is invalid"，请使用新的 scroll=true 请求重新开始 - 游标已失效。

总数

单个整数：当前存储区中的页面数量。

GEThttps://api.crawlbase.com/storage/total_count

curl 'https://api.crawlbase.com/storage/total_count?token=YOUR_TOKEN'

# Response
# { "totalCount": 5491078 }
import requests

res = requests.get('https://api.crawlbase.com/storage/total_count?token=YOUR_TOKEN')
print(res.json())
const res = await fetch('https://api.crawlbase.com/storage/total_count?token=YOUR_TOKEN');
console.log(await res.json());
require 'net/http'

puts Net::HTTP.get(URI('https://api.crawlbase.com/storage/total_count?token=YOUR_TOKEN'))

保留期与价格

存储的页面默认保留 30 天。企业版套餐可延长。
每次 store=true 调用都按一次请求计费 - 不另外收费。
检索（即此端点）免费。您可按需读取任意次数。
如果某个页面以 store=true 重新抓取，新版本将替换旧版本。

存储的用武之地

审计追踪（「我们抓取时该页面显示了什么？」）、再处理流水线（在抓取逻辑改进后重新解析已存储的 HTML），以及向读者提供缓存结果而无需重新抓取。