crawlbase文档
登录

API 用法

Crawling API 请求中添加 &scraper=stackexchange-thread。在 url 参数中对目标 URL 进行 URL 编码。

curl 'https://api.crawlbase.com/?token=YOUR_TOKEN' \
  --data-urlencode 'url=https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file' \
  --data-urlencode 'scraper=stackexchange-thread' -G
from crawlbase import CrawlingAPI

api = CrawlingAPI({'token': 'YOUR_TOKEN'})
res = api.get(
    'https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file',
    {'scraper': 'stackexchange-thread'}
)

import json
data = json.loads(res['body'])
const { CrawlingAPI } = require('crawlbase');
const api = new CrawlingAPI({ token: 'YOUR_TOKEN' });

const res = await api.get(
  'https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file',
  { scraper: 'stackexchange-thread' }
);
const data = JSON.parse(res.body);
require 'crawlbase'
api = Crawlbase::API.new(token: 'YOUR_TOKEN')

res = api.get('https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file', scraper: 'stackexchange-thread')
data = JSON.parse(res.body)

示例输入 URL

url 参数支持任意 Stack Exchange 问题 URL,来自网络中的任意站点。例如:

https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file
https://superuser.com/questions/441895/automate-opening-html-and-printing-to-pdf
https://askubuntu.com/questions/1206658/why-is-apt-held-back
https://serverfault.com/questions/439471/which-openvpn-cipher-should-i-use
https://mathoverflow.net/questions/44265/is-the-riemann-hypothesis

响应结构

JSON 响应体。当源页面省略对应值时,字段类型可能为 null

question
object
问题本身。
question.id
string
Stack Exchange 问题 id。
question.title
string
问题标题。
question.url
string
问题的规范 URL。
question.body
string
问题正文文本。
question.score
integer
问题得分(顶数减去踩数)。
question.viewCount
integer
问题的浏览数量。
question.tags
array
应用于问题的标签。
question.askedAt
string
问题创建时间(ISO 8601)。
question.author
object
问题的作者。
question.author.name
string
问题作者的显示名称。
question.author.url
string
问题作者的个人资料 URL。
question.author.reputation
integer
问题作者的声望分数。
question.comments
array
问题上的评论。
question.comments[].id
string
评论 id。
question.comments[].score
integer
评论得分。
question.comments[].body
string
评论文本。
question.comments[].author
string
评论作者的显示名称。
question.comments[].authorUrl
string
评论作者的个人资料 URL。
question.comments[].createdAt
string
评论创建时间(ISO 8601)。
answerCount
integer
answers 中返回的回答数量。
answers
array
问题的回答,按列表顺序排列。
answers[].id
string
回答 id。
answers[].score
integer
回答得分(顶数减去踩数)。
answers[].isAccepted
boolean
当这是已采纳的回答时为 true。
answers[].body
string
回答正文文本。
answers[].author
object
回答的作者。
answers[].author.name
string
回答作者的显示名称。
answers[].author.url
string
回答作者的个人资料 URL。
answers[].author.reputation
integer
回答作者的声望分数。
answers[].createdAt
string
回答创建时间(ISO 8601)。
answers[].comments
array
回答上的评论。
answers[].comments[].id
string
评论 id。
answers[].comments[].score
integer
评论得分。
answers[].comments[].body
string
评论文本。
answers[].comments[].author
string
评论作者的显示名称。
answers[].comments[].authorUrl
string
评论作者的个人资料 URL。
answers[].comments[].createdAt
string
评论创建时间(ISO 8601)。

示例响应

{
  "question": {
    "id": "2861071",
    "title": "How to modify a text file?",
    "url": "https://stackoverflow.com/questions/2861071/how-to-modify-a-text-file",
    "body": "I'm using Python and I need to insert a line at the start of a text file without loading the whole file into memory. Is there a way to do this in place, or do I have to rewrite the file?",
    "score": 312,
    "viewCount": 486201,
    "tags": ["python", "file", "text-files"],
    "askedAt": "2010-05-18T20:11:33Z",
    "author": {
      "name": "Nathan Fellman",
      "url": "https://stackoverflow.com/users/8460/nathan-fellman",
      "reputation": 127843
    },
    "comments": [
      {
        "id": "2954120",
        "score": 3,
        "body": "Do you need to preserve the original file, or is rewriting acceptable?",
        "author": "Greg Hewgill",
        "authorUrl": "https://stackoverflow.com/users/893/greg-hewgill",
        "createdAt": "2010-05-18T20:19:04Z"
      }
    ]
  },
  "answerCount": 2,
  "answers": [
    {
      "id": "2861108",
      "score": 401,
      "isAccepted": true,
      "body": "You cannot insert into the middle of a file without rewriting it. Read the file into a list of lines, insert your new line, then write it all back:\n\n    with open('file.txt') as f:\n        lines = f.readlines()\n    lines.insert(0, 'new first line\\n')\n    with open('file.txt', 'w') as f:\n        f.writelines(lines)\n",
      "author": {
        "name": "Roberto Bonvallet",
        "url": "https://stackoverflow.com/users/193568/roberto-bonvallet",
        "reputation": 30215
      },
      "createdAt": "2010-05-18T20:15:52Z",
      "comments": [
        {
          "id": "2954260",
          "score": 12,
          "body": "For very large files, stream through a temporary file instead of holding everything in memory.",
          "author": "John Machin",
          "authorUrl": "https://stackoverflow.com/users/253537/john-machin",
          "createdAt": "2010-05-18T21:02:11Z"
        }
      ]
    },
    {
      "id": "2861180",
      "score": 47,
      "isAccepted": false,
      "body": "If the file is large, use fileinput with inplace=True to edit it line by line without loading it all at once.",
      "author": {
        "name": "codeape",
        "url": "https://stackoverflow.com/users/18770/codeape",
        "reputation": 98412
      },
      "createdAt": "2010-05-18T20:24:39Z",
      "comments": []
    }
  ]
}