🌐 Web Scraper

Scrape web pages with full JavaScript execution support. Extract raw HTML, plain text, or use CSS selectors to target specific elements. Returns structured data from any modern web application.

Request Body

Parameter	Type	Required	Description	Default
`url`	string	required	URL of the webpage to scrape	—
`selector`	string	optional	CSS selector to extract specific elements	—
`format`	string	optional	Output format: html \| text \| json	`text`

Response Fields

Field	Type	Description
`content`	string	Extracted content in the requested format
`url`	string	Final URL (after redirects)
`title`	string	Page title
`elements_found`	number	Number of elements matched by selector

Code Examples

cURL

bash

curl -X POST https://sparkforge.sh/api/scrape \
  -H "Content-Type: application/json" \
  -H "X-PAYMENT: <x402-payment-token>" \
  -d '{
    "url": "https://news.ycombinator.com",
    "selector": ".titleline",
    "format": "text"
  }'

Python

python

import requests

response = requests.post(
    "https://sparkforge.sh/api/scrape",
    headers={"Content-Type": "application/json", "X-PAYMENT": "<token>"},
    json={"url": "https://news.ycombinator.com", "selector": ".titleline", "format": "text"}
)

data = response.json()
print(f"Found {data['elements_found']} elements")
print(data["content"][:500])

TypeScript / Node.js

typescript

const res = await fetch402("https://sparkforge.sh/api/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://news.ycombinator.com",
    selector: ".titleline",
    format: "text",
  }),
});

const { content, elements_found } = await res.json();

Example Response

json

{
  "content": "Ask HN: How do you handle database migrations?\nShow HN: I built a tool that...",
  "url": "https://news.ycombinator.com",
  "title": "Hacker News",
  "elements_found": 30
}

💳 Payment via x402

This endpoint costs $0.01 per call, paid in USDC on Base, Polygon, or Solana. Use the @x402/fetch library to handle payments automatically, or implement the x402 handshake manually.

bash

# Free demo mode (no payment)
curl https://sparkforge.sh/api/scrape?demo=true

# Or try via MCP (Claude Desktop, Cursor, Windsurf)
# Add to mcp config: { "url": "https://sparkforge.sh/api/mcp" }