WebCrawlerAPI
Site crawling with page scraping, monitored feeds, and structured extraction.
Crawls a site section, extracts the fields worth keeping, and watches it for later changes.
Use WebCrawlerAPI with Snow
Snow is a personal AI assistant that builds real apps for you, and they can put WebCrawlerAPI to work.
What Snow can do with WebCrawlerAPI
These are the WebCrawlerAPI operations Snow and the apps you build with it can run. Sign in to choose which ones Snow may run without asking.
Reads6
Look at your data without changing it.
- Get Crawl Job
Get a crawl job's status, configuration, per-page results, content URLs, errors, and observed costs. Respect recommended_pull_delay_ms when polling.
- Get Feed
Get one feed's configuration, lifecycle status, and recent run history, including per-run crawl counts and cost. Canceled feeds remain directly readable while retained by the provider.
- Get Organization Costs
Return current spendable balance plus request count and USD usage for a date range. The connected API key is used for both organization routes.
- List Feed Changes
Return one page of detected feed changes as structured JSON, with an opaque continuation cursor for older pages.
- List Feeds
List active and paused feeds for the connected organization, newest first. Logically canceled feeds are omitted.
- Scrape Page
Scrape one web page synchronously and return requested content or structured extraction. This consumes metered credits; supplying a prompt adds $0.002 per request.
Creates2
Add something new to your account.
- Create Crawl
Start a metered asynchronous crawl over a site and return its job ID. The crawl may visit and charge for up to items_limit pages; use GET_CRAWL_JOB to monitor it.
- Create Feed
Create a recurring website-change feed. Creation immediately schedules an initial crawl that may consume metered credits, and future scheduled runs may also consume credits; delete the feed when it is no longer needed.
Updates1
Change something that is already there.
- Set Feed State
Pause an active feed's future scheduled runs or resume a paused feed. Pausing does not stop a run that has already started, and resuming schedules future crawls that may consume metered credits.
Deletes1
Remove something. This often cannot be undone.
- Delete Feed
Permanently cancel a feed so it cannot be resumed. The feed is removed from feed listings, but retained history may remain directly readable for 30 days.
Related integrations
Ready to put WebCrawlerAPI to work?
Sign up free, build an app by chatting, and connect WebCrawlerAPI in minutes. No credit card required.