Web crawling
Overview
Crawl API provides methods to see the crawl result of a website. Result of this function is the HTML source code of webpage, screenshot of webpage, and page loading metric, under JSON format.
Parameters
Parameter | Description |
---|---|
ip or host or url | IP address or string IP address or hostname of the remote host. |
device | mobile for simulating mobile device, PC mode otherwise. |
width | width=1270 for example. This parameter is just used when device is not mobile |
height | height=1270 for example. This parameter is just used when device is not mobile |
Example:
curl -X POST -d '{"url":"http://scaleflex.com", "key":"API_KEY"}' http://eu-west.probe.onecheck.io/crawl
curl -X POST -d "url=http://scaleflex.com&key=API_KEY" http://eu-west.probe.onecheck.io/crawl
curl -X POST -d "url=http://scaleflex.com&key=API_KEY&device=mobile" http://eu-west.probe.onecheck.io/crawl
curl -X POST -d "url=http://scaleflex.com&key=API_KEY&width=2460&height=1120" http://eu-west.probe.onecheck.io/crawl
Exception
This function is just run on any zone having high performance servers. An error string would be returned when running on other zones: NO_HIGHPERF_UPSTREAM.