Web crawling

Overview

Crawl API provides methods to see the crawl result of a website. Result of this function is the HTML source code of webpage, screenshot of webpage, and page loading metric, under JSON format.

Parameters

Parameter	Description
ip or host or url	IP address or string IP address or hostname of the remote host.
device	mobile for simulating mobile device, PC mode otherwise.
width	width=1270 for example. This parameter is just used when device is not mobile
height	height=1270 for example. This parameter is just used when device is not mobile

Example:

curl -X POST -d '{"url":"http://scaleflex.com", "key":"API_KEY"}' http://eu-west.probe.onecheck.io/crawl
curl -X POST -d "url=http://scaleflex.com&key=API_KEY" http://eu-west.probe.onecheck.io/crawl
curl -X POST -d "url=http://scaleflex.com&key=API_KEY&device=mobile" http://eu-west.probe.onecheck.io/crawl
curl -X POST -d "url=http://scaleflex.com&key=API_KEY&width=2460&height=1120" http://eu-west.probe.onecheck.io/crawl

Exception

This function is just run on any zone having high performance servers. An error string would be returned when running on other zones: NO_HIGHPERF_UPSTREAM.