Web crawling

Overview

Crawl API provides methods to see the crawl result of a website. Result of this function is the HTML source code of webpage, screenshot of webpage, and page loading metric, under JSON format.

Parameters

ParameterDescription
ip or host or urlIP address or string IP address or hostname of the remote host.
devicemobile for simulating mobile device, PC mode otherwise.
widthwidth=1270 for example. This parameter is just used when device is not mobile
heightheight=1270 for example. This parameter is just used when device is not mobile

Example:

curl -X POST -d '{"url":"http://scaleflex.com", "key":"API_KEY"}' http://eu-west.probe.onecheck.io/crawl
curl -X POST -d "url=http://scaleflex.com&key=API_KEY" http://eu-west.probe.onecheck.io/crawl
curl -X POST -d "url=http://scaleflex.com&key=API_KEY&device=mobile" http://eu-west.probe.onecheck.io/crawl
curl -X POST -d "url=http://scaleflex.com&key=API_KEY&width=2460&height=1120" http://eu-west.probe.onecheck.io/crawl

Exception

This function is just run on any zone having high performance servers. An error string would be returned when running on other zones: NO_HIGHPERF_UPSTREAM.