Fetches the resource of a given url and checks if it is available. Only websites using HTTP/HTTPS are allowed. IP addresses and ports other than 80/443 are not permitted and will be ignored.
The API uses the POST request method to send data in the HTTP message body to avoid data being stored in the cache of the web server. A request is valid if all parameters are provided as described below (data type, minimum and maximum length). Please note that all data must be UTF-8 and URL encoded.
| Parameter | Mandatory | Data Type | Min. | Max. | Description |
|---|---|---|---|---|---|
| license | required | string | 29 | 29 | Your license key used for authentication - keep this key secret! |
| guid | required | string | 30 | 40 | Your GUID used for authentication - keep this key secret! |
| requestproof | optional | string | 0 | 1 | set to 1 if you need a unique request identifier as proof for your API request e.g. for VAT checks; default is 0 |
| returnavailablecredits | optional | string | 0 | 1 | set to 1 to get the number of data.mill credits available in your data.mill account; default is 0; a return value of -1 means that you have unlimited credits |
| responseformat | optional | string | 3 | 4 | format of returned data; possible values are xml or json (default) |
| url | required | string | 4 | 2048 | The url to be checked (e.g. any website) |
| max_redirects | optional | string | 0 | 100 | The maximum amount of redirects until the lookup for the root resource will be stopped (default 10) |
The API returns all data in valid JSON. A few programming languages include native support for JSON. For those that don't, you can find a suitable library at http://www.json.org. Valid requests always returns all response keys listed below. Depending on the request parameters some response keys may be empty.
| Name | Data Type | Description |
|---|---|---|
| valid | string | Flag if the root resource (website) is valid or not [0, 1] |
| url | string | The final url of the resource |
| http_code | string | Http Status Code |
| total_time | string | Total time of the request [seconds] |
| namelookup_time | string | Time until host name resolved [seconds] |
| connect_time | string | Time until connection established [seconds] |
| pretransfer_time | string | Time until file transfer began [seconds] |
| starttransfer_time | string | Time to first byte [seconds] |
| primary_ip | string | IP address of the most recent connection |
| primary_port | string | Destination port of the connection |
| download_content_length | string | Number of bytes to download from the resource (-1 means no information available) Deprecated, will be removed in Version 2.0.0 |
| content_type | string | Content type of the requested resource |
| redirects | string | Collection of all urls redirected |
| parameters | string | Collection of all parameters included in the requested url |
If the request fails the API returns a HTTP status code according to the reason. A reason may be a missing mandatory request parameter, invalid parameter credentials (minimum/maximum length) or invalid API keys. The response body is a valid JSON containing detail information about the reason (except for status code 500).
| HTTP Status Code | Reason |
|---|---|
| 401 | Unauthorized The license key and/or GUID key is missing, invalid or inactive. |
| 402 | Payment Required Your /data.mill credit quota is exceeded. You need to get more credits. |
| 403 | Forbidden You need to accept all required agreements of this function to be able to use it. |
| 404 | Not Found The request path in combination with the request method does not exist. |
| 405 | Method Not Allowed The request method is not allowed for the requested path. |
| 412 | Precondition Failed At least one of the given request parameter has invalid credentials (e.g. data type, length). |
| 428 | Precondition Required At least one of the mandatory request parameter is missing or has an empty value. |
| 500 | Internal Server Error An unexpected condition terminated the request. |
| 503 | Service Unavailable The service is currently in maintenance mode and will be alive later. |
The batch mode allows the execution of thousands of records (limited to 5,000) combined in a single request. This API endpoint as described above uses the single mode execution of records. To use the batch mode you need to combine all request parameters above (excluding license and guid) as array, where each array element represents a single record. This array is converted to a valid JSON string and set as value of the new request parameter called batch. The response of the batch mode is a valid JSON which contains the result for each record set in the request. The response keys of each result element are the same as for the single request mode described above.
In order to enable the payment of different currencies, we have introduced a system based on credit balance as payment for the individual functions. The currency of the account is called /data.mill credits. To use the functions, the account must be charged in advance. If the credit price of a function is higher than the current balance, the function can not be executed. The credits will be deducted from your balance after executing the function. Your credits have no expiration date.
We provide some code snippets for multiple programming languages to easily implement this function in your own application. The code snippets below uses the single request mode. If you want to use the batch mode instead of the single mode you need to modify the snippet. Feel free to copy the code snippets for your preferred programming language.
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "https://api.datamill.solutions/url/check");
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($curl, CURLOPT_POST, 7);
curl_setopt($curl, CURLOPT_POSTFIELDS, "license=&guid=&requestproof=&returnavailablecredits=&responseformat=&url=&max_redirects=");
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($curl, CURLOPT_TIMEOUT, 10);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$responseString = curl_exec($curl);
curl_close($curl);
$responseArray = json_decode($responseString, true);
jQuery.ajax({
// set the url to which the request is sent url : "https://api.datamill.solutions/url/check",// set the HTTP method to use for the request type: "POST",// data to be sent to the server (you may use a JavaScript object instead of a string) // TODO: append your API keys and value(s) and don't forget to urlencode() them data : "license=&guid=&requestproof=&returnavailablecredits=&responseformat=&url=&max_redirects=",// type of data that we are expecting back from the server dataType: "JSON",// set a timeout in milliseconds for the request timeout: 10000,// function to be called if the request succeeds success:function(data, textStatus, jqXHR) {// {object} data The data returned from the server, formatted according to the dataType parameter. // {string} textStatus String describing the status. // {object} jqXHR A XMLHttpRequest object describing the response. // TODO: process the data in the 'data' JavaScript object },// function to be called if the request fails error: function(jqXHR, textStatus, errorThrown) {// {object} jqXHR A XMLHttpRequest object describing the response. // {string} textStatus String describing the type of error occurred. // {string} errorThrown Text portion of the HTTP status, such as "Not Found" or "Internal Server Error". // TODO: process the error occurred }
});
using System.IO;
using System.Web;
using System.Net;
byte[] Data = Encoding.ASCII.GetBytes("license=&guid=&requestproof=&returnavailablecredits=&responseformat=&url=&max_redirects=");
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://api.datamill.solutions/url/check");
Request.Method = "POST";
Request.ContentType = "application/x-www-form-urlencoded";
Request.ContentLength = Data.Length;
Request.Timeout = 10000;
try {
using(var Stream = Request.GetRequestStream()) {Stream.Write(Data, 0, Data.Length);}
} catch (Exception ex) {
// TODO: process the exception thrown (e.g. url not found) return;
}
try {
HttpWebResponse ResponseObject = (HttpWebResponse)Request.GetResponse();StreamReader Reader = new StreamReader(ResponseObject.GetResponseStream());// read the whole response as string and store it String ResponseMessage = System.Net.WebUtility.HtmlDecode(Reader.ReadToEnd());// TODO: process the data in 'ResponseMessage'
} catch (WebException ex) {
// error occurred, http status code != 200 (e.g. timeout or parameter invalid) if(ex.Status == WebExceptionStatus.Timeout) {// TODO: process the timeout reached } else {// read the response from the web exception HttpWebResponse ResponseObject = (HttpWebResponse)ex.Response;StreamReader Reader = new StreamReader(ResponseObject.GetResponseStream());// read the whole response as string and store it String ResponseMessage = System.Net.WebUtility.HtmlDecode(Reader.ReadToEnd());// TODO: process the error response received from the server (stored in 'ResponseMessage') }
}
import java.util.List;
import org.apache.http.NameValuePair;
import java.util.ArrayList;
import org.apache.http.message.BasicNameValuePair;
import java.net.URLDecoder;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.HttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
List<NameValuePair> dataPairs = new ArrayList<NameValuePair>();
dataPairs.add(new BasicNameValuePair("license", URLDecoder.decode("enter value here", "UTF-8")));
dataPairs.add(new BasicNameValuePair("guid", URLDecoder.decode("enter value here", "UTF-8")));
dataPairs.add(new BasicNameValuePair("requestproof", URLDecoder.decode("enter value here", "UTF-8")));
dataPairs.add(new BasicNameValuePair("returnavailablecredits", URLDecoder.decode("enter value here", "UTF-8")));
dataPairs.add(new BasicNameValuePair("responseformat", URLDecoder.decode("enter value here", "UTF-8")));
dataPairs.add(new BasicNameValuePair("url", URLDecoder.decode("enter value here", "UTF-8")));
dataPairs.add(new BasicNameValuePair("max_redirects", URLDecoder.decode("enter value here", "UTF-8")));
HttpPost request = new HttpPost("https://api.datamill.solutions/url/check");
request.setEntity(new UrlEncodedFormEntity(dataPairs));
try {
// execute the request and read the response string HttpClient client = new DefaultHttpClient();String response = client.execute(request);// TODO: process the data in 'response'
} catch (IOException ex) {
// TODO: process the exception thrown
}
Enter some example data to test the result of the API.
If you have any questions about the implementation of this API or found an error please don't hesitate to contact our support team.