Information about Web Resources

Fetches the resource of a given url and checks if it is available. Only websites using HTTP/HTTPS are allowed. IP addresses and ports other than 80/443 are not permitted and will be ignored.

Request Url: https://api.datamill.solutions/url/check
Request Method: POST

Table of Contents

Request Parameters

The API uses the POST request method to send data in the HTTP message body to avoid data being stored in the cache of the web server. A request is valid if all parameters are provided as described below (data type, minimum and maximum length). Please note that all data must be UTF-8 and URL encoded.

POST /url/check HTTP/1.1
Host: https://api.datamill.solutions

name1=value1&name2=value2
Parameter Mandatory Data Type Min. Max. Description
license required string 29 29 Your license key used for authentication - keep this key secret!
guid required string 30 40 Your GUID used for authentication - keep this key secret!
requestproof optional string 0 1 set to 1 if you need a unique request identifier as proof for your API request e.g. for VAT checks; default is 0
returnavailablecredits optional string 0 1 set to 1 to get the number of data.mill credits available in your data.mill account; default is 0; a return value of -1 means that you have unlimited credits
responseformat optional string 3 4 format of returned data; possible values are xml or json (default)
url required string 4 2048 The url to be checked (e.g. any website)
max_redirects optional string 0 100 The maximum amount of redirects until the lookup for the root resource will be stopped (default 10)

Response

The API returns all data in valid JSON. A few programming languages include native support for JSON. For those that don't, you can find a suitable library at http://www.json.org. Valid requests always returns all response keys listed below. Depending on the request parameters some response keys may be empty.

HTTP/1.1 200 OK

Server: Apache
Content-Type: application/json; charset=UTF-8
Date: Sat, 09 Nov 2024 15:50:50 GMT
Content-Length: 33
Access-Control-Allow-Origin: *

{"key1":"value1","key2":"value2"}
Name Data Type Description
valid string Flag if the root resource (website) is valid or not [0, 1]
url string The final url of the resource
http_code string Http Status Code
total_time string Total time of the request [seconds]
namelookup_time string Time until host name resolved [seconds]
connect_time string Time until connection established [seconds]
pretransfer_time string Time until file transfer began [seconds]
starttransfer_time string Time to first byte [seconds]
primary_ip string IP address of the most recent connection
primary_port string Destination port of the connection
download_content_length string Number of bytes to download from the resource (-1 means no information available) Deprecated, will be removed in Version 2.0.0
content_type string Content type of the requested resource
redirects string Collection of all urls redirected
parameters string Collection of all parameters included in the requested url

Response Errors

If the request fails the API returns a HTTP status code according to the reason. A reason may be a missing mandatory request parameter, invalid parameter credentials (minimum/maximum length) or invalid API keys. The response body is a valid JSON containing detail information about the reason (except for status code 500).

HTTP/1.1 404 Not Found

Server: Apache
Content-Type: application/json; charset=UTF-8
Date: Sat, 09 Nov 2024 15:50:50 GMT
Content-Length: 71
Access-Control-Allow-Origin: *

{"errorcode":404,"errormessage":"transformation function name invalid"}
HTTP Status Code Reason
401 Unauthorized
The license key and/or GUID key is missing, invalid or inactive.
402 Payment Required
Your /data.mill credit quota is exceeded. You need to get more credits.
403 Forbidden
You need to accept all required agreements of this function to be able to use it.
404 Not Found
The request path in combination with the request method does not exist.
405 Method Not Allowed
The request method is not allowed for the requested path.
412 Precondition Failed
At least one of the given request parameter has invalid credentials (e.g. data type, length).
428 Precondition Required
At least one of the mandatory request parameter is missing or has an empty value.
500 Internal Server Error
An unexpected condition terminated the request.
503 Service Unavailable
The service is currently in maintenance mode and will be alive later.

Batch Mode

The batch mode allows the execution of thousands of records (limited to 5,000) combined in a single request. This API endpoint as described above uses the single mode execution of records. To use the batch mode you need to combine all request parameters above (excluding license and guid) as array, where each array element represents a single record. This array is converted to a valid JSON string and set as value of the new request parameter called batch. The response of the batch mode is a valid JSON which contains the result for each record set in the request. The response keys of each result element are the same as for the single request mode described above.

Batch Request

license=...
&guid=...
&batch=[{"requestproof":"...","returnavailablecredits":"...","responseformat":"...","url":"...","max_redirects":"..."},{"requestproof":"...","returnavailablecredits":"...","responseformat":"...","url":"...","max_redirects":"..."}]

Batch Response

[{"valid":"...","url":"...","http_code":"...","total_time":"...","namelookup_time":"...","connect_time":"...","pretransfer_time":"...","starttransfer_time":"...","primary_ip":"...","primary_port":"...","download_content_length":"...","content_type":"...","redirects":"...","parameters":"..."},{"valid":"...","url":"...","http_code":"...","total_time":"...","namelookup_time":"...","connect_time":"...","pretransfer_time":"...","starttransfer_time":"...","primary_ip":"...","primary_port":"...","download_content_length":"...","content_type":"...","redirects":"...","parameters":"..."}]

Pricing

In order to enable the payment of different currencies, we have introduced a system based on credit balance as payment for the individual functions. The currency of the account is called /data.mill credits. To use the functions, the account must be charged in advance. If the credit price of a function is higher than the current balance, the function can not be executed. The credits will be deducted from your balance after executing the function. Your credits have no expiration date.

This functions costs unknown /data.mill credits for each record set in the request.

Code Snippets

We provide some code snippets for multiple programming languages to easily implement this function in your own application. The code snippets below uses the single request mode. If you want to use the batch mode instead of the single mode you need to modify the snippet. Feel free to copy the code snippets for your preferred programming language.

// the following code requires cURL php extension installed on your server (http://php.net/manual/en/book.curl.php) // initialize a new cURL session and store the cURL handler object $curl = curl_init(); // set the url to which the request will be sent curl_setopt($curl, CURLOPT_URL, "https://api.datamill.solutions/url/check"); // set the HTTP method to use for the request curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST"); // set the number of parameters to be sent to the server curl_setopt($curl, CURLOPT_POST, 7); // data to be sent to the server // TODO: append your API keys and value(s) and don't forget to urlencode() them curl_setopt($curl, CURLOPT_POSTFIELDS, "license=&guid=&requestproof=&returnavailablecredits=&responseformat=&url=&max_redirects="); // set a timeout in seconds for the request curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10); curl_setopt($curl, CURLOPT_TIMEOUT, 10); // return the raw data from the server curl_setopt($curl, CURLOPT_RETURNTRANSFER, true); // execute the previously created cURL session $responseString = curl_exec($curl); // close the cURL session after execution curl_close($curl); // convert the response (string) to an array $responseArray = json_decode($responseString, true); // TODO: process the data in the response array (you may process the data in the response string too)
/* Note: We highly recommend to NOT use this code as it is written here. If you choose JavaScript as your preferred programming language, please change the following code to request your own server (without API keys) and further redirect the request from your own server to the API including your API keys. Otherwise everyone will be able to read your API keys and use them for their own purpose! */ // the following code requires jQuery JavaScript library (minimum version 1.8.1) added to your website (http://jquery.com/download/) // perform an asynchronous HTTP (Ajax) request jQuery.ajax({
// set the url to which the request is sent url : "https://api.datamill.solutions/url/check", // set the HTTP method to use for the request type: "POST", // data to be sent to the server (you may use a JavaScript object instead of a string) // TODO: append your API keys and value(s) and don't forget to urlencode() them data : "license=&guid=&requestproof=&returnavailablecredits=&responseformat=&url=&max_redirects=", // type of data that we are expecting back from the server dataType: "JSON", // set a timeout in milliseconds for the request timeout: 10000, // function to be called if the request succeeds success:function(data, textStatus, jqXHR) {
// {object} data The data returned from the server, formatted according to the dataType parameter. // {string} textStatus String describing the status. // {object} jqXHR A XMLHttpRequest object describing the response. // TODO: process the data in the 'data' JavaScript object
}, // function to be called if the request fails error: function(jqXHR, textStatus, errorThrown) {
// {object} jqXHR A XMLHttpRequest object describing the response. // {string} textStatus String describing the type of error occurred. // {string} errorThrown Text portion of the HTTP status, such as "Not Found" or "Internal Server Error". // TODO: process the error occurred
}
});
// the following code uses standard libraries included in Microsoft Visual Studio 2013 or higher using System.IO; using System.Web; using System.Net;
// create an array of bytes representing the data to be sent to the server // TODO: append your API keys and value(s) and don't forget to urlencode() them byte[] Data = Encoding.ASCII.GetBytes("license=&guid=&requestproof=&returnavailablecredits=&responseformat=&url=&max_redirects="); // create a new web request resource and set the url to which the request is sent HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://api.datamill.solutions/url/check"); // set the HTTP method to use for the request Request.Method = "POST"; // set the request content type Request.ContentType = "application/x-www-form-urlencoded"; // set the number of bytes being sent to the server Request.ContentLength = Data.Length; // set a timeout in milliseconds for the request Request.Timeout = 10000; // try to send all data to the newly created web request resource try {
using(var Stream = Request.GetRequestStream()) {
Stream.Write(Data, 0, Data.Length);
}
} catch (Exception ex) {
// TODO: process the exception thrown (e.g. url not found) return;
} // try to read the response from the web resource try {
HttpWebResponse ResponseObject = (HttpWebResponse)Request.GetResponse(); StreamReader Reader = new StreamReader(ResponseObject.GetResponseStream()); // read the whole response as string and store it String ResponseMessage = System.Net.WebUtility.HtmlDecode(Reader.ReadToEnd()); // TODO: process the data in 'ResponseMessage'
} catch (WebException ex) {
// error occurred, http status code != 200 (e.g. timeout or parameter invalid) if(ex.Status == WebExceptionStatus.Timeout) {
// TODO: process the timeout reached
} else {
// read the response from the web exception HttpWebResponse ResponseObject = (HttpWebResponse)ex.Response; StreamReader Reader = new StreamReader(ResponseObject.GetResponseStream()); // read the whole response as string and store it String ResponseMessage = System.Net.WebUtility.HtmlDecode(Reader.ReadToEnd()); // TODO: process the error response received from the server (stored in 'ResponseMessage')
}
}
// the following code requires at least Android API version 10 // import standard Java and Android libraries import java.util.List; import org.apache.http.NameValuePair; import java.util.ArrayList; import org.apache.http.message.BasicNameValuePair; import java.net.URLDecoder; import org.apache.http.client.methods.HttpPost; import org.apache.http.client.entity.UrlEncodedFormEntity; import org.apache.http.client.HttpClient; import org.apache.http.impl.client.DefaultHttpClient;
// create a list of key-value-pairs to store the data which will be sent to the server List<NameValuePair> dataPairs = new ArrayList<NameValuePair>(); // add each parameter to the key-value-pair list (value is decoded to UTF-8 standard) // TODO: append your API keys and value(s) dataPairs.add(new BasicNameValuePair("license", URLDecoder.decode("enter value here", "UTF-8"))); dataPairs.add(new BasicNameValuePair("guid", URLDecoder.decode("enter value here", "UTF-8"))); dataPairs.add(new BasicNameValuePair("requestproof", URLDecoder.decode("enter value here", "UTF-8"))); dataPairs.add(new BasicNameValuePair("returnavailablecredits", URLDecoder.decode("enter value here", "UTF-8"))); dataPairs.add(new BasicNameValuePair("responseformat", URLDecoder.decode("enter value here", "UTF-8"))); dataPairs.add(new BasicNameValuePair("url", URLDecoder.decode("enter value here", "UTF-8"))); dataPairs.add(new BasicNameValuePair("max_redirects", URLDecoder.decode("enter value here", "UTF-8"))); // create the web resource with pre-defined POST method and set the url to which the request is sent HttpPost request = new HttpPost("https://api.datamill.solutions/url/check"); // append the parameters to the web resource request.setEntity(new UrlEncodedFormEntity(dataPairs)); try {
// execute the request and read the response string HttpClient client = new DefaultHttpClient(); String response = client.execute(request); // TODO: process the data in 'response'
} catch (IOException ex) {
// TODO: process the exception thrown
}

Try it right now!

Enter some example data to test the result of the API.

Need help?

If you have any questions about the implementation of this API or found an error please don't hesitate to contact our support team.