Parameters
Parameters are the top-level settings that control the fundamental behaviour of your automation. These parameters define where your request goes, how it's routed, whether it runs synchronously or asynchronously, and how caching is handled.
Below you'll find detailed documentation for each available parameter.
Proxy servers
Parameter: proxy_location (string or null)
Gaffa makes it super simple to proxy your traffic through a global network of residential proxies. Setting proxy_location in your request will allow you to utilize one of our partner third-party proxy services to gain local access to a site.
Not setting a proxy_location will mean the request does not use a proxy server and will use a generic datacenter IP.
Available Locations
United States
us
Ireland
ie
Singapore
sg
France
fr
IP Types
Currently, all our IP addresses are residential IP addresses, which are procured through reputable third parties.
IP Rotation
IP rotation is an essential part of any web data scraping or automation task. In Gaffa, each browser request is treated as unique. We regularly rotate the IP addresses used, so you should assume each request is made from a different IP address than the last.
Restrictions
Whilst we'll do our best to provide access to as wide a range of sites as possible, we may have to restrict access to certain sites to prevent abuse of our service or of other services. Our proxy partners may also enforce restrictions on certain sites and categories of sites that we don't have any control over.
Synchronous & Asynchronous Requests
Parameter: async (boolean)
By default, browser requests run synchronously: the request waits and hands back the full result directly, with nothing further to check. Setting async: true instead returns an id immediately, and your automation checks back later to see when the result is ready.
Sync suits quick, predictable jobs where you'd rather not check back at all. Async is the better fit for anything that might run a while. A synchronous request holds a connection open for as long as it runs, and most tools time out well before a long job finishes, so a slow sync request risks getting cut off partway through even if Gaffa itself would have completed it. If a job might run long, use async.
When you set async: true, running the request gives you back just enough to track it: an id and a pending status, not a result yet.
Response:
Getting your result
Poll GET /v1/browser/requests/{id} — using the id from your original request — with your API key passed as the X-API-Key header:
Request:
Keep checking back until it's done. There are two states:
state: "pending"— still running. Check back again shortly.state: "completed"— finished. Your result is included, along with everything your actions produced.
state tracks the request's lifecycle within Gaffa, not the outcome of your automation. "pending" means it's still running. "completed" means Gaffa has finished processing it, your actions ran (or were cancelled), and the result is ready, either way.
Whether it actually succeeded is a separate question, answered by the error field, not by state. A request that failed or was cancelled partway through is still "completed".
A couple of things worth knowing
Your plan caps request runtime, and the cap differs for sync and async requests. Read more about Time Limit.
A request that gets cut off for running too long isn't billed for the attempt.
Results are kept for as long as your plan's data retention period, after which the
idcan no longer be used to fetch them. See data retention for details.
Caching
Parameter: max_cache_age integer (seconds)
When we were building Gaffa, we noticed that many existing scraping tools don't let users easily share their scraped web data, even though many users request the same pages on the same sites. Not only is this a waste of a user's allowance, but it also puts a burden on the site owners who are serving the same data to different users for the same purpose. Because of this, we have created a service-wide cache in Gaffa.
How it works
When making a browser request, you can provide a max_cache_age parameter that is a number in seconds equal to or greater than 0. This value denotes the maximum age of data you would accept from the API.
If another user of our service has requested the same URL with exactly the same parameters and actions as you in this timeframe, the response will be returned to you immediately and will not be processed by one of our browsers. If there are multiple identical requests in the given timeframe, then the most recent will be returned.
This will save you time waiting for a response and credits, because requests returned from the cache don't use any bandwidth.
Settings
The settings object allows you to configure how your browser requests behave. It currently supports six parameters that control request recording, media downloads, execution time limits, ad blocking, redirect logging, and browser actions.
You can read more about all available settings parameters here.
Last updated