Why You Can't Scrape Google and What to Use Instead
You are attempting to fetch search results using http://ajax.googleapis.com/ajax/services/search/web, a legacy public API that was deprecated years ago. The code snippet you provided is no longer functional for its intended purpose, even if it were; the endpoint has been shut down by Google. When you execute that script today, you will likely encounter an HTTP 403 Forbidden error or a JSON response indicating that the service is unavailable.
The limitation of getting only four results in your specific example isn't just about pagination parameters like start or count. It is a fundamental misunderstanding of how search engines operate. Google's primary goal is to prevent exactly what you are trying to do: automated scraping and bulk data harvesting. They have built sophisticated systems to detect bot traffic, fingerprint user agents, and block requests coming from data centers or known scraper patterns.
Attempting to bypass these restrictions by tweaking headers, rotating IPs, or using proxy services is an arms race that rarely favors the developer. Google's Terms of Service explicitly prohibit web crawling for commercial purposes without a specific license. Relying on the old AJAX API was always a temporary workaround that never offered a path to retrieving "all" results or even a reliable set of 100 unique entries. The architecture of modern search engines prioritizes relevance and anti-spam measures over open data access.
If your goal is to build an application that needs to display search results, the technical reality is that you must integrate with Google's official ecosystem. The current standard for programmatic access is the Google Custom Search JSON API. This is not a replacement for the general web search engine; it is a paid service where you create a custom search engine within your own project context.
To implement this correctly, you need to:
- Register for an API key in the Google Cloud Console.
- Create a Custom Search Engine (CSE) that indexes the domains you wish to query.
- Implement rate limiting logic in your code. The free tier offers limited queries per day (typically 100), while paid tiers allow significantly higher volumes, but you must respect the
X-Goog-Quotaheaders and implement exponential backoff strategies to avoid hitting quota limits.
Here is a conceptual look at how the request structure differs from your original attempt:
use Google\Cloud\Search\Client;
// Note: This requires installing the official client library
// composer require google/cloud-search
$client = new Client(['apiKey' => 'YOUR_API_KEY']);
$query = 'Nikita Platonenko';
$cseId = 'YOUR_CUSTOM_SEARCH_ENGINE_ID';
$results = $client->search($query, [
'limit' => 100, // Requesting up to 100 results per page
'cseId' => $cseId
]);
foreach ($results as $result) {
echo $result['title'] . "\n";
}
This approach is the only sustainable method. It provides structured JSON responses, handles pagination via nextPageToken, and operates within legal and technical boundaries. There is no secret parameter or hidden endpoint that will grant access to the full index of Google's main search engine via HTTP requests.
The trade-off here is cost and complexity. You are moving from a free, broken public API to a managed service with usage quotas. However, this grants you reliability, official support, and compliance with Google's security protocols. If you need to process massive datasets or require real-time freshness beyond what the index provides, consider whether your use case actually fits within the constraints of the Custom Search Engine or if an alternative data source might be more appropriate for your specific application requirements.