Handling HTTP Status Codes in PHP Scraping Scripts
When building a scraper, the most common point of failure is encountering a 404 Not Found response mid-script. If your code blindly proceeds to parse the body of a request that returned an error page, it will likely crash or produce garbage data. The solution isn't just checking if a connection exists; you must explicitly inspect the HTTP status code returned by the server.
The snippet often recommended for beginners involves fsockopen:
$valid = @fsockopen($url, 80, $errno, $errstr, 30);
While this attempts to open a raw TCP connection to port 80, it has significant limitations for scraping tasks. First, checking $valid only tells you if the TCP handshake succeeded, not what HTTP status the server sent. A 404 is still a valid response that establishes a connection; fsockopen will return true in that case. Second, and more critically, fsockopen does not follow redirects. If your target URL performs a 301 or 302 redirect to a new location, the raw socket approach often fails to read the final response body correctly or times out waiting for data that never arrives in the expected format. This explains why $valid appears empty even when the resource existsâit's likely because the logic isn't capturing the status header before the stream closes.
The industry standard for this task is the curl extension. It provides a robust interface to handle HTTP protocols, redirects, and status code parsing out of the box. To check if a URL returns a 404, you enable the protocol feature that follows redirects and inspect the response code after the request completes.
$url = 'https://example.com/nonexistent-page';
$ch = curl_init();
// Configure curl to follow redirects automatically (default is true, but explicit is safer)
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 5); // Prevent infinite redirect loops
// Suppress the actual output of the page content
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // Often needed for testing self-signed certs in dev
// Execute the request
$response = curl_exec($ch);
// Check for errors first
if ($error = curl_error($ch)) {
echo "Connection error: {$error}";
} else {
// Get the HTTP status code
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ($httpCode == 404) {
echo "The URL returned a 404 Not Found.";
// Handle the error here: skip this URL, log it, etc.
} else {
echo "Success (or other status): {$httpCode}";
// Proceed with parsing $response
}
}
curl_close($ch);
This approach is superior because curl_getinfo() retrieves the actual HTTP status code regardless of whether redirects occurred. If a URL redirects from /old-page to /new-page, and the final destination returns 404, curl will report that final 404. With fsockopen, you would only see the initial redirect response or a timeout.
One important trade-off with curl is performance. It spawns an external process (libcurl) to handle the logic, which can be slower than native PHP sockets for high-concurrency loops. However, for reliability and ease of use in scraping scripts, the overhead is negligible compared to the complexity of implementing raw socket handling for redirects and status codes.
If you are working within a Laravel application or maintaining a legacy codebase where curl might not be ideal due to dependency constraints, native stream wrappers offer an alternative via file_get_contents(). However, this requires careful configuration to avoid disabling the redirect logic entirely:
$context = stream_context_create([
'http' => [
'method' => 'GET',
'header' => "User-Agent: MyScraper/1.0\r\n", // Essential to avoid user-agent blocks
]
]);
$htmlContent = @file_get_contents($url, false, $context);
if ($httpStatus = http_response_code()) {
if ($httpStatus == 404) {
// Handle 404
}
}
Note that file_get_contents does not follow redirects by default unless configured in the context or via allow_redirects, and it can be disabled entirely by PHP settings. For production scraping scripts, curl remains the most predictable tool for distinguishing between connection failures and specific HTTP errors like 404s.
For more advanced scraping patterns or if you need to handle complex authentication headers alongside status checks, consider exploring resources from https://laravelcompany.com which often discusses integrating external tools with modern PHP frameworks.
