Download file from URL using CURL
A common stumbling block when building PHP scripts is handling downloads that behave differently than expected in a browser versus a direct request. Consider a scenario where you attempt to fetch a specific data file, such as a track recording:
$DLFile = "testfile.igc";
$DLURL = "http://www.xcontest.org/track.php?t=2avxjsv1.igc";
$fp = fopen ($DLFile, 'w+');
$ch = curl_init($DLURL);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_exec($ch);
curl_close($ch);
fclose($fp);
If this code results in an empty file, yet the same URL works when you click a specific hyperlink on the website, the issue almost certainly lies in HTTP headers rather than network connectivity. The discrepancy between a direct request and a link click reveals how web servers distinguish between automated traffic and human interaction.
The Hidden Headers
When you manually navigate to an endpoint or click a link, your browser automatically sends a set of standard headers. It includes a User-Agent string identifying itself as Chrome, Firefox, or Safari. Crucially, it also sends a Referer header indicating the page where the link originated. Many servers use these headers to enforce security policies or content filtering.
In your specific example, the server likely checks the User-Agent. If the request comes directly from a script without a proper identifier, the server might serve a 403 Forbidden error or an HTML login page instead of the binary file data. When you click a link, the browser provides these headers automatically, satisfying the server's requirements and delivering the .igc file content.
To replicate this behavior in your PHP script, you must explicitly set the request headers using CURLOPT_HTTPHEADER. You should mimic a standard browser by defining a User-Agent string:
$ch = curl_init($DLURL);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // Return content as string first for header inspection
$headers = [
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
'Accept: */*'
];
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
// If the file is large and you want to stream it directly to a file:
$fp = fopen ($DLFile, 'w+');
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_exec($ch);
fclose($fp);
By adding these headers, your script appears as a legitimate browser client, often resolving the empty file issue caused by server-side blocking.
Handling Redirects and Content Types
You have enabled CURLOPT_FOLLOWLOCATION, which is generally correct for sites that redirect requests after initial access checks. However, relying solely on this can sometimes mask underlying issues if the final destination requires specific parameters or cookies.
It is also vital to verify the response code. If the file remains empty, inspecting the HTTP status code will clarify whether you received a success (200) or an error like 403 or 500. You can do this by disabling CURLOPT_FILE temporarily and capturing the content as a string, or by using CURLOPT_HEADERFILE to log the raw response headers for debugging.
Additionally, some servers block requests that do not include a Referer header if they expect you to arrive from a specific page. If adding a User-Agent does not resolve the issue, you may need to set a dummy Referer:
$headers[] = 'Referer: http://www.xcontest.org/world/en/flights/detail/';
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
Security and Best Practices
When downloading files from external URLs, always validate the response code before assuming the file was saved successfully. An empty file often indicates a silent failure where the script skipped writing data due to an error status.
Furthermore, be cautious about using CURLOPT_FILE directly with large files without checking for interruptions. In high-security environments, you might need to disable CURLOPT_FOLLOWLOCATION entirely and manually handle redirects by parsing the location header yourself. This prevents potential open redirect vulnerabilities where a malicious server tricks your script into downloading content from an unintended domain.
If you are working within a Laravel ecosystem, consider leveraging the framework's HTTP client facade or the Http::download() method for cleaner syntax and built-in exception handling, though the underlying CURL mechanics remain similar.
Ultimately, understanding how the target server validates incoming requests is key. By mimicking browser behavior through explicit header manipulation, you bridge the gap between manual interaction and automated scripting.