The Conflict Between CURLOPT_FILE and CURLOPT_RETURNTRANSFER
You are encountering a fundamental conflict in how cURL handles output buffers. In your current implementation, you have enabled both CURLOPT_FILE and CURLOPT_RETURNTRANSFER. These two options cannot coexist effectively because they instruct the cURL handle to perform contradictory actions regarding the response body.
When you set CURLOPT_RETURNTRANSFER to true, cURL is instructed to return the content of the requested file as a string rather than writing it directly to the stream specified by CURLOPT_FILE. This behavior occurs immediately upon execution. Consequently, when you call curl_exec($curl), the data never leaves the internal buffer to be written into your cache/temp_file.txt file handle. Instead, cURL prepares that content as a string for the function's return value.
The side effect of enabling CURLOPT_RETURNTRANSFER is that curl_exec() returns an empty string when writing to a file, even if the request was successful. Your code executes without throwing an exception because cURL successfully fetched the data internally, but your local file remains empty. To resolve this and ensure the remote content actually lands in your local cache, you must disable the return transfer mode.
Disabling Return Transfer Mode
To fix the immediate issue with temp_file.txt, remove or unset CURLOPT_RETURNTRANSFER. This allows cURL to follow its primary directive: CURLOPT_FILE. When only CURLOPT_FILE is set (along with CURLOPT_URL), cURL streams the response directly to the file descriptor passed via the file pointer.
Here is the corrected logic flow:
function get_remote_file_to_cache()
{
$the_site = 'http://facebook.com'; // Note: Modern sites often require HTTPS and User-Agent headers
$curl = curl_init();
$fp = fopen('cache/temp_file.txt', 'w');
curl_setopt($curl, CURLOPT_URL, $the_site);
curl_setopt($curl, CURLOPT_FILE, $fp);
// CRITICAL FIX: Disable returning content as a string
curl_setopt($curl, CURLOPT_RETURNTRANSFER, false);
// Optional but recommended for debugging if the file is still empty
curl_setopt($curl, CURLOPT_VERBOSE, true);
curl_exec($curl);
curl_close($curl);
fclose($fp); // Always close the file handle to flush buffers
}
Handling HTTP Status Codes
Your original code attempts to determine success by checking for a 404 status using CURLINFO_HTTP_CODE. While this logic is sound, relying on a specific 404 response as your primary condition for creating an error log (404_err.txt) is fragile. Most modern websites, like Facebook, will return 200 OK or 301/302 Redirects, not 404s, unless the specific resource is missing.
Furthermore, using touch() to create a file with a random name based solely on success (rand(0, 99999)--all_good.txt) is an anti-pattern for caching. It overwrites the cache directory every time you run the script, destroying previous cached data and making it impossible to verify if a specific file was fetched or not later.
A better approach involves checking the HTTP status code explicitly after closing the handle (or using curl_getinfo before closing) and handling different scenarios:
- 2xx Status Codes: The request succeeded. Keep the content in
temp_file.txt. - 404 Status Code: The resource was not found. Create a specific marker file like
cache/not_found.txtbut do not overwrite existing cache files randomly. - Other Errors (5xx, etc.): Log these to a central error log or a dedicated error file rather than generating random filenames.
Managing File Handles and Buffers
Even after removing CURLOPT_RETURNTRANSFER, there is one final detail often overlooked: closing the file handle. In your original snippet, you never called fclose($fp). While PHP might close standard streams automatically upon script termination, explicit file handles require manual closure to ensure buffers are flushed and resources are released immediately. Always call fclose() after curl_exec().
Additionally, fetching from http://facebook.com without setting a User-Agent header will likely result in a connection refusal or an empty response (200 OK but no body) because modern servers reject requests that appear to be from bots or scripts lacking a browser signature. You should add:
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)');
Summary of Corrections
To successfully write remote content to a local file using cURL:
- Disable
CURLOPT_RETURNTRANSFER. - Ensure
CURLOPT_FILEpoints to an opened file handle. - Close the file handle explicitly with
fclose(). - Check the HTTP status code to categorize the result (Success vs. Not Found) without generating random filenames.
By aligning your cURL options to not fight for control of the response stream, you ensure the data flows exactly where you intend: from the network to your local disk.