Downloading a large file using curl

Downloading a large file using curl

  1. Blog
  2. 15 November 2026
  3. 4 min read

Stefan Bogdanescu

Founder & Senior Architect

Stream Large Files Directly to Disk with curl

Your current approach stores the entire remote payload in memory before writing it to a temporary file. While this works perfectly for small assets, it hits a wall immediately when processing gigabyte-sized datasets. The $st variable holds the full content of the response as a string, consuming RAM proportional to the file size plus overhead. Once that buffer exceeds available memory or PHP's memory_limit, the script crashes with an out-of-memory error long before the disk write completes.

To handle large files efficiently, you must bypass the in-memory storage entirely and pipe data directly from the network socket to the disk. This is achieved by setting CURLOPT_RETURNTRANSFER to false and piping the output of curl_exec directly into a file handle using process substitution or shell redirection.

Disabling Buffering

The first step is to remove the instruction that forces curl to return the content as a string. In your sample code, this line:

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

is the primary culprit for memory exhaustion. When enabled, PHP attempts to allocate space equal to the response size. For a 2GB file, that is two gigabytes of RAM just sitting there.

Instead, configure curl to stream the raw output to STDOUT:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
// Disable returning transfer as string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false); 

// Ensure we get headers if needed, but keep body streaming
curl_setopt($ch, CURLOPT_HEADER, false); 
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

curl_exec($ch);
curl_close($ch);

At this stage, the data flows through stdout but has nowhere to go yet. To persist it, you must redirect that stream into a file.

Piping Output to Disk

In a standard CLI shell, you can achieve streaming by chaining curl directly to a file descriptor using shell redirection (>). This allows the OS kernel to manage the buffering and writing to disk efficiently without PHP ever touching the full payload in memory.

If you are running this logic from a command line:

curl -L "$url" > "$tmp_name"

The -L flag handles automatic redirects, which is crucial for most web resources. The shell handles the streaming internally; as soon as curl writes a chunk to stdout, the OS writes it to $tmp_name.

However, if you must execute this within PHP (for example, inside a CLI script), you can use proc_open or simply rely on the fact that curl_exec outputs to STDOUT when CURLOPT_RETURNTRANSFER is false. You then capture that output using shell pipes if your environment supports it.

Handling Redirects and Headers

When dealing with large files, redirects are common. If a server returns a 302, curl will follow it only if you enable CURLOPT_FOLLOWLOCATION. Without this, the script might fail or hang trying to write to a URL that no longer exists.

Additionally, always verify the HTTP status code before assuming success. You can check the return value of curl_exec and inspect the response headers using curl_getinfo:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_HEADER, true); // Include headers in output if you need to parse them
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

$response = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);

if ($response === false) {
    echo "Error: " . curl_error($ch);
} elseif ($httpCode !== 200) {
    echo "Download failed with HTTP status $httpCode";
} else {
    // Streamed successfully to stdout (pipe to file in shell or handle via proc_open)
}

curl_close($ch);

Trade-offs and Limitations

Streaming directly to disk is not a silver bullet. The primary trade-off is that you lose the ability to inspect the full content after download without reading it back from the file system first. If your application requires parsing the content immediately after download, streaming might force you to re-read the file, doubling I/O operations.

Another consideration is error handling during the transfer. With in-memory downloads, you can check for partial writes or corruption before closing the file. In a pure stream scenario, detecting a network interruption mid-transfer is more complex. You often need to rely on HTTP range requests or retry logic at the application level if a download fails partway through.

For most use cases involving large static assets, logs, or backups, however, streaming via shell redirection remains the most robust and memory-efficient solution available in the curl ecosystem.

Our network