Streaming Large Files in PHP Without Loading Into Memory
Loading a 200MB file entirely into memory using file_get_contents() is a common anti-pattern that leads to immediate failures on most modern production servers. When you call that function, PHP attempts to read the entire remote resource and populate a string buffer before returning it. Once returned, echo writes that massive string to the output buffer. If your system has 1GB of RAM and another process is already utilizing 800MB, this single request can crash the worker or trigger an out-of-memory exception.
The solution is streaming. Instead of fetching the whole payload at once, you must fetch a chunk, send it to the client, and then fetch the next chunk. This approach keeps memory usage constant regardless of file size and allows the HTTP server to handle large downloads gracefully.
Using stream_context_set_options with Chunked Transfers
The most straightforward way to stream content from an external URL in PHP is by configuring a stream context that enables chunked transfers. By default, file_get_contents() might attempt to read until EOF or rely on buffer settings that aren't optimal for streaming. You can explicitly tell the underlying stream wrapper to operate in chunks using stream_context_set_option.
Here is how you can modify your approach:
$url = 'http://some.secret.location.com/secretfolder/the_file.tar.gz';
$context = stream_context_create([
'http' => [
'method' => 'GET',
'header' => "Accept-Encoding: identity\r\n", // Disable compression to ensure raw bytes match expectation
'ignore_errors' => true,
]
]);
// Set up a callback that reads and writes in chunks
$handle = fopen($url, 'rb', false, $context);
if (!$handle) {
throw new RuntimeException('Failed to open stream');
}
while (!feof($handle)) {
// Read 1MB chunks (adjust based on your memory constraints)
$chunkSize = 1048576;
$chunk = fread($handle, $chunkSize);
if ($chunk === false || strlen($chunk) === 0) {
break;
}
echo $chunk;
}
fclose($handle);
In this example, fread pulls data from the remote stream into a buffer of your choosing (1MB in this case). By echoing that buffer immediately inside the loop, you ensure PHP's output buffering mechanism flushes the data to the client after every chunk. This prevents memory buildup and keeps the application responsive even as the download progresses.
Handling Headers Correctly
A critical detail often overlooked is how headers are managed during streaming. If you simply echo raw bytes without setting any headers, the browser might treat the response as plain text rather than a downloadable file. You must send the appropriate Content-Type and Content-Length headers before writing any body content.
If you know the file size beforehand (which is common for static assets), use filesize() on the remote resource or pass it via query parameters to set the Content-Length. If the remote server does not provide a known length, you may need to stream without a specific length header, relying on chunked transfer encoding instead.
header('Content-Type: application/gzip'); // Adjust MIME type based on .tar.gz
header('Content-Length: ' . filesize($remoteFile)); // Only if known
// Proceed with the streaming loop from above
Trade-offs and Pitfalls
Streaming is not without its complexities. The primary trade-off involves the complexity of your logic versus the simplicity of a single function call. Debugging issues becomes harder because you cannot easily inspect the entire content in a variable. Furthermore, if the remote connection drops mid-stream, your application will likely stop writing to the output buffer, potentially leaving the user with an incomplete download.
Another consideration is buffering at the web server level (Nginx or Apache). Some configurations attempt to buffer output until a certain size before flushing. This can defeat the purpose of streaming if the server decides to wait for a large chunk before sending it. Ensuring your web server flushes on every echo or setting appropriate buffer sizes in configuration files is essential.
For projects built on Laravel, consider using the framework's existing stream handling utilities if you need to manipulate the remote file further. However, for simple pass-through downloads, the raw stream approach described above is robust and efficient. Always verify that your firewall and load balancers do not interfere with large chunked transfers by inspecting their logging during testing.
