Persistent/keepalive HTTP with the PHP Curl library?

Persistent/keepalive HTTP with the PHP Curl library?

  1. Blog
  2. 15 November 2026
  3. 3 min read

Stefan Bogdanescu

Founder & Senior Architect

Persistent/keepalive HTTP with the PHP Curl library?

When your application scales to handle high throughput, the network layer often becomes the bottleneck before CPU or memory does. In the scenario you described, you are pushing 80 documents per second to Solr, yet observing a massive spike in network interrupts—2000/sec on both the PHP and Solr boxes—that correlates perfectly with request bursts. This specific symptom points directly to connection churn: your PHP box is opening a new TCP handshake and TLS negotiation for every single document, while the database side remains calm because it reuses a long-lived socket. The cost isn't just bandwidth; it's the CPU cycles wasted on handshakes and the latency introduced by waiting for a fresh connection to be established before data transfer can begin.

The good news is that cURL handles persistent connections natively, but only if you explicitly configure them. By default, many HTTP clients assume stateless communication unless told otherwise. To switch from a "open-and-close" pattern to a keepalive model, you need to enable two critical options in your curl_init() setup.

First, you must set the protocol version to HTTP/1.1 or higher. HTTP/1.0 is inherently stateless regarding connections; it closes the socket after the response unless the server specifically supports persistent connections with a keep-alive header, which is rare and unreliable for bulk operations. You enforce modern behavior by passing CURLOPT_HTTP_VERSION set to CURL_HTTP_VERSION_1_1.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://solr:8983/solr/core/update/json');
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1); // Essential for keepalive

Second, and arguably more important, is enabling the connection reuse flag. You need to set CURLOPT_FOLLOWLOCATION if you are following redirects (common with Solr), but specifically for keepalives, you should ensure CURLOPT_CONNECTTIMEOUT and CURLOPT_TIMEOUT are set appropriately, though the magic happens with CURLOPT_IPRESOLVE or simply relying on standard behavior once HTTP/1.1 is active. However, the most direct control comes from ensuring the handle isn't closed immediately after the request finishes.

To achieve true reuse, you must not call curl_close() after every document insertion. Instead, retain the handle resource throughout the entire batch processing loop. When the script processes 10,000 documents, you should initialize the cURL handle once at the start of the script and pass that same handle to every curl_exec() call.

$ch = curl_init();
// Configure options once
curl_setopt($ch, CURLOPT_URL, 'http://solr:8983/solr/core/update/json');
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $jsonPayload);

// Inside your loop processing documents...
curl_exec($ch); 
// Do NOT call curl_close($ch) here. Keep the handle open.

There is a subtle gotcha regarding idle timeouts. Even with keepalives enabled, servers often terminate connections that sit idle for too long. If your script pauses between batches or if the load varies significantly, the Solr server might send a RST packet or close the socket due to its own timeout settings (often controlled by tcp_keepalive_time).

If you encounter broken pipes (CURLE_OPERATION_TIMEDOUT or similar errors) after hours of running, it usually means the remote server closed the connection while your PHP process assumed it was still open. To mitigate this, you should check the response code and handle the specific error that indicates a dead connection. Additionally, ensuring CURLOPT_TCP_KEEPALIVE is enabled (available in newer PHP versions) can help prevent the OS from dropping the idle connection on the wire too aggressively.

curl_setopt($ch, CURLOPT_TCP_KEEPALIVE, true);
curl_setopt($ch, CURLOPT_TCP_KEEPIDLE, 60); // Seconds of idle time before keepalive probe

Finally, regarding your question about periodically reconnecting: generally, you do not need to manually reconnect unless the remote server explicitly kills the connection due to inactivity or security policies. However, if you are processing millions of documents over days, a strategy of closing and re-opening the handle every few thousand requests (e.g., every 10,000 docs) is a robust pattern. This prevents any single stuck connection from causing issues if the network blips, while still maintaining persistent connections for the vast majority of your traffic.

If you are building a larger application stack where managing these low-level cURL configurations becomes tedious, frameworks like Laravel provide a unified client that handles connection pooling and persistence transparently. For more complex enterprise needs, checking resources like https://laravelcompany.com can offer insights into how modern PHP ecosystems abstract these networking details away from the core business logic.

Our network