PHP Multiple Curl Requests

PHP Multiple Curl Requests

  1. Blog
  2. 15 November 2026
  3. 3 min read

Stefan Bogdanescu

Founder & Senior Architect

Moving Beyond Single-Curl Loops

When you fetch roughly 100 pages sequentially using curl_init(), curl_exec(), and curl_close() inside a loop, you are bottlenecking your script on I/O latency. Each iteration waits for the previous request to complete before initiating the next one. If a remote server takes 500ms to respond, a naive loop will take at least 50 seconds just to establish connections and receive data, ignoring network overhead like DNS lookups and TCP handshakes.

To address this, you should switch to curl_multi_init(). This function creates a multi-handle that can manage multiple asynchronous requests simultaneously. The core principle here is non-blocking execution: instead of waiting for one request to finish before starting the next, you start many requests at once and then repeatedly poll the multi-handle to see which ones have completed while others are still in progress.

Setting Up the Multi-Handle

The setup phase differs significantly from single-curl logic. You initialize a multi-handle once, not per request.

$mh = curl_multi_init();

// Create an array of handles or iterate through your URLs
$urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    // ... up to 100+ URLs
];

$handles = [];

foreach ($urls as $url) {
    $ch = curl_init($url);
    
    // Standard options apply here, e.g., headers, timeouts
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_TIMEOUT, 30);

    $handle_count = curl_multi_add_handle($mh, $ch);
    if ($handle_count === CURLM_OK) {
        $handles[] = $ch;
    } else {
        // Handle initialization error appropriately
    }
}

Once the handles are added to the multi-handle, you must set up the execution loop. The critical function is curl_multi_exec(), which returns a status code indicating whether the internal state machine of all requests has been updated.

The Execution Loop

A common pitfall is calling curl_multi_select() without checking if curl_multi_exec() has fully processed the handle list. You should only call select() when exec() indicates that no more data is ready to be read, or when you want to block until something happens.

$running = null;

do {
    // Execute requests and get status
    $mrc = curl_multi_exec($mh, $running);

    if ($mrc === CURLM_CALL_MULTI_PERFORM) {
        // More data is expected, but we can't execute yet.
        // Usually safe to skip select here or call it briefly.
    } elseif ($mrc === CURLM_OK) {
        // Requests are running. Wait for activity.
        if ($running > 0) {
            curl_multi_select($mh, 1); // 1 second timeout max
        }
    } else {
        // Error occurred in the multi-handle
        break;
    }

    while ($done = curl_multi_info_read($mh)) {
        $handle = $done['handle'];
        
        if (curl_errno($handle)) {
            $error = curl_error($handle);
            echo "Error on handle: $error\n";
        } else {
            $response = curl_multi_getcontent($handle);
            // Process completed request
        }

        curl_multi_remove_handle($mh, $handle);
        curl_close($handle);
    }
} while ($running > 0);

This pattern ensures that you don't spin the event loop unnecessarily. The curl_multi_select() call with a timeout prevents your script from freezing if no network activity occurs for an extended period.

Handling Errors and Cleanup

One significant trade-off with multi-curl is error handling granularity. Since requests run asynchronously, a failure in one request does not automatically cancel others. You must explicitly check for errors using curl_multi_info_read() after every call to exec(). This allows you to gracefully handle partial failures without aborting the entire batch.

Additionally, ensure you remove and close handles immediately after processing them from the queue. Leaving handles attached to a multi-handle that is no longer needed can lead to resource leaks or unexpected behavior in subsequent iterations of your script.

If you are using this within a Laravel application, consider wrapping this logic in a service provider or job class to maintain separation of concerns. For high-volume scraping tasks where strict order doesn't matter, this asynchronous approach reduces total runtime by an order of magnitude compared to sequential loops.

Our network