Best way to manage long-running php script?

Best way to manage long-running php script?

  1. Blog
  2. 15 November 2026
  3. 4 min read

Stefan Bogdanescu

Founder & Senior Architect

Why PHP Isn't Ideal for Long-Running Crawler Tasks

If you are running a PHP script that takes between 5 and 30 minutes to complete, specifically because it relies on curl to wait for remote server responses, you are facing the classic "request timeout" problem. The core issue isn't just the duration; it is how web servers handle persistent connections. When your script initiates a request from a command-line interface or a scheduled task, the underlying HTTP server (like Apache or Nginx) often times out waiting for that process to finish writing output before releasing the connection. Even if you are bypassing the web server and running PHP directly via CLI, the operating system's default behavior regarding long-lived processes can lead to resource exhaustion or unexpected termination if not managed correctly.

The fundamental misunderstanding in asking "how to end the HTTP request before the script is finished" highlights a specific architectural flaw: treating a synchronous, blocking operation as an asynchronous task. When your script calls curl_exec (or similar) without a timeout limit, it sits idle waiting for the remote server to respond. If that remote server is slow or unresponsive, your PHP process hangs. You cannot simply "cancel" the TCP connection from within the PHP script once it has initiated; you can only interrupt the waiting part of the process by setting a strict CURLOPT_TIMEOUT. However, this doesn't solve the problem of the script still needing 30 minutes to scrape multiple pages sequentially.

The most robust solution is to decouple the execution logic from the web request lifecycle. Instead of trying to force a long-running synchronous script to fit within a standard HTTP request context (which is inherently limited), you should treat the scraping job as a background task. The pattern that solves this involves splitting the workflow: the initial trigger creates a record in your database and returns immediately, while the heavy lifting happens in a separate process or queue worker.

To implement this effectively, you should move the logic out of a direct CLI invocation into a message queue system like RabbitMQ or Redis. Your web request (the "initiator") pushes a job payload containing the list of URLs to scrape into the queue and then exits instantly. A dedicated worker process consumes these jobs one by one. This worker runs the PHP script, allowing it to handle the 5-30 minute duration without interfering with the web server's connection pool.

Here is a conceptual example of how this separation looks in code using a Redis queue:

// In your web controller or API route
$url = 'https://example.com';
$jobData = [
    'urls' => ['https://page1.com', 'https://page2.com'],
    'callback_url' => '/jobs/status/' . $uuid
];

// Push job to queue immediately
Redis::lpush('scrape_queue', json_encode($jobData));

// Return success instantly without waiting for completion
return response()->json(['status' => 'queued']);

The worker, running as a persistent daemon or via Laravel's queue:work command, picks up this job. Inside the worker, you can set per-request timeouts using CURLOPT_TIMEOUT to prevent any single page load from killing the entire worker process if the remote server hangs. If the script completes successfully, it updates the database flag; if it fails or times out on a specific page, that error is logged, and the worker moves to the next item in the queue or retries based on your strategy.

There are significant trade-offs to consider. Using a synchronous CLI script without a queue risks hitting PHP's memory_limit or the system's process limits if multiple instances run simultaneously. Furthermore, relying on a single long-running script makes debugging difficult; if it crashes at minute 25, you lose context unless you implement robust checkpointing (saving progress to the DB before processing each page).

If you are already using Laravel, leveraging the framework's built-in queue drivers simplifies this architecture significantly. You can define a Job class that encapsulates the scraping logic and handles the curl configuration with appropriate timeouts. This approach ensures that your web server remains responsive, your database flags are set reliably upon completion, and your infrastructure scales horizontally if you need to process more pages in parallel.

For high-volume scraping operations, this asynchronous pattern is the industry standard rather than attempting to force a synchronous script to run until completion within a single request cycle.

Our network