How do I make a simple crawler in PHP?

How do I make a simple crawler in PHP?

  1. Blog
  2. 15 November 2026
  3. 3 min read

Stefan Bogdanescu

Founder & Senior Architect

How do I make a simple crawler in PHP?

Building a web crawler in PHP starts with understanding that you are essentially acting as a client to a server, requesting resources and parsing the responses. The core mechanism is straightforward: use file_get_contents() or cURL to fetch an HTML document, then extract hyperlinks using a regular expression or a DOM parser. However, moving from a theoretical script to a robust tool requires navigating several technical pitfalls immediately.

Choosing Your Fetching Method

The most basic approach involves file_get_contents(). It is concise and works well for quick scripts running locally or in environments where SSL verification isn't a primary concern.

$url = 'https://example.com';
$html = file_get_contents($url);

While simple, this method has limitations. It does not allow you to set custom headers, manage timeouts explicitly without relying on the default system settings, or handle redirects gracefully in complex scenarios. For production-grade crawlers, cURL is the superior choice. It offers granular control over user agents, referers, and timeout durations, which are critical for avoiding being blocked by target servers or hanging indefinitely on slow connections.

$ch = curl_init('https://example.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10); // Prevent infinite hangs
$html = curl_exec($ch);
curl_close($ch);

Parsing the HTML Structure

Once you have the raw HTML string, you need to extract the links. Using a regular expression like /<a\s+href="([^"]+)"/i is tempting due to its brevity, but it is fragile. HTML is not always well-formed; attributes can span multiple lines or be unquoted. A regex solution will likely break on edge cases and might even execute unintended code if the target site includes malicious scripts (though PHP generally mitigates this by treating input as strings).

The standard practice is to use a DOM parser. The DOMDocument class comes built into PHP, but it struggles with malformed HTML common in real-world sites. For better resilience, consider using an external library like simple_html_dom. It offers a more forgiving parser that mimics the browser's behavior more closely without requiring you to install heavy dependencies like libxml2 from the command line.

Handling Redirects and User Agents

A simple crawler often gets rejected before it even fetches content. Many modern websites check the User-Agent header to identify bots. If your script sends a default or empty user agent, the server may return an error page instead of the actual content. You must explicitly set a realistic browser identifier in your cURL configuration:

curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)');

Furthermore, CURLOPT_FOLLOWLOCATION is essential. Without it, your crawler will crash when encountering a 302 redirect, leaving you with an incomplete dataset. Always ensure redirects are followed so your script captures the final destination of linked resources.

Managing Scale and Rate Limiting

If your goal is to dump data from "a bunch of links," you must consider rate limiting. Fetching hundreds of pages in rapid succession can trigger IP bans or overload the target server. A naive loop will ignore HTTP response codes; you must check for 4xx and 5xx errors to ensure you aren't crawling error pages.

Additionally, implement a sleep interval between requests. This respects the robots.txt protocol (though not strictly enforced by your code) and prevents Denial of Service accusations against the target site. A simple approach is to add usleep() or sleep() inside your fetch loop, perhaps randomized to mimic human browsing patterns.

Data Storage Strategy

Finally, decide how to store the extracted links. If you are writing to a local file, ensure you handle large datasets efficiently. Appending to a file (file_put_contents($file, $link . "\n", FILE_APPEND)) is standard, but for massive scales, consider buffering writes or switching to a database like SQLite if relationships between pages need tracking. Remember that file_get_contents() does not return data from the current file; it only fetches remote URLs. Always validate that the URL being requested is external and safe before processing.

Our network