Get DIV content from external Website

Get DIV content from external Website

  1. Blog
  2. 15 November 2026
  3. 3 min read

Stefan Bogdanescu

Founder & Senior Architect

Why file_get_contents Fails When Scraping External HTML

Attempting to pull a specific <div> from an external website using pure PHP often hits a wall immediately. The standard approach involves fetching the raw HTML string with file_get_contents, loading it into a DOMDocument, and then querying the DOM nodes. While this logic holds up for local files or same-origin requests, it breaks down when dealing with third-party sites like the example provided (isitdownrightnow.com).

The primary culprit is not your parsing logic; it is how modern web servers handle cross-origin resource fetching. When file_get_contents hits a URL from a different domain without the proper headers, the server often responds with an HTTP 500 error or simply blocks the request due to security restrictions implemented by the target site's robots.txt or specific anti-scraping middleware. The error message "Failed to load resource: the server responded with a status of 500" indicates that the PHP script was able to reach the network, but the remote server rejected the raw GET request intended for scraping.

To successfully extract the content <div class="statusup">The website is probably down just for you...</div> from an external source, you must mimic a real browser request. The solution lies in configuring the curl extension, which allows you to send custom headers and manage redirects effectively.

Here is a robust implementation using cURL:

$url = 'http://www.isitdownrightnow.com/youtube.com.html';
$ch = curl_init();

// Set necessary headers to mimic a browser request
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // Return response as string instead of outputting directly
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // Follow redirects if the target redirects

// Execute the request
$page = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$errno = curl_errno($ch);

if ($httpCode !== 200) {
    // Handle non-200 status codes gracefully
    die("Request failed with HTTP code: " . $httpCode);
}

curl_close($ch);

// Now that we have the HTML string, parse it safely
$dom = new DOMDocument();
libxml_use_internal_errors(true); // Suppress warnings from malformed HTML often found in scraped content
$dom->loadHTML($page);
libxml_clear_errors();

$divs = $dom->getElementsByTagName('div');
foreach ($divs as $div) {
    if ($div->getAttribute('class') === 'statusup') {
        echo $div->nodeValue;
    }
}

The Importance of Headers and Error Handling

The critical difference here is the User-Agent header. Many servers, including the one hosting isitdownrightnow.com, check the User-Agent string to identify automated bots. If it looks like a script or is missing entirely, they may serve an error page (500) instead of the actual content. By setting CURLOPT_USERAGENT to a standard browser string, you pass the initial security check.

Additionally, libxml_use_internal_errors(true) is essential when working with external HTML. Scrape targets rarely have perfectly valid HTML; they often contain unclosed tags or malformed attributes that would throw exceptions on a standard DOMDocument::loadHTML() call. Suppressing these internal errors allows the script to continue parsing and finding your target node without crashing.

Trade-offs and Limitations

While this approach works for simple text extraction, it is important to understand the legal and ethical implications. Scraping a third-party site extracts data you do not own. Always check the robots.txt file of the target domain (e.g., http://www.isitdownrightnow.com/robots.txt) before implementing this logic in production. If a site explicitly disallows crawling via robots.txt, bypassing that restriction can lead to IP bans or legal issues.

If you are building a larger application where reliability is paramount, consider using a dedicated scraping library like Goutte (which wraps Symfony's HttpClient) or an API if the site provides one. For simple, ad-hoc checks within a Laravel project, this cURL method remains the most lightweight and dependency-free solution available in core PHP.

Our network