How can I emulate a get request exactly like a web browser?

How can I emulate a get request exactly like a web browser?

  1. Blog
  2. 15 November 2026
  3. 4 min read

Stefan Bogdanescu

Founder & Senior Architect

The Hidden Layers Behind Browser GET Requests

When a web browser loads a page, it does more than just send an HTTP GET request to the server. It sends a complex envelope of metadata that tells the application exactly who is asking and what environment they are running in. If you attempt to replicate this behavior using curl, you often encounter situations where the server returns an error or a blank page, even though the exact same URL works perfectly fine in Chrome or Firefox.

This discrepancy usually stems from missing headers or specific user agent strings that older or stricter applications expect. In your specific scenario with the AOL endpoint (ScreenNameFlow), the server is likely checking the User-Agent header to determine if the request comes from a legitimate browser or an automated script. By default, curl sends a generic identifier like curl/7.x.x. If the backend logic explicitly blocks non-browser requests, your script will fail immediately.

Setting the User-Agent Correctly

The most common fix is overriding the User-Agent header in your curl configuration. However, simply changing this string isn't enough to emulate a browser perfectly. You must ensure you are passing the exact headers that a real browser sends for that specific site.

In PHP, you can achieve this by modifying the default options passed to CURLOPT_HTTPHEADER. Your current implementation uses ini_set('user_agent', ...), but this global setting does not automatically apply to curl_exec unless explicitly configured via the handle options.

Here is how you should structure the request to mimic a browser:

$url = 'https://new.aol.com/productsweb/subflows/ScreenNameFlow/AjaxSNAction.do?s=username&f=firstname&l=lastname';

// Initialize curl with specific headers mimicking a browser
$ch = curl_init();

// Set the User-Agent to match a modern or legacy browser as needed
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36');

// It is often necessary to set other standard headers browsers send automatically
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language: en-US,en;q=0.5',
    'Accept-Encoding: gzip, deflate',
    'Connection: keep-alive',
    'Upgrade-Insecure-Requests: 1'
));

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // Capture output instead of printing directly

// Optional: Some older servers require specific cookie handling or referer headers
// curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt"); 
// curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");

$result = curl_exec($ch);
curl_close($ch);

print $result;

Beyond the User-Agent: Session and Cookies

Even if you successfully mimic the browser's identity with the User-Agent string, the request might still fail due to session state. Many web applications, including older portals like the AOL example above, rely on cookies to maintain a session or validate that the request is part of an ongoing interaction.

If the server expects a specific cookie (like a session ID) that wasn't present in your initial curl call, it may return a 403 Forbidden or a generic error page. To emulate a browser fully, you often need to persist cookies between requests. You can do this by enabling CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE. This tells curl to save any response headers containing Set-Cookie into a file and read that file before sending subsequent requests, effectively simulating the browser's history of visiting the site.

The Limitations of Emulation

It is important to recognize that emulating a browser request has inherent limitations. While matching headers solves identity issues, it does not replicate JavaScript execution. If the target page relies on client-side scripts (like jQuery or React) to populate data before making an internal AJAX call, a simple curl GET request might return the initial HTML skeleton rather than the fully rendered content you see in the browser.

In such cases, tools like Puppeteer or Playwright are superior because they drive a real browser engine that executes JavaScript. However, for static endpoints or legacy APIs that do not require script execution, properly configured curl is a lightweight and efficient solution. Always check if the server returns different content based on the Accept header; some sites serve JSON to non-browser clients even when served HTML to browsers.

For developers working within the Laravel ecosystem, handling these scenarios often involves creating a dedicated service class that wraps Http\RemoteRequest or utilizes the Guzzle package, which provides a more ergonomic interface for managing cookies and headers compared to raw curl. If you need robust solutions for scraping or API integration in Laravel projects, exploring resources from https://laravelcompany.com can provide further insights into handling external requests within your framework.

Our network