How can I send cookies using PHP curl in addition to CURLOPT_COOKIEFILE?

How can I send cookies using PHP curl in addition to CURLOPT_COOKIEFILE?

  1. Blog
  2. 15 November 2026
  3. 4 min read

Stefan Bogdanescu

Founder & Senior Architect

Why CURLOPT_COOKIEFILE Alone Isn't Enough for Complex Scraping Targets

When scraping modern web applications, relying solely on CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR often leads to intermittent failures. The scenario you described—where a script fails roughly 20% of the time while a browser succeeds consistently—is a classic symptom of missing state that PHP's default cookie handling cannot capture.

The core issue lies in how cookies are managed between requests versus what the target server actually expects. In your specific case, the discrepancy arises because the server requires session tokens and tracking identifiers (like Google Analytics __utma or custom flags like www-47873) that are not persisted in a standard text-based cookie file by default.

The Limitation of Standard Cookie Files

Your current implementation correctly initializes the curl handle with both CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE. This tells cURL to read existing cookies from the file before making a request and write any new response cookies back to that file afterwards.

However, this mechanism has a critical blind spot: it cannot inject cookies that do not exist in the file prior to the request.

If a cookie like __utma is set via JavaScript on the client side (e.g., inside an <script> tag executing on the page) and then sent as part of the subsequent form submission, a purely server-side PHP script cannot "see" or "read" that cookie unless it has been explicitly written to the jar file in a previous step. If your scraper starts fresh or if the logic for reading cookies from the response body is missing, the cookie jar remains empty regarding those specific tracking tokens. Consequently, when you submit the form, cURL sends only the session ID (JSESSIONID) but misses the analytics and custom tracking cookies.

The Role of JavaScript-Generated Cookies

You asked if these missing variables are due to JavaScript setting them. In high-probability cases like yours, yes. Modern web frameworks often initialize analytics scripts or set session flags via client-side JavaScript immediately after page load.

A browser handles this seamlessly. When fetch or XMLHttpRequest runs in the browser, it executes the JS, reads the resulting cookie string from the browser's internal store, and automatically attaches it to the outgoing HTTP header. The server receives a complete Cookie header containing every identifier required for its logic.

PHP cURL, however, treats cookies strictly as opaque data stored in files or arrays. It does not execute JavaScript. If your scraper fetches the HTML, extracts the form, but never explicitly reads the Set-Cookie headers from the previous response to populate a cookie array, the subsequent request will lack those values. The fact that you see www-20480=MIFBNLFDFAAA in the browser but not in cURL suggests the server is rejecting requests where this specific key-value pair is absent, likely treating it as an invalid or missing session state.

Constructing a Robust Header List

To resolve this, you must move beyond passive cookie file management and actively construct the Cookie header based on known requirements. You can achieve this by manually building an array of headers and injecting them into your cURL handle using CURLOPT_HTTPHEADER.

Here is how you should structure the request to ensure all expected cookies are sent:

// 1. Initialize with a cookie jar to capture response cookies if needed later
$ckfile = tempnam("/tmp", 'cookiename');
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);

// 2. Define the headers array
$headers = [];

// 3. Add static cookies that are known to be required or can be scraped from HTML
// Note: You must extract these values from the previous response's Set-Cookie header
// or hardcode them if they are constant across sessions.
$knownCookies = [
    'subdomainPARTNER=NSINT',
    'BCPermissionLevel=PERSONAL' 
];

foreach ($knownCookies as $cookie) {
    $headers[] = 'Cookie: ' . $cookie;
}

// 4. If you have a session ID from the previous response (e.g., JSESSIONID), add it
// This prevents the server from seeing two different session IDs or missing one entirely.
session_id_from_prev_response = '7BC2A5277A4EB07D9A7237A707BE1366'; 
$headers[] = 'Cookie: JSESSIONID=' . session_id_from_prev_response;

// 5. Set the headers option
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

// ... rest of your request setup (URL, POST data, etc.)

Why Header Differences Matter

Beyond cookies, other header discrepancies between your script and a browser can trigger security filters or return incorrect content. The target server likely checks the Accept header to determine how to format the response. Your script defaults to Accept: */*, while the browser sends a specific list including text/html. If the server logic is strict, it might serve an error page or an API-only response to requests lacking the correct Accept profile.

Similarly, missing Referer headers can cause anti-fraud systems to block requests that appear to come from nowhere. While not always strictly necessary for form submission, mimicking the browser's header fingerprint (including User-Agent, Accept-Language, and Referer) significantly increases the success rate of your scraper by aligning its behavior with legitimate client traffic.

Conclusion

The most reliable approach for scraping complex targets is to combine a persistent cookie jar for dynamic session tokens with an explicit list of headers containing all known static or scraped cookies. Do not rely on CURLOPT_COOKIEFILE to magically inject cookies that were never written to the file; you must pro

Our network