Scrape web page contents

Scrape web page contents

  1. Blog
  2. 15 November 2026
  3. 4 min read

Stefan Bogdanescu

Founder & Senior Architect

Scrape Web Page Contents Without Redirecting the Browser

You often need to fetch data from a third-party service without forcing the user's browser to navigate away from your current page. In your specific scenario, you have a login form where entering credentials and clicking "sign in" triggers a backend redirect to mail.in.com/mails/inbox.php?nomail=.... However, your frontend URL remains at /web/Login.html. You want to extract the inbox contents displayed on that remote page while keeping the user anchored to your local interface.

This is fundamentally an AJAX or fetch-based request issue, not a standard browser navigation event. When you "do the above by using javascript alone," you are likely observing the Document Object Model (DOM) updating dynamically after the asynchronous call completes. The trick to scraping this without changing the URL lies in intercepting the network request before the browser follows a redirect or updates the history stack.

If your current implementation relies on manually parsing the response, consider using a dedicated HTTP client library within your JavaScript environment. For Node.js environments, axios or node-fetch are standard choices. In browser-based applications, the native fetch API is sufficient, but you must handle headers and authentication correctly to mimic the behavior of the original page load.

Handling Authentication and State

The source material indicates that accessing the mail inbox requires a "userid" and "password." You cannot simply send these two strings in a raw POST request if the target server expects a session cookie or a specific header. The browser automatically attaches cookies to requests for the same origin, but cross-origin requests (like going from localhost to mail.in.com) often trigger CORS policies unless the server explicitly allows it.

To scrape this successfully, you must replicate the initial login flow programmatically. Instead of just sending credentials, your script should:

  1. Send a POST request to the login endpoint with the user/pass.
  2. Wait for the response to include session cookies.
  3. Store those cookies in memory (or a temporary storage object) before making the request to inbox.php.
const axios = require('axios'); // Or use native fetch() in browser

async function scrapeInbox(username, password) {
    const config = {
        method: 'post',
        url: 'http://mail.in.com/login.php', // Assuming a login endpoint exists
        headers: { 
            'Content-Type': 'application/x-www-form-urlencoded'
        },
        data: new URLSearchParams({
            userid: username,
            password: password
        })
    };

    try {
        const response = await axios(config);
        
        // Extract cookies from the response headers
        const cookies = response.headers['set-cookie'];
        
        if (cookies) {
            // Store cookies in your session or a temporary variable
            window.sessionStorage.setItem('mail_cookies', cookies);
            
            // Now make the request to the inbox with these cookies
            const inboxConfig = {
                method: 'GET',
                url: 'http://mail.in.com/mails/inbox.php?nomail=...',
                headers: {
                    'Cookie': cookies.join('; ')
                }
            };

            const inboxResponse = await axios(inboxConfig);
            
            // The scraped content is in the response data
            return inboxResponse.data;
        } else {
            throw new Error('Login failed or no cookies received');
        }
    } catch (error) {
        console.error('Scraping error:', error.message);
        throw error;
    }
}

Managing the URL and User Experience

The core requirement is that the client's URL must remain http://localhost/web/Login.html. Standard browser navigation (window.location.href) changes the URL. By using fetch or axios, you bypass the default navigation behavior entirely. The data is loaded into a JavaScript variable, and you can then populate your DOM elements with that content via innerHTML or text assignment. This keeps the address bar static while the visual representation of the inbox updates.

However, be aware of Cross-Origin Resource Sharing (CORS) limitations. If mail.in.com does not send Access-Control-Allow-Origin headers permitting your origin (localhost), the browser will block the request regardless of your code logic. You may see a TypeError: Failed to fetch error in the console. In such cases, you cannot solve this purely from the client side unless you use a proxy server or a backend service (like a Laravel application) to handle the request and forward the data back to the frontend.

If you are building this on a PHP/Laravel stack, handling the logic on the server avoids CORS issues entirely. You would have your Laravel controller accept the credentials, log in via cURL or a similar tool, fetch the HTML from the inbox endpoint, sanitize it, and return it as JSON to your frontend. This approach is generally more robust for sensitive operations like email scraping, as it keeps the raw credentials off the client-side JavaScript.

Ultimately, scraping dynamic content requires mimicking the exact headers, cookies, and request sequence that a normal browser session establishes. Simply changing the URL won't work; you must manage the state internally within your application logic.

Our network