Detect encoding and make everything UTF-8

Detect encoding and make everything UTF-8

  1. Blog
  2. 15 November 2026
  3. 4 min read

Stefan Bogdanescu

Founder & Senior Architect

Detect encoding and make everything UTF-8

Handling text from external sources like RSS feeds introduces a classic headache: inconsistent character encodings. You might expect uniformity, but in reality, feeds often mix UTF-8, ISO-8859-1 (Latin-1), Windows-1252, and even legacy encodings like MacRoman. The specific symptoms you described—where an umlauted ß appears as Ÿ, then ß, or sometimes correctly as ß but with broken rendering downstream—are telltale signs of encoding mismatches between the source, your parsing logic, and your database storage layer.

The core issue is rarely just "different encodings"; it's a chain of double-encoding errors. When you see ß, that is almost certainly an ISO-8859-1 string (which contains the bytes for ß) that was incorrectly interpreted as UTF-8 and then saved into a database expecting UTF-8. The à character is actually the UTF-8 representation of the byte 0xC3, while the Ÿ pair often indicates Latin-1 bytes being read through a wrong filter.

To solve this, you need a robust strategy for detection and normalization. Your proposed function using mb_detect_encoding() combined with iconv() is a logical starting point, but it has a critical flaw: mb_detect_encoding() can return an empty string or guess incorrectly if the text contains mixed encodings (a common scenario in HTML content) or if the byte sequence is ambiguous. Furthermore, relying solely on detection without validation leaves your system vulnerable to silent corruption.

Here is a more robust approach using PHP's built-in functions:

function normalizeToUtf8(string $text): ?string {
    // 1. Detect the current encoding.
    // If mb_detect_encoding returns an empty string or 'unknown', 
    // we default to ISO-8859-1 as it is a very common fallback for legacy content.
    $detected = mb_detect_encoding($text, ['UTF-8', 'ISO-8859-1', 'Windows-1252'], true);

    if ($detected === false || $detected === null) {
        // Fallback strategy: Assume ISO-8859-1 or Windows-1252 for legacy text.
        // Note: mb_detect_encoding is not 100% reliable on mixed content.
        return iconv('ISO-8859-1', 'UTF-8//IGNORE', $text) ?? null;
    }

    // 2. Convert to UTF-8.
    // The '//IGNORE' modifier in iconv discards characters that cannot be converted,
    // preventing further corruption of the remaining valid data.
    return iconv($detected, 'UTF-8//IGNORE', $text);
}

Why mb_detect_encoding isn't a silver bullet

The function mb_detect_encoding() scans the byte stream to find a pattern that matches a known encoding signature. While effective for pure text, it struggles when an RSS feed contains HTML entities mixed with raw text or when the file starts with one encoding and switches mid-stream. In your specific case, if the source returns ß, mb_detect_encoding might correctly identify it as ISO-8859-1 because that sequence is valid in Latin-1 but invalid in UTF-8. The function then passes it to iconv, which correctly translates those bytes into the proper Unicode character ß.

However, if the source already returns correct UTF-8 (ß), mb_detect_encoding will identify it as such, and iconv will return the string unchanged. This is the desired behavior. The danger lies in the fallback logic. If detection fails, blindly assuming UTF-8 and passing it through iconv('UTF-8', 'UTF-8') works, but if you assume ISO-8859-1 when it is actually UTF-8, you will corrupt valid high-bit characters into garbage.

Handling the "Already Correct" Case

You asked when to do nothing with the input. The answer is: only after successful detection and conversion. If your database connection (e.g., MySQL) and table are strictly configured for utf8mb4 (the modern standard, as opposed to the deprecated utf8), storing data in any other encoding will result in data loss or display issues.

There are two main scenarios for your workflow:

  1. Strict Validation: You only process feeds that explicitly declare their charset via a Content-Type header (e.g., <meta charset="UTF-8">). In this case, you trust the header and skip detection, using iconv only if you suspect corruption.
  2. Aggressive Normalization (Recommended for RSS): You assume nothing until proven otherwise. Use the detection logic above. If iconv reports an error or returns a shorter string than expected, log it for manual inspection rather than silently overwriting valid data with garbage.

Database Configuration Matters

Finally, ensure your database engine is configured to reject non-UTF-8 data. In MySQL, this means setting the column charset to utf8mb4. If you are using Laravel or another modern framework, check your .env file for DB_CONNECTION and ensure no legacy encoding settings are forcing a different default.

By combining accurate detection with a safe conversion strategy that handles edge cases like double-encoding, you can ensure that "Fußball" always remains "Fußball" in your database, regardless of what the external feed throws at you. For more guidance on handling internationalization and text processing in modern PHP applications, see how LaravelCompany approaches these challenges in their documentation.

Our network