PHP: Convert any string to UTF-8 without knowing the original character set, or at least try

PHP: Convert any string to UTF-8 without knowing the original character set, or at least try

  1. Blog
  2. 15 November 2026
  3. 4 min read

Stefan Bogdanescu

Founder & Senior Architect

PHP: Convert any string to UTF-8 without knowing the original character set, or at least try

Handling international data is a perennial headache in application development. When building for a global audience, you need everything stored in your database to be UTF-8. However, relying on client-side directives like <form accept-charset="utf-8"> is often insufficient; if the user's browser sends headers incorrectly or if data comes from an uploaded text file with unknown origins, your application receives raw bytes that may not match your expected encoding.

The naive approach usually involves mb_detect_encoding(). While this function attempts to guess the source encoding, it is notoriously unreliable for mixed-content strings or when the input contains invalid byte sequences. A common failure mode is seen when processing French text: passing 'fiancée' through iconv(mb_detect_encoding($text), "UTF-8", $text) often results in truncated output like 'fianc'. This happens because the detection logic misidentifies the source encoding (often guessing Windows-1252 or ISO-8859-1) and then fails to map the specific accented character correctly during conversion, dropping it entirely rather than replacing it.

A more robust strategy requires moving away from single-pass detection and toward a multi-stage sanitization pipeline. The goal is not perfection—since some data might be truly unencodable—but rather resilience: ensuring that valid characters are preserved and invalid ones are handled gracefully so they don't corrupt the database schema.

The first step in this pipeline should always be to ensure the PHP runtime environment is configured correctly. Set mb_internal_encoding('UTF-8') or configure your application's configuration files to default to UTF-8. This ensures that strlen() and other string functions operate on a consistent basis, though it does not magically fix the input bytes themselves.

When handling file uploads where the encoding is unknown, you can implement a heuristic approach. Instead of relying solely on detection libraries which often fail with complex scripts or mixed encodings, consider using iconv with a specific strategy for error handling. In modern PHP versions (7.4+), you can pass the //IGNORE suffix to the target encoding in the iconv call.

function normalizeToUtf8($string) {
    // First, attempt to detect the source encoding.
    $detected = mb_detect_encoding($string, ['UTF-8', 'ISO-8859-1', 'Windows-1252', 'CP1250', 'CP1251', 'KOI8-R', 'BIG5', 'GB2312'], true);

    if ($detected === false || $detected === 'UTF-8') {
        // If detection fails or it's already UTF-8, try a direct conversion with strict checking first.
        return $string; 
    }

    // Attempt conversion from detected encoding to UTF-8.
    // The //IGNORE flag tells iconv to skip invalid byte sequences instead of returning false or throwing errors.
    return iconv($detected, 'UTF-8//IGNORE', $string);
}

However, mb_detect_encoding is not infallible and can misidentify encodings if the string is short or contains mixed scripts. A more aggressive fallback strategy involves iterating through common legacy encodings used by Windows and older Unix systems. If iconv fails to produce valid UTF-8 output (which you can check using mb_check_encoding()), apply a secondary conversion assuming ISO-8859-1, as this is the most common single-byte encoding for Western European languages that isn't ASCII.

function robustUtf8Converter($string) {
    // Ensure input is treated as UTF-8 initially to avoid double-encoding issues if it already was.
    $converted = iconv('ISO-8859-1', 'UTF-8//IGNORE//TRANSLIT', $string);
    
    // Check if the result is valid. If not, try falling back to other common encodings.
    if (mb_check_encoding($converted, 'UTF-8')) {
        return $converted;
    }

    // Fallback loop for other common single-byte encodings often found in legacy files
    $fallbackEncodings = ['Windows-1252', 'CP1250', 'CP1251', 'KOI8-R'];
    
    foreach ($fallbackEncodings as $fallback) {
        $temp = iconv($fallback, 'UTF-8//IGNORE', $string);
        if (mb_check_encoding($temp, 'UTF-8')) {
            return $temp;
        }
    }

    // If absolutely nothing works, return the original string. 
    // This is safer than returning a truncated version or throwing an exception.
    return $string;
}

This approach prioritizes data integrity over strict encoding purity. By using the //TRANSLIT flag in the conversion call, you allow PHP to attempt transliterating characters that cannot be represented in UTF-8 (e.g., converting a Cyrillic 'а' to an ASCII 'a' if no equivalent exists), though this should be used carefully as it changes the data. The //IGNORE flag is generally preferred for database storage to simply drop invalid sequences, ensuring the record remains intact rather than becoming NULL or causing a write error.

It is crucial to remember that there is no silver bullet for unknown encodings. As with file uploads, asking users to specify their encoding via a meta tag in an uploaded file is a good UX practice but offers zero defense against malicious actors attempting to inject data with unexpected byte sequences. Therefore, defensive programming must assume the worst-case scenario: the input contains mojibake (garbled text) and invalid byte clusters. Your code should be prepared to strip these out or normalize them without crashing.

Our network