UTF8 Encoding problem - With good examples

UTF8 Encoding problem - With good examples

  1. Blog
  2. 15 November 2026
  3. 3 min read

Stefan Bogdanescu

Founder & Senior Architect

Detecting Mixed Character Encodings in Legacy PHP Applications

Detecting whether a string requires encoding conversion is often more art than science, especially when dealing with legacy systems like PHP 5.2 and MyISAM tables. The core challenge illustrated by the sample data is that visual inspection of output strings is insufficient to determine internal byte representation. A string that appears corrupted in one context might be perfectly valid UTF-8 if interpreted correctly, while another might look correct but actually contain non-UTF-8 bytes masquerading as text.

The provided examples highlight a classic symptom of encoding drift: France Télécom and Condé Nast Publications. In the first case, the original string contains multi-byte sequences that are being misinterpreted when treated as single-byte Latin-1 (ISO-8859-1), resulting in artifacts like é. Conversely, the second example shows a string that appears correct visually but behaves inconsistently under different functions. The key insight is that utf8_encode() and utf8_decode() rely on strict assumptions about the input byte order. If you pass a Latin-1 encoded string to utf8_encode(), it doubles every non-ASCII byte, creating garbage like Télécom. If you pass a UTF-8 string to utf8_decode(), it strips valid multi-byte characters down to single bytes, corrupting the text.

To programmatically decide whether a string needs conversion, you must verify its current state against strict UTF-8 validation rather than guessing based on length alone. However, in PHP 5.2, the most reliable heuristic for mixed environments is checking if the byte count changes after encoding attempts, combined with strict validation functions available in that version.

The following implementation demonstrates a robust detection strategy. It attempts to validate the string as UTF-8 first. If validation fails, it assumes the data is likely Latin-1 and attempts conversion using utf8_encode().

<?php
function ensure_utf8($string) {
    // In PHP 5.2, mb_detect_encoding can identify the current encoding.
    $current = @mb_detect_encoding($string, 'UTF-8, ISO-8859-1', true);
    
    if ($current === 'UTF-8') {
        return $string;
    }

    // If detected as ISO-8859-1 or unknown, attempt strict conversion.
    // We use utf8_encode to map single-byte Latin characters to UTF-8 equivalents.
    $converted = utf8_encode($string);
    
    // Verification step: Ensure the result is actually valid UTF-8.
    if (function_exists('mb_check_encoding')) {
        if (!mb_check_encoding($converted, 'UTF-8')) {
            return $string; // Keep original if conversion creates invalid UTF-8.
        }
    }
    
    return $converted;
}

$text = "France Télécom"; // Example from source material
$result = ensure_utf8($text);
echo $result . "<br>"; 
// Output: France Télécom (correctly handled)
?>

This approach avoids the pitfalls of blindly applying iconv with flags like //TRANSLIT or //IGNORE. While those flags are useful for handling unknown characters gracefully, they can silently alter data. For instance, converting an accented character to its ASCII equivalent (é becomes e) changes the semantic meaning of the text. In a database schema where title is stored as varchar(255) with ENGINE=MyISAM DEFAULT CHARSET=utf8, preserving the original diacritics is critical for data integrity.

The length check mentioned in the update is also a valid heuristic but requires careful application. Comparing strlen($string) against strlen(utf8_encode($string)) works because UTF-8 strings maintain variable lengths (1 to 4 bytes per character), whereas Latin-1 always uses one byte per character. However, this method assumes that the input string does not already contain valid UTF-8 sequences that happen to match the byte count of their single-byte equivalents, which is rare but possible with specific characters. A safer bet in PHP 5.2 is to rely on mb_detect_encoding when available, as it inspects the byte patterns directly.

When migrating from MyISAM tables or legacy scripts, always ensure that your application logic treats all strings as UTF-8 internally. Relying on visual output to debug encoding issues often leads down a rabbit hole of trial and error. By implementing a function that detects the source encoding and converts only when necessary, you maintain consistency across your data layers without introducing silent corruption.

Our network