Detect file encoding in PHP

Detect file encoding in PHP

  1. Blog
  2. 15 November 2026
  3. 3 min read

Stefan Bogdanescu

Founder & Senior Architect

Detect file encoding in PHP

When combining files programmatically, invisible characters often become visible once they are concatenated into a single stream. The specific symptom of  appearing at the start of a file is a classic indicator that you are dealing with UTF-8 content that includes a Byte Order Mark (BOM), also known as EF BB BF in hex representation. These three bytes sit before the actual text data and are intended for applications reading binary files to determine endianness, but they cause corruption when treated as part of a valid UTF-8 string sequence in a plain text context.

The immediate temptation is to apply utf8_decode() to every file in the loop. This approach is flawed because it assumes all inputs are UTF-8, which might not be true. More importantly, utf8_decode() converts multi-byte UTF-8 characters into single-byte ISO-8859-1 (Latin-1) characters. It does not remove the BOM; in fact, applying it to a file with a UTF-8 BOM often results in garbled output or unexpected character mappings depending on how the decoder handles the initial bytes.

To solve this reliably, you must first detect the encoding of each file before processing it. PHP provides mb_detect_encoding() for this purpose, but relying solely on detection can be risky because the function requires a list of possible encodings to check against. If a file is actually UTF-8 with a BOM, mb_detect_encoding() will correctly identify it as UTF-8, but you still face the issue of the leading bytes.

The most robust strategy involves stripping the BOM if it exists, regardless of how the file was encoded. The standard way to handle this in PHP is to check for the specific byte sequence \xEF\xBB\xBF at the beginning of the string and remove it if found. This allows you to treat the content as standard UTF-8 without worrying about the hidden marker interfering with your combined output.

Here is a refined implementation that handles both detection and BOM removal:

$output = '';

foreach ($files as $filename) {
    $content = file_get_contents($filename);
    
    // Remove UTF-8 BOM if present
    if (substr($content, 0, 3) === "\xEF\xBB\xBF") {
        $content = substr($content, 3);
    }

    // Optionally verify encoding or handle other formats dynamically
    // For this specific problem, stripping the BOM is often sufficient.
    
    $output .= $content . "\n";
}

file_put_contents('combined.txt', $output);

This approach separates the concerns of data acquisition and data cleaning. By explicitly checking for the BOM sequence, you ensure that whether the file was originally UTF-8, ASCII, or something else, the output stream remains clean. If your application also needs to support other encodings like Windows-1252 or ISO-8859-1, you would need to inspect $content after stripping the BOM (if necessary) and apply mb_convert_encoding() based on the detected result. However, for simple concatenation tasks where UTF-8 is the target standard, removing the BOM is the critical step.

Be cautious when using mb_detect_encoding(). It performs a statistical analysis of the character set and may return "ISO-8859-1" for files that are actually UTF-8 if the content doesn't contain enough multi-byte characters to distinguish them statistically. In such cases, assuming UTF-8 is safer unless you have explicit metadata indicating otherwise. Additionally, always ensure your output file is saved with UTF-8 encoding without a BOM to maintain consistency across different platforms and editors.

If you are working within the Laravel ecosystem, consider using the Str facade which offers utilities for string manipulation that can be integrated into your service classes or controllers to handle these encoding issues cleanly.

Our network