Malformed UTF-8 characters error when trying to use json_encode
When json_encode throws a JSON_ERROR_UTF8 error, it is almost never because the JSON specification itself has changed or your data structure is invalid. The error explicitly states that the string contains malformed UTF-8 characters or incorrectly encoded sequences. This usually points to a mismatch between how your PHP strings are stored in memory and what encoding you expect them to be.
In the scenario described, the array structure looks perfectly valid: keys like 'name' hold strings such as 'Ile Verte', and numeric keys hold integers for distance and IDs. The fact that json_encode works on a similar structure in another script suggests the issue is isolated to how the specific data was created or acquired. The critical clue lies in the behavior when mb_detect_encoding returns different values. If the function detects ASCII, encoding succeeds. When it detects UTF-8, the error triggers. This indicates that while your strings might look like they contain standard characters, the underlying byte sequences are likely corrupted or inconsistent with a valid UTF-8 character set.
This often happens when data is pulled from a legacy system, an old database column defined as CHAR(255) rather than VARCHAR, or via a file import that wasn't properly converted. A string containing only ASCII characters (like "Bastille") works in both cases because ASCII is a subset of UTF-8 and has no byte-order marks to confuse the encoder. However, strings with accented characters like 'Ile Verte' or 'La Tronche Hôpital' expose the problem immediately if those characters are not stored as valid multi-byte UTF-8 sequences.
To resolve this, you must ensure that every string value in your array is a valid Unicode scalar value before passing it to json_encode. The most robust approach is to force a conversion rather than relying on detection. You can iterate through the array and sanitize each string using mb_convert_encoding. This function attempts to map characters from one encoding to another, handling invalid sequences gracefully or raising an error depending on your configuration.
function ensureUtf8(array $data) {
foreach ($data as &$item) {
if (is_array($item)) {
// Recursively process nested arrays
$item = ensureUtf8($item);
} elseif (is_string($item)) {
// Force conversion to UTF-8, ignoring invalid sequences or setting on_error
// The 'CP1252' or 'ISO-8859-1' fallback is common for legacy data
$item = mb_convert_encoding($item, 'UTF-8', 'UTF-8');
// Or if source might be Windows-1252:
// $item = mb_convert_encoding($item, 'UTF-8', 'CP1252');
}
}
return $data;
}
// Usage
$sanitizedData = ensureUtf8($yourArray);
$jsonOutput = json_encode($sanitizedData, JSON_UNESCAPED_UNICODE);
The key here is the second argument of mb_convert_encoding. If you pass 'UTF-8' as both source and target, PHP will attempt to re-normalize the string. If the original bytes were actually valid UTF-8 but flagged incorrectly by a previous detection step (perhaps due to BOMs or mixed encodings), this re-validation often fixes it. If the characters are truly corrupt (e.g., a standalone 0xE9 byte which is invalid in UTF-8), you might need to specify a fallback source encoding like 'ISO-8859-1' or 'CP1252' to see if those legacy encodings contain valid representations of the characters.
Another pitfall is invisible Byte Order Marks (BOM). Sometimes data retrieved from external sources includes a BOM at the start of strings, which json_encode treats as invalid control characters within the string value itself. You can strip these using mb_convert_encoding($string, 'UTF-8', 'UTF-8', MB_ERR_INVALID_CHAR) combined with regex removal if necessary, though usually specifying the correct source encoding handles this automatically.
If you are working in a Laravel application, ensure your Eloquent models and database migrations explicitly define columns as utf8mb4. Legacy utf8 or latin1 columns can silently truncate multi-byte characters, leading to exactly the inconsistencies seen here. While json_encode is a low-level PHP function, ensuring your data source adheres to modern UTF-8 standards (UTF-8) is essential for reliability.
Ultimately, the error is a signal that your string data is inconsistent. By normalizing all string values to strict UTF-8 using mb_convert_encoding before serialization, you eliminate the ambiguity and allow json_encode to produce valid JSON output without triggering JSON_ERROR_UTF8.
