PHP json encode - Malformed UTF-8 characters, possibly incorrectly encoded
When you call json_encode($data) on an array containing Russian text, and PHP throws a warning about "Malformed UTF-8 characters," it usually means one of two things: your string is not actually valid UTF-8, or the bytes are valid UTF-8 but represent a surrogate pair that needs special handling.
In your specific case, where mb_detect_encoding() reports UTF-8, the culprit is almost certainly an invalid byte sequence within the string. You mentioned seeing characters like "Ñаâ". That diamond with a question mark (or similar placeholder) is a classic symptom of how terminals or debuggers handle bytes that don't form a valid Unicode code point when interpreted as UTF-8.
Why utf8_encode Isn't the Solution
You noted that running utf8_encode() on the data bypasses the error, but the text looks wrong afterward. This is expected behavior because utf8_encode() assumes the input is ISO-8859-1 (Latin-1). If your source string contains a mix of valid Russian characters and invalid byte sequences, converting the whole block assuming Latin-1 will shift every byte to a different code point. You aren't fixing the encoding; you are corrupting it further by misinterpreting the existing bytes.
The real issue lies in how PHP's internal C-string functions handle invalid UTF-8 during serialization. The json_encode function passes your string to the underlying C library (libjson_c or similar), which is strict about validity. If a byte sequence looks like it starts a multi-byte character but fails to complete correctly (e.g., an extra high-bit byte at the end of a string, or a lone continuation byte), the encoder rejects it.
Validating and Cleaning the String
Before serialization, you must ensure the string contains only valid Unicode code points. The most robust approach is to use mb_convert_encoding to normalize the data into a known-good state.
If your source might contain invalid UTF-8 sequences masquerading as UTF-8, you can convert them to the UTF-8 encoding while replacing invalid characters:
$data['field'] = mb_convert_encoding(
$data['field'],
'UTF-8',
'UTF-8' // Attempting conversion forces strict validation
);
However, this specific signature often throws an error if the input is already declared as UTF-8 but contains bad bytes. A safer pattern when you know the source claims to be UTF-8 but might be malformed is to use mb_convert_encoding with a fallback or to explicitly handle the invalid byte sequences:
// Convert potentially invalid UTF-8 to valid UTF-8, replacing errors with replacement char
$data['field'] = mb_convert_encoding(
$data['field'],
'UTF-8',
'UTF-8'
);
// If the above doesn't work because the input is technically "UTF-8" but invalid,
// you might need to detect the actual encoding first or normalize to Unicode scalar values.
A more aggressive cleaning strategy involves converting the string to UTF-32 (which can represent any sequence), then back to UTF-8, effectively filtering out invalid byte sequences:
$cleaned = mb_convert_encoding($data['field'], 'UTF-32', 'UTF-8');
$data['field'] = mb_convert_encoding($cleaned, 'UTF-8', 'UTF-32');
This process forces PHP to interpret the bytes as valid UTF-32 units. If a sequence was invalid in UTF-8, it won't map correctly to a code point in UTF-32, and converting back will result in standard replacement characters (�) for those bad spots. This ensures json_encode receives a strictly valid string.
Handling Surrogate Pairs
There is a second edge case often confused with malformed UTF-8: surrogate pairs used by JavaScript or other languages to represent emojis or rare CJK characters. In PHP, strings are Unicode code points (UTF-32 internally in modern versions), but json_encode expects the output to follow JSON standards. If your input contains lone surrogates (U+D800 to U+DFFF) which are invalid in standard UTF-8 text streams, json_encode will fail.
If you are dealing with data coming from a JavaScript environment that uses surrogate pairs for characters outside the Basic Multilingual Plane, you must ensure those pairs are kept together as a single code point before encoding to JSON. Using mb_convert_encoding with 'UTF-8' as both source and target usually handles this correctly provided the input is valid UTF-8. If the input is truly broken bytes (like your "Ñаâ" example), the conversion-to-UTF-32-and-back method described above remains the most reliable fix.
The Trade-off of Replacement Characters
By cleaning the data, you are accepting that some characters might be replaced with � if they cannot be validly interpreted. This is a necessary trade-off when dealing with malformed input. You cannot magically reconstruct what those bytes were supposed to represent without knowing the original encoding scheme (e.g., Windows-1251 vs ISO-8859-7).
If your application requires strict data fidelity and you cannot tolerate replacement characters, you must trace back to the source of the invalid bytes. Is it a database column with mixed encodings? Is it an API receiving broken payloads? Fixing the corruption at the ingestion point is superior to patching it in PHP.
Conclusion on Encoding Safety
The error message "Malformed UTF-8 characters" is PHP's way of telling you that the byte sequence violates the UTF-8 specification. Running utf8_encode is a band-aid that changes the meaning of the text. The correct path is to validate or normalize the string using mb_convert_encoding to ensure only valid Unicode code points are passed to json_encode. This guarantees that your JSON output is both syntactically correct and semantically safe for downstream consumers.
