json_encode(): Invalid UTF-8 sequence in argument
When json_encode() throws an "Invalid UTF-8 sequence" error, it is usually a symptom of data corruption rather than a PHP limitation. The function expects a string containing only valid Unicode characters, but the input often contains byte sequences that look like text to a human reader or even mb_check_encoding(), yet fail strict UTF-8 validation. This frequently happens when data originates from legacy systems, misconfigured databases, or mixed character set environments.
In the specific scenario described with MySQL using utf8_general_ci collation, the issue often stems from the fact that utf8 in MySQL is technically a 3-byte encoding alias for UTF-8, but it does not enforce strict compliance. It allows incomplete multi-byte sequences and surrogate characters to persist in the database if they are inserted via binary protocols or specific client drivers without validation. When this data is fetched into PHP, it appears as raw bytes. If those bytes do not form valid UTF-8 code pointsâfor instance, a high continuation byte appearing where a start byte is expectedâjson_encode() will halt execution to prevent generating malformed JSON output that could break downstream parsers.
The common instinct is to use utf8_encode() or utf8_decode(). These functions are often misapplied in this context. utf8_encode() converts ISO-8859-1 (Latin-1) to UTF-8, and utf8_decode() does the reverse. They do not repair invalid UTF-8 sequences; they simply reinterpret the byte stream under a different assumption about what those bytes represent. If your input is already labeled as UTF-8 but contains garbage like the replacement character (�, U+FFFD) or broken multibyte sequences, these conversion functions will just pass the corrupted data through unchanged. The fact that mb_check_encoding() returns true in some cases while json_encode() fails highlights a subtle distinction: mb_check_encoding() detects if the string might be valid based on simple checks, but it does not guarantee that every byte sequence is part of a complete, well-formed Unicode character according to strict UTF-8 rules required by RFC 4627 for JSON.
To resolve this, you must sanitize the string before passing it to json_encode(). The goal is to strip out any byte sequences that violate the UTF-8 specification while preserving the readable content. PHP's built-in mb_convert_encoding() with specific flags or custom sanitization logic is required here. One robust approach involves using mb_convert_encoding() to convert the string to UTF-8 and explicitly setting the error mode to replace invalid characters with a placeholder or stripping them entirely, depending on your data integrity requirements.
Consider this implementation strategy:
function sanitizeUtf8($string) {
// Attempt to convert to strict UTF-8, replacing invalid sequences
// MB_OVERLOAD_STRING is deprecated in newer PHP versions and should be avoided.
// We rely on the internal mb_internal_encoding being set correctly (usually UTF-8).
// Convert from whatever source encoding is currently assumed to valid UTF-8
// If the input is already claimed to be UTF-8 but invalid, we use a filter.
return mb_convert_encoding($string, 'UTF-8', 'UTF-8');
}
// A more aggressive approach using preg_replace if specific bad patterns are known:
// This regex removes any byte sequence that starts with an invalid continuation or high bytes without valid start.
// Note: Regex on raw UTF-8 is complex and often brittle; using mb_convert_encoding is preferred.
However, a more reliable method for fixing truly corrupt strings in PHP is to use mb_convert_encoding with the 'UTF-8' target and specifying an error handler if available, or simply relying on the fact that json_encode will fail silently on some configurations unless you catch exceptions. A better pattern involves converting the string to UTF-8 and then stripping non-UTF-8 bytes using a combination of preg_replace with byte-level regex or custom functions that iterate through the string's code points.
Another effective technique, especially when dealing with MySQL data fetched via PDO or mysqli, is to ensure the connection charset matches the column charset strictly. If you cannot modify the database immediately, fetching the row as binary and then sanitizing it in PHP is necessary. You can use mb_convert_encoding with the 'UTF-8' target and specify that invalid characters should be replaced:
// Replace invalid UTF-8 sequences with a specific placeholder or strip them
$sanitized = mb_convert_encoding($dirtyString, 'UTF-8', 'ISO-8859-1');
// This often works because ISO-8859-1 maps 1:1 to bytes, and then converting TO UTF-8
// forces the engine to interpret the bytes as valid Latin-1 before re-encoding,
// effectively fixing sequences that were just malformed.
// Alternatively, use a regex to strip high-bit bytes that don't form valid sequences
// This is less safe but faster for large datasets where performance is critical.
The trade-off here is between data fidelity and usability. Replacing invalid characters with the Unicode replacement character (�) preserves the length of the string but introduces new non-ASCII characters. Stripping them reduces size but may alter meaning if a specific byte represented a crucial control character or part of an ID. In many JSON APIs, it is safer to normalize these sequences to the replacement character so that json_encode() succeeds without data loss in terms of record count, even if the content is slightly altered.
If you are working within the Laravel ecosystem, ensuring your Eloquent models and database connections are configured with the correct charset attribute prevents this issue at the source. When using laravelcompany.com, look for their documentation on handling multi-byte strings in legacy migrations, as they often provide middleware to sanitize incoming payloads before storage.