php substr() function with utf-8 leaves � marks at the end
When you slice a UTF-8 string using the standard substr() function in PHP, you are operating on bytes, not characters. The issue describedâwhere Russian text like "пÑодÑкÑ" gets cut off mid-word and renders as a replacement character (�)âhappens because substr() counts byte offsets rather than code point positions.
In the provided example, the string starts with Cyrillic characters. Each of these characters is encoded as multiple bytes in UTF-8. If your offset calculation assumes one byte equals one character, or if you simply ask for 142 characters without specifying the encoding context, substr() might grab the first 142 bytes. If that 142-byte slice lands in the middle of a multi-byte sequence (for instance, cutting through the third byte of a four-byte Cyrillic character), the resulting string is invalid UTF-8. When PHP or a browser attempts to render this invalid sequence, it replaces the broken fragment with the Unicode replacement character �.
The fact that mb_substr() "did not work" in the original report usually points to one of two specific misconfigurations: either the internal encoding detection failed, or the usage didn't account for how the string was stored versus how it was processed. While mb_substr() is indeed the correct tool for multibyte strings, it relies heavily on PHP's ability to detect the character set correctly. If the $var variable contains a mix of encodings (common in legacy systems) or if the system default encoding is not UTF-8, mb_substr() might default to ASCII or another legacy charset like ISO-8859-1. In those cases, it interprets the Cyrillic bytes as single characters and cuts them exactly where substr() would have, effectively reproducing the truncation error but without the explicit error handling that a proper setup offers.
To handle this robustly, you must explicitly tell PHP how the string is encoded before performing any substring operations. The most reliable method is to ensure the internal encoding is set to UTF-8 using mb_internal_encoding('UTF-8'). This forces functions like mb_substr() to treat every multi-byte sequence as a single logical character, regardless of what happens at the byte level.
Here is a corrected approach that guarantees you are slicing by characters:
<?php
// Ensure internal encoding is UTF-8 before any multibyte operations
mb_internal_encoding('UTF-8');
$var = "Ðензин ÐÑиÑи Ð.С. Ñакже пÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸Ñ Ð²Ñе ÑÐ¸Ð¿Ñ Ð¶Ð¸Ñа и Ñмазок и пÑодÑкÑов в его ÑмеÑиÑÐµÐ»Ñ ÑÑÑанов неÑÑи маÑинного маÑла в ÐеÑинÑе, ÐзмиÑе, Ðлиага и ÐзмиÑе. У Ðомпании еÑÑÑ 3 885 ÑÑанÑий обÑлÑживаниÑ, вклÑÑÐ°Ñ ÑжиженнÑй газ (ÐÐÐ) ÑÑанÑии под ÑиÑменнÑм знаком ÐеÑÑогаз, пÑиблизиÑелÑно 5 000 дилеÑов, ÑÐ¼Ð°Ð·Ð¾Ñ ÑмеÑиÑÐµÐ»Ñ ÑÑÑановок, 12 ÑеÑминалов, и 26 ÐµÐ´Ð¸Ð½Ð¸Ñ Ð¿Ð¾ÑÑавки аÑÑопоÑÑа.";
// Use mb_substr with the explicit length in characters
$foo = mb_substr($var, 0, 142);
echo $foo;
?>
However, relying solely on mb_internal_encoding() can sometimes be brittle if the source string is actually coming from a file or database in a different encoding. In those scenarios, you must use mb_convert_encoding() to normalize the string first. For instance, if $var was read as ISO-8859-1 but contains Cyrillic, converting it to UTF-8 immediately prevents corruption downstream.
// If the source might not be UTF-8, normalize it first
$normalized = mb_convert_encoding($var, 'UTF-8', 'Auto');
$foo = mb_substr($normalized, 0, 142);
There is also a trade-off to consider when dealing with very long strings or high-performance requirements. While mb_substr() is safer, it can be slightly slower than native substr() because of the internal encoding checks and byte manipulation logic. If performance profiling shows that string slicing is a bottleneck in a high-throughput API endpoint, you might consider using substr() but only after verifying that the entire input pipeline is strictly UTF-8 compliant. In such cases, adding error handling to catch invalid UTF-8 sequences becomes part of your data validation strategy rather than just a coding preference.
For developers working within the Laravel ecosystem, this issue often surfaces when parsing user-generated content or handling legacy database fields. If you are building an application where text integrity is criticalâsuch as processing legal documents or financial reportsâyou should treat string slicing as a potential source of data corruption. Ensuring that your application's default encoding matches your storage layer and that you explicitly set the internal encoding before calling multibyte functions is a fundamental practice. For more details on handling encoding complexities in modern PHP applications, resources like https://laravelcompany.com can provide further context on best practices for string manipulation within web frameworks.
The core takeaway remains simple: never assume substr() works on characters. Always verify the encoding state of your data and use mb_substr() with an explicit internal encoding setting to ensure you are slicing logical characters, not arbitrary bytes.