How to remove accents and turn letters into "plain" ASCII characters?

How to remove accents and turn letters into "plain" ASCII characters?

  1. Blog
  2. 15 November 2026
  3. 4 min read

Stefan Bogdanescu

Founder & Senior Architect

Normalizing Strings: Removing Accents and Converting to ASCII

Stripping diacritics from text—transforming ÈâuÑ into Eaun—is a common requirement in data processing, search indexing, and user input validation. While it seems like a simple string manipulation task, the underlying approach varies significantly depending on your programming language, performance constraints, and how strictly you need to handle edge cases. There is no single "magic" regex that works universally across all scripts without some caveats.

The Unicode Reality

The fundamental issue lies in how Unicode represents characters. A character like È might be stored as a single code point (composed form) or as two separate code points: the base letter E followed by a combining grave accent. If you simply iterate over a string and replace specific characters, you risk leaving behind hidden combining marks that still affect sorting or matching.

The most robust technical approach involves Unicode normalization. Specifically, you should normalize the string to NFD (Normalization Form Decomposition) before stripping accents. NFD splits composed characters into their base character and combining marks. Once decomposed, you can filter out any characters classified as "Mark" types.

Python Implementation

In Python, the unidecode library is often cited, but it relies on a massive lookup table which can be heavy for memory-constrained environments. For a dependency-free solution, leveraging the standard library's unicodedata module is more efficient and transparent.

Here is a function that demonstrates the NFD approach:

import unicodedata

def remove_accents(input_str):
    # Normalize the string to NFD (decomposed form)
    normalized = unicodedata.normalize('NFD', input_str)
    
    # Filter out combining characters (category starts with 'M')
    stripped = ''.join(
        char for char in normalized 
        if unicodedata.category(char)[0] != 'M'
    )
    
    return stripped

# Example
print(remove_accents("ÈâuÑ")) # Output: Eaun

This method works because unicodedata.category returns a two-letter string representing the Unicode general category. Combining marks always start with 'M'. By removing them, you are left only with base letters. Note that this process does not automatically convert Cyrillic or Greek characters to Latin equivalents; it strictly removes diacritics. If you need transliteration (e.g., turning Ñ into N), a different algorithm or library like unidecode is required, though the normalization step remains a best practice for handling composed characters first.

PHP and Laravel Contexts

In PHP, developers often reach for regular expressions to solve this, such as /[^a-zA-Z0-9 ]/. While this removes non-ASCII symbols, it fails on composed characters because the regex engine sees E and the accent as two distinct entities, but a simple character class might not catch every variant of the combining mark correctly depending on the PHP version and locale settings.

A more reliable pattern in PHP mirrors the Python logic: normalize first, then filter.

function removeAccents(string $text): string {
    $normalized = normalizer::normalize($text, Normalizer::FORM_DECOMPOSE);
    
    return preg_replace('/\p{M}+/', '', $normalized);
}

// Example
echo removeAccents("ÈâuÑ"); // Outputs: Eaun

Using the Normalizer class ensures you are dealing with canonical representations. The regex \p{M} matches any character within the General Category "Mark, Nonspacing". This approach is generally safer than trying to enumerate every possible accented character in a character class. If you are working within the Laravel ecosystem, you might find utility in packages that wrap these native functions for convenience, but understanding the underlying normalization logic helps when debugging why a specific edge case fails.

Trade-offs and Pitfalls

The primary trade-off with Unicode normalization is performance. Decomposing every string adds computational overhead compared to a simple replace or regex substitution on a static list. However, this cost is negligible for typical text processing but becomes significant in high-throughput systems like real-time chat logs or massive log aggregators.

Another pitfall is the handling of characters that have no direct ASCII equivalent. The NFD + filter method will keep the base character even if it is not a Latin letter (e.g., converting Ñ to N requires transliteration, not just accent removal). If your use case demands strict ASCII-only output where non-Latin letters are dropped entirely, you must add an additional filter step after removing accents to exclude any remaining characters outside the A-Z and 0-9 range.

Finally, be wary of relying solely on regex patterns like [\u00C0-\u017F] without normalization. These ranges often cover precomposed characters but miss combining marks that sit above or below standard ASCII codes. Normalization is the only way to guarantee consistency across different input methods and operating systems.

Our network