Erlang Unicode Bug: Bin_is_7bit/1 Issue & Solutions

Aug 5, 2025 by Luna Greco 52 views

Unicode bin_is_7bit/1 Bug Investigation and Solutions

Hey guys! Today, we're diving deep into a quirky bug in Erlang's :unicode.bin_is_7bit/1 function. This function is supposed to check if a binary string contains only ASCII characters (0-127), but it seems like it's not behaving as expected. Let's break down the issue, explore the code, and figure out what's going on and what we can do about it. Buckle up, it's gonna be a fun ride!

Understanding the Bug

So, what's the deal with this bug? The :unicode.bin_is_7bit/1 function is designed to return true if a binary string consists entirely of 7-bit ASCII characters. These characters have values ranging from 0 to 127 in decimal. Anything beyond that range isn't considered a standard ASCII character. However, the function consistently returns false, even for strings that clearly contain only ASCII characters. This is super unexpected and can lead to some serious head-scratching when you're trying to work with character encodings in Erlang.

To put it simply, the core problem lies in how the function determines whether a binary string is 7-bit ASCII. The current implementation has a flaw in its logic, causing it to always return false. This can be particularly frustrating when you have binaries that you know are 7-bit ASCII, but the function disagrees. Think about scenarios where you're processing text files, network data, or any other kind of input where character encoding matters. If you rely on :unicode.bin_is_7bit/1 to validate your data, you might end up with incorrect results and potentially break your application's functionality.

Let's look at some examples to drive this point home. Imagine you have a binary string like <<65, 66, 67>>, which represents the ASCII characters 'A', 'B', and 'C'. You'd expect :unicode.bin_is_7bit/1 to return true for this string, right? But it doesn't! It stubbornly returns false. Similarly, a simple string like "Hello, World!" (represented as <<72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 108, 100, 33>>) also gets a false verdict. This consistent misbehavior highlights a significant issue that needs addressing.

Reproducing the Bug

Alright, let's get our hands dirty and see this bug in action. The easiest way to reproduce it is by using the Erlang shell, IEx (Interactive Elixir), which allows you to execute Erlang code interactively. Fire up your IEx session, and let's try some test cases.

Here’s the code snippet we’ll use to demonstrate the bug:

iex(1)> :unicode.bin_is_7bit(<<1, 2, 3>>).
false
iex(2)> :unicode.bin_is_7bit(<<"Hello, World!">>).
false

As you can see, even though the binary <<1, 2, 3>> clearly contains only characters within the 7-bit ASCII range (0-127), the function returns false. The same thing happens with the "Hello, World!" string. This consistently incorrect behavior is the heart of the problem we're investigating.

These examples make it crystal clear that something is amiss. The function isn't correctly identifying 7-bit ASCII strings. To further confirm this, you can try various other ASCII strings or binaries containing only characters from 0 to 127. You'll find that :unicode.bin_is_7bit/1 stubbornly refuses to acknowledge them as 7-bit ASCII.

This consistent failure to correctly identify 7-bit ASCII strings can have significant implications. If you're relying on this function to validate or process text data, you might end up with unexpected results and potentially introduce bugs into your application. Therefore, understanding why this bug occurs and how to work around it is crucial.

Diving into the Code

Okay, now for the juicy part! Let's crack open the source code and see what's causing this misbehavior. By examining the Erlang source code, we can pinpoint the exact location of the bug and understand the flawed logic behind it. This deep dive will give us the insight we need to propose a fix.

The buggy code resides in the erl_unicode.c file, which is part of the Erlang runtime system. The specific function we're interested in is unicode_bin_is_7bit_1. Here’s the snippet of code that's causing the trouble:

BIF_RETTYPE unicode_bin_is_7bit_1(BIF_ALIST_1)
{
    Sint need;
    if(!is_bitstring(BIF_ARG_1)) {
        BIF_RET(am_false);
    }
    need = latin1_binary_need(BIF_ARG_1);
    // The aligned_binary_size function returns the number of bits, and need represents the number of bytes. The two are of different magnitudes, so this if conditional branch will never be true.
    if(need >= 0 && aligned_binary_size(BIF_ARG_1) == need) {
        BIF_RET(am_true);
    }
    BIF_RET(am_false);
}

static Sint aligned_binary_size(Eterm binary)
{
    Uint size = bitstring_size(binary);

    if (TAIL_BITS(size) == 0 && size <= ERTS_SINT_MAX) {
        return (Sint)size;
    }

    return -1;
}

static Sint latin1_binary_need(Eterm binary)
{
    const byte *temp_alloc = NULL, *bytes;
    Uint size;
    Sint need;
    Sint i;

    bytes = erts_get_aligned_binary_bytes(binary, &size, &temp_alloc);
    if (bytes == NULL) {
        return -1;
    }

    for(i = 0, need = 0; i < size; ++i) {
        if (bytes[i] & ((byte) 0x80)) {
            need += 2;
        } else {
            need += 1;
        }
    }

    erts_free_aligned_binary_bytes(temp_alloc);
    return need;
}

Let's break this down step by step. The unicode_bin_is_7bit_1 function first checks if the input is a bitstring. If not, it immediately returns false. Fair enough. Then, it calls latin1_binary_need to calculate the number of bytes needed to represent the binary as Latin-1. This is where things get interesting.

The latin1_binary_need function iterates through each byte in the binary. If a byte has its most significant bit set (i.e., the value is greater than 127), it adds 2 to the need counter; otherwise, it adds 1. This is because Latin-1 characters above 127 require two bytes in UTF-8 encoding. So, this function is essentially calculating the UTF-8 encoded size.

Now, look closely at the conditional statement in unicode_bin_is_7bit_1:

if(need >= 0 && aligned_binary_size(BIF_ARG_1) == need) {
    BIF_RET(am_true);
}

Here's the crux of the bug. The aligned_binary_size function returns the size of the binary in bits, while need represents the number of bytes required to represent the binary in Latin-1. These two values are of different magnitudes, meaning the condition aligned_binary_size(BIF_ARG_1) == need will almost never be true. This is why the function almost always returns false.

For example, if you have a 3-byte binary, aligned_binary_size will return 24 (3 bytes * 8 bits/byte), while latin1_binary_need will return 3 if all characters are within the 7-bit ASCII range. Clearly, 24 is not equal to 3, so the condition fails.

This deep dive into the code reveals a fundamental flaw in the comparison logic. The function is comparing apples and oranges, leading to the incorrect behavior we've observed.

Proposed Solutions

Okay, we've identified the bug! Now, let's brainstorm some solutions to fix it. The goal is to make the unicode_bin_is_7bit/1 function accurately determine whether a binary string consists of 7-bit ASCII characters. We need to modify the code so that it correctly compares the size of the binary with the number of bytes required for 7-bit ASCII representation.

Here are a couple of approaches we can take:

Solution 1: Correct the Comparison

The most straightforward solution is to fix the comparison in the unicode_bin_is_7bit_1 function. Instead of comparing the bit size with the byte count from latin1_binary_need, we should compare the byte size of the binary with the result of latin1_binary_need. If latin1_binary_need returns a value equal to the byte size, it means all characters are within the 7-bit ASCII range.

To implement this, we need to get the size of the binary in bytes. We can use the bitstring_size function and divide the result by 8 (since there are 8 bits in a byte). Here’s how the modified code would look:

BIF_RETTYPE unicode_bin_is_7bit_1(BIF_ALIST_1)
{
    Sint need;
    if(!is_bitstring(BIF_ARG_1)) {
        BIF_RET(am_false);
    }
    need = latin1_binary_need(BIF_ARG_1);
    // Compare byte size with the result from latin1_binary_need
    if(need >= 0 && bitstring_size(BIF_ARG_1) / 8 == need) {
        BIF_RET(am_true);
    }
    BIF_RET(am_false);
}

This change ensures that we're comparing byte sizes, which is the correct approach for determining if a binary is 7-bit ASCII.

Solution 2: Simplify the Logic

Another approach is to simplify the logic and directly check each byte in the binary to see if it's within the 7-bit ASCII range (0-127). This avoids the need for the latin1_binary_need function and the potentially confusing size comparison. We can iterate through the bytes and return false immediately if we encounter a byte outside the range. If we make it through the entire binary without finding any non-ASCII characters, we return true.

Here’s how this solution would look in code:

BIF_RETTYPE unicode_bin_is_7bit_1(BIF_ALIST_1)
{
    if(!is_bitstring(BIF_ARG_1)) {
        BIF_RET(am_false);
    }

    const byte *temp_alloc = NULL, *bytes;
    Uint size;
    bytes = erts_get_aligned_binary_bytes(BIF_ARG_1, &size, &temp_alloc);
    if (bytes == NULL) {
        BIF_RET(am_false);
    }

    for (Uint i = 0; i < size; ++i) {
        if (bytes[i] > 127) {
            erts_free_aligned_binary_bytes(temp_alloc);
            BIF_RET(am_false);
        }
    }

    erts_free_aligned_binary_bytes(temp_alloc);
    BIF_RET(am_true);
}

This solution is more direct and easier to understand. It iterates through the binary, checks each byte, and returns false if a non-ASCII character is found. If the loop completes without finding any non-ASCII characters, it returns true.

Both of these solutions should correctly address the bug in unicode_bin_is_7bit/1. The choice between them depends on factors like performance considerations and code readability. But, at their core, both solutions aim to accurately determine if a binary string is 7-bit ASCII.

Affected Versions

It's important to know which versions of Erlang are affected by this bug. As reported, the issue is present in OTP 27 and OTP 28. This means that if you're using either of these versions, you'll encounter the incorrect behavior of :unicode.bin_is_7bit/1.

This information is crucial for developers and system administrators who rely on this function for character encoding validation. If you're working with OTP 27 or 28, you'll need to be aware of this bug and consider implementing a workaround or applying a patch once a fix is available. This could involve using an alternative method for checking 7-bit ASCII or incorporating one of the solutions we discussed earlier into your code.

Keeping track of affected versions helps prevent unexpected issues in production environments. If you're planning to upgrade your Erlang installation, it's always a good idea to check for known bugs and their fixes in the release notes. This proactive approach can save you from potential headaches down the road.

Conclusion

So, there you have it, guys! We've taken a deep dive into the :unicode.bin_is_7bit/1 bug, reproduced it, dissected the code, and proposed a couple of solutions. The key takeaway here is that a seemingly small bug can have a significant impact on your code if you're not aware of it.

The root cause of this bug lies in the incorrect comparison between bit size and byte count within the function. By understanding this flaw, we can implement effective solutions to ensure accurate 7-bit ASCII detection.

Whether you choose to correct the comparison or simplify the logic, the goal remains the same: to provide a reliable way to check if a binary string contains only 7-bit ASCII characters. And remember, staying informed about affected versions is crucial for maintaining the stability and correctness of your Erlang applications.

Hopefully, this exploration has been helpful and insightful. Keep those coding skills sharp, and stay tuned for more deep dives into the fascinating world of software bugs and solutions! Peace out!