none
Concerns about endianness RRS feed

  • Question

  • I have an application which needs to convert between types of PCM audio. I've written code for my application which will read a file into a buffer, and extract samples. Where the samples are 24 bit integers, I have a problem, because I have not been able to find a data type to hold them.

    The below code works on my development machine. I got it mainly by trial and error. I am concerned will go badly wrong on other machines. Am I right to be concerned? How might I make it safer?

    struct Int24
    {
        unsigned char one;
        unsigned char two;
        unsigned char three;
    };

    // If samples need to be scaled down if (bytesPerSample == 3) { ASSERT(sizeof(Int24) == 3); ASSERT(sizeof(int) == 4); unsigned char buff[4] = "\0\0\0"; int *value = reinterpret_cast<int*>(&buff[0]); Int24 *threeBytes = reinterpret_cast<Int24*>(&buff[0]); for (unsigned int count = 0; count < samples; count++) { *threeBytes = *reinterpret_cast<Int24*>(&buffer[count * bytesPerSample]); // Sign extend if (buff[2] >= 128) buff[3] = 255; else buff[3] = 0; // Scale to 16 bit representation for (int power = numSignificantBits - 16; power > 0; power--) { *value /= 2; } newSamples[count] = *value; } }

    Before someone tells me to read a book, I don't have time. This is supposed to be being used by real people within days.

    Tuesday, June 5, 2012 6:32 AM

Answers

  • "Where the samples are 24 bit integers, I have a problem, because I have not been able to find a data type to hold them."

    Use an int or whatever other integral type that you prefer and has more than 24 bits.

    "The below code works on my development machine"

    Well, this code is endian sensitive indeed because you are using reinterpret_cast<int *> to extract an integer from a byte buffer. AFAIR PCM files are always little endian and if you run this code on a big endian machine you won't get the correct results. It's best to build the integer by using shift operations, here's an example:

    #include <cstdlib>
    #include <cstdint>
    int main(int argc, char* argv[])
    {
        uint32_t bytesPerSample = 3;
        uint8_t buffer[] = { 1, 2, 3, 4, 5, 0xff };
        for (uint32_t count = 0; count < _countof(buffer) / bytesPerSample; count++) {
            int32_t value = 
                // first byte contains bits 0-7 of the integer
                  buffer[count * bytesPerSample + 0]
                // second byte contains bits 8-15
                | buffer[count * bytesPerSample + 1] << 8
                // sign extended the last byte to int16 to fill bits 16-31
                | static_cast<int16_t>(static_cast<int8_t>(buffer[count * bytesPerSample + 2])) << 16;
            printf("%08x\n", value);
        }
    	return 0;
    }
    Note that I find preferable to use stdint types in such cases, you don't have to worry about the sizeof(int) if you do that.
    • Marked as answer by JosephFox Tuesday, June 5, 2012 1:40 PM
    Tuesday, June 5, 2012 7:31 AM
    Moderator
  • MFC doesn't have an endian-ness check because it only runs on little-endian platforms.

    There aren't a whole lot of big-endian platforms left in common circulation.

    At compile time the "system" should know; the compiler indeed MUST know.  But there's no standard for communicating that to the preprocessor.

    Performance is going to be limited by memory bandwidth on just about any practical platform, so explicitly shifting bytes will probably be as fast as the non-portable solution in practice.  You are only wasting cycles if there is something else you could be doing with them, and you are going to be stalling for cache fills anyway.

    If you wanted to write a compile-time check for endian-ness it MIGHT work to test

    A runtime check for endian-ness would be

    long x;

    if ( &(short)x == &x ) // then little-endian.

    Developing a compile-time test is harder because most addresses are not compile-time constants.  Also, I don't think casts are valid in preprocessor conditional expressions.

    It appears there is no portable compiler-time test: http://stackoverflow.com/questions/1001307/detecting-endianness-programmatically-in-a-c-program


    Stephen W. Nuchia StatSoft, Inc. Tulsa, Oklahoma USA

    • Marked as answer by JosephFox Tuesday, June 5, 2012 3:12 PM
    Tuesday, June 5, 2012 2:35 PM
  • JosephFox wrote:
    >
    >Brilliant. Makes sense. I was wondering why the MFC framework doesn't
    >support an easy way to check endianess; I was hoping there would be
    >some low level operations ...
     
    There is no need.  Every version of Windows ever made runs exclusively on
    little-endian machines.  That's even true for Windows CE.
    --
    Tim Roberts, timr@probo.com
    Providenza & Boekelheide, Inc.
     

    Tim Roberts, VC++ MVP Providenza & Boekelheide, Inc.
    • Marked as answer by JosephFox Thursday, June 7, 2012 8:29 AM
    Thursday, June 7, 2012 4:37 AM
  • There is no guarantee that buff will be properly aligned for an int.  Your attempt to cast &buf[0] to int* will cause undefined behavior if it is not.  One way to insure that it is properly aligned is to include it in a union with an int.
    • Marked as answer by JosephFox Monday, June 11, 2012 8:41 AM
    Monday, June 11, 2012 5:54 AM

All replies

  • "Where the samples are 24 bit integers, I have a problem, because I have not been able to find a data type to hold them."

    Use an int or whatever other integral type that you prefer and has more than 24 bits.

    "The below code works on my development machine"

    Well, this code is endian sensitive indeed because you are using reinterpret_cast<int *> to extract an integer from a byte buffer. AFAIR PCM files are always little endian and if you run this code on a big endian machine you won't get the correct results. It's best to build the integer by using shift operations, here's an example:

    #include <cstdlib>
    #include <cstdint>
    int main(int argc, char* argv[])
    {
        uint32_t bytesPerSample = 3;
        uint8_t buffer[] = { 1, 2, 3, 4, 5, 0xff };
        for (uint32_t count = 0; count < _countof(buffer) / bytesPerSample; count++) {
            int32_t value = 
                // first byte contains bits 0-7 of the integer
                  buffer[count * bytesPerSample + 0]
                // second byte contains bits 8-15
                | buffer[count * bytesPerSample + 1] << 8
                // sign extended the last byte to int16 to fill bits 16-31
                | static_cast<int16_t>(static_cast<int8_t>(buffer[count * bytesPerSample + 2])) << 16;
            printf("%08x\n", value);
        }
    	return 0;
    }
    Note that I find preferable to use stdint types in such cases, you don't have to worry about the sizeof(int) if you do that.
    • Marked as answer by JosephFox Tuesday, June 5, 2012 1:40 PM
    Tuesday, June 5, 2012 7:31 AM
    Moderator
  • Brilliant. Makes sense. I was wondering why the MFC framework doesn't support an easy way to check endianess; I was hoping there would be some low level operations (bit masks or shift operations, which I don't have much practice in) about which someone could enlighten me. Well exemplified.
    Tuesday, June 5, 2012 1:40 PM
  • MFC doesn't have an endian-ness check because it only runs on little-endian platforms.

    There aren't a whole lot of big-endian platforms left in common circulation.

    At compile time the "system" should know; the compiler indeed MUST know.  But there's no standard for communicating that to the preprocessor.

    Performance is going to be limited by memory bandwidth on just about any practical platform, so explicitly shifting bytes will probably be as fast as the non-portable solution in practice.  You are only wasting cycles if there is something else you could be doing with them, and you are going to be stalling for cache fills anyway.

    If you wanted to write a compile-time check for endian-ness it MIGHT work to test

    A runtime check for endian-ness would be

    long x;

    if ( &(short)x == &x ) // then little-endian.

    Developing a compile-time test is harder because most addresses are not compile-time constants.  Also, I don't think casts are valid in preprocessor conditional expressions.

    It appears there is no portable compiler-time test: http://stackoverflow.com/questions/1001307/detecting-endianness-programmatically-in-a-c-program


    Stephen W. Nuchia StatSoft, Inc. Tulsa, Oklahoma USA

    • Marked as answer by JosephFox Tuesday, June 5, 2012 3:12 PM
    Tuesday, June 5, 2012 2:35 PM
  • I consider myself 'schooled' re. MFC's innate endianness. Very neat run time test.

    Tuesday, June 5, 2012 3:19 PM
  • JosephFox wrote:
    >
    >Brilliant. Makes sense. I was wondering why the MFC framework doesn't
    >support an easy way to check endianess; I was hoping there would be
    >some low level operations ...
     
    There is no need.  Every version of Windows ever made runs exclusively on
    little-endian machines.  That's even true for Windows CE.
    --
    Tim Roberts, timr@probo.com
    Providenza & Boekelheide, Inc.
     

    Tim Roberts, VC++ MVP Providenza & Boekelheide, Inc.
    • Marked as answer by JosephFox Thursday, June 7, 2012 8:29 AM
    Thursday, June 7, 2012 4:37 AM
  • There is no guarantee that buff will be properly aligned for an int.  Your attempt to cast &buf[0] to int* will cause undefined behavior if it is not.  One way to insure that it is properly aligned is to include it in a union with an int.
    • Marked as answer by JosephFox Monday, June 11, 2012 8:41 AM
    Monday, June 11, 2012 5:54 AM