none
Can use 32 bit size_t in x64 VC2008 STL?

    Question

  • When compiling an x64 application that also runs in x86, is there a way to make the x64 version of STL use 32 bit sizes for container lengths (e.g., #define/compiler switch/etc)?  I don't need STL containers to hold more than 2^32 amount of data so it would make some code changes much simpler if I could use something like a size_32_t (32 bit) type in x64 rather than size_t (64 bit) type.

    If this is not possible, can I change the allocators to do this and if so will there be any significant problems?

    Thanks

    Monday, January 25, 2010 11:45 PM

Answers

  • Header-hacking isn't supported. Macro-stomping is forbidden by the Standard and isn't supported. (We document a few macros, such as _SECURE_SCL, as being controllable by users. <xmemory>'s _SIZT isn't one of them. Please don't stomp on the code that Dinkumware and I maintain; we don't stomp on your code.)

    Custom STL allocators are supported, so you could use one that typedefs size_type to unsigned long and difference_type to signed long.  See http://blogs.msdn.com/vcblog/archive/2008/08/28/the-mallocator.aspx for an example, but note that it will require some reworking because I baked the assumption that size_type == size_t into it.

    vector and the other containers are pretty careful to work in terms of their allocator's size_type instead of size_t.  However, because this assumption is easy to make and we don't have test coverage for size_type != size_t, there may be bugs in this area.  They would probably take the form of spurious warnings, but other problems are possible.

    Thursday, January 28, 2010 9:49 PM

All replies

  • You can't change the definition of size_t without risking breaking library code, if that's what you mean.

    Exactly what are you trying to achieve? Why do you care whether your STL container holds more than 2^32 or not? Are you worried that your application is going to waste some space in some respect?

    You may be needlessly worrying about a problem that isn't a problem.
    Tuesday, January 26, 2010 12:15 AM
  • Iirc, containers don't allocate the full amount right from the start, that is an awful waste of space.
    I know one strategy that is used is to allocate a small default amount (maybe two elements) and then when it you try to put something in after it becomes full then it allocates a new block of memory which is larger, then copy all of the old elements over and finally add the new one to the end.

    The increase in size is usually large enough but not too large.

    On a side note, after looking at the shared source for the .net framework libraries the way it works is similar.
    For them the starting size is 2 elements, when it needs to reallocate it doubles it to 4, if it needs to reallocate then it doubles it to 8 and so on.
    Visit my (not very good) blog at http://c2kblog.blogspot.com/
    Tuesday, January 26, 2010 1:44 AM
  • I have an x86 program that contains 2X10^6 lines of code that we're trying to get to run natively in x64.  In that many lines of code, you have many, many locations where you have issues with int and other type comparisons and assignments with size_t that issue compiler errors or warnings.  If I could center the STL containers around a 32 bit size rather than a 64 bit size (since in most cases I don't need containers that large), I could eliminate many of these compiler issues.
    Tuesday, January 26, 2010 2:18 PM
  • I don't think that is the best way to go. The problem is not the doubling of the data path, its code that assumes it knows better than the compiler what the size of an integer is.

    I have done this sort of modification before, and...in that case, it was a lot more than a simple matter of controlling the library code. (In fact, that has already been done...its the code you are working on that is the "issue".) Lying to the runtime won't do anything for code that assumes the size of an integer without asking. In code of that size, I imagine there are some lines that make those sort of assumptions.

    I still don't understand why the warnings you mentioned are acceptable...I attempt to eliminate all of them in a way that won't come back to bite me later. The example you cite is an excellent reason. I would fix all of those in the 32-bit side before even attempting to port the code. Yes, its a lot of code, but I'd do it anyway. I never accept a warning wrt sizeof; I fix the error that causes it.

    It is not efficient to port code that is throwing compiler warnings, and it will likely create a lot of bugs in the process. I am thinking now about programmers that I've worked with that thought that was me being too picky. (The compiler complainis for a reason; I listen to those complaints as much as possible.)

    It would not have been simple without this concern, however. It is trying to be clever that causes this type of nightmare in the first place...its not a good answer to double down on that at this point, imo.

    Tuesday, January 26, 2010 4:15 PM
  • The warnings (and obviously the errors) are not acceptable.  They do not occur in x86 but only in x64 builds that I am now trying to make.  For example, the following code works perfectly in 32 bit code but will issue a warning in 64 bit:

    std::vector<int> a;

    DWORD s = a.size();

    Many of these exist because the Windows API requires a DWORD for certain calls.  However, if I can change STL to use 32 bit size types rather than 64 bit size types, the code will work in both x86 and x64.  If I'm going to pass the size to a Win API that uses a DWORD argument, the buffer obviously cannot be greater than DWORD size thus validating my statement indicating only 2^32 size buffers are needed.

    Obviously, I can fix the code using casts.  I'm just looking for minimal effort due to the large size of the code base.  The modifications will require a significant number of casts to prevent the errors and warnings.
    Tuesday, January 26, 2010 8:28 PM
  • It's an interesting problem.

    Personally, I think you should be using explicit casting. I'm wondering if you can somehow partially automate this, since there will be patterns in the code which could be leveraged using a regular expression engine.

    The example you gave is a good one. You can use Perl to change all instances of

    DWORD = ____.size();

    to

    DWORD = reinterpret_cast<DWORD> (_____.size());

    Tuesday, January 26, 2010 9:21 PM
  • Hello Simdoc,

    first of all I admire your will to remove not only errors, but also warnings. Every Developer should strive for a clean build. However the suggestion of changing standard headers to redefine size_t fills me with unease. This could not only cause errors then you are linking vs C++ runtime enviroments, but linking in general versus code using STL could be a dangerous thing to do.  I can imagine this going well, but I can also see it produce some really hard to find bugs. I wouldn't recommend messing with this forces. As far as quitening the warnings using a static_cast goes, this seems to me like a very bad idea too. Warnings are not evil in themselfs, but the problems they point at are. Sure you won't get a warning because you did just tell the compiler to shut up, but your code hasn't gotten more robust or safe. On the contrary, if you should get a bug involving an overflow error you just quitend all the warnings and lost the information there the dangerous places in code are.

    The underlying problem is, that this code needs an additional runtime check then converting size_t down to DWORD and throw an exception if it fails. Unless you're willing ( or rather paid ) to put the effort into it, I would leave the warnings there they are, to remind future generations that there is work for them.

    I'm pretty sure you already did know everything that I've written you. Your suggestions seems to me born out of despair, but don't go down that road. There is no easy way out. Either renew the code to be truly 64Bit or just ignore the warnings and fix the errors for the time being.

    best regards,
    Markus
    Tuesday, January 26, 2010 9:47 PM
  • It's a template library.  The code is generated and instantiated at compile time.  Nothing is linked.  It should be somewhat immune to underlying size changes (as it obviously is if you compile x86 rather than x64).  I was hoping it would be as simple as throwing a switch or using a #define for size selection.

    Unfortunately, bulk code changes in perl or other utilities are not very useful as I have many more examples of similar issues.  Remember, this is a 2 million line program.  In programs that size, you encounter almost every language use.
    Wednesday, January 27, 2010 2:01 AM
  • Hello,

    size_t is a 64-bit value on 64-bit Windows operating systems.
    http://msdn.microsoft.com/en-us/library/3b2e7499.aspx
    http://www.viva64.com/content/articles/64-bit-development/?f=size_t_and_ptrdiff_t.html&lang=en&content=64-bit-development

    Not sure where the code generate from(IDL?), is it possible for you to change the original code that generate size_t?

    Thanks,
    Rong-Chun Zhang
    MSDN Subscriber Support in Forum
    If you have any feedback on our support, please contact msdnmg@microsoft.com
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Wednesday, January 27, 2010 9:31 AM
  • Hello simdoc,

    you are right, of course you don"t link vs the STL. The problems I thought of can occure if you link vs code which happens to use the STL itself. Imagine a library written in C++ using STL. If this library is compiled for a 64 Bit platform the binary code of the library contains concrete template instanciations in their binary form using size_t with 64 Bit. Then you go ahead now and compile your code using a 32Bit size_t, the standard containers from the code you link against have a different size_t from what your compiler would expect. Of course the game changes a bit if you have the option of recompiling every library with the new size_t.

    best regards,
    Markus Klein
    Wednesday, January 27, 2010 1:15 PM
  • Markus:

    I do have the option of recompiling everything.

    Wednesday, January 27, 2010 2:23 PM
  • Well, if you do have the option of recompiling everything then open up your Visual Studio install directory, navigate to VC\crt\src and start editing. You need to change the basic definition of size_t under x64, then from there go through the entire CRT modifying all of it to work with the 32 bit size_t since x64 code assumes that size_t is 64 bit. Once that is done, build the CRT to a custom name and there you go, then you have to test it to make sure nothing weird happens.
    As you an see, you will have to do work either way, so it would probably be easier to just fix your own code to be properly x64 compatible.
    Visit my (not very good) blog at http://c2kblog.blogspot.com/
    Wednesday, January 27, 2010 3:20 PM
  • That's not a very good alternative as every release (and possibly patches) of VC++ I would have to rework STL.  I'd rather modify my code.  Further, it's not the CRT I'm interested in.  It is STL.

    I've looked through the code for this type of switch and have not found it.  It probably does not exist even though I think it would have been relatively easy to support.
    Wednesday, January 27, 2010 5:23 PM
  • You would have to fix the entire CRT though since the STL depends on the CRT. For example, if you look at xmemory it depends on cstdlib (which as we should know is a wrapper around stdlib.h). So you would need to change the size_t from the CRT otherwise there will be problems with communication between STL and CRT too.
    Visit my (not very good) blog at http://c2kblog.blogspot.com/
    Wednesday, January 27, 2010 5:54 PM
  • I guess more specifically, does anyone have experience with the following statements added before including any STL header files:

    #define _FARQ
    #define _PDFT
    int
    #define
    _SIZT
    unsigned



    This will allow the definitions to be override in the include file xmemory.  However, I don't know if there are any side effects.

    Thursday, January 28, 2010 12:29 AM
  • simdoc, try posting in the microsoft.public.vc.stl newgroup. P.J. Plauger, author of the Dinkumware STL (which was licensed to Microsoft) might respond directly.
    Thursday, January 28, 2010 12:32 AM
  • simdoc -

    Send a short email to

    Stephan T. Lavavej
    Visual C++ Libraries Developer
    stl@microsoft.com

    and ask him if he will look at this thread and comment.
    Include a link to this thread so he can come here directly.

    - Wayne
    Thursday, January 28, 2010 2:00 AM
  • Use static_cast not reinterpret_cast however be aware of possible overflow problems.  Only use DWORD in the parts of your code that actually involve a Windows API call otherwise use container::size_type as your variable type.  Add an assert before any assigns to a DWORD variable to ensure overflows are highlighted. If you are not confident that such a problem will be highlighted during development then write a helper template function called "no_overflow_cast" or something which will throw an exception on overflow terminating your program.

    struct overflow {};

    template <typename To, typename From>
    inline To no_overflow_cast(From from)
    {
      if (from > std::numeric_limits<To>::max())
        throw overflow();
      return static_cast<To>(from);
    }

    int main()
    {
      unsigned char ch = 255;
      char bad = no_overflow_cast<char>(ch);
    }
    Thursday, January 28, 2010 3:01 PM
  • Header-hacking isn't supported. Macro-stomping is forbidden by the Standard and isn't supported. (We document a few macros, such as _SECURE_SCL, as being controllable by users. <xmemory>'s _SIZT isn't one of them. Please don't stomp on the code that Dinkumware and I maintain; we don't stomp on your code.)

    Custom STL allocators are supported, so you could use one that typedefs size_type to unsigned long and difference_type to signed long.  See http://blogs.msdn.com/vcblog/archive/2008/08/28/the-mallocator.aspx for an example, but note that it will require some reworking because I baked the assumption that size_type == size_t into it.

    vector and the other containers are pretty careful to work in terms of their allocator's size_type instead of size_t.  However, because this assumption is easy to make and we don't have test coverage for size_type != size_t, there may be bugs in this area.  They would probably take the form of spurious warnings, but other problems are possible.

    Thursday, January 28, 2010 9:49 PM
  • Use 32 bit size_t in x64 - it's the wrong way. It is increase number of errors in code and hide some other 64-bit errors.

    Clean compiler errors and warnings. Use Viva64 for complete check code.

    See also articles:

    1. Seven Steps of Migrating a Program to a 64-bit System
    2. A 64-bit horse that can count
    3. Problems of testing 64-bit applications
    4. About size_t and ptrdiff_t
    5. Optimization of 64-bit programs
    6. 20 issues of porting C++ code on the 64-bit platform
    Saturday, January 30, 2010 7:46 AM