none
LINK.EXE BUG: /OPT:NOREF option doesn't work!

    Question

  • My program contains static objects whose constructors are supposed to run at the beginning of the program, but otherwise those variables are unreferenced. The problem is, when I put these variables into a .lib file, they get completely optimized away and never appear in the resulting .exe!

    My understanding is that /opt:noref is supposed to cure this, however, it has absolutely no effect. I also tried __declspec(selectany) on the vars, also no effect. The same behaviour occurs with VC6, 7 (2003) and now also 8 (2005 Express).

    This looks very much like a linker bug. Is there any workaround to this problem? (Such as additional linker flag, __declspec, pragma or anything? Referencing the vars from other files or using /INCLUDE:symbol for each var is not doable, because those vars are created by macros and supposed to be invisible to the user.)

    I'm attaching a little program to demonstrate the problem. When built using build.bat (below), it should print "Noisy created" -- but nothing gets printed. And if you search for the text "Noisy created" in main.exe, it's not contained in there.

    noisy.cpp:

    #include <stdio.h> 
    class Noisy {
    public:
    Noisy() {printf("Noisy created!\n");}
    };
    Noisy noisy;

    main.cpp:

    int main(int argc, char **argv) { return 0; } 

    build.bat:

    cl /c noisy.cpp 
    lib /out:noisy.lib noisy.obj
    cl /c main.cpp
    link /out:main.exe main.obj /opt:noref noisy.lib

     

    Thursday, November 17, 2005 6:04 PM

Answers

  • This bug drove me nuts a while back. As far as I know, there's no way to correct this generally, the linker just doesn't seem to understand that it can't discard static constructors.

    For a while I would make an explicit reference to a dummy function which referenced the variables from my main project, but that was obviously not an ideal solution.

    My current solution (hack?) is to use a Ruby script as a post build step on my libraries. This script uses the dumpbin.exe tool to find all symbols containing the word 'forceLink', and it outputs a header file containing
    #pragma comment(linker, "/include:symbol")
    for each of the symbols. I then include that header file from my main project, and then everything just works.

    Niall.
    Thursday, November 17, 2005 7:19 PM
  • Discarding unreferenced symbols is what linkers do and it's "a good thing."  Your technique of using #pragma comment(linker,...) is the correct approach.  MFC does this also (see the header afx.h).

    Brian
    Thursday, November 17, 2005 9:35 PM
    Moderator
  • This is not a linker bug and the linker isn't discarding anything.  This is just the way .libs work. 

    An object in a library isn't pulled into the final image unless a symbol definition in the object is needed to satisfy an unresolved referenced in the image.  Otherwise, when linking in the CRT or any other lib, you'd get the whole lib pulled in instead of just the needed pieces.

    So you do need to had something to force the linker to pull in that object.  You could add dummy references, or use /include.

    Hope this helps,

    Louis Lafreniere
    VC++ Development
    Thursday, November 17, 2005 9:48 PM
  • The linker only pulls in objects which have referenced symbols.

    Would it be possible to add a function to these types which does nothing, and reference that via a #pragma comment() within a header file that your user's include?

    e.g.
    noisy.cpp

    #include <stdio.h>

    class Noisy
    {
    public:

    Noisy()
    {
       printf(
    "Noisy created!\n");
    }
    void emptyreference();

    };

    void Noisy::emptyreference(){}

    Noisy noisy;

    mylib.h

       #pragma comment(linker, "/include:?emptyreference@Noisy@@QAEXXZ")

    main.cpp

    #include "mylib.h"

    int main(int argc, char **argv) { return 0; }

    Thursday, November 17, 2005 10:00 PM
  • Louis: I don't mind that the linker does optimizations, that's part of its job, I just mean that /OPT:NOREF doesn't do what it is supposed to do. Please have a look at the following MSDN page which documents the linker /OPT options:

    http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore/html/_core_.2f.opt.asp

    A quote from this page:

    "REF | NOREF
    /OPT:REF eliminates functions and/or data that are never referenced while /OPT:NOREF keeps functions and/or data that are never referenced."
    This is quite clear. Still, in the above example, the linker discards the global "Noisy" object (its constructor doesn't get called at startup, and the code even doesn't appear in the executable -- the exe file doesn't contain the string "Noisy created".)

    How can this NOT be a bug, when it is clearly the contrary of what the documentation says?

    BTW, the equivalent option in the Linux world is "-z allextract" (Sun) or "--whole-archive" (GNU), they work as expected.

    Niall: thanks for the suggestion to collect the list of variables from dumpbin output. I think I'll to do that. Not very elegant, but at least works :)

    Friday, November 18, 2005 10:07 AM
  • Andras,

    You are mixing 2 concepts:
    * When using /OPT:REF linker removes unreferenced code and data from object files it decides to add to the executable. That is the optimization, and you can turn it off by using /OPT:NOREF
    * When searching libraries, linker picks up only object files that contain explicitly referenced code or data -- so it will not pick up some variable with global constructor if there is no reference from your code (or from other object file, or from object file included in some library if linker decides to include that object file). That is NOT the optimization, linker always works this way, and that behavior is NOT controlled by /OPT:NOREF.

    You can do what you want by including explicit reference to the global variable from your code, or by explicitly putting object file  containing the variable into linker command line.

    You are right that MSDN documentation may be clearer on that issue. I'll let our documentation people know, so hopefully they will improve that topic for the next VS release.

    Thanks,
    Eugene
    Friday, November 18, 2005 7:38 PM

All replies

  • This bug drove me nuts a while back. As far as I know, there's no way to correct this generally, the linker just doesn't seem to understand that it can't discard static constructors.

    For a while I would make an explicit reference to a dummy function which referenced the variables from my main project, but that was obviously not an ideal solution.

    My current solution (hack?) is to use a Ruby script as a post build step on my libraries. This script uses the dumpbin.exe tool to find all symbols containing the word 'forceLink', and it outputs a header file containing
    #pragma comment(linker, "/include:symbol")
    for each of the symbols. I then include that header file from my main project, and then everything just works.

    Niall.
    Thursday, November 17, 2005 7:19 PM
  • Discarding unreferenced symbols is what linkers do and it's "a good thing."  Your technique of using #pragma comment(linker,...) is the correct approach.  MFC does this also (see the header afx.h).

    Brian
    Thursday, November 17, 2005 9:35 PM
    Moderator
  • This is not a linker bug and the linker isn't discarding anything.  This is just the way .libs work. 

    An object in a library isn't pulled into the final image unless a symbol definition in the object is needed to satisfy an unresolved referenced in the image.  Otherwise, when linking in the CRT or any other lib, you'd get the whole lib pulled in instead of just the needed pieces.

    So you do need to had something to force the linker to pull in that object.  You could add dummy references, or use /include.

    Hope this helps,

    Louis Lafreniere
    VC++ Development
    Thursday, November 17, 2005 9:48 PM
  • The linker only pulls in objects which have referenced symbols.

    Would it be possible to add a function to these types which does nothing, and reference that via a #pragma comment() within a header file that your user's include?

    e.g.
    noisy.cpp

    #include <stdio.h>

    class Noisy
    {
    public:

    Noisy()
    {
       printf(
    "Noisy created!\n");
    }
    void emptyreference();

    };

    void Noisy::emptyreference(){}

    Noisy noisy;

    mylib.h

       #pragma comment(linker, "/include:?emptyreference@Noisy@@QAEXXZ")

    main.cpp

    #include "mylib.h"

    int main(int argc, char **argv) { return 0; }

    Thursday, November 17, 2005 10:00 PM
  • Louis: I don't mind that the linker does optimizations, that's part of its job, I just mean that /OPT:NOREF doesn't do what it is supposed to do. Please have a look at the following MSDN page which documents the linker /OPT options:

    http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore/html/_core_.2f.opt.asp

    A quote from this page:

    "REF | NOREF
    /OPT:REF eliminates functions and/or data that are never referenced while /OPT:NOREF keeps functions and/or data that are never referenced."
    This is quite clear. Still, in the above example, the linker discards the global "Noisy" object (its constructor doesn't get called at startup, and the code even doesn't appear in the executable -- the exe file doesn't contain the string "Noisy created".)

    How can this NOT be a bug, when it is clearly the contrary of what the documentation says?

    BTW, the equivalent option in the Linux world is "-z allextract" (Sun) or "--whole-archive" (GNU), they work as expected.

    Niall: thanks for the suggestion to collect the list of variables from dumpbin output. I think I'll to do that. Not very elegant, but at least works :)

    Friday, November 18, 2005 10:07 AM
  • Andras,

    You are mixing 2 concepts:
    * When using /OPT:REF linker removes unreferenced code and data from object files it decides to add to the executable. That is the optimization, and you can turn it off by using /OPT:NOREF
    * When searching libraries, linker picks up only object files that contain explicitly referenced code or data -- so it will not pick up some variable with global constructor if there is no reference from your code (or from other object file, or from object file included in some library if linker decides to include that object file). That is NOT the optimization, linker always works this way, and that behavior is NOT controlled by /OPT:NOREF.

    You can do what you want by including explicit reference to the global variable from your code, or by explicitly putting object file  containing the variable into linker command line.

    You are right that MSDN documentation may be clearer on that issue. I'll let our documentation people know, so hopefully they will improve that topic for the next VS release.

    Thanks,
    Eugene
    Friday, November 18, 2005 7:38 PM
  • Regardless whether or not there is an issue with /OPT, this version of main.cpp should pull in the global Noisy instance:


    #pragma comment(linker, "/include:??0Noisy@@QAE@XZ")

    int main(int argc, char **argv) { return 0; }

     

    Saturday, November 19, 2005 1:22 AM
  • OK, I realize there isn't much willingness to fix it properly in the linker, so I'm not pushing it any further.

    Thanks to everybody suggesting the dumpbin+grep+pragma comment(linker,...) workaround.
    Tuesday, November 22, 2005 11:31 AM
  • Mr. Lafreniere,

    This is actually a real problem.

    ANY OBJECT IN THE GLOBAL SCOPE OF A .LIB IS IN THE GLOBAL SCOPE OF THE TARGET .EXE. It is incorrect to treat objects in the global scope of a .lib any differently than you treat objects in the global scope of the .exe.

    The /OPT:NOREF linker option should have the same effect on objects in the global scope of .libs as it has on objects in the global scope of the .exe. IT IS THE SAME SCOPE. To sound like a broken record, OBJECTS IN THE GLOBAL SCOPE OF A LIB ARE IN THE SAME "GLOBAL" SCOPE OF THE .EXE. THERE'S A REASON ITS CALLED "GLOBAL".

    lemme know when its fixed =)

    Trey
    Wednesday, November 23, 2005 2:34 AM
  • I'm sorry but this is just not the way libraries work.  A lib contains a collection of features supported by a set of .obj, which themselves contain a set of functions and objects.

    When the linker has a reference it can't resolves, it looks in the libs available.  If it finds a definition for that symbol, it pulls the .obj defining that reference out of the .lib and links it in.

    Only the .objs containing the definitions necessary to resolve all references are pulled in.  This is nothing new and has been this way since we added support libraries a long long time ago.

    This is by design and for very good reasons:  you don't want to blow up the image size with a lot of unnecessary objects (code and data) which are not used.  For example, you don't want the whole CRT and the whole MFC in your application when creating a simple MFC app.  That's exactly why libraries were added in the first place!  Yes, all these functions and objects are at global scope, but the whole concept of libraries was added to avoid including the unnecessary ones, and only include code which supports the features used by your app.

    But even if I agreed with you, we couldn't change the behavior of libs since most library providers (including Microsoft) rely heavily on the current behavior.  For example, a library can contain multiple versions of a function or object (single vs multi-threaded, ansi vs unicode, etc).  Depending on your app configuration, only one of the .objs will be pulled in by a reference and you'll get that particular version of the function.

    In C/C++ 7.0, long after library support was added, we added the concept of COMDATs, which allows us to package functions and objects individually.  This was to provide a way to merge duplicate functions in order to support inline functions in C++.  It then was natural to add the -Gy switch to the compiler and the /OPT:ref switch to the linker to allow us to exclude unreferenced COMDATs within an object.

    To get a particular object pulled in, you need to fake a reference to it, or even better, have the object in a .obj which you link directly.

    Louis Lafreniere
    VC++ Development
    Thursday, November 24, 2005 12:41 AM
  • Dear Mr Lafreniere,
    Perhaps it would help a little to shed some light on why this feature is needed, and why a few of us are desparately asking for it.

    Those global variables are normally used to register various components, classes, features etc inside the application. For example, to be able to create a C++ object from its class name as string (coming from a config file, etc), one could use an global variable called classRegistry, with the type std::map<std::string,FactoryFunction>. Then, to dynamically register a class Foo, one would create a global object whose constructor performs {classRegistry["Foo"]=&createFoo;}, with createFoo() being defined  as {return new Foo();}. Hope it makes sense so far. In the main program, when I want to create a blank object given its class name, I'd just look up its factory function in classRegistry and call it. Perhaps you recognize the C++ replacement of Java's Class.forName method here.

    Now, suppose have a bunch of those dynamically registered classes, and instead of linking them as .obj files, I want to put all those classes into a lib file. (To make some sense of it, suppose those classes come from a 3rd party vendor as a library, and I want to add them into my app.) It won't work, because  I'd end up with an empty class registry in the main program. I can't add references to those global objects in the main program, because I don't know those classes beforehand. That's exactly the point of dynamic registration and factories.

    This was just one example, but it wouldn't be difficult to bring a number of other applications of the concept.

    What you described above is the normal way the linker works. That is correct. Except that there should be an option to pull in ALL symbols, to enable the above scenario. I thought it was supposed to be  /OPT:NOREF, but if it isn't, then there should be another one, /OPT:PULL_IN_ALL_SYMBOLS or whatever. Without that option, the only solution left is to run dumpbin on the lib, then get perl or some other script language to turn it into a .cpp file with heaps of #pragma comment(linker,...) and link that into the application. Pretty lame.

    Please see the Unix/GNU "-z allextract" or "--whole-archive" linker options for reference.

    Regards...
    Thursday, November 24, 2005 11:05 PM
  • Andras99, you describe *exactly* the scenario I am dealing with. I have a static library of components linked to my main .exe that the .exe creates and refers to by name (char *) and by abstract (pure virtual) C++ interfaces. Each component implementation registers via a global object in the .lib that as of now is never getting constructed because of the problems we and Mr. Lafreniere have brought to light.

    Mr. Lafreniere, we both agree with you that importing only referenced symbols should be the default behavior.

    But there SHOULD ALSO BE AN OPTION TO OVERRIDE THIS BEHAVIOR. Either OPT:NOREF should be extended to apply to objects in the global scope of .libs being linked in, or a new option should be added.

    Please take these concerns to heart and bring them up with the VC team. The current behavior of the linker is a serious roadblock to all similarly designed dynamic class registration systems using static .libs.

    Trey

    Sunday, November 27, 2005 7:24 PM
  • I had the same problem as Andras few yers ago when I implement a factory.

    The only solution I found, is to use DLL instead of LIB. It's different and more restrictive in certain case, but it works !

    regards,
    François.
    Wednesday, November 30, 2005 2:58 PM
  • We will consider what we can do for this in the next version of the product.

    Mark Lacey
    Visual C++ Development
    Thursday, December 01, 2005 9:34 AM
  •  

    Has there been any progress on this OLD topic ?
    Monday, February 25, 2008 4:36 PM
  • Codewarrior also provived a solution to solve the probleme with #pragma force_active on/off or better with modifying .lcf and just added "__declspec (My_AutoRegisterObject)" before each static objects (where "My_AutoRegisterObject" is a section in lcf to keep all symbols of this section").

    Is any plan to integrate such things in Visual Studio product ?
    I hope it will be the case !

    kinds regards,
    François.
    Monday, April 21, 2008 9:49 PM
  • co-ask

     

    Has there been any progress on this OLD topic ?
    Thursday, March 24, 2011 9:10 AM
  • co-ask

     

    Has there been any progress on this OLD topic ?
    co-ask
    Thursday, September 01, 2011 4:40 AM
  • I am suffering similar problems trying to compute good statistics on code coverage. The BlocksNotCovered was always too small as many symbols was excluded allready when linking the test program.

    I tried another approach not mentioned here which at least at first glance looks like it have worked.

    I combined the /OPT:NOREF flag with "Use Library Dependency Inputs" set to true (found in Properties -> Common Properties/Framework and References). This is for VS2010, I don't remember if this view was present in the earlier versions of VS.

     

    Thursday, December 22, 2011 3:53 PM
  • I have similar problems with IBM/Rational PureCoverage.  I tried the /OPT:NOREF flag with "Use Library Dependency Inputs" set to true in VS2005, but no effect.  Have others verified this to work with VS2010? 
    Friday, August 03, 2012 6:11 PM
  • I upgraded to VS2010 and verified the /OPT:NOREF flag with "Use Library Dependency Inputs" set to true does work.  There does not appear to be an analogue to "Common Properties/Framework and References" in VS2005.
    Tuesday, August 21, 2012 8:28 PM
  • I am suffering similar problems trying to compute good statistics on code coverage. The BlocksNotCovered was always too small as many symbols was excluded allready when linking the test program.

    I tried another approach not mentioned here which at least at first glance looks like it have worked.

    I combined the /OPT:NOREF flag with "Use Library Dependency Inputs" set to true (found in Properties -> Common Properties/Framework and References). This is for VS2010, I don't remember if this view was present in the earlier versions of VS.

     

    The "Use Library Dependency Inputs" option did the trick in VS2005!  The option can be found under Configuration Properties -> Linker -> General -> Use Library Dependency Inputs.

    On a note, you can declare the following to get the symbol to export from the DLL.  Define LIB_EXPORTS in the library project and nothing in either the DLL project or the DLL client project.

    #ifdef LIB_EXPORTS
    #define DLLAPI __declspec(dllexport)
    #else
    #define DLLAPI __declspec(dllimport)
    #endif
    Turns out there is no need #include any headers from the LIB project when compiling the DLL project.  However, if you need to make use of the LIB code from within the DLL, you will need to #define DLLAPI as an empty macro.
    • Edited by James Hugard Wednesday, September 12, 2012 7:51 PM
    • Proposed as answer by James Hugard Wednesday, September 12, 2012 7:51 PM
    Wednesday, September 12, 2012 7:38 PM
  • Louis,
    Thanks for your explanation. However, what would be required here doesn't mean importing the whole .obj symbols from a .lib. Would it possible to consider linker directives from .libs? Even if the .obj in the .lib is not reachable?
    That would allow for #pragma comment(linker, "/INCLUDE:MySymbol") to work. Right now, that #pragma isn't even parsed because the .obj isn't reachable.
    Would that break compatibility?
    Thanks,
    Alex.


    Wednesday, September 10, 2014 2:25 PM
  • co-ask
    Monday, June 27, 2016 7:42 AM