LINK.EXE BUG: /OPT:NOREF option doesn't work!

Question

My program contains static objects whose constructors are supposed to run at the beginning of the program, but otherwise those variables are unreferenced. The problem is, when I put these variables into a .lib file, they get completely optimized away and never appear in the resulting .exe!

My understanding is that /opt:noref is supposed to cure this, however, it has absolutely no effect. I also tried __declspec(selectany) on the vars, also no effect. The same behaviour occurs with VC6, 7 (2003) and now also 8 (2005 Express).

This looks very much like a linker bug. Is there any workaround to this problem? (Such as additional linker flag, __declspec, pragma or anything? Referencing the vars from other files or using /INCLUDE:symbol for each var is not doable, because those vars are created by macros and supposed to be invisible to the user.)

I'm attaching a little program to demonstrate the problem. When built using build.bat (below), it should print "Noisy created" -- but nothing gets printed. And if you search for the text "Noisy created" in main.exe, it's not contained in there.

noisy.cpp:

#include <stdio.h> 
class Noisy { 
 public: 
 Noisy() {printf("Noisy created!\n");} 
}; 
Noisy noisy; 

main.cpp:

int main(int argc, char **argv) { return 0; } 

build.bat:

cl /c noisy.cpp 
lib /out:noisy.lib noisy.obj 
cl /c main.cpp 
link /out:main.exe main.obj /opt:noref noisy.lib 

 

Answer 1

This bug drove me nuts a while back. As far as I know, there's no way to correct this generally, the linker just doesn't seem to understand that it can't discard static constructors.

For a while I would make an explicit reference to a dummy function which referenced the variables from my main project, but that was obviously not an ideal solution.

My current solution (hack?) is to use a Ruby script as a post build step on my libraries. This script uses the dumpbin.exe tool to find all symbols containing the word 'forceLink', and it outputs a header file containing
#pragma comment(linker, "/include:symbol")
for each of the symbols. I then include that header file from my main project, and then everything just works.

Niall.

Answer 2

Discarding unreferenced symbols is what linkers do and it's "a good thing."  Your technique of using #pragma comment(linker,...) is the correct approach.  MFC does this also (see the header afx.h).

Brian

Answer 3

This is not a linker bug and the linker isn't discarding anything.  This is just the way .libs work. 

An object in a library isn't pulled into the final image unless a symbol definition in the object is needed to satisfy an unresolved referenced in the image.  Otherwise, when linking in the CRT or any other lib, you'd get the whole lib pulled in instead of just the needed pieces.

So you do need to had something to force the linker to pull in that object.  You could add dummy references, or use /include.

Hope this helps,

Louis Lafreniere
VC++ Development

Answer 4

The linker only pulls in objects which have referenced symbols.

Would it be possible to add a function to these types which does nothing, and reference that via a #pragma comment() within a header file that your user's include?

e.g.
noisy.cpp

#include <stdio.h>

class Noisy
{
public:

Noisy()
{
   printf("Noisy created!\n");
}
void emptyreference();

};

void Noisy::emptyreference(){}

Noisy noisy;

mylib.h

   #pragma comment(linker, "/include:?emptyreference@Noisy@@QAEXXZ")

main.cpp

#include "mylib.h"

int main(int argc, char **argv) { return 0; }

Answer 5

Louis: I don't mind that the linker does optimizations, that's part of its job, I just mean that /OPT:NOREF doesn't do what it is supposed to do. Please have a look at the following MSDN page which documents the linker /OPT options:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore/html/_core_.2f.opt.asp

A quote from this page:

"REF | NOREF

/OPT:REF eliminates functions and/or data that are never referenced while /OPT:NOREF keeps functions and/or data that are never referenced."

This is quite clear. Still, in the above example, the linker discards the global "Noisy" object (its constructor doesn't get called at startup, and the code even doesn't appear in the executable -- the exe file doesn't contain the string "Noisy created".)

How can this NOT be a bug, when it is clearly the contrary of what the documentation says?

BTW, the equivalent option in the Linux world is "-z allextract" (Sun) or "--whole-archive" (GNU), they work as expected.

Niall: thanks for the suggestion to collect the list of variables from dumpbin output. I think I'll to do that. Not very elegant, but at least works :)

Answer 6

Andras,

You are mixing 2 concepts:
* When using /OPT:REF linker removes unreferenced code and data from object files it decides to add to the executable. That is the optimization, and you can turn it off by using /OPT:NOREF
* When searching libraries, linker picks up only object files that contain explicitly referenced code or data -- so it will not pick up some variable with global constructor if there is no reference from your code (or from other object file, or from object file included in some library if linker decides to include that object file). That is NOT the optimization, linker always works this way, and that behavior is NOT controlled by /OPT:NOREF.

You can do what you want by including explicit reference to the global variable from your code, or by explicitly putting object file  containing the variable into linker command line.

You are right that MSDN documentation may be clearer on that issue. I'll let our documentation people know, so hopefully they will improve that topic for the next VS release.

Thanks,
Eugene

Answer 7

Regardless whether or not there is an issue with /OPT, this version of main.cpp should pull in the global Noisy instance:


#pragma comment(linker, "/include:??0Noisy@@QAE@XZ")

int main(int argc, char **argv) { return 0; }

 

Answer 8

OK, I realize there isn't much willingness to fix it properly in the linker, so I'm not pushing it any further.

Thanks to everybody suggesting the dumpbin+grep+pragma comment(linker,...) workaround.

Answer 9

Mr. Lafreniere,

This is actually a real problem.

ANY OBJECT IN THE GLOBAL SCOPE OF A .LIB IS IN THE GLOBAL SCOPE OF THE TARGET .EXE. It is incorrect to treat objects in the global scope of a .lib any differently than you treat objects in the global scope of the .exe.

The /OPT:NOREF linker option should have the same effect on objects in the global scope of .libs as it has on objects in the global scope of the .exe. IT IS THE SAME SCOPE. To sound like a broken record, OBJECTS IN THE GLOBAL SCOPE OF A LIB ARE IN THE SAME "GLOBAL" SCOPE OF THE .EXE. THERE'S A REASON ITS CALLED "GLOBAL".

lemme know when its fixed =)

Trey

Answer 10

I'm sorry but this is just not the way libraries work.  A lib contains a collection of features supported by a set of .obj, which themselves contain a set of functions and objects.

When the linker has a reference it can't resolves, it looks in the libs available.  If it finds a definition for that symbol, it pulls the .obj defining that reference out of the .lib and links it in.

Only the .objs containing the definitions necessary to resolve all references are pulled in.  This is nothing new and has been this way since we added support libraries a long long time ago.

This is by design and for very good reasons:  you don't want to blow up the image size with a lot of unnecessary objects (code and data) which are not used.  For example, you don't want the whole CRT and the whole MFC in your application when creating a simple MFC app.  That's exactly why libraries were added in the first place!  Yes, all these functions and objects are at global scope, but the whole concept of libraries was added to avoid including the unnecessary ones, and only include code which supports the features used by your app.

But even if I agreed with you, we couldn't change the behavior of libs since most library providers (including Microsoft) rely heavily on the current behavior.  For example, a library can contain multiple versions of a function or object (single vs multi-threaded, ansi vs unicode, etc).  Depending on your app configuration, only one of the .objs will be pulled in by a reference and you'll get that particular version of the function.

In C/C++ 7.0, long after library support was added, we added the concept of COMDATs, which allows us to package functions and objects individually.  This was to provide a way to merge duplicate functions in order to support inline functions in C++.  It then was natural to add the -Gy switch to the compiler and the /OPT:ref switch to the linker to allow us to exclude unreferenced COMDATs within an object.

To get a particular object pulled in, you need to fake a reference to it, or even better, have the object in a .obj which you link directly.

Louis Lafreniere
VC++ Development

Answer 11

Dear Mr Lafreniere,
Perhaps it would help a little to shed some light on why this feature is needed, and why a few of us are desparately asking for it.

Those global variables are normally used to register various components, classes, features etc inside the application. For example, to be able to create a C++ object from its class name as string (coming from a config file, etc), one could use an global variable called classRegistry, with the type std::map<std::string,FactoryFunction>. Then, to dynamically register a class Foo, one would create a global object whose constructor performs {classRegistry["Foo"]=&createFoo;}, with createFoo() being defined  as {return new Foo();}. Hope it makes sense so far. In the main program, when I want to create a blank object given its class name, I'd just look up its factory function in classRegistry and call it. Perhaps you recognize the C++ replacement of Java's Class.forName method here.

Now, suppose have a bunch of those dynamically registered classes, and instead of linking them as .obj files, I want to put all those classes into a lib file. (To make some sense of it, suppose those classes come from a 3rd party vendor as a library, and I want to add them into my app.) It won't work, because  I'd end up with an empty class registry in the main program. I can't add references to those global objects in the main program, because I don't know those classes beforehand. That's exactly the point of dynamic registration and factories.

This was just one example, but it wouldn't be difficult to bring a number of other applications of the concept.

What you described above is the normal way the linker works. That is correct. Except that there should be an option to pull in ALL symbols, to enable the above scenario. I thought it was supposed to be  /OPT:NOREF, but if it isn't, then there should be another one, /OPT:PULL_IN_ALL_SYMBOLS or whatever. Without that option, the only solution left is to run dumpbin on the lib, then get perl or some other script language to turn it into a .cpp file with heaps of #pragma comment(linker,...) and link that into the application. Pretty lame.

Please see the Unix/GNU "-z allextract" or "--whole-archive" linker options for reference.

Regards...

Answer 12

Andras99, you describe *exactly* the scenario I am dealing with. I have a static library of components linked to my main .exe that the .exe creates and refers to by name (char *) and by abstract (pure virtual) C++ interfaces. Each component implementation registers via a global object in the .lib that as of now is never getting constructed because of the problems we and Mr. Lafreniere have brought to light.

Mr. Lafreniere, we both agree with you that importing only referenced symbols should be the default behavior.

But there SHOULD ALSO BE AN OPTION TO OVERRIDE THIS BEHAVIOR. Either OPT:NOREF should be extended to apply to objects in the global scope of .libs being linked in, or a new option should be added.

Please take these concerns to heart and bring them up with the VC team. The current behavior of the linker is a serious roadblock to all similarly designed dynamic class registration systems using static .libs.

Trey

Answer 13

I had the same problem as Andras few yers ago when I implement a factory.

The only solution I found, is to use DLL instead of LIB. It's different and more restrictive in certain case, but it works !

regards,
François.

Answer 14

We will consider what we can do for this in the next version of the product.

Mark Lacey
Visual C++ Development

Answer 15

 

Has there been any progress on this OLD topic ?

Answer 16

Codewarrior also provived a solution to solve the probleme with #pragma force_active on/off or better with modifying .lcf and just added "__declspec (My_AutoRegisterObject)" before each static objects (where "My_AutoRegisterObject" is a section in lcf to keep all symbols of this section").

Is any plan to integrate such things in Visual Studio product ?
I hope it will be the case !

kinds regards,
François.

Answer 17

co-ask

 

Has there been any progress on this OLD topic ?

Answer 18

co-ask

 

Has there been any progress on this OLD topic ?

co-ask

Answer 19

I am suffering similar problems trying to compute good statistics on code coverage. The BlocksNotCovered was always too small as many symbols was excluded allready when linking the test program.

I tried another approach not mentioned here which at least at first glance looks like it have worked.

I combined the /OPT:NOREF flag with "Use Library Dependency Inputs" set to true (found in Properties -> Common Properties/Framework and References). This is for VS2010, I don't remember if this view was present in the earlier versions of VS.

 

Answer 20

I have similar problems with IBM/Rational PureCoverage.  I tried the /OPT:NOREF flag with "Use Library Dependency Inputs" set to true in VS2005, but no effect.  Have others verified this to work with VS2010? 

Answer 21

I upgraded to VS2010 and verified the /OPT:NOREF flag with "Use Library Dependency Inputs" set to true does work.  There does not appear to be an analogue to "Common Properties/Framework and References" in VS2005.

Answer 22

I am suffering similar problems trying to compute good statistics on code coverage. The BlocksNotCovered was always too small as many symbols was excluded allready when linking the test program.

I tried another approach not mentioned here which at least at first glance looks like it have worked.

I combined the /OPT:NOREF flag with "Use Library Dependency Inputs" set to true (found in Properties -> Common Properties/Framework and References). This is for VS2010, I don't remember if this view was present in the earlier versions of VS.

 

The "Use Library Dependency Inputs" option did the trick in VS2005!  The option can be found under Configuration Properties -> Linker -> General -> Use Library Dependency Inputs.

On a note, you can declare the following to get the symbol to export from the DLL.  Define LIB_EXPORTS in the library project and nothing in either the DLL project or the DLL client project.

#ifdef LIB_EXPORTS
#define DLLAPI __declspec(dllexport)
#else
#define DLLAPI __declspec(dllimport)
#endif

Turns out there is no need #include any headers from the LIB project when compiling the DLL project.  However, if you need to make use of the LIB code from within the DLL, you will need to #define DLLAPI as an empty macro.

Answer 23

Louis,
Thanks for your explanation. However, what would be required here doesn't mean importing the whole .obj symbols from a .lib. Would it possible to consider linker directives from .libs? Even if the .obj in the .lib is not reachable?
That would allow for #pragma comment(linker, "/INCLUDE:MySymbol") to work. Right now, that #pragma isn't even parsed because the .obj isn't reachable.
Would that break compatibility?
Thanks,
Alex.

Answer 24

co-ask

Answer 25

I was searching for a way to do the same thing and came across this thread. Perhaps I'm not the only one. In case someone else stumbles on this thread, here's the answer for more current MS toolchains (at least 2015 and newer) ...

MS has added a linker option similar to the one in GCC referenced in this thread:

https://docs.microsoft.com/en-us/cpp/build/reference/wholearchive-include-all-library-object-files