none
How to calculate dependencies at build time

    Question

  • I am creating a custom build rule for VS2010.  How do I make my rule depend on files referenced inside my custom "source files"?  I want to avoid having an "Additional Dependencies" field.  I'd like the build process to essentially figure out the dependencies as if they were "#inlcude"ed.

    ...Matt

    Thursday, April 22, 2010 7:50 PM

All replies

  • Hi Matt,

    Thanks for your post.

    I'm not sure I understand you clearly, if I misunderstood you, please feel free to let me know.

    Do you mean how to create a custom build rule for VS 2010? If so, in VS2010, due to the migration to MSBuild, the information in the rules file is represented by three files: .XML, .props and .targets files. And  there is no Build Customization editor in VS 2010 yet.  The only simple way would be to use VS 2008 editor to create custom build rule .rules file and then convert it to 2010. 

    And you can use <Link> and <Lib> dependency to include some source file or lib file in custom build rule file.For more information, please see:

    Quick Help on VS2010 Custom Build Rule

    But if you want to use custom build rule to include some files but not need to specify under "Additional Dependencies", you can use Library tool as Li Shao said in this thread: Visual Studio 2010 Link dependency for more information. 

    Regards,

    Nancy Shao [MSFT]
    MSDN Subscriber Support in Forum
    If you have any feedback on our support, please contact msdnmg @ microsoft.com


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Monday, April 26, 2010 3:24 AM
  • For example,

    Let's say I have my own compiler which compiles files from .x to .y. 

    I create a custom build rule for my .x files.

    However, my .x files "include" other files (for example .hx).  So how do I tell MSBuild what .hx files it needs to look at to determine if it needs to compile the .x file.

    ...Matt

    Monday, April 26, 2010 2:27 PM
  • Hi Matt,

    As far as I know Custom build rules, just like custom build steps, are commonly used for invoking external tools that translate files from one format to the other, so if you .x files "include" other files, it's not related to MSBuild, but related to the external tool.

    If these custom build rules depends on some dependencies, you can add these dependencies to Additional dependencies box in Custom Build Rules Dialog when you add this new custom build rule to Visual C++ project. For more information, please refer to:

    Understanding Custom Build Rules in Visual C++ 2005: Building Flex and Bison Files from the IDE

    Regards,

    Nancy Shao [MSFT]
    MSDN Subscriber Support in Forum
    If you have any feedback on our support, please contact msdnmg @ microsoft.com


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Wednesday, April 28, 2010 9:21 AM
  • I understand that the tool is required to do the actual including and that it is not MSBuild's job to do that.  But somehow, MSBuild "knows" when .h files change to have .cpp files recompiled.  How can I get the same to happen for my own files?

    I don't want to use "AdditionalDependencies"... .cpp files don't require this.

    Or is MSBuild just lucky that it and cl.exe are made by the same company so it gets some special treatment?

    Is there anyway for me to instruct MSBuild to run a program/load a DLL/execute some code to "calculate" the dependencies and put them in some variable?

    ...Matt

    Wednesday, April 28, 2010 9:36 PM
  • Hi Matt,

    MSBuild engine compares the timestamp between inputfiles and outputfiles. Once the timestamp of inputfiles is newer than outputfiles, it will trigger to be rebuilded. If you use MSBuild to build your own files, you don't need to do this by yourself.

    ->Is there anyway for me to instruct MSBuild to run a program/load a DLL/execute some code to "calculate" the dependencies and put them in some variable?

    Do you mean you want to specify performing a link witout building sources in MSBuild command line? If so, in VCBuild.exe, you can use /link switch, but in MSBuild, there is no corresponding switch to do this, you can try to use /t:BuildLink. For more information, please see:

    VCBuild vs. C++ MSBuild on the Command Lin

    Regards,

    Nancy Shao [MSFT]
    MSDN Subscriber Support in Forum
    If you have any feedback on our support, please contact msdnmg @ microsoft.com


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Wednesday, May 05, 2010 4:48 AM
  • Hi Nancy,

    I don't think you're understanding what I am trying to accomplish.

    Let's think about the C++ compiler.  When you have a cpp file that includes a header file (.h), the whole system (whether it's MSBuild, cl, etc.) knows that if the timestamp on the header file is more recent than the output file, then it will recompile the cpp file.  It does this without the user having to specify the header file specially anywhere in the build system.  MSBuild just "knows" somehow.  Somewhere, some process in MSBuild looks at the cpp file, sees the #include for the .h file then checks the timestamp on that .h file.  Something in the MSBuild process examines the contents of the cpp file to look for additional dependencies.

    How can I accomplish this same thing for my own .x and .xh files using my own compiler?  It has to be part of the MSBuild process otherwise my compiler will never to told to recompile the .x file when the .xh file is modified.

    I know that "AdditionalDependencies" will solve this, but this requires manual work for every .xh file by the programmer, when cpp has it done automatically.

    ...Matt

    Wednesday, May 05, 2010 3:51 PM
  • I'm very interested in the answer to this, too. We also have a case where it's not feasible to use AdditionalDependencies, for similar reasons.

    We eventually just hacked around this problem by:

    1. Forcing our tool to run on every build, by listing an output file that isn't ever actually created.
    2. Executing an extremely fast dependency calculation phase at the beginning of our custom tool. If no dependencies have changed, we exit immediately.

    But we'd rather have an elegant solution that relies on MSBuild's existing dependency calculation.

    Thursday, May 06, 2010 7:21 PM
  • I am also interested in this, I can see that Matt's D solution would work, but it's definitely not ideal.

    I'm also afraid that Nancy is not understanding what Matt C was trying to ask, so I'll restate the problem.

    We want to be able to programatically set the "Additional Dependencies" for a file (not the link dependencies, I think maybe this is where Nancy was getting confused) that is being built by a custom build step / rule.  At the moment you can only supply a static list of dependency files.

    Imagine that you are compiling an ASM file using custom build rules, this assembler allows you to #include a header file that has some defines shared between it and a C file.  Now sure I could add the ASM file and the H file to the "Additional Dependancies" but what if the header includes other headers, and what if those headers include other headers, and what if someone changes one of those headers to include more headers, each time someone does this we'd manually have to add the new dependencies in the "Additional Dependencies" box.  In fact Microsofts own custom build rule for MASM files suffers this problem, and its a serious one.

    Build systems get around this by using a program to calculate the dependencies for a file, like using GCC's -M option, when any of the dependencies change this program is run again to cache the new dependency list.  Visual Studio does this for C/CPP files, but doesn't expose that functionality to the user.

    A static list of dependencies specified by the user is only useful in very simple cases.

    Friday, May 21, 2010 7:54 PM
  • Has there been any development on this? Is there a way to do what Matt Houser suggests?
    Tuesday, December 07, 2010 11:24 PM
  • Bump,

    I just wrote a very poor and hacky bit of C++ to go through a C++ style file and traverse #includes, respecting some degree of macros.

    In my case I want to make FXC, the shader compiler, respect its header dependencies and in shader files we'll be keeping the pre-processor simple enough that my code will hopefully work as part of a wrapper. It's ugly though, I'd rather have some C++ interface for invoking the pre-processor for this type of stuff.

    Another headache on timestamps... I have P4, so I was going to edit a header and then reverted. All this updated the timestamp of the header, but such action does not trigger MSVC8 at least to rebuild, even though the modified date/time on the file is newer than the .cpps it's in, the libs, and the exe. My code doesn't behave the same way because I compare the dates and the .h file is legitimately newer. MSBuild and/or MSVC8 must be doing something either smarter or in such a way that it doesn't make it to the dependencies. Any light that can be shed on this would help me in my hacky emu endeavors, thanks!

    Friday, January 14, 2011 3:31 AM
  • Hi igHunterKiller

     

    Have you made any improvements? Where can I find a sample of your hacky C++ snippets?

    Saturday, March 05, 2011 4:47 PM
  • How to calculate dependencies at build time in VC2010 (with msbuild) , in a nutshell:

     

    1. On your build target, use makedepend or gcc -M to build a ".dep" file for every input file, right after the input file was built successfully.
    2. On your build target, translate (using gawk or anything else) the newly created ".dep" file into a file which contains a simple list of dependent files (lets call it ".msdep" file).
    3. These ".msdep" files should be defined as the "Outputs" of your Build target.
    4. Create a new target (let's call it "Prepare"), on which your Build target depends.
    5. The "Prepare" target will create a list of dependencies per input file from the relevant ".msdep" file (use "ReadLinesFromFile" action with msbuild batching).
    6. Your Build target "Inputs" should contain both the input file and the dependencies calculated by Prepare.
    What actually happens:
    1. Build target is called. It depends on target "Prepare" for each input file.
    2. Target prepare is called for each input file and reads the ".msdep" file which specifies the dependencies, from which it generates a list of dependencies for that input file.
    3. Build target now checks the list of dependencies generated by "Prepare". If the date of any of them (or of the input file) is newer than the date of the ".msdep" file, Build target will run.
    4. When the Build target runs it processes the input file, and if it succeeds it generates (and overwrites) the ".msdep" file. The ".msdep" file will be used both as "Output" and as a list of dependencies for the next build.
    It is also worth mentioning that:
    1. "Build" and "Prepare" targets run for each input file individually.
    2. ".msdep" file is created for each input file.
    3. Every time Build is attempted, all ".msdep" files of all input files are read.
    4. Every time an input file is built successfully, an ".msdep" file is generated for it and overwrites the old ".msdep" file (if exists).
    5. At the first build the ".msdep" files don't exist, so all input files + their ".msdep" files will be built.
    This technique works well for us, and is actually fast enough although all .msdep files are read for every build.
    I hope this summary will help someone.

    Amir

     

     

    Sunday, March 13, 2011 2:55 PM
  • Sorry scandella, I finally figure out something in 2008 and didn't check posts for a while. The C++ traversal thing ended up being a bit wonky because with Perforce there were legitimate cases where the dates would not imply a rebuild, but indeed a rebuild was needed - so the coding way wasn't a good idea.

    Now I'm back because I, like the OP, am in VS2010, floundering in the UI-less definition of a custom build step for FXC. But I've almost got it working. For MS Support folks:

    I have my custom build rule called FXC. I have a lot of shaders (~20 with more to come) I'd like to share settings such as include paths and "Additional Dependencies". I tried emulating VS2008 practice by adding the values to the Project's version of FXC rules, but in VS2010 it isn't coming through.

    My .target file has this section:

      <FXC
       Condition="'@(FXC)' != '' and '%(FXC.ExcludedFromBuild)' != 'true'"
       CommandLineTemplate="%(FXC.CommandLineTemplate)"
       NoLogo="%(FXC.NoLogo)"
       Includes="%(FXC.Includes)"
       Defines="%(FXC.Defines)"
       EntryPoint="%(FXC.EntryPoint)"
       FXCOutput="%(FXC.FXCOutput)"
       ByteCodeVariableName="%(FXC.ByteCodeVariableName)"
       ShaderModel="%(FXC.ShaderModel)"
       DebugInformationFormat="%(FXC.DebugInformationFormat)"
       AdditionalOptions="%(FXC.AdditionalOptions)"
       Inputs="%(FXC.Identity)" />
    
    

    It appears this is saying "Get my value from the FXC rule". Is there a way to also get the values from the current project? Or is that supposed to happen inside the %() variable?

    For example how can I get this working?

    Includes="%(FXC.Includes);%(CURRENT_PROJECT.Includes)"
    Please advise.

    Friday, May 13, 2011 10:23 PM
  • /bump

    Is there a way in 2010 to do what I could do in 2008, which is for custom build rules set build options in the project that would by default apply to all files built by that rule in the project?

     

    Thanks!

    Thursday, July 07, 2011 5:36 PM
  • Hi Amir

    Could you please provide a sample project/solution which implements your method described above? That would be great!

    Thanks!

    Wednesday, April 04, 2012 12:20 PM
  • scandella -

    Unfortunately I cannot provide you with actual code since it is part of the company's IP.

    However if you follow my explanation, I could try to help you if you have any specific questions.

    btw, my post above is about 1 year old. Since then we found another more general way to handle build dependencies, using the Tracker utility. It is used by Microsoft build tools to record every file which is read or written while running some application. In other words - you can track all your inputs and outputs during compilation and based on that you can build incrementally.

    Today we are using both methods in our build system.

    Amir


    • Edited by Amir Gonnen Wednesday, April 04, 2012 1:44 PM typo
    Wednesday, April 04, 2012 1:37 PM
  • I just tested it with VS 2012. Shader compilation with fxc is now integrated and seems to work perfectly. Also changing an include file (e.g. fxh) works correctly.

    Saturday, September 08, 2012 10:42 PM