none
How to compile files using multiple processes or threads RRS feed

  • Question

  • I have an MSBuild file that I use to compile over 700 header files (i.e. *.h) in a directory. It works great, however it runs only one process at a time. I would like to use multiple processes to run CL.exe on many files at one time. Anyone have any ideas on how to do that?

    Here is a simple version of my build script: 

    <?xml version="1.0" encoding="utf-8"?> <Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" > <PropertyGroup> ... </PropertyGroup>

    <ItemDefinitionGroup>
    <SDKFiles>
    <WarningLevel>Level4</WarningLevel>
    <MultiProcessorCompilation>true</MultiProcessorCompilation>
    </SDKFiles>
    </ItemDefinitionGroup>

    <ItemGroup> <SDKFiles Include="$(CompileDir)\**\*.h" Exclude="$(CompileDir)\**\...; ... </ItemGroup> <UsingTask TaskName="CL" AssemblyFile="$(VCTargetsPath)\Platforms\x64\Microsoft.Build.CPPTasks.x64.dll" Condition="'$(Platform)'=='x64'" /> <UsingTask TaskName="CL" AssemblyFile="$(VCTargetsPath)\Platforms\Win32\Microsoft.Build.CPPTasks.Win32.dll" Condition="'$(Platform)'=='Win32'" /> <Target Name="CompileStuff"> ... <!-- This CL.exe compiles all the files --> <CL Sources="@(SDKFiles)" CompileAs="CompileAsCpp" SuppressStartupBanner="true" ObjectFileName="$(IntDir)" WarningLevel="$(Warnings)" ExceptionHandling="Sync" /> ... </Target> </Project>

    I have tried passing in /M:n to MSBuild.exe on the command line, but that doesn't seem to help. 

    Thanks for any help in advance.


    • Edited by Chris P Johnson Monday, December 31, 2012 2:13 PM Added ItemDefinitionGroup
    Tuesday, December 4, 2012 2:28 PM

Answers

  • Hi Chris,

    I figured out why C++ parallel build doesn't work. There're two project setting conflict with /mp. If you turn on Diagnostic logging, you can find below in VS Output window:

    cl : Command line warning D9030: '/Gm' is incompatible with multiprocessing; ignoring /MP switch

    1>cl : Command line warning D9030: '/Yc' is incompatible with multiprocessing; ignoring /MP switch

    Turning off the above two switch will kick off the parallel build. They corresponds to:

    1. Project property page > C/C++ > Code Generation > Enable Minimal Rebuild(/Gm)
    2. C/C++ > Precompiled Header

    Configured three parts to make C++ project build in parallel:

    1.        Visual Studio 2012 Tools menu > Options > Project and Solutions > Build and Run: default to CPU cores
    2.        Project and Solutions > VC++ Project Settings > Maximum concurrent C++ compilation: 10
    3.        Project property page > Configuration Properties > C/C++ > Multi-processor Compilation : Yes (/mp)

    I think you can leave your task extension now. Please let me know if you have further questions.


    Forrest Guo | MSDN Community Support | Feedback to manager

    Friday, January 4, 2013 3:14 AM
    Moderator

All replies

  • Hi Chris,

    Thank you for your question.

    I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.
     
    Thank you for your understanding and support.

    Best regards,


    Ego [MSFT]
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Wednesday, December 5, 2012 8:30 AM
    Moderator
  • Hi,

    If you have only one project file, the /m switch use only one processor. refer to this msdn page. Note that, the example in the article specify file *.proj, this work like solution file.  This applys to non-C++ solution, like C#.  C++ solution can build parallel even it's one project.

    While you want to accelerate the build, you need to organize the solution in multiple projects.

    Tuning C++ Build Parallelism blog

    thanks.


    Forrest Guo | MSDN Community Support | Feedback to manager


    Thursday, December 6, 2012 4:57 AM
    Moderator
  • Thanks, I tried everything listed in these two articles, and nothing worked. I've tried all sorts of parameters to MSBuild, and other build properties. Nothing worked for this. 
    Thursday, December 27, 2012 4:15 PM
  • Bump. Can I get an MSBuild Guru to look at this? I'd like to know if this is possible without having to write my own task extension.
    Friday, December 28, 2012 5:06 PM
  • Hi Chris,

    I have made update to my first reply in this thread. C++ project build parallel works for single project also. It doesn't limited by project.

    My further investigation shows build in parallel can be triggered in one test project, but does not trigger in another. Difference is project A has more C++, and header files than project B. So I believe there is a trigger condition, but I haven't figured out yet.

    I'll consult the topic internally. Please wait a few days.  Happy holiday.

    thanks.


    Forrest Guo | MSDN Community Support | Feedback to manager

    Sunday, December 30, 2012 9:42 AM
    Moderator
  • One thing I did was try was to create an empty project and modify it so that the <ClCompile> items were header files, instead of *.cpp files. But when I tried to load the project in visual studio 2010 and 2012 it would not load. I'm not sure why though...

    <?xml version="1.0" encoding="utf-8"?>
    <Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    	<ItemGroup Label="ProjectConfigurations">
    		<ProjectConfiguration Include="Debug|Win32">
    			<Configuration>Debug</Configuration>
    			<Platform>Win32</Platform>
    		</ProjectConfiguration>
    		<ProjectConfiguration Include="Release|Win32">
    			<Configuration>Release</Configuration>
    			<Platform>Win32</Platform>
    		</ProjectConfiguration>
    	</ItemGroup>
    	<PropertyGroup Label="Globals">
    		<ProjectGuid>{0434DA11-4E54-467C-AA66-A2D0FE287FB8}</ProjectGuid>
    		<Keyword>Win32Proj</Keyword>
    		...
    		<SDK>..\..\..\..\..\src\SDK</SDK>
    		<SdkDir>$(SDK)\include</SdkDir>
    	</PropertyGroup>
    	<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
    	...
    	<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
    	<ItemDefinitionGroup>
    		<ClCompile>
    			<WarningLevel>Level4</WarningLevel>
    			<MultiProcessorCompilation>true</MultiProcessorCompilation>
    		</ClCompile>
    		<Link>
    			<SubSystem>Console</SubSystem>
    		</Link>
    	</ItemDefinitionGroup>
    	<ItemGroup>
    		<ClInclude Include="targetver.h" />
    	</ItemGroup>
    	<ItemGroup>
    		<ClCompile Include="foo.cpp" />
    		<ClCompile Include="$(SdkDir)\**\*.h" 
    					Exclude="$(SdkDir)\**\bla.h;
    							$(SdkDir)\**\foo.h;
    							$(SdkDir)\**\baz.h;
    							$(SdkDir)\**\etc.h;
    		/>
    	</ItemGroup>
    	<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
    </Project>



    Monday, December 31, 2012 2:08 PM
  • So I have written a C# Task extension that solves this problem for me. If anyone is interested in it, respond to this, and I'll send it to them.
    Monday, December 31, 2012 2:32 PM
  • Hi Chris,

    I figured out why C++ parallel build doesn't work. There're two project setting conflict with /mp. If you turn on Diagnostic logging, you can find below in VS Output window:

    cl : Command line warning D9030: '/Gm' is incompatible with multiprocessing; ignoring /MP switch

    1>cl : Command line warning D9030: '/Yc' is incompatible with multiprocessing; ignoring /MP switch

    Turning off the above two switch will kick off the parallel build. They corresponds to:

    1. Project property page > C/C++ > Code Generation > Enable Minimal Rebuild(/Gm)
    2. C/C++ > Precompiled Header

    Configured three parts to make C++ project build in parallel:

    1.        Visual Studio 2012 Tools menu > Options > Project and Solutions > Build and Run: default to CPU cores
    2.        Project and Solutions > VC++ Project Settings > Maximum concurrent C++ compilation: 10
    3.        Project property page > Configuration Properties > C/C++ > Multi-processor Compilation : Yes (/mp)

    I think you can leave your task extension now. Please let me know if you have further questions.


    Forrest Guo | MSDN Community Support | Feedback to manager

    Friday, January 4, 2013 3:14 AM
    Moderator
  • Wow Forest, that totally worked!

    Here is XML that calls CL:

    <UsingTask TaskName="CL" AssemblyFile="$(VCTargetsPath)\Platforms\x64\Microsoft.Build.CPPTasks.x64.dll"   Condition="'$(Platform)'=='x64'" />

    <UsingTask TaskName="CL" AssemblyFile="$(VCTargetsPath)\Platforms\Win32\Microsoft.Build.CPPTasks.Win32.dll" Condition="'$(Platform)'=='Win32'" />

    <!-- Uses normal means to compile the files specified -->
    <Target Name="CompileFiles">
    <!-- Delete any left over intermediate files from previous attempts, if there are any -->
    <RemoveDir Directories="$(IntDir)" Condition="exists($(IntDir))" />
    <!-- If we don't create the directory first, it will give an error when the .obj file can't be written -->
    <MakeDir Directories="$(IntDir)" /> <!-- This CL.exe compiles all the files -->
    <!-- In order to compile in parallel and take advantage of /Mp, and get a big performance boost, we have to specify false for Minimal rebuild and to not use a precompiled header. -->
    <CL Sources="@(SDKFiles)" MultiProcessorCompilation="true" CompileAs="CompileAsCpp" SuppressStartupBanner="true" MinimalRebuild="false" PrecompiledHeader="NotUsing" ObjectFileName="$(IntDir)" WarningLevel="$(Warnings)" ExceptionHandling="Sync" />   
    </Target>


    And here is how I call it from the command line:

    call "%VS100COMNTOOLS%..\..\VC\vcvarsall.bat" x64

    set TARGET=/target:CompileFiles
    set MISC=/m /nologo /verbosity:Normal
    set PROPERTY=/property:Platform=x64;Configuration=Debug
    msbuild %PROPERTY% %MISC% %TARGET% BuildSomething.xml



    Thursday, May 16, 2013 5:06 PM
  • It's nice to know it works!

    Forrest Guo | MSDN Community Support | Feedback to manager

    Friday, May 17, 2013 12:44 AM
    Moderator