none
VS2017 brings new parallel build synchronization issues where VS2015 was stable (C++)

    Question

  • Howdy all (especially VS2017 authors),

    I've spent the better part of last two weeks trying to pinpoint down a build synchronization issue that was brought by using VS2017 instead of VS2015: build steps of one shared project can be, under some conditions, launched two or more times during the same build - on top of each other. This almost always leads to multiple processes trying to access the same file and build ultimately failing. Hopefully I've now created a repro issue that should reproduce it on most other computers as well.

    The problem of not waiting for a shared dependency to build and launching multiple instances of it happens only in VS2017 and only if projects in solution utilize custom MSBuild targets that bring additional project references during build time (I wrote such script for my company which at build time adds <ProjectReference> elements into projects so we don't have to have every single dependency inside the same solution). The exact same script was building our projects flawlessly in VS2012, VS2013, VS2015 a command-line MSBuild for several years already and I can still compare with VS2015 (working) behavior as our projects still utilize VS2015 runtime at this moment.

    What actually happens is that under normal conditions, a MSBuild Project Target is always executed once and only once during one build if it is launched with the exact same set of properties each time. If multiple projects built in parallel decide to run the same target on the same project, they all wait until it is completed. However, VS2017 sometimes decides to not wait and run a second instance of it, ruining the build since both instances access the same files and do the same thing. This does not happen if run via MSBuild in command-line or in any other Visual Studio. It seems to be a timing issue and it's quite hard to reproduce consistently as it sometimes decides to wait, sometimes not. It is "fixed" by adding something as a reference to one project and then removing it (essentially reverting to original state) which gives a feel there is something fishy going behind the scenes in VS2017.

    I've now created a reliable repro scenario and would like to ask you to try it so it can be isolated and fixed. It should be consistently reproducible on a clean machine with only VS2017 Studio + VS2015 C++ compiler installed using the following steps:

    0) Download https://drive.google.com/a/avast.com/file/d/1q66ZavFWY933ZFKtTN9OvzFuEJ9E7k37/view?usp=sharing

    1) run build_system.bat which builds a .NET Task dll used by the build script
    2) open solution/repro/repro2.sln in VS2017 (do not upgrade toolset). Make sure you have parallel build enabled in 'Build and Run'.
    3) run "Clean Solution" command (no need to build).
    You should see that both projects in solution run the same Clean targets on their (hidden) dependencies such as:

    1>------ Clean started: Project: project_2, Configuration: Debug Win32 ------
    1>Cleaning shared_1(Debug Lib|Win32).
    1>Cleaning shared_2(Debug Lib|Win32).
    1>Cleaning shared_3(Debug Lib|Win32).
    1>Cleaning project_2(Debug|Win32).
    2>------ Clean started: Project: project_1, Configuration: Debug Win32 ------
    2>Cleaning shared_2(Debug Lib|Win32).
    2>Cleaning shared_1(Debug Lib|Win32).
    2>Cleaning shared_3(Debug Lib|Win32).
    2>Cleaning project_1(Debug|Win32).
    ========== Clean: 2 succeeded, 0 failed, 0 skipped ==========

    8 lines starting with 'Cleaning' means reproduction is successful (there are only 5 if build is property synchronized, each clean referencing a unique project). If you get only 5, please see the second to last paragraph.
    Now the fun part - you can change and 'fix' the build output by doing the following (ultimately null) operation:
    4) Add one project as a reference of another (doesn't matter which).
    5) Clean Solution (the amount lines should change)
    6) Revert the reference by removing it. You're now in the same state you started.
    7) Clean Solution - you will still get the changed result with only 5 'Cleaning' lines. You reverted the state of the solution to original yet Clean Solution now produces different results than before. I'm not kidding.

    If you turn on build logging to at least detailed, you'll see that same targets are executed on shared_* projects two times, sometimes even in parallel. This should be enough of a lead to get this fixed, I hope, as it never happens when run via MSBuild command-line or older Visual Studios which you can use to compare the two behaviors on this sample.

    Note: Unfortunately, I also encountered some computer environments that will not reproduce it using this repro, possibly due to differences in VS2017+VS2015 configurations and/or plugins. This will manifest as having only one line in second Clean output (Cleaning project_2). Using a clean physical/virtual machine should reproduce it most reliably.

    I'm here if you need any more information about this as it is really stalling our adoption of VS2017 and its runtime, having us to stay with VS2015 until we can work around it.

    Thanks for any help,

    Ondrej Dobias
    Avast Software






    • Edited by Ondrej Dobias Wednesday, December 13, 2017 11:45 PM Added parallel build requirement clarification
    Wednesday, December 13, 2017 4:52 PM

Answers

  • Hi All,

    I help paste the simple solution for this issue here, so it could help other community members who get the same issues to get the solution directly:

    The temporary workaround is disable project caching in Tools - Options - Projects and Solutions - VC++ Project Settings. 

    Besides, Visual Studio team have fixed the problem in an upcoming release.

    If you want check more info about this issue, please check below link:

    https://developercommunity.visualstudio.com/content/problem/169941/vs2017-brings-new-parallel-build-synchronization-i.html


    MSDN Community Support Please remember to click &quot;Mark as Answer&quot; the responses that resolved your issue, and to click &quot;Unmark as Answer&quot; if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Ondrej Dobias Tuesday, January 2, 2018 4:58 PM
    Friday, December 22, 2017 9:39 AM

All replies

  • Hi Ondrej Dobias,

    Thanks for posting here.

    Would you please share me the test sample via onedrive? That would be more convenient for us to download it. I will check if I could reproduce this issue on my side.


    MSDN Community Support Please remember to click &quot;Mark as Answer&quot; the responses that resolved your issue, and to click &quot;Unmark as Answer&quot; if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Thursday, December 14, 2017 9:20 AM
  • Hi Leo, of course, here you go: https://1drv.ms/u/s!Aoth89aSIxgLgaETkrQvxC2VPNUmWg

    Please let me know if there are any issues with reproduction. I can also share you a virtual machine image that consistently reproduces it (but most physical machines do as well). The issue is quite important to us as it breaks our builds.

    Thank you.

    Thursday, December 14, 2017 10:52 AM
  • @Ondrej, thanks for your sharing, I could reproduce this issue according to your test sample. But I could not find any solution and workaround at this moment. I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience. Besides, since this issue can be reproduced steadily, I suggest you can report this issue to VS development team on Developer Community.

    More community members and MVP on that forum may further look at your issue and provide more suggestions.

    Thanks for your understanding!


    MSDN Community Support Please remember to click &quot;Mark as Answer&quot; the responses that resolved your issue, and to click &quot;Unmark as Answer&quot; if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    Friday, December 15, 2017 8:46 AM
  • Hi Leo,

    thanks for your time, reported the issue here as well: https://developercommunity.visualstudio.com/content/problem/169941/vs2017-brings-new-parallel-build-synchronization-i.html. So far, no response, I'll appreciate if you can let the devs know.

    Thank you,

    Ondrej

    Tuesday, December 19, 2017 10:15 AM
  • Hi All,

    I help paste the simple solution for this issue here, so it could help other community members who get the same issues to get the solution directly:

    The temporary workaround is disable project caching in Tools - Options - Projects and Solutions - VC++ Project Settings. 

    Besides, Visual Studio team have fixed the problem in an upcoming release.

    If you want check more info about this issue, please check below link:

    https://developercommunity.visualstudio.com/content/problem/169941/vs2017-brings-new-parallel-build-synchronization-i.html


    MSDN Community Support Please remember to click &quot;Mark as Answer&quot; the responses that resolved your issue, and to click &quot;Unmark as Answer&quot; if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Ondrej Dobias Tuesday, January 2, 2018 4:58 PM
    Friday, December 22, 2017 9:39 AM