locked
SYSTEM VARIABLES lower case ??? Ant issue RRS feed

  • Question

  • It appears that Team Build interprets Windows system variables as all lower case. We use ant to build across multiple platforms and each platform/OS utilizes its own system variables.  Our build on Windows is failing as the variables defined in our ANT scripts are not in sync with the system variables interpreted through Team Build 

     

    When I run msbuild from the command line with /v: diag the output shows system variables correct. 

     

    D:\temp>msbuild.exe build.proj /v: diag
    Microsoft (R) Build Engine Version 2.0.50727.832
    [Microsoft .NET Framework, Version 2.0.50727.832]
    Copyright (C) Microsoft Corporation 2005. All rights reserved.

    Build started 1/24/2008 8:38:37 AM.
    __________________________________________________
    Project "D:\temp\build.proj" (default targets):

    Initial Properties:
    ALLUSERSPROFILE                = C:\Documents and Settings\All Users
    ANT_HOME                       = D:\apache-ant-1.6.2
    APPDATA                        = C:\Documents and Settings\g004574\Application Data
    BUILD_HOME                     = D:\NovusBuild
    BUILD_LOG_HOME_DEV             = D:\build_batch\logs\Dev
    BUILD_LOG_HOME_QC              = d:\build_batch\logs\qc
    CLASSPATH                      = C:\Program Files\IBM\WebSphere MQ\Java\lib\prov
    iderutil.jar;C:\Program Files\IBM\WebSphere MQ\Java\lib\com.ibm.mqjms.jar;C:\Pro
    gram Files\IBM\WebSphere MQ\Java\lib\ldap.jar;C:\Program Files\IBM\WebSphere MQ\
    Java\lib\jta.jar;C:\Program Files\IBM\WebSphere MQ\Java\lib\jndi.jar;

    ..

    ..

    ..

     

    Running the same msbuild command through Team Build:

     

    Target CoreCompile:

    msbuild.exe d:\temp\build.proj /v: diag

    Microsoft (R) Build Engine Version 2.0.50727.832

    [Microsoft .NET Framework, Version 2.0.50727.832]

    Copyright (C) Microsoft Corporation 2005. All rights reserved.

    Build started 1/24/2008 10:19:14 AM.

    __________________________________________________

    Project "d:\temp\build.proj" (default targets):

    Initial Properties:

    allusersprofile = C:\Documents and Settings\All Users

    ant_home = D:\apache-ant-1.6.2

    appdata = C:\Documents and Settings\g004574\Application Data

    build_home = D:\NovusBuild

    build_log_home_dev = D:\build_batch\logs\Dev

    build_log_home_qc = d:\build_batch\logs\qc

    classpath = C:\Program Files\IBM\WebSphere MQ\Java\lib\providerutil.jar;C:\Program Files\IBM\WebSphere MQ\Java\lib\com.ibm.mqjms.jar

    ..

    ..

     

    Any ideas on how to get Team Build to interpret the system variables accurately?

     

    Thursday, January 24, 2008 6:20 PM

Answers

  • Right then.  After much digging around I think I have this figured out.

     

    It is true to say that Windows is case-insensitive to environment variable names, however it is case insensitive in the same way that the file system is case insensitive.  The case of the environment variable name is preserved as it is initially created.  This can be demonstrated as follows.  At a command shell type:

     

    set Test_VAR_1=foobar

     

    This will create the environment variable in the current cmd.exe process.  Then type:-

     

    set test_var_1

     

    You will be returned the key and value stored (even thought you are using a different case to query for it), i.e:-

     

    Test_VAR_1=foobar

     

    Then, change the value by typing

     

    set test_var_1=foobar2

     

    Then

     

    set test_var_1

     

    To display the value.  This will give you

     

    Test_VAR_1=foobar2

     

    I.e. the case of the initial set has been preserved.

     

    The problem is due to the fact that when the Build Agent service runs the MSBuild process, it does so using the standard mechanism for doing this in a bit of .NET code which is to use the System.Diagnostics.Process class.  The build agent passes through the environment variables into that process (and optionally adds some of its own) using the standard way of doing this which is to use the System.Diagnostics.ProcessInfo class.

     

    This is where the problem comes in.  System.Diagnostics.ProcessStartInfo has the EnvironmentVariables property to add or remove environment variables, but the problem is uses a StringDictionary to store the environment variables.  Famously, StringDictionary stores it's keys in lowercase which catches many people out (including myself from time to time).

     

    Therefore, if you start a process in .NET using the ProcessStartInfo class and you do anything to it's environment variables then it will lowercase all the keys.  Ouch.

     

    I've logged a bug about this behaviour in the ProcessStartInfo class as it differs from the underlying behaviour of the operating system.  For more information see the following connect bug:-

     

    https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=326163

     

    However.  As this bug is in System.Diagnostics this isn't going to help you much in the short term.

     

    I think I have figured out a bit of a hack to get around this.  In .NET 2.0 Microsoft introduced the ability to get and set environment variables using the System.Environment class.  This crucially has the ability to get environment variables out the user and machine settings as well as from the current running process.

     

    Thefore, you can wrap the call to Ant in a custom MSBuild task (I have an example of such a task on my blog at http://www.woodwardweb.com/teamprise/000406.html).  During this call you can "fix" the current processes environment variables by comparing the environment variables in the current process with those defined at the user or machine level and then if the key of that variable differs by case alone then you reset the case.  Below is an example of how to do that, taken from the source code of my Ant wrapper task

     

    Code Snippet

    private void fixupEnvironmentVariables()

    {

      StringDictionary envVars = new StringDictionary();

      foreach (string key in Environment.GetEnvironmentVariables(EnvironmentVariableTarget.Machine).Keys)

      {

        envVars.Add(key, key);

      }

      foreach (string key in Environment.GetEnvironmentVariables(EnvironmentVariableTarget.User).Keys)

      {

        envVars.Remove(key);

        envVars.Add(key, key);

      }

      foreach (string key in Environment.GetEnvironmentVariables(EnvironmentVariableTarget.Process).Keys)

      {

      if (envVars.ContainsKey(key) && !envVars[key].Equals(key, StringComparison.Ordinal))

      {

        // We have an env var in the process that is in a different case at the user or machine level.

        // Add the correct case version.

        // Simply resetting the name will have no effect because Windows will preserve the case in which

        // it was initially created. We must remove it and then re-add it.

        // To remove a environment variable in a process, you must set the value of it to empty string.

        string value = Environment.GetEnvironmentVariable(key, EnvironmentVariableTarget.Process);

        Environment.SetEnvironmentVariable(envVars[key],

          "",

          EnvironmentVariableTarget.Process);

        // Now set the environment variable using the correct case.

        Environment.SetEnvironmentVariable(envVars[key],

          value,

          EnvironmentVariableTarget.Process);

        }

      }

    }

     

     

    This should have not affect to the rest of the process, because MSBuild and Win32 code treat environment variables as case insensitive (i.e. Environment.GetEnvironmentVariable("foobar") and Environment.GetEnvironmentVariable("FOOBAR") both return a result).

     

    As I say, I think I have this fixed in an TeamBuild wrapper task for MSBuild that you can download from my blog.  I'd love to get some feedback if this works for you.

     

    Good luck,

     

    Martin.

    Friday, February 1, 2008 11:51 AM
    Moderator

All replies

  • How is msbuild being invoked? Is this using an Exec task to invoke the msbuild command line? Also, how is the case change affecting your build process? Environment variables are not case sensitive in windows, along with many other artifacts that exist within the operating system.

     

    Patrick

     

    Friday, January 25, 2008 12:16 AM
  • In the Team Build proj file  we call <Exec Command='msbuild.exe d:\temp\build.proj /v: diag'/> in CoreCompile.

     

    Our Build compiles java, c, c++ on Mutiple OS Build Machines (including windows) utilizing the same build scripts.  Linux and Unix and ANT are all case sensative. 

     

    I have a couple ideas on ways to get around this issue but I would like clarification that this Team Build behavior is exepcted:

     

    running msbuild from comand line - your system variables are interpreted as defined

    running msbuild from comand line(Exec task) through Team Build - your system variables are interpreted all lower case

    Friday, January 25, 2008 2:33 PM
  • Two more notes on this problem.

     

    1) same lower case issue occurs with Team System 2008.

     

    2)  The variables (see below) set by Team Build at Build time appear to be in case sensitive format. 

     

    Entering the CoreCompile Target
    Microsoft (R) Build Engine Version 3.5.21022.8
    [Microsoft .NET Framework, Version 2.0.50727.1433]
    Copyright (C) Microsoft Corporation 2007. All rights reserved.
      
    Build started 1/28/2008 11:59:04 AM.

    Project "d:\temp\build.proj" (default targets):

    Initial Properties:

    allusersprofile = C:\Documents and Settings\All Users

    ant_home = D:\apache-ant-1.6.2

    appdata = C:\Documents and Settings\g004574\Application Data

    build_home = D:\NovusBuild

    build_log_home_dev = D:\build_batch\logs\Dev

    build_log_home_qc = d:\build_batch\logs\qc

    classpath = C:\Program Files\IBM\WebSphere MQ\Java\lib\providerutil.jar;C:\Program Files\IBM\WebSphere MQ\Java\lib\com.ibm.mqjms.jar;C:\Program Files\IBM\WebSphere MQ\Java\lib\ldap.jar

    clusterlog = C:\WINDOWS\Cluster\cluster.log

    com_ibm_mq_dir = C:\Program Files\IBM\WebSphere MQ\Java\lib

    com_ibm_mq_jar = C:\Program Files\IBM\WebSphere MQ\Java\lib\com.ibm.mq.jar

    commonprogramfiles = C:\Program Files\Common Files

    computername = D1-S707-S2K3A

    comspec = C:\WINDOWS\system32\cmd.exe

    cvsroot = Stick out tongueserver:cmadmin@mora.westlan.com:/pkgs/cvs_master/easel

    db2instance = DB2

    db2tempdir = C:\PROGRA~1\IBM\SQLLIB\

    fp_no_host_check = NO

    homedrive = I:

    homepath = \

    homeshare = \\homedir\g004574

    include = C:\Program Files\Microsoft Visual Studio\VC98\atl\include;C:\Program Files\Microsoft Visual Studio\VC98\mfc\include;C:\Program Files\Microsoft Visual Studio\VC98\include

    java_cc_home = D:\NovusBuild\javacc-4.0\bin\lib

    java_classpath = C:\Program Files\IBM\sqllib\java\db2java.zip;D:\NovusBuild\links\db2jcc.jar;D:\NovusBuild\links\ojdbc14.jar

    java_home = D:\j2sdk1.4.2_10\

    lib = C:\Program Files\Microsoft Visual Studio\VC98\mfc\lib;C:\Program Files\Microsoft Visual Studio\VC98\lib

    logonserver = \\EG-TLRDC-A05

    mq_java_data_path = C:\Program Files\IBM\WebSphere MQ

    mq_java_install_path = C:\Program Files\IBM\WebSphere MQ\Java

    ms_visual_studio_home = c:\Program Files\Microsoft Visual Studio

    MSBuildBinPath = C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727

    MSBuildExtensionsPath = C:\Program Files\MSBuild

    MSBuildProjectDefaultTargets = dist

    MSBuildProjectDirectory = d:\temp

    MSBuildProjectExtension = .proj

    MSBuildProjectFile = build.proj

    MSBuildProjectFullPath = d:\temp\build.proj

    MSBuildProjectName = build

    msdevdir = C:\Program Files\Microsoft Visual Studio\Common\MSDev98

    number_of_processors = 2

    os = Windows_NT

    path = c:\Perl1\bin;C:\Perl\bin\;C:\Program Files\IBM\WebSphere MQ\Java\lib;

    Monday, January 28, 2008 6:54 PM
  • I'm able to reproduce this as well on both Team Build 2005 and Team Build 2008.  Full disclosure: SJK2555 and I work for the same company.

     

    All you have to do to reproduce the error is copy this into your TFSBuild.proj file.  Any environment variables that were set (e.g. ANT_HOME=...) come out as lowercase (e.g. ant_home).  That's bad as Ant is case sensitive.

     

     <Target Name="CoreCompile">

     

        <!-- This command will spit out the system enivronments so you can see they come out as lower case. -->
        <Exec WorkingDirectory="." Command="cmd.exe /c SET"/>
       
      </Target>

     

     

    Wednesday, January 30, 2008 8:33 PM
  • Right then.  After much digging around I think I have this figured out.

     

    It is true to say that Windows is case-insensitive to environment variable names, however it is case insensitive in the same way that the file system is case insensitive.  The case of the environment variable name is preserved as it is initially created.  This can be demonstrated as follows.  At a command shell type:

     

    set Test_VAR_1=foobar

     

    This will create the environment variable in the current cmd.exe process.  Then type:-

     

    set test_var_1

     

    You will be returned the key and value stored (even thought you are using a different case to query for it), i.e:-

     

    Test_VAR_1=foobar

     

    Then, change the value by typing

     

    set test_var_1=foobar2

     

    Then

     

    set test_var_1

     

    To display the value.  This will give you

     

    Test_VAR_1=foobar2

     

    I.e. the case of the initial set has been preserved.

     

    The problem is due to the fact that when the Build Agent service runs the MSBuild process, it does so using the standard mechanism for doing this in a bit of .NET code which is to use the System.Diagnostics.Process class.  The build agent passes through the environment variables into that process (and optionally adds some of its own) using the standard way of doing this which is to use the System.Diagnostics.ProcessInfo class.

     

    This is where the problem comes in.  System.Diagnostics.ProcessStartInfo has the EnvironmentVariables property to add or remove environment variables, but the problem is uses a StringDictionary to store the environment variables.  Famously, StringDictionary stores it's keys in lowercase which catches many people out (including myself from time to time).

     

    Therefore, if you start a process in .NET using the ProcessStartInfo class and you do anything to it's environment variables then it will lowercase all the keys.  Ouch.

     

    I've logged a bug about this behaviour in the ProcessStartInfo class as it differs from the underlying behaviour of the operating system.  For more information see the following connect bug:-

     

    https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=326163

     

    However.  As this bug is in System.Diagnostics this isn't going to help you much in the short term.

     

    I think I have figured out a bit of a hack to get around this.  In .NET 2.0 Microsoft introduced the ability to get and set environment variables using the System.Environment class.  This crucially has the ability to get environment variables out the user and machine settings as well as from the current running process.

     

    Thefore, you can wrap the call to Ant in a custom MSBuild task (I have an example of such a task on my blog at http://www.woodwardweb.com/teamprise/000406.html).  During this call you can "fix" the current processes environment variables by comparing the environment variables in the current process with those defined at the user or machine level and then if the key of that variable differs by case alone then you reset the case.  Below is an example of how to do that, taken from the source code of my Ant wrapper task

     

    Code Snippet

    private void fixupEnvironmentVariables()

    {

      StringDictionary envVars = new StringDictionary();

      foreach (string key in Environment.GetEnvironmentVariables(EnvironmentVariableTarget.Machine).Keys)

      {

        envVars.Add(key, key);

      }

      foreach (string key in Environment.GetEnvironmentVariables(EnvironmentVariableTarget.User).Keys)

      {

        envVars.Remove(key);

        envVars.Add(key, key);

      }

      foreach (string key in Environment.GetEnvironmentVariables(EnvironmentVariableTarget.Process).Keys)

      {

      if (envVars.ContainsKey(key) && !envVars[key].Equals(key, StringComparison.Ordinal))

      {

        // We have an env var in the process that is in a different case at the user or machine level.

        // Add the correct case version.

        // Simply resetting the name will have no effect because Windows will preserve the case in which

        // it was initially created. We must remove it and then re-add it.

        // To remove a environment variable in a process, you must set the value of it to empty string.

        string value = Environment.GetEnvironmentVariable(key, EnvironmentVariableTarget.Process);

        Environment.SetEnvironmentVariable(envVars[key],

          "",

          EnvironmentVariableTarget.Process);

        // Now set the environment variable using the correct case.

        Environment.SetEnvironmentVariable(envVars[key],

          value,

          EnvironmentVariableTarget.Process);

        }

      }

    }

     

     

    This should have not affect to the rest of the process, because MSBuild and Win32 code treat environment variables as case insensitive (i.e. Environment.GetEnvironmentVariable("foobar") and Environment.GetEnvironmentVariable("FOOBAR") both return a result).

     

    As I say, I think I have this fixed in an TeamBuild wrapper task for MSBuild that you can download from my blog.  I'd love to get some feedback if this works for you.

     

    Good luck,

     

    Martin.

    Friday, February 1, 2008 11:51 AM
    Moderator
  • Like most of your stuff, works great for us!  Just another reason I continue to promote Teamprise.

    Friday, February 1, 2008 4:24 PM
  • Hello,

    I would like to force user in active directory to login with only lower case. do you have any recommendation ?

    Thanks

    Wednesday, December 5, 2012 6:53 PM