none
Java SDK authentication - native_auth library doesn't load

    Question

  • I have written some code using the TFS Java SDK to talk to TFS version control. The code works in isolation in my automated tests, but when I deploy it into my application it fails to authenticate. I would appreciate any help you can give me in debugging the problem.

    Both cases are running under Java 1.6 on Linux.

    I have turned on debug logging in both cases, which allows me to see the details of native library loading and the HTTP conversation.

    The successful case looks like this:

      request: POST /tfs/DefaultCollection/Services/v1.0/Registration.asmx
      response: 401, specifying NTLM scheme
    
      Client loads native_auth library, selecting JavaNTLM implementation.
    
      request: repost, starting NTLM auth handshake
      reponse: 401, with NTLM challenge
    
      request: repost, with NTLM reponse
      response: 200
    

    The failing case looks like this:

      request: POST /tfs/DefaultCollection/Services/v1.0/Registration.asmx
      response: 401, specifying NTLM scheme
    
    No log messages about library loading.

    request: repost, starting NTLM auth handshake (a short message anyway, I haven't decoded it) reponse: 401, with NTLM challenge request: repost, unknown NTLM message (another, different, short message) response: 401, specifying NTLM, no message

     

    Do you have any ideas what the problem might be?

    Thanks

    -Ben



    Thursday, January 26, 2012 1:09 PM

Answers

  • I've written a wrapper around your SDK which runs in its own classloader. That solves the problem and it's now working fine.

    This is a practical solution for me at the moment because I'm only using a very small subset of your API. But for a more complex use of the library it would be extremely painful to have marshal data across the interface between classloaders.

    I see that both HttpClient 3 and 4 support a plugin mechanism for authentication. Is there any reason why your NTLMv2 client implementation can't be rewritten as a plugin? In fact, I see that HttpClient 4 supports NTLMv2 and Kerberos natively.

    -Ben

     

    Tuesday, January 31, 2012 7:22 PM

All replies

  • The log messages are a bit misleading in this case.  JavaNTLM doesn't require the native_auth library (NTLM is implemented in pure Java), but we always try to load that library so other mechanisms can be used if the system libraries are present (Kerberos, for example).  NTLM authentication is always available, regardless of whether libnative_auth.so can be loaded.

    Are you using the same credentials for your tests vs. deployment?  The final 401 in the failed case could point to a misspelled password, or perhaps TFS permissions don't allow this account access to the project collection.  You might check the server's event log for details.

    Generally, you really want the native libraries to load for your SDK applications.  Certain version control features won't work correctly if they don't.  Make sure you're defining the "com.microsoft.tfs.jni.native.base-directory" system property to point where the redistributable native libraries are located.  The readme.txt files that come with the SDK samples show an example of this.

    Thursday, January 26, 2012 2:52 PM
    Owner
  • Thanks for replying so quickly, Shaw.

    I am using the same credentials (and the same URL and project) in both cases. The native libraries are loading fine as far as I can see from other logs.

    My understanding of NTLM is that the second message that the client sends to the server (the response to the challenge) should be considerably longer than the earlier messages in the handshake. I see this in the successful case, but in the failing case the response is far shorter

    The logs on the server show an error "TF53008: The authentication type  is not supported."

    -Ben

    Friday, January 27, 2012 12:24 PM
  • Thanks for the details, the TF53008 error is good info.

    Which Java runtime are you using on the deployment platform?  Linux distributions often install GNU GCJ by default, which historically did not support the cryptography libraries that TEE needs, and is not supported.  Oracle's runtime is recommended and supported.

    Friday, January 27, 2012 1:22 PM
    Owner
  • We're using Oracle Java in both places.

    It might be worth pointing out that, while I'm developing this stuff, the deployment platform is actually the same phyisical box as the development platform. The only difference is that there are a whole bunch more libraries loaded and maybe some different environment variables.

    -Ben

     

    Friday, January 27, 2012 2:07 PM
  • Hi Ben-

    We ship a modified version of Apache Commons HTTPClient 3, which includes NTLMv2 and Kerberos (SPNEGO) authentication mechanisms.  (The stock version of HTTPClient includes only NTLMv1 compatibility, which is unsuitable for authentication.)

    However, we still include the package as "org.apache.commons.httpclient" so that SDK users can swap ours out their own HTTPClient mechanism for whatever reason.

    Is it possible that your deployment platform also includes org.apache.commons.httpclient?

    Friday, January 27, 2012 3:04 PM
    Moderator
  • Hello Edward

    We do indeed already have HttpClient (3.1) in the environment. That seems very likely to be the problem. I shall try to prove it by hacking the environment to remove that other HttpClient jar.

    If that confirms it, I'm not sure how to proceed.

    -Ben

     

    Friday, January 27, 2012 3:25 PM
  • You might try putting the TFS SDK JARs on the classpath before the other HttpClient contributor.  I don't know how your application is deployed, but some application containers let you use different classloaders for each component, to keep their dependencies isolated.  That might also work, but I don't have any specific advice in that area.
    Friday, January 27, 2012 4:19 PM
    Owner
  • I've tested using the TFS SDK to provide the Apache Commons HTTPClient to the whole system. That works fine, confirming that this is indeed the problem.

    I am not terribly happy with the idea of using the TFS SDK as a source of the HTTPClient to the whole system, particularly as you've modified it. The stuff I'm working on is a small corner, integrating TFS with a large, existing system. Apart from anything else, it would make us reliant on you to propagate any security patches to that library.

    I will investigate the possibility of running the TFS integration inside an isolated classloader.

    Have you had any luck contributing your changes back to the core Apache project?

    -Ben

     

    Friday, January 27, 2012 5:49 PM
  • Hi Ben-

    It's good to know that we've found the cause.  I wouldn't recommend replacing your existing implementation with our HTTPClient.  While it's undergone thorough testing in our product, we will not support it as a general replacement for Apache Commons HTTPClient and do not test it as such.

    I believe that a classloading strategy will suffice -- please do let us know the results of your testing.

    As for contributing back:  while I'd love to see more packages adopt a proper NTLM2 implementation, we've decided not to contribute our code back to HttpClient for three major reasons:  first, we've removed NTLMv1 as an authentication option in HTTPClient in order to meet our security standards and while we absolutely feel that this is an appropriate decision for our product, we do not feel that we should make this decision for all library users in general.  Second, HttpClient 3 is being replaced by HttpComponents HttpClient 4 and while we could target that for patching, it's simply not a priority for us to move to HttpClient 4 at the moment.  Finally, our Kerberos implementation relies heavily on native method calls using JNI, as does our NTLM implementation on Windows.  We do not feel that making HttpClient depend on native libraries for authentication is appropriate for a general-purpose library.

    In any case, we're looking at this more for our next release to see how we can minimize your pain here.  We simply hadn't considered some of the inventive ways people would want to use our Java SDK.

    -ed

    Friday, January 27, 2012 6:23 PM
    Moderator
  • I've written a wrapper around your SDK which runs in its own classloader. That solves the problem and it's now working fine.

    This is a practical solution for me at the moment because I'm only using a very small subset of your API. But for a more complex use of the library it would be extremely painful to have marshal data across the interface between classloaders.

    I see that both HttpClient 3 and 4 support a plugin mechanism for authentication. Is there any reason why your NTLMv2 client implementation can't be rewritten as a plugin? In fact, I see that HttpClient 4 supports NTLMv2 and Kerberos natively.

    -Ben

     

    Tuesday, January 31, 2012 7:22 PM
  • We'll consider releasing our authentication code in the future.  It currently contains some other changes to support special authentication scenarios required by TFS (in addition to the NTLM/Negotiate providers).  We also took a different approach for Kerberos integration, using the OS's GSS libraries directly instead of Java GSS, so the user doesn't have to configure Java GSS during installation.
    Thursday, February 02, 2012 7:25 PM
    Owner
  • I've written a wrapper around your SDK which runs in its own classloader. That solves the problem and it's now working fine.

    This is a practical solution for me at the moment because I'm only using a very small subset of your API. But for a more complex use of the library it would be extremely painful to have marshal data across the interface between classloaders.

    I think I'm seeing the same problem as you; my code runs fine in isolated tests, but authentication to TFS fails when deployed inside a larger application (which includes Apache Commons HttpClient on its classpath).

    Can you please share some information on how you've implemented the wrapper around the SDK? I also only need a small subset of the API; retrieve a list of team project collections and projects, and submit work items.

    Edit: Never mind, after delving into Classloader documentation I already crafted my own solution. My code was already abstracted by an interface, so I now simply instantiate my code using a custom classloader like so:
    return (<iface>)new ParentLastURLClassLoader(<impl-jar>).loadClass(<impl>).newInstance();

    The code for ParentLastURLClassLoader is documented here: http://stackoverflow.com/questions/5445511/how-do-i-create-a-parent-last-child-first-classloader-in-java-or-how-to-overr. <iface> and all of its dependencies are available in the parent classloader, <impl> is available in <impl-jar> and of course implements <iface>, and <impl-jar> contains the following entry in its Manifest.mf file to load the TFS API:  
    Class-Path: com.microsoft.tfs.sdk-10.1.0.jar

    Thank you,
    Ruud



    • Edited by Ruud1234 Friday, April 06, 2012 4:17 PM
    Friday, April 06, 2012 2:42 PM