locked
"Lost in (32/64 bit) Translation:" Passing Pointer from 64bit RPC server to 32bit RPC client and back. RRS feed

  • Question

  • I need some assistance with the changes I need to make here with my 64 bit RPC server and 32 bit RPC client system.

    Background:

    I have an extensive 32 bit RPC client/server platform which I have been porting to 64 bit.  I have been focused on creating a 64 bit server first, keeping with the 32 bit clients for now for backward compatibility.   I have been remarkably surprised in how well the 64 bit server and 32 bit clients has been working together with relatively little change.

    But I just came across a situation where the 64 bit RPC server generated thread session context is created and returned to calling a 32 bit RPC client, saved in TLS for the client thread, is now getting lost in the 32/64 bit translations when the context handle is passed back to the server for a session "cloning" operation.

    Let me show it in unchanged 32bit code to see what needs to be changed. First the IDL for the functions in questions:

    // Every client must call this to establish a RPC server context
    
      error_status_t WildcatServerCreateContext([out, ref] TWildcatContextHandle *wch,
                                                [in, string] const char *username,
                                                [in, string] const char *computer,
                                                [in, string] const char *program);
    
    // Used under special 32 bit shelled exe apps, i.e. 3rd party games, to allow
    // the compliant app to create a new session context and clone the current
    // session context data.
    
      error_status_t WildcatServerCreateContextFromHandle([out, ref] TWildcatContextHandle *wch,
                                                          [in, string] const char *username,
                                                          [in, string] const char *computer,
                                                          [in, string] const char *program,
                                                          [in] DWORD context);
    

    The server-side stub functions are:

    /* [fault_status][comm_status] */ error_status_t WcsWildcatServerCreateContext(
        /* [ref][out] */ TWildcatContextHandle __RPC_FAR *wch,
        /* [string][in] */ const char __RPC_FAR *username,
        /* [string][in] */ const char __RPC_FAR *computer,
        /* [string][in] */ const char __RPC_FAR *program)
    {
        TClientContext *cc = new TClientContext(username, computer, program);
        *wch = cc;
        return 0;
    }
    
    /* [fault_status][comm_status] */ error_status_t WcsWildcatServerCreateContextFromHandle(
        /* [ref][out] */ TWildcatContextHandle __RPC_FAR *wch,
        /* [string][in] */ const char __RPC_FAR *username,
        /* [string][in] */ const char __RPC_FAR *computer,
        /* [string][in] */ const char __RPC_FAR *program,
    	 /* [in] */ unsigned long context)
    {
        *wch = NULL;
        TClientContext *cc = (TClientContext *)context;   <<<< NOT VALID POINTER UNDER 64 BIT
    
    #ifndef _WIN64
        // 454.8, MSDN says IsBadWritePtr() is obsolete, gpf under 64bit, ignore for now
        if (IsBadWritePtr(cc, sizeof(TClientContext))) {
            return ERROR_INVALID_PARAMETER;
        }
    #endif
    
        if (typeid(*cc) != typeid(TClientContext)) {
            return ERROR_INVALID_PARAMETER;
        }
    	 *wch = cc->Clone(username, computer, program);
        return 0;
    }

    The clone function creates a new TClientContext instance and copies some session data:

    TClientContext *TClientContext::Clone(const char *username, const char *computer, const char *program)
    {
        TClientContext *cc = new TClientContext(username, computer, program);
        ... some parent session data is copied to new cc instanced ....
        return cc;
    }

    The client side functions with the TLS logic:

    BOOL APIENTRY WildcatServerCreateContext()
    {
        TWildcatContextHandle wch = TlsGetValue(tlsContext);
        if (!wch) wch = GetGrantedContext();
        if (wch) {
           SetLastError(WC_CONTEXT_ALREADY_INITIALIZED);
           return FALSE;
        }
        TLocalComputerInfo lci = {0};
        GetLocalComputerInfo(lci);
        DWORD status = WcsWildcatServerCreateContext(&wch,
                                                     lci.UserName,
                                                     lci.CompName,
                                                     lci.ProgName);
        TlsSetValue(tlsContext, wch);
        if (status) {
          SetLastError(status);
          return FALSE;
        }
        return TRUE;
    }
    
    BOOL APIENTRY WildcatServerCreateContextFromHandle(DWORD context)
    {
        TWildcatContextHandle wch = TlsGetValue(tlsContext);
        if (!wch) wch = GetGrantedContext();
        if (wch) {
            SetLastError(WC_CONTEXT_ALREADY_INITIALIZED);
            return FALSE;
        }
        TLocalComputerInfo lci = {0};
        GetLocalComputerInfo(lci);
        DWORD status = WcsWildcatServerCreateContextFromHandle(&wch,
                                                   lci.UserName,
                                                   lci.CompName,
                                                   lci.ProgName,
                                                   context);
        TlsSetValue(tlsContext, wch);
        if (status) {
            SetLastError(status);
            return FALSE;
        }
        return TRUE;
    }
    

    Under pure 32bit client/server, it has been a long time engineered and solid system since 1996.  The cloning logic is basically done to allow 3rd party 32 bit "shelled" programs (i.e. legacy console games, etc) to reestablish a parent session information.  This is done by creating a batch file with an environment string to pass the context "handle."

       FILE *fv = fopen("PreDoor.Bat", "wt");
       fprintf(fv, "@echo off\n");
       ...
       fprintf(fv, "set WildcatServerContext=%d\n", GetWildcatServerContextHandle());
       ...
       fclose(fv);


    The client GetWildcatServerContextHandle() API is:

    DWORD APIENTRY GetWildcatServerContextHandle()
    {
        InitContext(0);
        DWORD handle;
        DWORD status = WcsGetWildcatServerContextHandle(wch, &handle);
        if (status) {
            SetLastError(status);
            return 0;
        }
        return handle;
    }


    and the server side function:

    /* [fault_status][comm_status] */ error_status_t WcsGetWildcatServerContextHandle(
        /* [in] */ TWildcatContextHandle wch,
        /* [ref][out] */ unsigned long __RPC_FAR *handle)
    {
        *handle = DWORD(wch);
        return 0;
    }


    When the app is shelled, the compliant app must do the following:

       DWORD context =  GetEnvInt("wildcatservercontext");
       if (context) WildcatServerCreateContextFromHandle(context);


    So overall, under pure 32 bit client/server, it works as expected, but when the server is running as 64 bit, the context is not passed back correctly to the WildcatServerCreateFromHandle(context) call:

        TClientContext *cc = (TClientContext *)context;   <<<< NOT VALID POINTER UNDER 64 BIT

    Debugger shows context as invalid.

    So before I begin any extensive changes to the RPC interfacing, between the IDL, the client API and server functions, I would like to get a "solid handle" of what I need to consider here.

    Can someone assist with what changes I need to make or consider?

    What is vitally importantly is that the #1 goal is to maintain 32 bit Client API compatibility.  I don't want existing client applications to be required to recompile, i.e. forcing a change to a new RPC binding handle.   If that is not possible, I will need to put aside the 64 bit server project until this is resolved.

    Thanks in advance.

      

    Hector Santos, CTO Santronics Software, Inc. http://www.santronics.com


    Thursday, May 9, 2019 3:17 PM

All replies

  • /* [fault_status][comm_status] */ error_status_t WcsWildcatServerCreateContextFromHandle(
        /* [ref][out] */ TWildcatContextHandle __RPC_FAR *wch,
        /* [string][in] */ const char __RPC_FAR *username,
        /* [string][in] */ const char __RPC_FAR *computer,
        /* [string][in] */ const char __RPC_FAR *program,
    	 /* [in] */ unsigned long context)
    {
        *wch = NULL;
        TClientContext *cc = (TClientContext *)context;   <<<< NOT VALID POINTER UNDER 64 BIT
    
    The unsigned long type for context will cause pointer truncation in 64 bit.  Can you use ULONG_PTR instead?
    Thursday, May 9, 2019 7:00 PM
  • Well,  These links helps.  The MSDN MIDL 64 bit migration pages helps a lot.  It is 2006 information and I wonder if it is still up to par.

    https://www.viva64.com/en/k/0005/
    https://docs.microsoft.com/en-us/previous-versions/ms810720(v=msdn.10)
    In the FAQ section, it provides the basic things to consider.  


    Hector Santos, CTO Santronics Software, Inc. http://www.santronics.com

    Thursday, May 9, 2019 7:53 PM
  • You're right.  I would need to go 64 bit here.  I will probably create extended API functions using 64 bit pointers and go from there:

    Current 32 bit API Interface:

    BOOL  APIENTRY WildcatServerCreateContextFromHandle(DWORD context);
    DWORD APIENTRY GetWildcatServerContextHandle();

    Extended functions 32/64 compatible:

    BOOL  APIENTRY WildcatServerCreateContextFromHandleEx(TWildcatContextHandle context);
    TWildcatContextHandle APIENTRY GetWildcatServerContextHandleEx();

    plus a new one:

    BOOL  APIENTRY WildcatServerCloneContext(TWildcatContextHandle *ctx = NULL);

    The default will clone the current session context.

    The extended functions can be used to create a compatibility module.


    Hector Santos, CTO Santronics Software, Inc. http://www.santronics.com


    Friday, May 10, 2019 2:41 AM
  • Can you illustrate how I would use the ULONG_PTR here on both ends?

    I've been having trouble using the void * (TWildcatContextHandle) for the extended functions which is causing a fault in the RPC server (32 or 64 bit).  I believe I have a type casting issue.  But I don't see it as we are just assigning values of the same type.  MIDL 64 bit migration says using a void * for the context handle is appropriate and in fact, I believe it say is a  requirement for the context handle.

    In the original 32 bit GetWildcatServerContextHandle(), the server stub is:

    /* [fault_status][comm_status] */ error_status_t WcsGetWildcatServerContextHandle(
        /* [in] */ TWildcatContextHandle wch,
    	/* [ref][out] */ unsigned long __RPC_FAR *handle)
    {
    	*handle = DWORD(wch); 
        return 0;
    }


    For the extended,  what I did was basically do this:

    /* [fault_status][comm_status] */ error_status_t WcsGetWildcatServerContextHandleEx(
        /* [in] */ TWildcatContextHandle wch,
    	/* [ref][out] */ TWildcatContextHandle __RPC_FAR *handle)
    {
    	*handle = wch; 
        return 0;
    }


    But this somehow is not correct and I believe is trashing the original wch.   If I avoid the server call, and just return the input for the output on the client side, then it appears to be ok:

    TWildcatContextHandle APIENTRY GetWildcatServerContextHandleEx()
    {
    	InitContext(0);
    #if 1
    	return wch;
    #else
    	TWildcatContextHandle handle = NULL;
    	DWORD status = WcsGetWildcatServerContextHandleEx(wch, &handle);
    	if (status) {
    		SetLastError(status);
    		return 0;
    	}
    	return handle;
    #endif
    }
    


    Can anyone explain this?  Its not a 64 bit compile issue.


    Hector Santos, CTO Santronics Software, Inc. http://www.santronics.com

    Monday, May 13, 2019 3:23 PM
  • Hi,

    /* [fault_status][comm_status] */ error_status_t WcsGetWildcatServerContextHandleEx(
        /* [in] */ TWildcatContextHandle wch,
    	/* [ref][out] */ TWildcatContextHandle __RPC_FAR *handle)
    {
    	*handle = wch; 
        return 0;
    }

    The wch in this function is just a formal parameter, and it doesn't change the value of an external variable unless the parameter type is "TWildcatContextHandle &".

    Best regards,

    Drake


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, May 14, 2019 9:50 AM
  • Thank you for your input.

    The calling client functions for the server functions are all passing wch for the client's context handle that was first generated at the server, returned to the client and kept in the client proxy TLS manager.   Once the context handle is created for the thread, subsequent API access must pass the wch.  This is the generic outline:

    BOOL APIENTRY RPC_FUNC_XXX(....)
    {
        // InitContext(X) is a macro to get/check 
        // the current thread TWildcatContextHandle wch.
        // If context already exist. return X
    
        InitContext(FALSE);   
        DWORD status = WcsRPC_FUNC_XXX(wch, ....);
        if (status) {
            SetLastError(status);
            return FALSE;
        }
        return TRUE;
    }
    


    But there is something more fundamental here that I am missing that may be related to MIDL and RPC when it comes to a context handle.  I could be missing a type cast,  an attribute or something.   But for sure, this is not 64 bit related as I initially thought.

    The function used to return the context handle from the server is:

    DWORD APIENTRY GetWildcatServerContextHandle()
    {
        InitContext(FALSE);   
    DWORD handle = 0; DWORD status = WcsGetWildcatServerContextHandle(wch, &handle); if (status) { SetLastError(status); return FALSE; } return handle; }


    This whole issue started because I thought returning DWORD would need to be changed to an opaque pointer for a 64 bit server.   So I added an extended function which returns TWildcatServerHandle:

    TWildcatServerHandle APIENTRY GetWildcatServerContextHandleEx()
    {
        InitContext(FALSE);   
        TWildcatServerHandle handle = 0;
        DWORD status = WcsGetWildcatServerContextHandleEx(wch, &handle);
        if (status) {
            SetLastError(status);
            return FALSE;
        }
        return handle;
    }
    

    What happened is that I didn't get much further, because this function now is somehow corrupting/changing the initial context handle somehow.  The server side extended function is similar to the original but outputs TWildcatContextHandle instead of DWORD.

    /* [fault_status][comm_status] */ error_status_t WcsGetWildcatServerContextHandleEx(
        /* [in] */ TWildcatContextHandle wch,
        /* [ref][out] */TWildcatContextHandle __RPC_FAR *handle)
    {
        *handle = TWildcatContextHandle(wch);
        return 0;
    }



    So basically, these two functions SHOULD return the same value:

    DWORD dwwch = GetWildcatServerContextHandle();
    TWildcatContextHandle wch = GetWildcatServerContextHandleEx();

    The value of wch and dwwch SHOULD be the same.   Is that correct?  They are not, not even when both ends are 32 bit, and that is where I am stuck at.  Once I use the extended function,  I am getting an a context delete/rundown problem.

    The problem is exasperated because I can't get the Microsoft PDB symbols loaded under the debugger so I can trace down the marshaling. :(

    Thanks for tying to help.


    Hector Santos, CTO Santronics Software, Inc. http://www.santronics.com

    Tuesday, May 14, 2019 2:05 PM
  • Try using the context_handle attribute where necessary. https://docs.microsoft.com/en-us/windows/desktop/Midl/context-handle
    Tuesday, May 14, 2019 2:24 PM
  • Yes, that's the issue, RLWA32.   It already was.  I've been deep into that 2006 link, trying a number of the suggestions.    Still at it.


    Hector Santos, CTO Santronics Software, Inc. http://www.santronics.com

    Wednesday, May 15, 2019 1:58 AM
  • Hi,

    How do you check the value of following code is different in 32-bit?

    DWORD dwwch = GetWildcatServerContextHandle();
    TWildcatContextHandle wch = GetWildcatServerContextHandleEx();

    I can get the same result in this sample(platform = x86):

    #include <windows.h>
    typedef void* TWildcatContextHandle;
    typedef void* TWildcatServerHandle;
    TWildcatContextHandle wch = (TWildcatContextHandle)0x12345678;
    error_status_t WcsGetWildcatServerContextHandle(
    	TWildcatContextHandle wch,
    	unsigned long __RPC_FAR *handle)
    {
    	*handle = DWORD(wch);
    	return 0;
    }
    DWORD APIENTRY GetWildcatServerContextHandle()
    {
    	DWORD handle = 0;
    	DWORD status = WcsGetWildcatServerContextHandle(wch, &handle);
    	if (status) {
    		SetLastError(status);
    		return FALSE;
    	}
    	return handle;
    }
    
    error_status_t WcsGetWildcatServerContextHandleEx(
    	TWildcatContextHandle wch,
    	TWildcatContextHandle __RPC_FAR *handle)
    {
    	*handle = TWildcatContextHandle(wch);
    	return 0;
    }
    TWildcatServerHandle APIENTRY GetWildcatServerContextHandleEx()
    {
    	TWildcatServerHandle handle = 0;
    	DWORD status = WcsGetWildcatServerContextHandleEx(wch, &handle);
    	if (status) {
    		SetLastError(status);
    		return FALSE;
    	}
    	return handle;
    }
    int main()
    {
    	DWORD dwwch = GetWildcatServerContextHandle();
    	TWildcatContextHandle wch = GetWildcatServerContextHandleEx();
    
    	return 0;
    }

    All the difference is just the type of them in 32-bit version(One is an unsign long value and the other is hexadecimal). Where and how do you use it?

    Best regards,

    Drake


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Wednesday, May 15, 2019 9:45 AM
  • HI,  yes,  I see the same thing too until the context handle is "marshalled" thru the RPC layer.  In other words, the WcsXXXXX functions are in a different process - the client talks to the server via RPC using a marshalled layer created by MIDL compiler.

    It wasn't a problem until I created the extended function where the In and Out parameters are both context handle types.  

    How I got to this point is that in one particular function:

    BOOL WildcatServerCreateContextFromHandle(DWORD wch)

    the 32 bit client passed back a handle as a DWORD to a 64 bit server which the server-side stub then type cast to the context class, and this pointer was wrong.   

    So I created the two extended functions, the first to get the handle as TWilddcatContextHandle from the server to give to an external EXE process.  So a client would call the extended now:

    TWildcatContextHandle wch = GetWildcatServerContextHandleEx();

    This wch is now saved in an environment string "WidlcatServerContext" or passed via command line to a compliant 3rd party app which would read the wch and passes it to the 2nd extended function to reestablish the parent session context:

    BOOL WildcatServerCreateContextFromHandleEx(wch)

    The problem is that first function, getting the handle from the server, I can't get pass that because it is now corrupted something.  The addresses and value appear ok, but something is not right and I think its the RPC Marshalling.  

    The 2006 link sort of talks about this, the potential problems because the RPC marshalling of the context-handle is always 32 bit, if I read it right.   So unless you specifically designed a persistent 64 bit object and maintain that on both ends, possibly via a struct, which then breaks the compatibility of existing 32 bit clients,  I may have to begin using a DWORD::POINTER map array on the server side.

    Fortunately, in order to move forward,  there is another method of using an unique session Challenge String (guid) created when the context-handle is created   So instead of saving and passing a handle (number) to a 3rd party EXE, the challenge string is passed and used with the function:

    BOOL WildcatServerCreateContextFromChallenge(const char *sz);

    This calls a server side stub that does a string::pointer map lookup.  This works perfectly and the 2006 links indicates this is ultimately how you will want to address the  32/64bit RPC marshaling translation and truncation potential issue.    I have updated my 3rd party APP DLLs to prepare the challenge method instead until the handle method can be resolved.     I am going to try looking at creating the dword:pointer map  because right now, its just does a type cast when the DWORD is passed. 

    So this:

        TClientContext *cc = (TClientContext *)wch;

    Will change to:
      
        TClientContext *cc = TClientContext::GrabByHandle(wch);   

    I appreciate your interest and effort to assist. :)


    Hector Santos, CTO Santronics Software, Inc. http://www.santronics.com


    Wednesday, May 15, 2019 12:44 PM
  • Hi,  I resolved it.

    I got away from using extended functions using context handles for both [in] and [out, ref] parameters.  I used the original IDL prototype but instead of DWORD for returning a cloned context-handle, I used DWORD_PTR.   This allows for mixed client 32/64 bit API functionality with 32/64 bit servers.  MIDL handling the marshaling.   But that wasn't enough because I did try DWORD_PTR before.    For a 64 bit server, I had to use a map/array lookup (TClientContext::GrabByHandle()) rather than using the simple type cast.     

    Thanks everyone for their input.








    Hector Santos, CTO Santronics Software, Inc. http://www.santronics.com

    Thursday, May 16, 2019 2:33 AM