locked
A call to a method in an out of process COM (ATL) server blocks when called from a C# method RRS feed

  • Question

  • I have a 64 bit [and also a separate 32 bit] out of process COM server written in ATL, named ProcessMonitor, the purpose of which is to inject a DLL into other processes for monitoring keystrokes and mouse movements. I need to be able to shut down this server upon certain events. I send a WM_QUIT message to its window and handle the shut down in the message loop of the window. The method which does that is as follows

     public void ShutdownMonitor()
            {
                try
                {
                    if (processMonitorX64 != null)
                    {
                        logger.Info("Shutting down ProcessMonitorX64");
                       // Thread thread = new Thread(()=> processMonitorX64.Shutdown());
                       // thread.Start();
                        processMonitorX64.Shutdown();
                        logger.Info("Shut down ProcessMonitorX64");
                    }
                    else
                    {
                        logger.Info("ProcessMonitorX64 was null");
                    }
                }
                catch (Exception e)
                {
                    logger.Error("Exception shutting down ProcessMonitorX64");
                    logger.Error(JsonConvert.SerializeObject(e));
                }
                try
                {
                    if (processMonitorX86 != null)
                    {
                        logger.Info("Shutting down ProcessMonitorX86");
                        processMonitorX86.Shutdown();
                        logger.Info("Shut down ProcessMonitorX86");
                    }
                    else
                    {
                        logger.Info("ProcessMonitorX86 was null");
                    }
                }
                catch (Exception e)
                {
                    logger.Error("Exception shutting down ProcessMonitorX86");
                    logger.Error(JsonConvert.SerializeObject(e));
                }
            }

    It basically checks if the object has been instantiated and if so, attempts to shut it down. The Shutdown method of ProcessMonitor is 

    STDMETHODIMP CProcessMonitorX64::Shutdown()
    {
    	SendMessage(hWndMonitor, WM_COMMAND, MAKEWPARAM(ID_MENU_EXIT, 0), NULL);
    	return S_OK;
    }
    

    This works well enough in a console C# app I use for testing. But the actual project is larger, and uses a plugin style design. [Plugins are loaded at runtime based on a configuration file] .This code is called from a plugin [in a separate app domain]. When I stepped through the code in the debugger, I found that the execution stops when this method is called. I have enabled native code debugging, and I also attach the process [the COM server] before calling the shutdown method on it. I have also placed a break point on the Shutdown method, but it is never reached. 

    However, this process also has a system tray icon, with an exit menu. When I click on the exit menu in the system tray icon, the break point is hit and the process shuts down properly

    Also, if I shut down Process Monitor by clicking on the exit menu after the execution has blocked, there is a COM exception in the C# code, which says "An RPC called failed".

    I cannot figure out why the execution blocks when I call the method. Other methods called on ProcessMonitor do not block,and work properly [I call a method to inject a hook into another process when the plugin is loaded, and that gets done properly]. 

    Please help.

    Thanks

    Saturday, May 11, 2019 3:24 PM

Answers

  •       I called from the same thread

          I havent specified a threading model in the plugin. [Its a dll]. The threading model in the COM  server is apartment


    I suggest you test by calling from a different thread to more closely imitate the behavior of the C# plugin.

    You can determine the threading model of the client by calling CoGetApartmentType function



    Sunday, May 12, 2019 4:16 PM

All replies

  • Try using PostMessage instead of SendMessage in the COM object's Shutdown() method.

    You might also consider carving the shutdown procedure out of the COM object and using a different technique ( for example, P/Invoke to send the window message).

    • Edited by RLWA32 Saturday, May 11, 2019 5:01 PM added comment
    Saturday, May 11, 2019 4:47 PM
  • RLWA,

    The problem does not seem to be SendMessage. I have placed a break point on the method in the C++ code.. The execution stops in the C# code and does not proceed inside the C++ code at all. However, if I click on a menu in the system tray, then the compiler steps into the method in the C++ code. So the problem seems to be outside the method, perhaps because of the COM proxy.

    I also thought about PostMessage or the possibility of sending the message using P/Invoke.  I would need the window handle of the process. So I would have to expose a hwnd property on the object. However, before I do that, I would like this issue to be resolved if possible.

    Sunday, May 12, 2019 2:47 AM
  • If ShutdownMonitor function is part of a control or form, have you tried BeginInvoke to call processMonitorX64.Shutdown and processMonitorX86.Shutdown?

    Maybe the problem is related to “Apartment”.

    Sunday, May 12, 2019 6:02 AM
  • Viorel_

    It is not part of a control or form. But it could definitely have something to do with apartments

    Sunday, May 12, 2019 6:14 AM
  • I suggested carving out the shutdown request because from what you have shown us the C# client is telling the COM server to shutdown while there are still instantiated objects and it is holding references to them.

    Is it possible that this code is getting called more than once?


    There are other ways to communicate beside sending a message to a specific window handle.  For example, you could broadcast a registered private message.  Or possibly have a COM server thread wait on an event.  The C# client could signal the event to initiate shutdown.

    • Edited by RLWA32 Sunday, May 12, 2019 10:20 AM added suggestions
    Sunday, May 12, 2019 10:13 AM
  • RLWA32,

    I did think of raising events from the COM server to notify the C# client that the server was shutting down, but havent done that yet. At present, I allow an exception to happen, and then handle it, in case the COM server has shut down but the C# client holds a reference to it.

    Carving out the shut down request is what I will try next, because  I need to resolve this issue. I need to expose the handle of the window as a property, which will require me to write code. I wanted to avoid doing that, since there is already a method which does that. 

    The other ways to communicate would require modifying a lot of the logic in the COM server, which is not feasible at this point of time, especially since the logic works fine in a console app. 

    Could apartments have something to do with the fact that code execution stops when this method is called from the C# client? The C# client is a plugin running in a separate thread in a different app domain

    I am really interested in figuring out why does the break point in the C++ code is not getting hit. 


    Sunday, May 12, 2019 11:17 AM
  • Does the C# code call the COM object methods from the thread that created it or from a different thread?

    If you tested with a native C++ client to exercise the out-of-process server by instantiating the COM object, calling a few methods and then calling Shutdown what happens?

    Sunday, May 12, 2019 11:57 AM
  • The c# code calls it from a different thread

    I have tested with a C# console client, not a native C++ client, and it works fine

    Sunday, May 12, 2019 3:02 PM
  • The c# code calls it from a different thread

    I have tested with a C# console client, not a native C++ client, and it works fine

    When you tested with your C# console client did you call from a different thread?

    Is the threading model (Apartment) used in your tests the same as the C# plugin?


    BTW, what are the threading models of the COM Server and of the client?
    • Edited by RLWA32 Sunday, May 12, 2019 3:34 PM added question
    Sunday, May 12, 2019 3:27 PM
  • The c# code calls it from a different thread

    I have tested with a C# console client, not a native C++ client, and it works fine

    When you tested with your C# console client did you call from a different thread?

    Is the threading model (Apartment) used in your tests the same as the C# plugin?

          I called from the same thread

          I havent specified a threading model in the plugin. [Its a dll]. The threading model in the COM  server is apartment


    Sunday, May 12, 2019 3:42 PM
  • Would making the COM server's threading model free threaded help?
    Sunday, May 12, 2019 3:49 PM
  •       I called from the same thread

          I havent specified a threading model in the plugin. [Its a dll]. The threading model in the COM  server is apartment


    I suggest you test by calling from a different thread to more closely imitate the behavior of the C# plugin.

    You can determine the threading model of the client by calling CoGetApartmentType function



    Sunday, May 12, 2019 4:16 PM
  • Am going off for the day, have to sleep
    Sunday, May 12, 2019 4:17 PM
  • Would making the COM server's threading model free threaded help?

    I cannot predict if that is the solution to the problem.

    However, it would require you make sure that your COM object can properly handle being in the MTA and called from multiple threads.

    Sunday, May 12, 2019 4:20 PM
  • I got it to work, but I am not sure I understand why.

    I tried out the C# console app again and found two things

    a. If I dont use a separate thread to call the Hook method, the execution stops at the hook method

    b. If I use a separate thread to call the hook method, execution proceeds, but stops at the next call to the Shutdown method, the same as in the plugin.

    So I figured it must be blocking on something. I made the following change to the C++ code

    typedef struct InjectParams { LONG processId; LPWSTR processName; }; DWORD WINAPI InjectThreadProc(void* lParams) { InjectParams params = *(InjectParams*)lParams; LONG processId=params.processId; LPWSTR processName=params.processName; //Find the window, and inject the Hook BOOL result = FindTargetWindowAndInjectHook(processId, processName); cout << "Result:" << result << endl; return result; } STDMETHODIMP CProcessMonitorX64::HookProcess(LONG processId, BSTR strProcessName, LONG idleTime) { InitializeLoggerConsole(L"LogConsole"); CComBSTR temp(strProcessName); processName = _tcsdup(temp.operator LPWSTR()); //Accept the Arguements into variables dwTargetProcessId = processId; //ZeroMemory(processName, sizeof(processName)); //strcpy_s(processName, sizeof(processName), argv[2]); nIdleTime = idleTime; LogC(wstring(L"IdleTime"), nIdleTime); LogC(wstring(L"TargetProcessId"), (LONG)dwTargetProcessId); LPWSTR path = GetCurrentPath(); cout << "Path:" << path << endl; //wstring strPath(path); //MessageBox(NULL,path, L"PROCESSMONITOR", MB_OK); //Load the DLL to be injected HINSTANCE hinst = LoadLibrary(_T("HookDLL_x64.dll")); cout << "LoadLibrary:" << hinst << endl; LogC(L"LoadLibrary returned hInstance:", (LONG)hinst); if (hinst) { //Get the addresses/ function pointer to the methods install and uninstall install = (Install)GetProcAddress(hinst, "install"); uninstall = (Uninstall)GetProcAddress(hinst, "uninstall"); cout << "Install Address:" << install << endl; cout << "Uninstall Address:" << uninstall << endl; LogC(L"Install Address", (LONG)install); LogC(L"Uninstall Address", (LONG)uninstall); } else { return S_FALSE; }

    //Changed this- Called FindTargetWindowAndInjectHook  via InjectThreadProc InjectParams params; params.processId = processId; params.processName = processName; DWORD dwThreadId = 0; HANDLE hThread = CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)InjectThreadProc, &params, NULL, &dwThreadId); if (hThread) { CloseHandle(hThread); return S_OK; } else { return S_FALSE; } }


    I moved the following line into a separate thread 

    BOOL result = FindTargetWindowAndInjectHook(processId, processName);

    and it started working. It seems that FindTargetWindowAndInjectHook waits for an object and hence this blocking behavior

    Thanks RLWA32 and Viorel_. Your answers got me thinking and allowed me to resolve the issue

    Monday, May 13, 2019 12:42 AM