How to hit test non-client buttons in Vista?

Question

The following code was added to a Win32 application's WndProc to hit test the minimize, maximize, and close buttons.  The buttons are detected if Desktop Composition is disabled, but not when it is enabled. Other non-client regions are detected when Desktop Composition is enabled.  Does anyone know of a work-around for this issue?

    case WM_NCHITTEST:
        LRESULT lResult;
        lResult = DefWindowProc(hWnd, message, wParam, lParam);
        if (lResult==HTCLOSE)
            OutputDebugString(L"HTCLOSE\n");        //detected if Desktop composition is disabled
        else if (lResult==HTMINBUTTON)
            OutputDebugString(L"HTMINBUTTON\n");    //detected if Desktop composition is disabled
        else if (lResult==HTMAXBUTTON)
            OutputDebugString(L"HTMAXBUTTON\n");   //detected if Desktop composition is disabled
        else OutputDebugString(L"HT other\n");
        return lResult;
        break;

Answer 1

 

Instead of using DefWindowProc, when composition is enabled use DwmDefWindowProc, which will perform NC hit testing for you for the buttons only   ( but works even if you've set your nc area to 0 and used DwmExtendFrameIntoClientArea to get min/max/close buttons on your client area).   If it returns without detecting a btn, pass it along to DefWindowProc as well for caption, sizing, etc testing.

 

 

Answer 2

Thanks for the suggestion, but I still can't detect the buttons.  I tried the following - the documentation for DwmDefWindowProc is a bit thin so I'm not sure if this is what you meant:

    case WM_NCHITTEST:
        LRESULT lResult;
        HRESULT hr;
        BOOL bDwmCompositionEnabled;
        hr = DwmIsCompositionEnabled( &bDwmCompositionEnabled );
        if ( SUCCEEDED( hr ) && bDwmCompositionEnabled )
        {
           hr = DwmDefWindowProc( hWnd, message, wParam, lParam, &lResult );
           if ( SUCCEEDED( hr ) )
           {
                if ( lResult==HTCLOSE )
                    OutputDebugString( L"DWM HTCLOSE\n" );    //never detected
                else
                    OutputDebugString( L"DWM Not HTCLOSE\n" );
           }
        }
        else ...

When desktop composition is enabled there are no WM_NCHITTEST messages when the arrow is over the buttons so the DwmDefWindowProc never gets called.  There are messages over other non-client areas - DwmDefWindowProc returns S_OK but the debugger shows lResult with the uninitialized value of 0xcccccccc.  (note: I am using RC1 build 5600) 

Answer 3

 

Actually, I test for HTCLIENT rather than the return of DwmDefWindowProc, but that's pretty much exactly what I meant.

So you have the standard titlebar - you don't have the frame extended in or drawn yourself?  I haven't used DwmDefWindowProc under those circumstances.  

Right now, my usage is (when Dwm enabled)

1) respond to wm_nccalcsize and set nc top area to 0 (client starts at the top)
2) call DwmExtendFrameIntoClientArea, extending it down the height of a titlebar.
3) respond to wm_nchittest and pass it to DwmDefWindowProc first.  if htclient is passed back I do my own testing to see if it should be htcaption

Note that step 2 above *will* show the control menu (icon), caption text, and min/max/close buttons as if it were in the standard NC area.  if you pass wm_nchittest to dwmdefwindowproc it will handle the "glow" and hit testing of the buttons for you.

 

However, I'm not sure how to go about testing in your scenario - I'm afraid I'm stumped! sorry to lead you down the wrong path.  I'm still struggling with proper handling of tricks in the NC area under vista as well (drawing, etc)

 

Answer 4

Thanks for the info on the usage of DwmDefWindowProc - that's the first description of its usage that I've seen. 

I am getting the impression that when the buttons are in the non-client area, DWM is gobbling up the WM_NCHITTEST messages to make the buttons glow and not passing them along to the WndProc.  If this is true it sounds like a design flaw to me - Vista should either pass them along or use a different message so that an app can still hit test them when desktop composition is enabled.

Answer 5

A work-around for this issue is to use WM_NCMOUSEMOVE instead of WM_NCHITTEST. wParam contains HTMINBUTTON, HTMAXBUTTON, and HTCLOSE whether desktop composition is enabled or disabled.

Answer 6

Thanks for the follow up with an answer!