none
C#, transparency, rendering videos, WindowsForms

    Question

  • Hello everyone,

    I have the following problem:
    I want to play a video and it should be possible to mark some stuff on it, i.e. whenever the user clicks on a certain place I want to draw a cross there. I based my code on the media player shown here:
    http://www.codeproject.com/KB/directx/directshowmediaplayer.aspx

    First I tried to place a drawing panel above the video panel. Just setting it to transparent only gave me the background color of the form, since WindowsForms has a weird definition of transparency.
    Placing the drawing panel inside the video panel doesn't work because the video somehow pushes my drawing panel in the background. As long as no video is loaded, everything is fine, but as soon as the video runs I only see the video and no more markers.

    And here's the most curious thing: If I call "invalidate" on the video panel, then all of a sudden I can see my transparent drawing panel again (until the next frame is rendered anyway), but the drawing panel doesn't show me the video in the background. Instead I see the background color of the video panel, covering the video itself.

    So the only way I can think of right now is not to draw any lines but to use a bunch of very small non-transparent panels for lines instead. Since I only need vertical and horizontal lines that should be fine but it looks extremely workaround-ish to me. Does anyone have a better idea?
    I also read that WPF supports true transparency, but I don't want to change the entire environment every time I find out that something doesn't work, because other environments have other problems. So, unless it's absolutely necessary, I don't want to change to WPF just for this one little (but vitally important) thing.

    Kind regards
    Squall Leonheart

    • Moved by CoolDadTxMVP Saturday, July 16, 2011 3:30 PM winforms related (From:Visual C# General)
    Friday, July 15, 2011 10:06 AM

Answers

  • Squall,

     

    Hey, I’m the team’s DShow expert. Trevor asked me to take a look at your post and give my two cents. From looking at the DShow code that you are using in your winforms application I just want you to be aware that by including quartz.dll as a dependency you are using the DirectShow 8 OLE automation objects. These objects have been deprecated for years and are certainly not recommended. At this time Microsoft does not have a supported solution for calling DirectShow code from C# (or any managed language).  Please see the “note” at the top of the page at the link below for documented confirmation of this. Because the technology is not supported from the winforms environment it is not possible for us to suggest a supported workaround from managed code.

     

    That said it should be possible to facilitate the functionality that you are looking for by creating a custom EVR presenter. By using a custom presenter you can get direct access to the D3D surface. You can then use the standard D3D constructs to draw directly to the same D3D surface that the EVR is using to blit the video. There are two things to keep in mind about this solution. First you must code this solution in unmanaged C++. Again this is due to the fact that DirectShow is not supported from managed code. Second, this solution is extremely complex and difficult to implement even for the most experienced DirectShow / D3D expert. Because of these two factors it is recommended that you take a serious look at the MediaElement in WPF.

     

    As you know the WPF environment is constructed from the ground up to offer developers a very rich “graphics first” environment. The MediaElement in particular was designed to allow you to mix video with various other UI components seamlessly. This solution will give you the flicker free, “draw over video” solution that you are looking for. The best part is you can do all of this in C#. The bad part of this solution is that the MediaElement is not designed for displaying time sensitive media content. In other words, the MediaElement is notorious for dropping and delaying the display of video frames. There are ways to minimize this such as using SD rather than HD content, use a video accelerated codec, etc. However, you will never get the same high quality video experience that you find with DirectShow.

     

    I hope this will help you understand the current shortcomings of the technologies that you have chosen and help you to focus your efforts on a fully supported and viable solution. If you need any additional clarification please let us know.

     

    Reference:

    FilgraphManager Object

    http://msdn.microsoft.com/en-us/Library/dd375648.aspx

     

    MediaElement Class

    http://msdn.microsoft.com/en-us/library/system.windows.controls.mediaelement.aspx

     

    Multimedia Overview .net Framework 4

    http://msdn.microsoft.com/en-us/library/aa970915.aspx

     

    Thanks,

     

    James

     

    Global Technical Escalations Lead

    Windows Media SDK Technologies

    Microsoft Developer Services

     

    http://blogs.msdn.com/mediasdkstuff/

    Wednesday, July 27, 2011 10:54 PM

All replies

  • Opacity turns the whole object partially transparent, but that's not what I want. I need the video and my markers to be 100% opaque, while the panel I want to use to draw the markers in must be 100% transparent, so I can see only the markers and the video. But even though I can get this part to work, I have the problem that the video goes in front of my marking panel, even if I explicitly use the "send to back" command in my code.

    Monday, July 18, 2011 6:49 AM
  • Hi Squall_Leonheart,

    Welcome to the MSDN Forum.

    After I read your quesitons, I think what you want is a transparent panel, have a look at the following sample:

    AlphaGradientPanel, an extended panel:
    http://www.codeproject.com/KB/cpp/AlphaGradientPanel.aspx

    If you have any questions, please feel free to tell us.

    Best Regards


    Neddy Ren [MSFT]
    MSDN Community Support | Feedback to us
    Get or Request Code Sample from Microsoft
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Wednesday, July 20, 2011 5:38 AM
  • Hello Neddy,

    thanks for the link, but that's not my problem. I have a video panel to play a video in and a transparent panel in which I paint things in to mark stuff in frames of the video. As soon as I load a video, the video panel covers the transparent panel no matter what I do. This happens only if the transparent panel is a child of the video panel. If it's not a child, then the transparent panel is not really transparent, because it shows the color of its parent (i.e. the main form in my case) and not what's really behind it (i.e. the video).

    Also I do not want the video to be transparent.

    Wednesday, July 20, 2011 9:36 AM
  • Sorry for the misunderstanding.

    So, could you please upload your reproduced projects for us so that we can help you to resolve your issue?


    Neddy Ren [MSFT]
    MSDN Community Support | Feedback to us
    Get or Request Code Sample from Microsoft
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Thursday, July 21, 2011 3:15 AM
  • No problem, any help is appreciated. :-)

    I got a little closer to the problem: It's not the video panel which pushes in front of my other elements, it's only the video itself. I found this out because I don't receive any more panel_click events when a video is loaded and I click the video with the mouse. Therefore a video is apparently using some sort of extra layer. I wonder how all those Media Players manage to go to full screen whenever their video window is double-clicked or switch between play and pause on a single-click, because I don't receive any events.

    I uploaded an example project at the following link: http://www.file-upload.net/download-3601983/TimeLineWithPanelClickEventHandler.7z.html

    It's basically the TimeLine project you can download at http://www.codeproject.com/KB/graphics/TimeLine.aspx. I opened it in Visual Studio and added a click event to the panel. I entered some random command into the function that was generated and set a debug break point there. Now if you start the project in debug mode and click on the panel, the breakpoint will trigger. If a video is running, then it won't. Note that if the video doesn't fill the whole panel you can still trigger the breakpoint by clicking on the panel at a place which doesn't contain the video.

    Currently I am working around the problem by catching keyboard events instead. With them I can still get information about mouse position and place my markers accordingly.
    Btw: My markers are also workarounds. I am using 2 small panels per marker and combine them to make a cross. It's kinda restrictive that I can work only with rectangles but it's the only way I found until now.

    Thursday, July 21, 2011 6:13 AM
  • There is no good way to place overlays over a video in WinForms.  DirectShow is not designed for managed code and cannot work well.  The best you will achieve is a layer of hacks which will mostly work in simple scenarios.  You will probably be much better off moving to a different toolkit.

     

    Please see:

    http://blogs.msdn.com/b/mediasdkstuff/archive/2009/04/01/calling-the-format-sdk-directshow-media-foundation-or-the-wasapi-from-managed-code-c-vb-net.aspx

     

    http://msdn.microsoft.com/en-us/library/dd375463(VS.85).aspx

     

    As mentioned, Winforms child control transparency is more of a chameleon effect rather than a true transparency.  The control asks its parent to paint its background on the child.  This won't work for the parent's foreground or for an animated background.

     

     

    To make a truly transparent window it will need to be a top-level window rather than a child window.  Then it can use the ColorKey properties to make everything that is painted a specific color transparent.

     

    The video itself likely plays in a separate window created just for that (the details will depend on how you initialized DirectShow).  Since that isn't a Winforms window you won't be able to interact with it via Winforms events.  It may itself be an owned top-level window rather than a child window, in which case it will always be above any child windows in the Form (you can confirm this in Spy++ or may be able to deduce it from how your existing marker panels work).


    --Trevor H.
    Send files to Hotmail.com: "MS_TREVORH"
    Monday, July 25, 2011 1:35 PM
    Owner
  • > There is no good way to place overlays over a video in WinForms.  DirectShow is not designed for managed code and cannot work well.
    Thanks, that's good to know.

    I looked around for other toolkits and I found:
    - C++ Winforms is the same as C# and VB, only slightly more complex (uses managed code as well, probably with the same transparency problem)
    - Win32 API: It sounds like it can do everything I want and is very fast, but it's way more complex. I used it here and there already (i.e. MFC, which is a Win32 API wrapper if I understood it correctly), but never used it to play videos in a frame and I never tried out transparency with it.
    - wxWidgets: Not available in LGPL, i.e. I can't make commercial apps with it.
    - QT: LGPL, unmanaged code and hopefully easier than Win32 API. Also the code is cross-platform. Sounds like the best alternative. Any other opinions on that one?

    > To make a truly transparent window it will need to be a top-level window rather than a child window.
    > Then it can use the ColorKey properties to make everything that is painted a specific color transparent.
    Will that window be truly transparent then? I tried to set a form's background color to transparent, but it didn't work. But I would've found it quite odd if it had worked anyway, what with all the info about C# transpareny not being real transparency.

    > The video itself likely plays in a separate window created just for that (the details will depend on how you initialized DirectShow).
    I have an IVideoWindow-object and I call its "put_owner" method to give a handle of one of my panels to it. Now that I've read the classname again it's suddenly quite obvious that I created a new window there. :-)

    • Edited by __Alexander__ Tuesday, July 26, 2011 9:03 AM formatting
    Tuesday, July 26, 2011 9:02 AM
  • I cannot comment on 3rd party libraries.  DirectShow can be used from unmanaged code or WPF may be an option for managed code.

     

    Setting the ColorKey or Opacity properties on a top level window use the Layered Windows system (http://msdn.microsoft.com/en-us/library/ms632599(VS.85).aspx#layered ) to create real transparency.  This is unrelated to setting the BackgroundColor to transparent.


    --Trevor H.
    Send files to Hotmail.com: "MS_TREVORH"
    Tuesday, July 26, 2011 8:42 PM
    Owner
  • DirectShow itself can be used for GUI creation? How? I only found ways of creating DS-filters and even their GUIs (i.e. property pages) are done in MFC in my case.

    I tried setting the ColorKey and it worked except for a flickering effect whenever the form lies above the video. Non-transparent forms don't flicker. Does WPF have that problem, too? If not then I'll keep it in mind and use it as soon as I have enough problems with C# to leave it.

    Wednesday, July 27, 2011 3:20 PM
  • Squall,

     

    Hey, I’m the team’s DShow expert. Trevor asked me to take a look at your post and give my two cents. From looking at the DShow code that you are using in your winforms application I just want you to be aware that by including quartz.dll as a dependency you are using the DirectShow 8 OLE automation objects. These objects have been deprecated for years and are certainly not recommended. At this time Microsoft does not have a supported solution for calling DirectShow code from C# (or any managed language).  Please see the “note” at the top of the page at the link below for documented confirmation of this. Because the technology is not supported from the winforms environment it is not possible for us to suggest a supported workaround from managed code.

     

    That said it should be possible to facilitate the functionality that you are looking for by creating a custom EVR presenter. By using a custom presenter you can get direct access to the D3D surface. You can then use the standard D3D constructs to draw directly to the same D3D surface that the EVR is using to blit the video. There are two things to keep in mind about this solution. First you must code this solution in unmanaged C++. Again this is due to the fact that DirectShow is not supported from managed code. Second, this solution is extremely complex and difficult to implement even for the most experienced DirectShow / D3D expert. Because of these two factors it is recommended that you take a serious look at the MediaElement in WPF.

     

    As you know the WPF environment is constructed from the ground up to offer developers a very rich “graphics first” environment. The MediaElement in particular was designed to allow you to mix video with various other UI components seamlessly. This solution will give you the flicker free, “draw over video” solution that you are looking for. The best part is you can do all of this in C#. The bad part of this solution is that the MediaElement is not designed for displaying time sensitive media content. In other words, the MediaElement is notorious for dropping and delaying the display of video frames. There are ways to minimize this such as using SD rather than HD content, use a video accelerated codec, etc. However, you will never get the same high quality video experience that you find with DirectShow.

     

    I hope this will help you understand the current shortcomings of the technologies that you have chosen and help you to focus your efforts on a fully supported and viable solution. If you need any additional clarification please let us know.

     

    Reference:

    FilgraphManager Object

    http://msdn.microsoft.com/en-us/Library/dd375648.aspx

     

    MediaElement Class

    http://msdn.microsoft.com/en-us/library/system.windows.controls.mediaelement.aspx

     

    Multimedia Overview .net Framework 4

    http://msdn.microsoft.com/en-us/library/aa970915.aspx

     

    Thanks,

     

    James

     

    Global Technical Escalations Lead

    Windows Media SDK Technologies

    Microsoft Developer Services

     

    http://blogs.msdn.com/mediasdkstuff/

    Wednesday, July 27, 2011 10:54 PM
  • Thanks for taking the time, that answers my question indeed. Also don't worry, I would've marked it as an answer, too. ;-) I just needed to read up a bit on WPF first (http://www.wpftutorial.net) and do various other things.

    In my case, a frame-delay is ok, but a drop isn't. Can I configure this kind of behavior in WPF?

    Thursday, July 28, 2011 3:48 PM
  • Squall,

    Unfortunately you can’t really tell the WPF MediaElement to never drop frames. The term we use for this class of issues is “disparate clocks”. In this case WPF is updating the screen at a certain rate (clock 1). The MediaElement (based on WMP) is cranking out video frames at a slightly different rate (clock 2). Given the underlying technologies there is currently no way to synchronize the two clocks and force them to “tick” at the same rate. Since the display will only be updated according to the WPF clock, multiple frames of video may be sent from the MediaElement to WPF between clock ticks. Because of this the MediaElement may appear to drop frames.  This is a very common problem in multimedia development and there is no simple solution.

    So if you absolutely need frame accuracy in your application then using the MediaElement probably won’t work for you. That said, there are some things that you can do to improve the chances of your content dropping as few frames as possible. Modify your content so that it uses either the h.264 or VC1 codec. Require your users to have modern video HW capable of advanced video acceleration. Use the MPEG 4 or ASF file container. When encoding your content set your frame rate at or below 25 frames per second. Set the resolution of your content to something like 720x480. Set the bitrate to VBR constrained and set an upper limit of between 500 Kbps and 2.5 Mbps.

    If you use the guidelines above you will minimize the number of frames that get dropped but you will never be able to completely eliminate them. Also keep in mind that the same frames may not be dropped. For example: if you play video1.asf the first time you might drop frames 200 and 375. On the next run of the same file you may drop frames 143, 678 and 901. This is due to the relatively nondeterministic nature of the Windows OS.

    I hope this helps.

    -James

    Friday, July 29, 2011 1:23 AM
  • Thanks for those tips, James. I'll keep them in mind.

    Does the ticking speed of these two clocks vary a lot? I mean, if they had a fairly constant speed then all I had to do would be to make sure the WPF clock ticks a lot faster, e.g. by lowering the frame rate of the video.

    edit: I just read in a blog from a few years ago that it is difficult to create a custom DS filter graph in WPF. Is that true today?
    Blog: http://jmorrill.hjtcentral.com/Home/tabid/428/EntryId/15/WPF-Hackery-Part-I.aspx
    Because that'd be way more important than real transparency.

    Monday, August 01, 2011 12:52 PM
  • We're well beyond the scope of a forum post here, but... the render thread isn't strictly timer based and there is no direct control over when it will tick.  If you need frame accuracy then WPF will not be a solution and you'll need to implement this in native code; however, implementing sprite overlays in native DirectShow is a fairly advanced topic and much more difficult than adding overlays in WPF. WPF still doesn't support implementing DS filter graphs.

    If you need more help on this issue beyond this, consider opening a support case with us. Visit this link to see the various support options that are available to better meet your needs:  http://support.microsoft.com/default.aspx?id=fh;en-us;offerprophone.


    --Trevor H.
    Send files to Hotmail.com: "MS_TREVORH"
    Tuesday, August 02, 2011 7:47 PM
    Owner
  • Okay, I guess in that case there really isn't any way to guarantee frame accuracy. Also if I can't build custom DS filter graphs, then I can't use WPF, because that's essential for the project. Thank you for all the help!
    Wednesday, August 03, 2011 7:48 AM
  • I'm quite disappointed in what appears to be Microsoft experts in this somewhat old thread weaseling out of helping a Windows platform developer because they are forced by some bizarre policy to spend a lot of words that essentially say "we won't help you."  Because the product is no longer supported.

    The Microsoft responses negligently fail to include any mention of the DirectShow.NET library http://directshownet.sourceforge.net/.  Why?  And shame on them for failing to do so.  This library helps you use DirectShow in a managed context.  There are plenty of code samples to be found.

    As for using a click event from the drawing surface, which you should also find in the code samples, it is possible to receive the event on your winform using the iVideoWindow.put_MessageDrain property:

             hr = m_objVideoWindow.put_MessageDrain(Me.Handle) ' Send events to containing form message pump
              DsError.ThrowExceptionForHR(hr) ' Throws exception on non-zero reult

    Look for code samples that use this property.    I hope I've been more helpful than the average MS response.  You might also also find samples to draw text or graphics.  I have not had need for it, but I did find this thread looking for a solution that gives me hot spot areas instead of the entire surface.  I'd like to just put a PictureBox on top as a child, but we all know about the transparency problems.  I wanted to put the video drawing surface on the background of the form but looks like that's not possible.



    • Proposed as answer by JWPlatt Wednesday, November 14, 2012 3:36 AM
    • Edited by JWPlatt Wednesday, November 14, 2012 3:52 AM
    Wednesday, November 14, 2012 3:36 AM