locked
Custom MediaStreamSource and Memory Leaks During SampleRequested

    Question

  • Greetings,

    I have a nasty memory leak problem that is causing me to pull my hair out.

    I'm implementing a custom MediaStreamSource along with MediaTranscoder to generate video to disk. The frame generation operation occurs in the SampleRequested handler (as in the MediaStreamSource example). No matter what I do - and I've tried a ton of options - inevitably the app runs out of memory after a couple hundred frames of HD video. Investigating, I see that indeed GC.GetTotalMemory reports an increasing, and never decreasing, amount of allocated RAM. 

    The frame generator in my actual app is using RenderTargetBitmap to get screen captures, and is handing the buffer to MediaStreamSample.CreateFromBuffer(). However, as you can see in the example below, the issue occurs even with a dumb allocation of RAM and no other actual logic. Here's the code:

            void _mss_SampleRequested(Windows.Media.Core.MediaStreamSource sender, MediaStreamSourceSampleRequestedEventArgs args)
            {
                if ( args.Request.StreamDescriptor is VideoStreamDescriptor )
                {
                    if (_FrameCount >= 3000) return;
                    var videoDeferral = args.Request.GetDeferral();
                    var descriptor = (VideoStreamDescriptor)args.Request.StreamDescriptor;
                    uint frameWidth = descriptor.EncodingProperties.Width;
                    uint frameHeight = descriptor.EncodingProperties.Height;
                    uint size = frameWidth * frameHeight * 4;
                    byte[] buffer = null;
                    try
                    {
                        buffer = new byte[size];
                        // do something to create the frame
                    }
                    catch
                    {
                        App.LogAction("Ran out of memory", this);
                        return;
                    }
                    args.Request.Sample = MediaStreamSample.CreateFromBuffer(buffer.AsBuffer(), TimeFromFrame(_FrameCount++, _frameSource.Framerate));
                    args.Request.Sample.Duration = TimeFromFrame(1, _frameSource.Framerate);
                    buffer = null;  // attempt to release the memory
                    videoDeferral.Complete();
                    App.LogAction("Completed Video frame " + (_FrameCount-1).ToString() + "\n" +
                                    "Allocated memory: " + GC.GetTotalMemory(true), this);
                    return;
                }

    It usually fails around frame 357, with GC.GetTotalMemory() reporting 750MB allocated.

    I've tried tons of work-arounds, none of which made a difference. I tried putting the code that allocates the bytes in a separate thread - no dice.  I tried Task.Delay to give the GC a chance to work, on the assumption that it just had no time to do its job. No luck.

    As another experiment, I wanted to see if the problem went away if I allocated memory each frame, but never assigned it to the MediaStreamSample, instead giving the sample (constant) dummy data. Indeed, in that scenario, memory consumption stayed constant. However, while I never get an out-of-memory exception, RequestSample just stops getting called around frame 1600 and as a result the transcode operation never actually returns to completion.

    I also tried taking a cue from the SDK sample which uses C++ entirely to generate the frame. So I passed the buffer as a Platform::Array<BYTE> to a static Runtime extension class function I wrote in C++. I won't bore you with the C++ code, but even directly copying the bytes of the array to the media sample using memcpy still had the same result! It seems that there is no way to communicate the contents of the byte[] array to the media sample without it never being released.

    I know what some will say: the difference between my code and the SDK sample, of course, is that the SDK sample generates the frame _entirely_ in C++, thus taking care of its own memory allocation and deallocation. Because I want to get the data from RenderTargetBitmap, this isn't an option for me. (As a side note, if anyone knows if there's a way to get the contents of an RT Window using DirectX, that might work too, but I know this is not a C++ forum, so...). But more importantly, MediaStreamSource and MediaStreamSample are managed classes that appear to allow you to generate custom frames using C# or other managed code. The MediaStreamSample.CreateFromBuffer function appears to be tailored for exactly what I want. But there appears to be no way to release the buffer when giving the bytes to the MediaStreamSample. At least none that I can find.

    I know the RT version of these classes are new to Windows 8.1, but I did see other posts going back 3 years discussing a similar issue in Silverlight. That never appears to have been resolved.

    I guess the question boils down to this: how do I safely get managed data, allocated during the SampleRequested handler, to the MediaStreamSample without causing a memory leak? Also, why would the SampleRequested handler just stop getting called out of the blue, even when I artificially eliminate the memory leak problem?

    Thanks so much for all input!

    Thursday, May 22, 2014 5:04 PM

Answers

  • Ugh, forget it. Even when it doesn't crash, the transcode operation just randomly stops, totally silently, in the middle. No exception, no warning, and no rhyme or reason.

    I'm done. Sorry but MediaTranscoder and MediaStreamSource are just too unreliable for me. (And if you Google them, others have similar horror stories it turns out). 

    As it turns out, using Media Foundation to output rendered video frame by frame is pretty much a breeze. It's ugly dealing with all the COM calls, but you get used to it. And it's faster and way more reliable than the transcoder. I'm already well on the way to making a totally custom transcoder class in C++ and so far, knock on wood, it's working fine.

    Thanks to James and Rob for the responses. Maybe one of you will figure out what, if anything, I was doing wrong, but the Media Foundation method seems to be the much better approach. 

    • Marked as answer by pnm655 Tuesday, May 27, 2014 10:26 PM
    Tuesday, May 27, 2014 10:25 PM

All replies

  • Just a quick correction: the Windows Runtime (including RenderTargetBitmap and MediaStreamSource) is native code, not managed. It can be called from native C++ with deterministic memory management.

    When you project it into C# your C# code is managed and garbage collected and the GC handles when to free the native runtime components. Setting the buffer to null will release the reference, but not immediately the memory.

    If you examine this in the memory profiler is the memory rooted? You should be able to see if there are outstanding references holding on to it. If you force a GC does it clean up?

    --Rob 

    Friday, May 23, 2014 3:24 PM
    Owner
  • Hello,

    If you want to actively manage your sample memory you should create a sample pool. In other words, you should pre allocate a pool of samples that is just large enough to support your scenario. You should then cycle through the pool round robin style, reusing samples as appropriate. This will allow you to pre allocate and know exactly how much sample memory your app will be using.

    I hope this helps,

    James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Friday, May 23, 2014 6:07 PM
    Moderator
  • Hi Rob - 

    Thanks for your quick reply and for clarifying the terminology. 

    In the Memory Usage test under Analyze/Performance and Diagnostics (is that what you mean?) it's clear that each frame of video being created is not released from memory except when memory consumption gets very high. GC will occasionally kick in, but eventually it succumbs.

    Interestingly, if I reduce the frame size substantially, say 320x240, it never runs out of RAM no matter how many frames I throw at it. The Memory Usage test, however, shows the same pattern. But this time the GC can keep up and release the RAM.

    After playing with this ad nauseum,  I am fairly convinced I know what the problem is, but the solution still escapes me. It appears that the Transcoder is requesting frames from the MediaStreamSource (and the MediaStreamSource is providing them via my SampleRequested handler) faster than the Transcoder can write them to disk and release them. Why would this be happening? The MediaStreamSource.BufferTime property is - I thought - used to prevent this very problem. However, changing the BufferTime seems to have no effect at all - even changing it to ZERO doesn't change anything. If I'm right, this would explain why the GC can't do its job - it can't release the buffers I'm giving to the Transcoder via SampleRequested because the Transcoder won't give them up until it's finished transcoding and writing them to disk. And yet the transcoder keeps requesting samples until there's no more memory to create them with.

    The following code, which I made from scratch to illustrate my scenario, should be air-tight according to everything I've read. And yet, it still runs out of memory when the frame size is too large. 

    If you or anyone else can spot the problem in this code, I'd be thrilled to hear it. Maybe I'm omitting a key step with regard to getting the deferral? Or maybe it's a bug in the back-end? Can I "slow down" the transcoder and force it to release samples it's already used?

    Anyway here's the new code, which other than App.cs is everything. So if I'm doing something wrong it will be in this module:

    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Threading.Tasks;
    using System.Linq;
    using System.Runtime.InteropServices.WindowsRuntime;
    using System.Diagnostics;
    using Windows.Foundation;
    using Windows.Foundation.Collections;
    using Windows.UI.Xaml;
    using Windows.UI.Xaml.Controls;
    using Windows.UI.Xaml.Controls.Primitives;
    using Windows.UI.Xaml.Data;
    using Windows.UI.Xaml.Input;
    using Windows.UI.Xaml.Media;
    using Windows.UI.Xaml.Navigation;
    using Windows.UI.Popups;
    using Windows.Storage;
    using Windows.Storage.Pickers;
    using Windows.Storage.Streams;
    using Windows.Media.MediaProperties;
    using Windows.Media.Core;
    using Windows.Media.Transcoding;
    
    
    // The Blank Page item template is documented at http://go.microsoft.com/fwlink/?LinkId=234238
    
    namespace MyTranscodeTest
    {
        /// <summary>
        /// An empty page that can be used on its own or navigated to within a Frame.
        /// </summary>
        public sealed partial class MainPage : Page
        {
            MediaTranscoder _transcoder;
            MediaStreamSource _mss;
            VideoStreamDescriptor _videoSourceDescriptor;
    
            const int c_width = 1920;
            const int c_height = 1080;
            const int c_frames = 10000;
            const int c_frNumerator = 30000;
            const int c_frDenominator = 1001;
            uint _frameSizeBytes;
            uint _frameDurationTicks;
            uint _transcodePositionTicks = 0;
            uint _frameCurrent = 0;
            Random _random = new Random();
            
            
    
            public MainPage()
            {
                this.InitializeComponent();
            }
    
            private async void GoButtonClicked(object sender, RoutedEventArgs e)
            {
                Windows.Storage.Pickers.FileSavePicker picker = new Windows.Storage.Pickers.FileSavePicker();
                picker.FileTypeChoices.Add("MP4 File", new List<string>() { ".MP4" });
                Windows.Storage.StorageFile file = await picker.PickSaveFileAsync();
                if (file == null) return;
    
                
                Stream outputStream = await file.OpenStreamForWriteAsync();
    
                var transcodeTask = (await this.InitializeTranscoderAsync(outputStream)).TranscodeAsync();
    
                transcodeTask.Progress = (asyncInfo, progressInfo) =>
                {
                    Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
                    {
                        _ProgressReport.Text = "Sourcing frame " + _frameCurrent.ToString() + " of " + c_frames.ToString() +
                                            " with " + GC.GetTotalMemory(false).ToString() + " bytes allocated.";
                    });
                };
    
                await transcodeTask;
                MessageDialog dialog = new MessageDialog("Transcode completed.");
                await dialog.ShowAsync();
                
            }
    
            async Task<PrepareTranscodeResult> InitializeTranscoderAsync (Stream output)
            {
                _transcoder = new MediaTranscoder();
                _transcoder.HardwareAccelerationEnabled = false;
    
                _videoSourceDescriptor = new VideoStreamDescriptor(VideoEncodingProperties.CreateUncompressed( MediaEncodingSubtypes.Bgra8, c_width, c_height ));
                _videoSourceDescriptor.EncodingProperties.PixelAspectRatio.Numerator = 1;
                _videoSourceDescriptor.EncodingProperties.PixelAspectRatio.Denominator = 1;
                _videoSourceDescriptor.EncodingProperties.FrameRate.Numerator = c_frNumerator;
                _videoSourceDescriptor.EncodingProperties.FrameRate.Denominator = c_frDenominator;
                _videoSourceDescriptor.EncodingProperties.Bitrate = (uint)((c_width * c_height * 4 * 8 * (ulong)c_frDenominator) / (ulong)c_frNumerator);
                _frameDurationTicks = (uint)(10000000 * (ulong)c_frDenominator / (ulong)c_frNumerator);
                _frameSizeBytes = c_width * c_height * 4;
                
                _mss = new MediaStreamSource(_videoSourceDescriptor);
                _mss.BufferTime = TimeSpan.FromTicks(_frameDurationTicks);
                _mss.Duration = TimeSpan.FromTicks(  _frameDurationTicks * c_frames );
    
                _mss.Starting += _mss_Starting;
                _mss.Paused += _mss_Paused;
                _mss.SampleRequested += _mss_SampleRequested;
    
                MediaEncodingProfile outputProfile = MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Ntsc);
                outputProfile.Audio = null;
    
                return await _transcoder.PrepareMediaStreamSourceTranscodeAsync(_mss, output.AsRandomAccessStream(), outputProfile);
            }
    
            void _mss_Paused(MediaStreamSource sender, object args)
            {
                throw new NotImplementedException();
            }
    
    
            void _mss_Starting(MediaStreamSource sender, MediaStreamSourceStartingEventArgs args)
            {
                args.Request.SetActualStartPosition(new TimeSpan(0));
            }
    
    
            /// <summary>
            ///  This is derived from the sample in "Windows 8.1 Apps with Xaml and C# Unleashed"
            /// </summary>
            /// <param name="sender"></param>
            /// <param name="args"></param>
            void _mss_SampleRequested(MediaStreamSource sender, MediaStreamSourceSampleRequestedEventArgs args)
            {
                if (_frameCurrent == c_frames) return;
    
                var deferral = args.Request.GetDeferral();
    
                byte[] frameBuffer;
                try
                {
                    frameBuffer = new byte[_frameSizeBytes];
                    this._random.NextBytes(frameBuffer);
                }
                catch
                {
                    throw new Exception("Sample source ran out of RAM");
                }
    
                args.Request.Sample = MediaStreamSample.CreateFromBuffer(frameBuffer.AsBuffer(), TimeSpan.FromTicks(_transcodePositionTicks)); 
                args.Request.Sample.Duration = TimeSpan.FromTicks(_frameDurationTicks);
                args.Request.Sample.KeyFrame = true;
                
                _transcodePositionTicks += _frameDurationTicks;
                _frameCurrent++;
    
                deferral.Complete();
            }
    
    
        }
    }
    

    Again, I can't see any reason why this shouldn't work. You'll note it mirrors pretty closely the sample in the Windows 8.1 Apps With Xaml Unleashed book (Chapter 14). The difference is I'm feeding the samples to a transcoder rather than a MediaElement (which, again should be no issue).

    Thanks again for any suggestions!

    Peter

    Friday, May 23, 2014 6:31 PM
  • Hi James -

    That's a very interesting idea thank you. Here's one problem I would have though. I want to be able to generate long videos (maybe an hour or more) so obviously I cannot allocate an hour's worth of uncompressed samples, nor would I know in advance how long the output will have to be.

    I think what you're saying, though, is that I would allocate a much smaller pool of frames - say 30 - and cycle through them, and when I hit 30, go back to 1. But how will I know for sure that any particular sample is free for re-use? Also, even a buffer of 30 could be huge - for 1080/30p video, that's ~200MB/s, so really something like no more than 5 or 10 samples would be what I would want to have on deck at any given time. 

    As it happens, I actually tried a version of your suggestion yesterday - I pre-allocated one frame buffer that I would give to MediaStreamSample.CreateFromBuffer each time, copying the new data into that frame buffer with each SampleRequested call. 

    Sure enough, no crash. I thought I'd solved the problem until I looked at the video, though. It effectively looked like approximately 1 frame per second video. What I think was happening in reality was that the MediaStreamSample buffers were all being linked to the same block of bytes - my frame buffer - and so whenever I updated those bytes, multiple samples got affected.

    Bottom line - if I use your approach, how do I know when it's safe for me to start re-using old samples? 

    Also, please see my last post with the new code sample that reduces the problem to the bare minimum. Maybe you'll see something else that I can't. 

    Thanks so much. Yours might be the answer if I can solve these questions.


    • Edited by pnm655 Friday, May 23, 2014 6:57 PM
    Friday, May 23, 2014 6:41 PM
  • More info, and this warrants its own post for sure. 

    Thanks to James' suggestion to use a revolving buffer I looked closer at the Processed handler for MediaStreamSample, answering my own question about how to know when the sample is available to be reused. It looks like the right way to do this is to maintain a list of, custom structures containing MediaStreamSamples and "isFree" flags, use the Process handlers to set samples as free, and then create new samples from free ones by calling MediaStreamSample.CreateFromBuffer, passing the buffer from the old sample. Perfect solution. EXCEPT:

    Before implementing the idea fully I wanted to get an idea of how many samples I would need to create, so I decided to just maintain a counter of outstanding samples, incrementing the counter on MediaStreamSample.CreateFromBuffer and decrementing it during the Processed handler. So,

    In the class:

            int _samplesInProcess = 0;
            int _samplesInProcessMax = 0;

    And in _mss_SampleRequested:

    MediaStreamSample targetSample = MediaStreamSample.CreateFromBuffer(frameBuffer.AsBuffer(), TimeSpan.FromTicks(_transcodePositionTicks));
                targetSample.Processed += (MediaStreamSample sampleSender, object sampleProcessedArgs) =>
                {
                    _samplesInProcess--;
                };
                targetSample.Duration = TimeSpan.FromTicks(_frameDurationTicks);
               
                _samplesInProcess++;
                _samplesInProcessMax = Math.Max(_samplesInProcess, _samplesInProcessMax);
    
                args.Request.Sample = targetSample;

    The idea is that _samplesInProcessMax would tell me the maximum number of samples that would ever  be outstanding, which I had assumed, and figured I was confirming, would correspond to the buffer size set by MediaStreamSource.BufferTime. That turned out not to be the case at all.

    Rather, after processing 10,000 frames at low-res (so it wouldn't crash), _samplesInProcessMax reported 2,944. In other words, there was at one point a 2,944 sample lag between SampleRequested getting called, and the sample being processed. That's about 1 GB worth of my 320x240x32 frames.

    NO WONDER it's running out of RAM!

    Further poking with breakpoints surprised me even more. The first time MediaStreamSample.Processed was called always seems to be based on the BufferTime, which is what I'd expect. That is, if I set the BufferTime to 1s, MediaStreamSample.Processed would be called with 30 _samplesInProcess. But after that things got very ugly. MediaStreamSample.Processed would only get called maybe 10 more times before the next sample was requested (in other words the buffer was not clear before the next sample was requested), but after that there would be a skip of 60 samples before MediaStreamSample.Process was called again, at which point there was an 80 sample backlog. Then Processed would be called a bunch more times, maybe 20, but then there would be a skip of 120 samples requested, so now the backlog was 180 samples. And so on. However, eventually the backlog plateaus at around the point that corresponds to 1GB of RAM. 

    There's clearly some kind of accumulating error here that is causing the transcoder not to process a sufficient number of frames before it asks for the new ones. It doesn't seem like a coincidence that the backlog roughly doubles for the first several rounds until critical memory limits are reached. 

    The question remains whether the bug is in my code or the framework. I'm certainly not above stupid and obvious mistakes, but I've been pouring over this problem for a week and in my experience when something is this hard to resolve it means there's a bug in the OS.

    I sure hope I'm wrong! But since this is stuff is new to Windows 8.1, it's entirely plausible this is a bug that no one's yet caught because no one's tried to use a MediaStreamSource to create HD renderings with big enough frames to cause a crash. 

    If it is a bug, what are my options?   Also what's my prize for finding it? :)

    Peter



    Friday, May 23, 2014 8:55 PM
  • Hello,

    I'm not sure what is occurring in your code but hopefully it is just something simple that you are missing. I can look at this further but to expedite the process I would ask that you please create a Visual Studio 2013 sample project that is a simple as possible but still reproduces the problem. Zip up the project and upload it to your OneDrive. Once up loaded please post a link here. I will grab the project and take a look at it next week.

    Thanks,

    James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Friday, May 23, 2014 11:37 PM
    Moderator
  • James and all -

    SUCCESS at last (I think). The solution was, in fact, to set MediaStreamSource.BufferTime to zero ticks. If BufferTime is set to zero ticks, MediaStreamSource seems never to ask for more than 1, or at most 2, samples before processing the sample it already has. However, if BufferTime is anything other than zero, MediaStreamSource will continue to ask for samples via SampleRequested, WITHOUT processing (and thus releasing) the ones it already has, at an exponentially increasing rate. This still clearly seems like a bug to me. Maybe it's limited to using MediaStreamSource with the transcoder. Perhaps the MediaStreamSource documentation omitted something the MS programmers just assumed, i.e., that when transcoding there's no need to buffer the MediaStreamSource. If that's the case, it's probably something that needs to be documented. :)

    Anyway, I just thought I'd share the balance of my experience for the benefit of others. What I did was make a SamplePool class that was basically a list containing structures consisting of a MediaStreamSample and a"bool IsAvailable" member. The SamplePool also has a member "MakeSample." That member does the following: (1) look to see if there are any available samples in the pool; (2) if not, add one using MediaStreamSource.CreateFromBuffer; (3) copy the new sample data into the buffer of the available one, then re-create it CreateFromBuffer, re-using the old buffer and the new sample time; (4) mark the sample as unavailable.

    The other critical step is to handle the Processed event; what that does is go back to the sample pool and free up (by changing IsAvailable) the sample that was released , PLUS (and this seems to be critical too) any earlier-timed sample in the pool. The reason for the latter step is that MediaStreamSource does indeed seem occasionally to skip releasing samples. If a sample's been released via the Processed handler, it appears safe to assume any sample with an earlier presentation time should also be released.

    And so, with that, no more memory leak, and smooth video output.

    But wait, there's more. As is well known, some graphics algorithms use an inverted Y axis (rather than top down, positive coordinates point up). Guess what? My video was turning out UPSIDE DOWN. I could manually flip the buffer, but the problem is some video cards do not do this, and I can't figure out a way to know from any C#-friendly API. Thus, I re-wrote my SamplePool class in C++ and now create the internal media buffers myself, using MFCreate2DMediaBuffer, and setting the "fBottomUp" flag to false. For some unfathomable reason, using MediaStreamSource.CreateFromBuffer does NOT allow you to control whether the buffer is bottom up or top down, but using the media foundation functions does allow this. 

    And now, at last, I can properly create computer-rendered video using MediaTranscoder and MediaStreamSource!

    I'll post my code shortly after it's cleaned up and more presentable for you and all else to see. Anyway, thanks again for the info on re-using the buffer; I'd have never of figured that out otherwise. 

    Peter


    • Edited by pnm655 Sunday, May 25, 2014 3:33 PM
    Sunday, May 25, 2014 3:29 PM
  • Alrighty, here's the link to the code:

    MyTranscodeExample

    Now, currently it seems to be working. However, I'm a little less optimistic than I was in my last post because I'm still not exactly understanding what's going on. First and foremost, it seems to work fine with the C++ sample pool I wrote, or without it (i.e. creating a new MediaStreamSample each time). The key seems to be: (1) making the BufferTime 0, and (2) forcing a garbage collection at the end of each SampleRequest. If I do those two things, it apparently doesn't matter whether I use the SamplePool or just create a new sample each frame. Either way memory seems to stay steady and it doesn't crash.

    I guess using the C++ SamplePool is better because I can force it not to make the image upside down when I create the MF2DMediaBuffer. But there's a lot about MF I don't understand yet, so I'm a little uneasy using the code in a production environment. 

    So, whoever is willing to take a look at the project I posted, if you happen to see anything wrong in the C++ SamplePool please let me know. Or, if you know a way to prevent the image inversion without resort to the C++ class (besides manually flipping it), I'd also love to hear it.

    But, at least we seem to be making progress. One thing is certain - BufferTime MUST BE ZERO when transcoding from a MediaStreamSource.

    Sunday, May 25, 2014 11:08 PM
  • Ugh, forget it. Even when it doesn't crash, the transcode operation just randomly stops, totally silently, in the middle. No exception, no warning, and no rhyme or reason.

    I'm done. Sorry but MediaTranscoder and MediaStreamSource are just too unreliable for me. (And if you Google them, others have similar horror stories it turns out). 

    As it turns out, using Media Foundation to output rendered video frame by frame is pretty much a breeze. It's ugly dealing with all the COM calls, but you get used to it. And it's faster and way more reliable than the transcoder. I'm already well on the way to making a totally custom transcoder class in C++ and so far, knock on wood, it's working fine.

    Thanks to James and Rob for the responses. Maybe one of you will figure out what, if anything, I was doing wrong, but the Media Foundation method seems to be the much better approach. 

    • Marked as answer by pnm655 Tuesday, May 27, 2014 10:26 PM
    Tuesday, May 27, 2014 10:25 PM
  • Hello,

    Thanks for the feedback on the MediaTranscoder. We will put this on the list to investigate further.

    I'm glad that you were able to find a solution. Let us know if you run into any other problems along the way.

    -James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Thursday, May 29, 2014 10:01 PM
    Moderator
  • For sure! Actually my C++ video writer is basically done so I thought I'd share a couple more observations. I realize this really isn't for this forum but I'm not sure how to PM you. :)

    I noticed in the MF documentation that you can enable or disable "throttling" on the media sink, and the docs say this is normally enabled to prevent the app from delivering too many samples. That sure seems to be exactly what was going wrong with the transcoder/MediaStreamSource.

    The other thing is that if I don't release both the sample and the buffer after IMFSinkWriter->WriteSample, the nasty memory leaks return with a vengence. Fortunately I don't need asynchronous functionality for this particular application (it's a dedicated video render and the app informs the user it's going to be unresponsive until it's done but for the cancel button), but I can easily imagine if you did use MF in asynchronous mode then trying to coordinate the release of used samples, and not overwhelm the garbage collector in the C# thread at the same time, could be a nightmare. Again, just the sort of problem I was experiencing with the RT transcoder.

    Hope this helps. If you'd like I can privately send you my C++ code for my frame writer. (I sortof feel like it's a trade secret since I spent so much damn time figuring it out! :))

    Friday, May 30, 2014 3:26 PM
  • Please feel free to contact me through my personal blog:

    http://blogs.msdn.com/b/mediasdkstuff/

    -James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Monday, June 02, 2014 11:36 PM
    Moderator
  • I am running in to the same exact scenario you are, especially with too many requests for samples before they are processed and then too much RAM being consumed before the whole process craps out due to insufficient RAM, even with constant calls to GC.Collect. I think this is a bug and hope they fix it soon.
    Monday, July 07, 2014 3:16 AM
  • Hey Scot - Glad to know it wasn't just me. :) For what it's worth, though, using C++ it's working fine. I know the MF extensions are a pain to use with all the COM lingo, but while the code is tedious it's nothing particularly complex. The key (for me at least; recall I was transcoding) was that it let me allocate only one frame, ever, processing it and overwriting it once processing was done. Because I was controlling the encoder directly, I knew exactly when it was done with a frame and when it was safe to overwrite. You just have to make sure to do all the correct COM release calls.

    The one drawback is that it has to be done synchronously, which obviously kills your app's performance and is exactly what you want to avoid in a tablet app. It's also inefficient since in theory you should be making the next frame while you're waiting for the GPU/disk to process the previous one. And it's probably not an option if you're trying to feed a MediaElement. The only way I know how to do that is the way they do it in the sample, feeding the sample requested args straight to a C++ module that does all the heavy lifting.

    Either way, the bottom line appears to be that you just can't implement a MediaStreamSource entirely in C#. Between the non-deterministic memory allocation and the (apparently) non-deterministic nature of the media foundation libraries insofar as you can't seem to control how many frames it requests before it finishes with the ones it's already been given, you'll always have those two forces competing.

    Monday, July 07, 2014 9:54 PM
  • you deserve a medal!!
    Monday, March 30, 2015 12:28 PM
  • Haha, well thank you that's very kind. But amazingly coincidentally, I just had to revisit this problem this past week. It turns out even my C++ based transcoding was having problems when things were going too fast. I was pulling my hair out for several days, trying everything I could think of. Searching the Internet, it's clear that lots of people are experiencing this issue, but no one has an answer.

    In the end, nothing could stop the memory leaks. Media Foundation has basically proven unusable to me for writing (it still seems to work ok for read ops, for now). So, just this weekend, I went through the painstaking process of building an LGPL-compatible build of the ffmpeg libraries, and will be using that for my write operations. And gee, what do you know, no memory leaks whatsoever. (Fortunately they released a WinRT build not too long ago).

    So that's that. 


    Monday, March 30, 2015 6:52 PM