none
[Lumia SKD 3.0][UWP]Applying Lumia Imaging SDK ChromaKeyEffect to a video stream RRS feed

  • Question

  • Hi,

    I am using the Lumia Imaging SDK VideoEffectSample code to apply an effect to a MediaCapture video stream. It is pretty simple with simple effects such a black and white or sepia, which work beautifully. But I cannot figure out how to get the ChromaKeyEffect to work so as to show an image underneath the processed video stream. I *think* it is working half way. I can get the ChromaKeyEffect to be applied to my live video stream and key out my selected color. However, the resulting alpha area is always black. I can't figure out how to make an image show up underneath. Of course the documentation is useless, and none of the samples actually use the ChromaKeyEffect with another image. Even the Lumia Imaging SDK EditShowcase sample app, which does contain the ChromaKeyEffect, keys to black. It does not attempt to set a Source image, if that is what you are supposed to do. So the sample is no help with this puzzle either.

    Here is a code snippet of my stumblings in the undocumented dark:

                Uri u = new Uri("ms-appx:///Assets/MoabBackground.jpg");
                StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(u);
                m_chromaKeyEffect = new ChromaKeyEffect(Color.FromArgb(255, 60, 215, 50), 0, 0, false);
                m_chromaKeyEffect.Source = new StorageFileImageSource(file);
                await CleanupCameraAsync();
                await InitializeWebcamAsync(true, false);

    If I replace the ChromaKeyEffect code above with a simple SeipaEffect, it works perfectly. Again, this code DOES produce a ChromaKeyEffect on the live video stream and it DOES key out the desired color, but the keyed areas are always black. (I have sliders that change the ColorDistance and NoiseSupression values, and so I know that it all appears to work...except for the background. The blackness correctly follows the contours of my key color in the live video preview.)

    Then it occurred to me that maybe the StorageFileImageSource didn't contain the image data when it was needed, so I tried this code, which was based on code from the EditShowcase sample.

                using (IRandomAccessStream photoStream = await file.OpenAsync(FileAccessMode.Read))
                {
                    var decoder = await BitmapDecoder.CreateAsync(photoStream);
                    using (var inMemoryStream = new InMemoryRandomAccessStream())
                    {
                        var encoder = await BitmapEncoder.CreateForTranscodingAsync(inMemoryStream, decoder);
                        encoder.BitmapTransform.ScaledWidth = (uint)PreviewControl.Width;
                        encoder.BitmapTransform.ScaledHeight = (uint)PreviewControl.Height;
                        await encoder.FlushAsync();
                        inMemoryStream.Seek(0);
                        RandomAccessStreamImageSource imageStream = new RandomAccessStreamImageSource(inMemoryStream);
                        //m_chromaKeyEffect = new ChromaKeyEffect(Color.FromArgb(255, 60, 215, 50), 0, 0, false);
                        m_chromaKeyEffect = new ChromaKeyEffect(clr, 0, 0, false);
                        m_chromaKeyEffect.SetSource(0, imageStream);

    But this doesn't work either. Same result. Apparently this is simply not how one sets the image that one wants to appear behind the keyed video stream, because it is always black.

    It also occurred to me that maybe the video stream itself has transparent regions, so I tried putting my background image underneath it in the XAML, but that wasn't the solution either.

    Can someone show me how it is supposed to be done?

    Thank you for your help!


    Thaine Norris


    • Edited by Fred Bao Thursday, November 26, 2015 8:58 AM add the tag
    Sunday, November 22, 2015 6:23 AM

Answers

  • Hi Thaine,

    I learned something new today, and hopefully I understand it well enough to be able to explain it to others. :)

    It seems that the best way to chain effects to meet your needs, is to use the EffectGroupBase base class to wrap around the ChromaKeyEffect and the BlendEffect. So, with the help of some folks from the Lumia Imaging SDK team, I created this class:

    class ChromaKeyBlendEffect : EffectGroupBase
    {
    	private BlendEffect _blendEffect;
    	private ChromaKeyEffect _chromakeyEffect;
    
    	public ChromaKeyBlendEffect(Color color, double colorDistance, double noiseSuppression)
    	{
    		// ChromaKeyEffect will operate on the camera, making color pixels transparent
    		_chromakeyEffect = new ChromaKeyEffect(color, colorDistance, noiseSuppression);
    
    		// Blend effect will use output of ChromaKeyEffect as the foreground
    		_blendEffect = new BlendEffect();
    		_blendEffect.ForegroundSource = _chromakeyEffect;
    	}
    
    	public IImageProvider BackgroundSource
    	{
    		get { return _blendEffect.Source; }
    		set { _blendEffect.Source = value; }
    	}
    
    	protected override IImageProvider PrepareGroup(IImageProvider groupSource)
    	{
    		// Connect the group source (this would be the camera feed) to the chromakey effect, the "input"
    		_chromakeyEffect.Source = groupSource;
    
    		// Return the "output" effect
    		return _blendEffect;
    	}
    }

    Which I then add to my MediaCapture instance like so:

    private async Task AddChromaKeyBlendEffect()
    {
    	var file = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///Assets/your_background_image_here.jpg"));
    	var storageFileImageSource = new StorageFileImageSource(file);
    
    	_effect = new ChromaKeyBlendEffect(Color.FromArgb(255, 206, 85, 104), 0.05, 0.1);
    	_effect.BackgroundSource = storageFileImageSource;
    
    	var propertySet = new PropertySet();
    	propertySet.Add(new KeyValuePair<string, object>("Effect", _effect));
    
    	var definition = new VideoEffectDefinition("Lumia.Imaging.VideoEffect", propertySet);
    	await _mediaCapture.AddVideoEffectAsync(definition, MediaStreamType.VideoPreview);
    }

    Then, whenever color RGB(206, 85, 104) is visible in the camera feed, it will instead be replaced with the pixels from your_background_image_here.jpg.

    There may be some issues with the aspect ratio being distorted if the image's and camera feed's don't match, but you should be able to control that through other means (GetMediaStreamProperties can tell you the aspect ratio of your camera feed).

    Hopefully this will meet your needs! Please upvote this post if it was helpful, and accept it as an answer if it solves your initial problem. If not, I'd be glad to help you further, and you're always welcome to start a new thread for other issues. See you around!

    • Marked as answer by dtnorris Sunday, December 13, 2015 12:54 AM
    Friday, December 11, 2015 6:33 PM

All replies

  • What's your OS version? wp8.1 or windows 10 mobile?

    This document might be helpful to you: Chroma Key Demo for Windows Phone

    Tuesday, November 24, 2015 9:43 AM
  • Hi,

    Thank you for your reply. I hope I have posted to the correct forum, "Developing for Universal Windows Apps," since I am writing a UWP app for Windows 10 and Windows 10 Mobile. The behavior I described is the same on the Windows 10 desktop and the Windows 10 Mobile environments.

    I looked at that sample already. It is for Windows 8.0 and it works completely differently than the Lumia Imaging SDK VideoEffectSample (for UWP apps) does, and is not compatible with UWP apps at all. For example, the Windows 8.0 Lumia Imaging SDK 2.0 uses ChromaKeyFilter. The new 3.0 SDK uses ChromaKeyEffect.

    The two UWP samples; VideoEffectSample and EditShowcase get very close to answering the question. As I said, EditShowcase implements the ChromaKeyEffect effect, but it does not complete the sample, does not attempt to set a background image, and also keys to black just like my code.


    Thaine Norris

    Tuesday, November 24, 2015 3:29 PM
  • Hello dtnorris,

    Instead of using the Lumia SDK, you could have a try with the win32 api to make the chroma key effect:

    https://msdn.microsoft.com/en-us/library/windows/desktop/dn890715(v=vs.85).aspx

    Regards.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, November 26, 2015 9:04 AM
  • Hi Fred,

    Thank you for your reply. That is an interesting option. It may be what is actually behind the Lumia SDK ChromaKeyEffect. But the Lumia SDK has a very nice C# interface and my app already uses dozens of its filters, which are buttery smooth and fast. If someone familiar with the Lumia SDK could simply show me how to make the background image on the existing ChromaKeyEffect work, it would be just a few lines of code I'm sure. As I said, the key effect works beautifully, only it keys to black.

    However, to use the Win32 effect in my C# UWP app would require a C# wrapper around a separate C++ project in my solution, and it is not my area of expertise. The previous version of my project had exactly this, with a custom MFT written in C++ which is applied to the video stream with AddEffectAsync. It worked, but it was nowhere near as smooth and fast as the Lumia SDK, which is why I was SO EXCITED to use the toolkit. Is there no one who can tell me how to use it properly? What about the engineers who wrote the EditShowcase Lumia SDK 3.0 sample?

    That said, do you have an example of how I would use the Win32 CLSID_D2D1ChromaKey effect in my C# project?

    Thank you for your help.


    Thaine Norris

    Thursday, November 26, 2015 3:23 PM
  • >> That said, do you have an example of how I would use the Win32 CLSID_D2D1ChromaKey effect in my C# project?

    Unfortunately, I do not have such a sample, you could post a sample request to:

    https://code.msdn.microsoft.com/site/requests


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, November 30, 2015 9:41 AM
  • Hi Fred,

    I appreciate the various suggestions of how to do the Chroma Key effect without the Lumia SDK. But can't someone find out how the Lumia SDK 3.0 is supposed to be used? Why won't anyone answer my question directly?

    It's not like the Lumia SDK is some third party project. This is a Microsoft product! Pressing F1 in Visual Studio on "ChromaKeyEffect" takes you to the MSDN website! 

    Please help! Please show me how to use the ChromaKeyEffect in the Lumia SDK 3.0 for a UWP app, not something else.


    Thaine Norris



    • Edited by dtnorris Wednesday, December 2, 2015 11:31 PM
    Wednesday, December 2, 2015 3:04 PM
  • Hello,

    For lumia imageing sdk issue, I am trying to invoke someone experienced to have a check it and as soon as we get any result, we will post a update to here.

    Regards.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, December 3, 2015 5:54 AM
  • Hi Thaine!

    It sounds like you're trying to create an app that can do something similar to what the weather report software does on TV: replace a green screen with an image of your choosing, correct? If so, it sounds like the ChromaKeyEffect is a good starting point.

    I created a test project to see how it behaves, and I can confirm your findings, the color appears to be replaced with black. I'm not sure this is an issue with the ChromaKeyEffect itself, because it appears to behave correctly in the EditShowcase SDK sample: Just change the background color on the FilterExplorerView.xaml page, and you'll see. I believe this is a limitation of either the camera pipeline or the CaptureElement.

    Have you tried looking into the BlendEffect? It sounds to me that if you chain it after the ChromaKeyEffect, and then manage to set your desired replacement pixels as the background and the camera feed as the foreground, it might do what you need. That being said, I was unable to do so, in my first time ever playing with the Lumia Imaging SDK. You might be more lucky.

    If that fails too, give the Lumia Imaging SDK Win2D Demo sample a look. I'm pretty sure Win2D allows complex compositing of frames, and it should allow you to operate on pixels in your own code. Worst case scenario: you replace all black pixels with pixels from your desired background image.

    Monday, December 7, 2015 9:53 PM
  • Hi Mike,

    Thank you for your reply! I sure appreciate your trying this out yourself. I can try experimenting with the blend effect. I have several chained effects in my app already, but they are all effects applied to still images. Maybe there is some hope there.

    Regarding the Win2D demo. In looking at the code, it seems like it works pretty much the same way that my code does when it applies the effect to the video stream. It uses AddVideoEffectAsync and the video effect class uses an effect from the same Lumia.Imaging.Adjustments namespace. The Win2D aspect of the sample is when it applies a 3D transform to the video stream. Maybe there is something I can do there in the frame by frame processing.

    Is there anyone who actually worked on the Lumia SDK sample code or on the SDK who could definitively answer the question of whether the ChromaKeyEffect should work with AddEffectAsync on a webcam stream? Yes or no?

    Thank you again for your help,


    Thaine Norris

    Tuesday, December 8, 2015 4:13 AM
  • Hi Thaine,

    I'm reaching out to the team that owns the Lumia Imaging SDK, and we'll keep investigating on how to cover this scenario. I'll let you know when we have an update.

    Thursday, December 10, 2015 11:05 PM
  • Hi Thaine,

    I learned something new today, and hopefully I understand it well enough to be able to explain it to others. :)

    It seems that the best way to chain effects to meet your needs, is to use the EffectGroupBase base class to wrap around the ChromaKeyEffect and the BlendEffect. So, with the help of some folks from the Lumia Imaging SDK team, I created this class:

    class ChromaKeyBlendEffect : EffectGroupBase
    {
    	private BlendEffect _blendEffect;
    	private ChromaKeyEffect _chromakeyEffect;
    
    	public ChromaKeyBlendEffect(Color color, double colorDistance, double noiseSuppression)
    	{
    		// ChromaKeyEffect will operate on the camera, making color pixels transparent
    		_chromakeyEffect = new ChromaKeyEffect(color, colorDistance, noiseSuppression);
    
    		// Blend effect will use output of ChromaKeyEffect as the foreground
    		_blendEffect = new BlendEffect();
    		_blendEffect.ForegroundSource = _chromakeyEffect;
    	}
    
    	public IImageProvider BackgroundSource
    	{
    		get { return _blendEffect.Source; }
    		set { _blendEffect.Source = value; }
    	}
    
    	protected override IImageProvider PrepareGroup(IImageProvider groupSource)
    	{
    		// Connect the group source (this would be the camera feed) to the chromakey effect, the "input"
    		_chromakeyEffect.Source = groupSource;
    
    		// Return the "output" effect
    		return _blendEffect;
    	}
    }

    Which I then add to my MediaCapture instance like so:

    private async Task AddChromaKeyBlendEffect()
    {
    	var file = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///Assets/your_background_image_here.jpg"));
    	var storageFileImageSource = new StorageFileImageSource(file);
    
    	_effect = new ChromaKeyBlendEffect(Color.FromArgb(255, 206, 85, 104), 0.05, 0.1);
    	_effect.BackgroundSource = storageFileImageSource;
    
    	var propertySet = new PropertySet();
    	propertySet.Add(new KeyValuePair<string, object>("Effect", _effect));
    
    	var definition = new VideoEffectDefinition("Lumia.Imaging.VideoEffect", propertySet);
    	await _mediaCapture.AddVideoEffectAsync(definition, MediaStreamType.VideoPreview);
    }

    Then, whenever color RGB(206, 85, 104) is visible in the camera feed, it will instead be replaced with the pixels from your_background_image_here.jpg.

    There may be some issues with the aspect ratio being distorted if the image's and camera feed's don't match, but you should be able to control that through other means (GetMediaStreamProperties can tell you the aspect ratio of your camera feed).

    Hopefully this will meet your needs! Please upvote this post if it was helpful, and accept it as an answer if it solves your initial problem. If not, I'd be glad to help you further, and you're always welcome to start a new thread for other issues. See you around!

    • Marked as answer by dtnorris Sunday, December 13, 2015 12:54 AM
    Friday, December 11, 2015 6:33 PM
  • Wow! Thank you for working on this.

    I have successfully implemented this in my UWP app and verified that it works on both Windows Mobile and Windows Desktop. It is super smooth and fast!

    Now that I see how it's done, it makes sense. But I don't know that I would have ever figured it out from the existing documentation and samples.

    Thanks again!


    Thaine Norris


    • Edited by dtnorris Sunday, December 13, 2015 12:57 AM
    Friday, December 11, 2015 8:52 PM