none
How to pass parameters to GetColorPixelCoordinatesFromDepthPixel RRS feed

  • Question

  • Can GetColorPixelCoordinatesFromDepthPixel be used to map the player index data (when using 'UseDepthAndPlayerIndex') to the rgb video?  From all of the research I've done, it seems this method is only useful for mapping the skeleton into the video space, but not the player index data? 

    I've tried a number of combinations, but cannot seem to get the depth with player index data lined up with the rgb video so can mask out the background and display only the player (i.e. simulate green screen).

    -Mike


    Solution Architect, Microsoft IEB Manufacturing, Supply Chain, and Information Solutions
    Monday, September 12, 2011 5:16 AM

Answers

  • There shouldn't be a parallax effect left over after using GetColorPixelCoordinatesFromDepthPixel method properly, but this method has proven notoriously confusing to call correctly. If you send me a code snippet of what you're doing, I can likely help you out. Have you seen the thread http://social.msdn.microsoft.com/Forums/en-US/kinectsdknuiapi/thread/c5c00b90-28d7-49f5-b451-d79609604ec0, which has a detailed discussion on this, with lots of sample code?

    Eddy


    I'm here to help
    Tuesday, September 13, 2011 12:48 AM
  • XNA uses the RGBA texture format while Kinect output is BGRA.

    You can either copy the Image data to a new array, switching the order. Or you can just use this shader to draw the image (SpriteBatch.Begin allows you to specify an Effect)

    sampler sprite : register(s0);
    
    /*
     * Convert from BGRA to RGBA
     */
    float4 BGR2RGB(float2 texCoord : TEXCOORD0) : COLOR0
    {
        float4 tex = tex2D(sprite, texCoord);
        return float4(tex.b, tex.g, tex.r, 1.0);
    }
    
    technique KinectVideo
    {
        pass KinectVideo
        {
            PixelShader = compile ps_2_0 BGR2RGB();
        }
    }
    

    Tuesday, September 13, 2011 5:52 AM

All replies

  • Mike, could you post some code showing us how you're calling this method?

    Using the SkeletalViewer sample as a starting point, you can modify the convertDepthFrame as follows to get the colorX/colorX coordinates in RGB image that correspond to depthX/depthY coordinates in depth image:

    byte[] convertDepthFrame(byte[] depthFrame16)
    {
        for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length; i16 += 2)
        {
            int depthPixelValue = (depthFrame16[i16 + 1] << 8) | depthFrame16[i16];
            
            int depthX = (i16 / 2) % 320;
            int depthY = (i16 / 2) / 320;
            int colorX, colorY;
            
            nui.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(ImageResolution.Resolution640x480, new ImageViewArea(), depthX, depthY, (short)depthPixelValue, out colorX, out colorY);
    
            colorX = Math.Max(0,Math.Min(639, colorX));
            colorY = Math.Max(0, Math.Min(479, colorY));
    }

    Does that make sense?
    Eddy


    I'm here to help
    Monday, September 12, 2011 6:35 PM
  • For what it's worth, I have tried a similar experiment as Mike, using code just about exactly like yours, Eddy.  (SkeletalViewer was my starting point as well.)

    Unfortunately, when the depth data is used to mask out the video image, there is a visible offset.  It looks like some kind of lateral parallax -- looking at the head, the depth pixels tagged with player 1 are left-shifted by about 1/3 the width of the head, resulting in a black fringe to the left of the head, and a chopped-off right third of the head.

    I am sorry I do not have a screen capture to share; I will work on getting one.

    I am (obviously) using UseDepthAndPlayerIndex, so it is not related to the "<< 3" issue with UseDepthIndex.

    Cheers,

    Rob


    Rob Jellinghaus
    Monday, September 12, 2011 10:06 PM
  • There shouldn't be a parallax effect left over after using GetColorPixelCoordinatesFromDepthPixel method properly, but this method has proven notoriously confusing to call correctly. If you send me a code snippet of what you're doing, I can likely help you out. Have you seen the thread http://social.msdn.microsoft.com/Forums/en-US/kinectsdknuiapi/thread/c5c00b90-28d7-49f5-b451-d79609604ec0, which has a detailed discussion on this, with lots of sample code?

    Eddy


    I'm here to help
    Tuesday, September 13, 2011 12:48 AM
  • I converted the sample code into an XNA app, and indeed it seems to work, providing an aligned rgb image with the depth map and player index.  The quality of the video data in the XNA version is poor (blue-tinged) compared to the WPF version - any ideas why?

    Thanks Eddy!     

    Here's the XNA version:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using Microsoft.Xna.Framework;
    using Microsoft.Xna.Framework.Audio;
    using Microsoft.Xna.Framework.Content;
    using Microsoft.Xna.Framework.GamerServices;
    using Microsoft.Xna.Framework.Graphics;
    using Microsoft.Xna.Framework.Input;
    using Microsoft.Xna.Framework.Media;
    using Microsoft.Research.Kinect.Nui;
    using System.Threading;
    
    namespace ColorDepthSync
    {
        /// <summary>
        /// This is the main type for your game
        /// </summary>
        public class Game1 : Microsoft.Xna.Framework.Game
        {
            GraphicsDeviceManager graphics;
            SpriteBatch spriteBatch;
    
            Runtime nui = null;
            Dictionary<int, ImageFrame> DepthList = new Dictionary<int, ImageFrame>();
            Queue<ImageFrame> VideoQ = new Queue<ImageFrame>();
            const int RED_IDX = 2;
            const int GREEN_IDX = 1;
            const int BLUE_IDX = 0;
            byte[] depthFrame32 = new byte[320 * 240 * 4];
            //private byte[] colorBuffer;
            Texture2D depthTexture;
            //Texture2D videoTexture;
            int windowWidth = 640;
            int windowHeight = 480;
    
            public Game1()
            {
                graphics = new GraphicsDeviceManager(this);
                graphics.PreferredBackBufferWidth = windowWidth;
                graphics.PreferredBackBufferHeight = windowHeight;
                Content.RootDirectory = "Content";
            }
    
            /// <summary>
            /// Allows the game to perform any initialization it needs to before starting to run.
            /// This is where it can query for any required services and load any non-graphic
            /// related content.  Calling base.Initialize will enumerate through any components
            /// and initialize them as well.
            /// </summary>
            protected override void Initialize()
            {
                nui = new Runtime();
    
                try
                {
                    nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);
                }
                catch (InvalidOperationException ex)
                {
                    throw ex;
                }
    
    
                try
                {
                    nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
                    nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);
                }
                catch (InvalidOperationException ex)
                {
                    throw ex;
                }
    
    
                base.Initialize();
            }
    
            ImageFrame FindMatchingVideo(int DepthFrameNumber)
            {
                // find the CLOSEST frame # in video. since we can drop frame, they won't always match.
                // If your machine can't process 100% of the frames, the queue will grow pretty large.
    
                if (VideoQ.Count == 0)
                    return null;
    
                ImageFrame qFrame = VideoQ.Dequeue();
                int qFrameNum = qFrame.FrameNumber;
                int MinDelta = Math.Abs(DepthFrameNumber - qFrameNum);
    
                while (VideoQ.Count > 0)
                {
                    // we found a video frame. But are there even newer frames in the queue that
                    // are even closer in time? If so, throw out the stale frame and use the newer one instead
    
                    ImageFrame qPeek = VideoQ.Peek();
                    int qPeekNum = qPeek.FrameNumber;
                    int PeekDelta = Math.Abs(DepthFrameNumber - qPeekNum);
    
                    if (PeekDelta < MinDelta)
                    {
                        qFrame = VideoQ.Dequeue();
                        qFrameNum = qFrame.FrameNumber;
                        MinDelta = PeekDelta;
                    }
                    else
                    {
                        // We peeked ahead. nothing is better than what we currently have. Stop looking.
                        break;
                    }
                }
    
                return qFrame;
            }
    
    
    
    
            /// <summary>
            /// LoadContent will be called once per game and is the place to load
            /// all of your content.
            /// </summary>
            protected override void LoadContent()
            {
                // Create a new SpriteBatch, which can be used to draw textures.
                spriteBatch = new SpriteBatch(GraphicsDevice);
    
                depthTexture = new Texture2D(GraphicsDevice, 320, 240, false, SurfaceFormat.Color);
                //videoTexture = new Texture2D(GraphicsDevice, 640, 480, false, SurfaceFormat.Color);
            }
    
            /// <summary>
            /// UnloadContent will be called once per game and is the place to unload
            /// all content.
            /// </summary>
            protected override void UnloadContent()
            {
                // TODO: Unload any non ContentManager content here
            }
    
            /// <summary>
            /// Allows the game to run logic such as updating the world,
            /// checking for collisions, gathering input, and playing audio.
            /// </summary>
            /// <param name="gameTime">Provides a snapshot of timing values.</param>
            protected override void Update(GameTime gameTime)
            {
                // Allows the game to exit
                if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed)
                    this.Exit();
    
                ImageFrame depthFrame = nui.DepthStream.GetNextFrame(0);
                ImageFrame videoFrame = nui.VideoStream.GetNextFrame(0);
    
    
    
                if (depthFrame != null)
                {
                    ImageFrame matchingVideo = FindMatchingVideo(depthFrame.FrameNumber);
                    if (matchingVideo != null)
                    {
                        updateDisplay(depthFrame, matchingVideo);
                    }
                    else if (!DepthList.ContainsKey(depthFrame.FrameNumber))
                    {
                        DepthList.Add(depthFrame.FrameNumber, depthFrame);
                    }
                }
    
    
                if (videoFrame != null)
                {
                    ImageFrame matchingDepth;
                    if (DepthList.TryGetValue(videoFrame.FrameNumber, out matchingDepth))
                    {
                        updateDisplay(matchingDepth, videoFrame);
                    }
                    else
                    {
                        // VideoList.Add(videoFrame.FrameNumber, videoFrame);
                        VideoQ.Enqueue(videoFrame);
                    }
                }
    
                base.Update(gameTime);
            }
    
            // Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
            // that displays different players in different colors
            byte[] convertDepthFrame(byte[] depthFrame16, byte[] videoFrame32)
            {
                for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length && i32 < depthFrame32.Length; i16 += 2, i32 += 4)
                {
                    int player = depthFrame16[i16] & 0x07;
                    int depthPixelValue = (depthFrame16[i16 + 1] << 8) | depthFrame16[i16];
                    int realDepth = depthPixelValue >> 3;
                    // transform 13-bit depth information into an 8-bit intensity appropriate
                    // for display (we disregard information in most significant bit)
                    byte intensity = (byte)(255 - (255 * realDepth / 0x0fff));
    
                    if (player == 0 || player == 7)
                    {
                        depthFrame32[i32 + RED_IDX] = (byte)(intensity / 2);
                        depthFrame32[i32 + GREEN_IDX] = (byte)(intensity / 2);
                        depthFrame32[i32 + BLUE_IDX] = (byte)(intensity / 2);
                    }
                    else
                    {
                        int depthX = (i16 / 2) % 320;
                        int depthY = (i16 / 2) / 320;
                        int colorX, colorY;
                        nui.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(ImageResolution.Resolution640x480, new ImageViewArea(), depthX, depthY, (short)depthPixelValue, out colorX, out colorY);
                        colorX = Math.Max(0, Math.Min(639, colorX));
                        colorY = Math.Max(0, Math.Min(479, colorY));
    
                        int colorIdx = 4 * (colorY * 640 + colorX);
    
                        depthFrame32[i32 + RED_IDX] = videoFrame32[colorIdx + RED_IDX];
                        depthFrame32[i32 + GREEN_IDX] = videoFrame32[colorIdx + GREEN_IDX];
                        depthFrame32[i32 + BLUE_IDX] = videoFrame32[colorIdx + BLUE_IDX];
    
                    }
                }
                return depthFrame32;
            }
    
            void updateDisplay(ImageFrame depthFrame, ImageFrame videoFrame)
            {
                PlanarImage Image = depthFrame.Image;
                byte[] convertedDepthFrame = convertDepthFrame(Image.Bits, videoFrame.Image.Bits);
    
                if (GraphicsDevice == null) return;
                GraphicsDevice.Textures[0] = null;
                depthTexture.SetData(convertedDepthFrame);
                //videoTexture.SetData(videoFrame.Image.Bits);
                
            }
    
            
    
            /// <summary>
            /// This is called when the game should draw itself.
            /// </summary>
            /// <param name="gameTime">Provides a snapshot of timing values.</param>
            protected override void Draw(GameTime gameTime)
            {
                
                GraphicsDevice.Clear(Color.Black);
    
                //Video image
                spriteBatch.Begin();
                spriteBatch.Draw(depthTexture, new Rectangle(0, 0, GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height), Color.White);
                //spriteBatch.Draw(videoTexture, new Rectangle(0, 0, GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height), Color.White);
                spriteBatch.End();
    
                base.Draw(gameTime);
            }
    
            protected override void OnExiting(object sender, EventArgs args)
            {
                nui.Uninitialize();
                base.OnExiting(sender, args);
            }
        }
    }
    
    

     


    Solution Architect, Microsoft IEB Manufacturing, Supply Chain, and Information Solutions

    • Edited by Mike Hatch Tuesday, September 13, 2011 4:49 AM
    Tuesday, September 13, 2011 4:37 AM
  • XNA uses the RGBA texture format while Kinect output is BGRA.

    You can either copy the Image data to a new array, switching the order. Or you can just use this shader to draw the image (SpriteBatch.Begin allows you to specify an Effect)

    sampler sprite : register(s0);
    
    /*
     * Convert from BGRA to RGBA
     */
    float4 BGR2RGB(float2 texCoord : TEXCOORD0) : COLOR0
    {
        float4 tex = tex2D(sprite, texCoord);
        return float4(tex.b, tex.g, tex.r, 1.0);
    }
    
    technique KinectVideo
    {
        pass KinectVideo
        {
            PixelShader = compile ps_2_0 BGR2RGB();
        }
    }
    

    Tuesday, September 13, 2011 5:52 AM
  • Excellent, thanks Dennis!  worked like a charm.

    For those unfamiliar with shaders:

    1. Save Dennis' shader code above into a '.fx' file in the Content folder (I called it BGR2RGB.fx)

    2. Declare an Effect variable, and set it in the LoadContent method

    protected override void LoadContent()
            {
                // Create a new SpriteBatch, which can be used to draw textures.
                spriteBatch = new SpriteBatch(GraphicsDevice);
    
                effectBGR2RGB = Content.Load<Effect>("BGR2RGB");
    
    ...
    

    3. In the Draw method, modify the spriteBatch begin call to use the effect:

    spriteBatch.Begin(SpriteSortMode.Texture, null, null, null, null, effectBGR2RGB);
    

    Note: I used SpriteSortMode.Texture as a random selection because XNA 4.0 does not accept null - not sure what the default value is (docs for XNA 4.0 are inaccurate - they show the 3.0 signature).




    Solution Architect, Microsoft IEB Manufacturing, Supply Chain, and Information Solutions
    Wednesday, September 14, 2011 4:45 AM