none
Kinect Sensors, Velocity and Acceleration Vector Fields RRS feed

  • General discussion

  • Greetings, I would like to describe some exciting research underway into velocity vector fields with Kinect (http://www.computing.surrey.ac.uk/personal/pg/S.Hadfield/sceneparticles.html).  I found an ICCV 2011 paper, Kinecting the Dots: Particle Based Scene Flow from Depth Sensors, by Simon Hadfield at the University of Surrey (http://personal.ee.surrey.ac.uk/Personal/R.Bowden/publications/2011/ICCV/Hadfield_ICCV11.pdf) and a video (http://www.youtube.com/watch?v=10CWOZcGMv0).  The video illustrates velocity vector fields from Kinect data and, when describing a CPU implementation, reports 9 seconds of processing per frame while related literature describes that a parallel GPU implementation with real-time performance is anticipated.

    Some topics pertaining to vector fields from Kinect sensors include the spatiotemporal superresolution of video, RGB+Velocity or RGBV video, and possibly other enhancements to video compression techniques.  Velocity and acceleration vector fields can open new avenues for algorithm design, including computer vision and video-related, while greatly enhancing the features available to Kinect developers.

    Kind regards,

    Adam Sobieski


    Wednesday, October 17, 2012 8:21 PM

All replies

  • Better video?

    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://www.computerprofessions.co.nr

    Thursday, October 18, 2012 12:29 PM
  • Some digital video topics include motion estimation and motion compensation.  Motion estimation determines vectors between bitmaps, and is useful when compressing video.  Challenges in motion estimation have included that motion was three-dimensional while video images were projections onto two-dimensional bitmaps.  Motion compensation utilizes motion data, motion vectors, to transform between bitmaps.  Contemporary formats such as H.264/MPEG-4 AVC and Dirac utilize motion-based digital video techniques.

    The super-resolution of video includes sharpening video, interpolating or extrapolating pixels when magnifying video without introducing blur, and increasing the FPS of video.  By means of super-resolution techniques, the Kinect sensor's RGB video, 640x480 @ 30 FPS, could be processed into HD video @ 60 FPS.  In addition to previous techniques for super-resolution, the depth stream data and computed data, such as surface geometry and velocity, can be of use to heuristics to super-resolve the video stream.

    Additionally, the raw Bayer stream data can be demosaicked for display modes with 8, 10, or more bits per channel.  Demosaicking and super-resolution are related topics; some demosaicking heuristics can make use of depth sensor data or computed data including surface geometry and velocity.

    Velocity and digital video are interrelated topics and the real-time velocity vector field developments with Kinect sensors are exciting with regard to both computer vision and digital video technologies.

    Kind regards,

    Adam Sobieski


    Friday, October 19, 2012 12:43 AM
  • Are you talking about in simpler terms making a 3d movie or video or about greenscreening? I just wanted to make sure I understood the general subject.

    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://www.computerprofessions.co.nr


    • Edited by The Thinker Friday, October 19, 2012 7:46 PM
    Friday, October 19, 2012 7:46 PM
  • In addition to 3D video and greenscreening, interesting scenarios include: video calls, video conferences, video blogs and video forums.  With the super-resolution of video possible from Kinect sensor data, video streaming and storage efficiencies can be enhanced.  Encoding raw Bayer streams into video is discussed in H.264-Based Compression of Bayer Pattern Video Sequences by Doutre, C., Nasiopoulos, P., and Plataniotis, K. N.  With super-resolution techniques, Kinect sensor streams, and 2d-plus-depth video, can be processed into a larger number of DirectShow and Media Foundation video formats.

    With regard to 3D video, some existing technologies include MPEG multiview coding, free viewpoint video, and 2D-plus-depth video.  2D-plus-depth video resembles the data from the Kinect sensor and 2D-Plus-Depth Based Resolution and Frame-rate Up-conversion Technique for Depth Video by Jinwook Choi, Dongbo Min, and Kwanghoon Sohn discusses related super-resolution topics.

    With regard to greenscreening, edge detection, silhouettes, visual hull construction, and topics in Silhouette-based 3D Model Reconstruction from Multiple Images by Adem Yasar Mulayim, Ulas Yılmaz, and Volkan Atalay, and Shape-From-Silhouette of Articulated Objects and its Use for Human Body Kinematics Estimation and Motion Capture by German K. M. Cheung, Simon Baker, and Takeo Kanade could enhance RGBD-based greenscreening techniques.

    Kind regards,

    Adam Sobieski

    Wednesday, October 24, 2012 3:59 PM
  • Kinect does have a green screen application and you might be able to play around with that to obtain a 3d green screening app. I would first take your video content and shot normally then transfer over to 3d. I have noticed that the 3d movies usually are the same movies with glasses to make it look 3d.

    P.S. Somewhere in a science magazine I saw an article on a company that is researching a hologram projector similar to the ones talked about in starwars.  I would help you experiment with the greenscreening app to help but I have some heaving coding going on myself at work (I do IT work in addition to my regular programming load at work). 


    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://www.computerprofessions.co.nr


    • Edited by The Thinker Wednesday, October 24, 2012 9:46 PM
    Wednesday, October 24, 2012 9:45 PM
  • I also saw that Kinect hologram projector project and research into holovideo; that is a cool project.

    In addition to stereoscopic 3D video (e.g. MPEG MVC), possible are 3D videos, calls and conferences, interoperable with head tracking, and 3D video formats based upon, instead of sequences of bitmaps, sequences of point clouds or 3D surfaces.

    While we have discussed super-resolving RGB video utilizing depth data, shape from shading is a technique to super-resolve depth data utilizing RGB video.  See also: Shape from Shading: A Survey by Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah and Recovering surface reflectance and multiple light locations and intensities from image data by S. Xu and A. M. Wallace.

    With velocity vector fields, and utilizing the timestamps from both the RGB and depth streams, point clouds or depth maps can be synchronized to the RGB data stream.

    With regard to the numerical precision of depth measurements and of velocity and acceleration vector fields, functions to obtain 3D probability distributions around each 3D point are interesting.  Accuracy Analysis of Kinect Depth Data by K. Khoshelham discusses the numerical precision of Kinect sensor depth measurements.  Accuracy analysis of depth measurements and of the timestamps of the depth and RGB data streams can be of use to numerical algorithms obtaining or utilizing depth measurements and velocity and acceleration vector fields.

    Kind regards,

    Adam Sobieski

    Thursday, October 25, 2012 2:52 AM
  • 3D with Kinect by Jan Smisek, Michal Jancosek and Tomas Pajdla also discusses the calibration of and accuracy analysis of Kinect sensors as well as structure from motion.

    Structure from motion techniques could additionally enhance depth data utilizing RGB video.  Some approaches to structure from motion could utilize depth measurements and velocity and acceleration vector fields.

    Kind regards,

    Adam Sobieski

    Friday, October 26, 2012 6:27 PM