none
Getting colors from ColorArray RRS feed

  • Question

  • Hi, 

    I am using the Unity SDK to build a point cloud from Kinects' data. I have succesfully set the positions of a particle system's particles from depth data but I have trouble extracting color. I am starting from the CoordinateMapperManager.cs file.

    This is my code:

    void ProcessFrame() { var pDepthData = GCHandle.Alloc(pDepthBuffer, GCHandleType.Pinned); var pDepthCoordinatesData = GCHandle.Alloc(m_pDepthCoordinates, GCHandleType.Pinned); var pColorData = GCHandle.Alloc(pColorBuffer, GCHandleType.Pinned); // Vertex and color lists List <Vector3> vertexList = new List<Vector3>(); List <Color32> colorList = new List<Color32>(); for (int i = 0; i < cDepthWidth; i++) { for(int j = 0 ; j < cDepthHeight ; j++){ int body_index = j*cDepthWidth + i; // If theres a user in the pixel if( pBodyIndexBuffer[body_index] != 0xff){ int depth = pDepthBuffer[ body_index]; // Depth information Vector3 vertex = new Vector3( depth * depthLookupTableVec2[body_index].x , depth * depthLookupTableVec2[body_index].y , depth) ; vertexList.Add ( vertex ); // Color information DepthSpacePoint depthPoint; depthPoint.X = i; depthPoint.Y = j; ColorSpacePoint colorPoint = m_pCoordinateMapper.MapDepthPointToColorSpace(depthPoint, (ushort)depth); int color_index_x = (int) (colorPoint.X + 0.5f); int color_index_y = (int) (colorPoint.Y + 0.5f); byte r = pColorBuffer[color_index_y*cColorWidth + color_index_x]; byte g = pColorBuffer[color_index_y*cColorWidth + color_index_x + 1]; byte b = pColorBuffer[color_index_y*cColorWidth + color_index_x + 2]; byte a = pColorBuffer[color_index_y*cColorWidth + color_index_x + 3]; Color32 color = new Color32(r, g, b, a); colorList.Add(color); } } } //print(sample_byte); // Particle system ParticleSystem.Particle[] particles = new ParticleSystem.Particle[ vertexList.Count]; myParticleSystem.GetParticles(particles); for(int i = 0 ; i < vertexList.Count ; i++){ particles[i].position = vertexList[i]* 0.01f; particles[i].startColor = colorList[i]; setDefaultsForParticle(particles[i]); } if (getSnapshot == true) { // Grab 30 values for(int i = 0; i < 30 ; i++){ int particleIndex = UnityEngine.Random.Range(0, particles.Length); print ( particles[particleIndex].position ); print( "Colors"); print(particles[i].startColor.r); print(particles[i].startColor.g); print(particles[i].startColor.b); print(particles[i].startColor.a); } getSnapshot = false; } myParticleSystem.SetParticles (particles, particles.Length); myParticleSystem.Emit (particles.Length);

    }


    I wonder if I am sampling incorrectly the color image. I am obtaining red, green and blue lines of color:

    https://imgur .com/a/e3Hy9

    Any clues on how to solve this?


    Monday, September 11, 2017 1:35 PM

Answers

  • Actually they are.These calculations always confuse me for some time(1D access of a 2D resource).

    Your pColorBuffer is of a type that has a size of 4bytes right? Like an int[] or a float[]. If so, then accessing it using pixel coordinates(color_index_..) would give you an int or a float back, the whole color's worth of bytes.

    You will have to convert it to bytes with something like this, or perhaps careful use of byte operations.

    If pColorBuffer is a byte array though then , I think it's color_y*cColorWidth*cColorBytePerPixel +color_x*cColorBytePerPixel ,where cColorBytePerPixel is 4,to access the red channel and +X for g,b,a. You only calculated the pixel part but forgot to take into account the size in bytes of the pixel for all those bytes you're passing over.


    Tuesday, September 12, 2017 8:22 AM

All replies

  • First off, why are you pinning memory inside ProcessFrame? You want to do it on Awake/Start , not on a per-frame call.

    Secondly, when copying over the new frame's color to the buffer, are you doing a CopyRaw... or a CopyConverted... call? I mean, are you sure the data copied over from Kinect are converted to an RGBA format?

    Monday, September 11, 2017 2:16 PM
  • Thank you Nikolau.

    1. While I do understand that the 

    var pDepthData = GCHandle.Alloc(pDepthBuffer, GCHandleType.Pinned);

    line avoids the garbage collector to collect pDepthBuffer, it is not clear for me why this should be done or not in a per-frame basis. I kept it here because this is how it is done in the examples. I just added the line for the color data.

    2. I am indeed converting to RGBA format in the update function:

    var pColorData = GCHandle.Alloc (pColorBuffer, GCHandleType.Pinned);
    pColorFrame.CopyConvertedFrameDataToIntPtr(pColorData.AddrOfPinnedObject(), (uint)pColorBuffer.Length, ColorImageFormat.Rgba);
    pColorData.Free();

    I wonder if my lines for accessing the color data are correct...

    Monday, September 11, 2017 3:24 PM
  • 1) Since C# is managed, everything can be collected by the GC. Every once in a while it collects stuff. To avoid data being collected without your permission you pin memory, meaning you notify the GC that this chunk of memory is not to be touched until further notice. The ID that describes that memory is the GCHandle. Ideally you want to pin memory once on initialization(In Unity Awake/Start) and free it once on destroy(OnDestroy/OnApplicationQuit).

    I asked because ProcessFrame is probably a function that is being called in either Update or LateUpdate. Even in the samples, it's a function that is called at least one per frame. So what you're doing is spamming the GC about memory being pinned and being freed a lot. So it doesn't make much sense. And there is the hypothetical scenario that the GC does collect the data and incurs a spike because it just so happened in between Free and Alloc.

    Always Free as many as you Alloc!!!

    2)CopyConvertedFrame seems ok to me.

    Have you tried dumping the buffer in a texture and checking the image?

    Monday, September 11, 2017 6:31 PM
  • Thank you Nikolaos!

    I'll correct the GCHandle line.

    I tried dumping the color array in a texture and it is correct. I wonder if the problem lies in how i am accesing the pixels:



    byte r = pColorBuffer[color_index_y*cColorWidth + color_index_x];
    byte g = pColorBuffer[color_index_y*cColorWidth + color_index_x + 1];
    byte b = pColorBuffer[color_index_y*cColorWidth + color_index_x + 2];
    byte a = pColorBuffer[color_index_y*cColorWidth + color_index_x + 3];

    Color32 color = new Color32(r, g, b, a);


    Tuesday, September 12, 2017 4:37 AM
  • Actually they are.These calculations always confuse me for some time(1D access of a 2D resource).

    Your pColorBuffer is of a type that has a size of 4bytes right? Like an int[] or a float[]. If so, then accessing it using pixel coordinates(color_index_..) would give you an int or a float back, the whole color's worth of bytes.

    You will have to convert it to bytes with something like this, or perhaps careful use of byte operations.

    If pColorBuffer is a byte array though then , I think it's color_y*cColorWidth*cColorBytePerPixel +color_x*cColorBytePerPixel ,where cColorBytePerPixel is 4,to access the red channel and +X for g,b,a. You only calculated the pixel part but forgot to take into account the size in bytes of the pixel for all those bytes you're passing over.


    Tuesday, September 12, 2017 8:22 AM
  • Can't believe I made such a silly mistake!

    Thanks Nikolaos! I am able to get correct colors now. Quite surprinsingly, when I query the number of bytes per pixel of the color frame, the output is 2. Is this a mistake from the library?

    pColorFrame.FrameDescription.BytesPerPixel;


    Tuesday, September 12, 2017 2:44 PM
  • Not a mistake. I asked you if you used CopyConverted to see if you knew about YUV2.

    When you grab the immediate frame description of the Color source, it describes YUV2 which is half of RGBA , 2bytes.

    The raw data you get from the sensor are in YUV2 format due to its size. Faster to get through from the sensor to the Kinect service.

    Then you convert it to RGBA and do whatever you want.

    To get a proper frame description about the RGBA format you have to call CreateFrameDescription and give it the format you wish to query for. It will return the description for it.


    UPDATE: The samples query the color format of the source first and act accordingly. But they query for Bgra first. Supposedly there have been sensors that used a different source format so you have to query first and see whether the source outputs the format you want, in case you can call CopyRaw.. ,which is faster than CopyConverted since there's no conversion involved.

    Tuesday, September 12, 2017 3:50 PM
  • Nikolaus,

    Sorry to jump into this thread again. I'm trying to clean the code as much as I can. The update function of the samples uses several allocations and deallocations, both in Update() and later in ProcessFrame().


    	
    	void Update()
    	{
    		// Get FPS
    		elapsedCounter+=Time.deltaTime;
    		if(elapsedCounter > 1.0)
    		{
    			fps = frameCount / elapsedCounter;
    			frameCount = 0;
    			elapsedCounter = 0.0;
    		}
    
    		if (m_pMultiSourceFrameReader == null) 
    		{
    			return;
    		}
    
    		var pMultiSourceFrame = m_pMultiSourceFrameReader.AcquireLatestFrame();
    		if (pMultiSourceFrame != null) 
    		{
    			frameCount++;
    			nullFrame = false;
    
    			using(var pDepthFrame = pMultiSourceFrame.DepthFrameReference.AcquireFrame())
    			{
    				using(var pColorFrame = pMultiSourceFrame.ColorFrameReference.AcquireFrame())
    				{
    					using(var pBodyIndexFrame = pMultiSourceFrame.BodyIndexFrameReference.AcquireFrame())
    					{
    						// Get Depth Frame Data.
    						if (pDepthFrame != null)
    						{
    							var pDepthData = GCHandle.Alloc (pDepthBuffer, GCHandleType.Pinned);
    							pDepthFrame.CopyFrameDataToIntPtr(pDepthData.AddrOfPinnedObject(), (uint)pDepthBuffer.Length * sizeof(ushort));
    							pDepthData.Free();
    						}
    						
    						// Get Color Frame Data
    						if (pColorFrame != null)
    						{
    							var pColorData = GCHandle.Alloc (pColorBuffer, GCHandleType.Pinned);
    							pColorFrame.CopyConvertedFrameDataToIntPtr(pColorData.AddrOfPinnedObject(), (uint)pColorBuffer.Length, ColorImageFormat.Rgba);
                                pColorData.Free();
                            }
                            
                            // Get BodyIndex Frame Data.
                            if (pBodyIndexFrame != null)
                            {
    							var pBodyIndexData = GCHandle.Alloc (pBodyIndexBuffer, GCHandleType.Pinned);
    							pBodyIndexFrame.CopyFrameDataToIntPtr(pBodyIndexData.AddrOfPinnedObject(), (uint)pBodyIndexBuffer.Length);
    							pBodyIndexData.Free();
                            }
    
    
    					}
    				}
    			}
    
    			ProcessFrame();
            }
            else
    		{
    			nullFrame = true;
    		}
    	}
    
    	void ProcessFrame()
    	{
    		var pDepthData = GCHandle.Alloc(pDepthBuffer, GCHandleType.Pinned);
    		var pDepthCoordinatesData = GCHandle.Alloc(m_pDepthCoordinates, GCHandleType.Pinned);
    
    		m_pCoordinateMapper.MapColorFrameToDepthSpaceUsingIntPtr(
    			pDepthData.AddrOfPinnedObject(), 
    			(uint)pDepthBuffer.Length * sizeof(ushort),
    			pDepthCoordinatesData.AddrOfPinnedObject(), 
    			(uint)m_pDepthCoordinates.Length);
    
    		pDepthCoordinatesData.Free();
    		pDepthData.Free();
    
    		m_pColorRGBX.LoadRawTextureData(pColorBuffer);
    		m_pColorRGBX.Apply ();
    	}
    	
    	

    So, just to be sure, I should declare the handles in the global scope, allocate in Start() or Awake(), and free in OnApplicationQuit(). Am I right? (and get rid of the allocations and deallocations in frame and process frame)

    Something like this:

    	GCHandle pDepthData;
    	GCHandle pDepthCoordinatesData;
    	GCHandle pColorData;
    	GCHandle pBodyIndexData;
    
    	void Awake ()
    	{...
                    pDepthData = GCHandle.Alloc (pDepthBuffer, GCHandleType.Pinned);
    		pDepthCoordinatesData = GCHandle.Alloc (m_pDepthCoordinates, GCHandleType.Pinned);
    		pColorData = GCHandle.Alloc (pColorBuffer, GCHandleType.Pinned);
    		pBodyIndexData = GCHandle.Alloc (pBodyIndexBuffer, GCHandleType.Pinned);
    
    }
    ...
    
    
    	void OnApplicationQuit ()
    	{	
    
    		pDepthCoordinatesData.Free ();
    		pDepthData.Free ();
    		pColorData.Free ();
    		pBodyIndexData.Free ();
    }



    Friday, November 3, 2017 4:04 AM
  • Yap. You should be able to see the result in the Profiler Window. This script should be 0 bytes most of the time(Unless you do something else you can't avoid allocating for).
    Friday, November 3, 2017 8:07 AM