camera video + overlayed DOM elements => video file, there must be a way! (HTML/JS)

Question

My goal is for my HTML/JS Windows Store app to do this:  1) render a video element that is previewing from the device/attached camera, 2) overlay DOM elements/controls; could be simple elements like DIVs with text/img or might be complex third party controls like animated SVG based graphs, charts, gauges, etc. 3) and here's the kicker, encode what you see to a video file.

The first two requirements are easy and look fantastic, but how do I get that great looking, real time 1080p imagery encoded to a file?  I've looked at writing an MFT in C++ to overlay the video, looked at drawing frames to a canvas and encoding to WebM, but the hang up is always the overlay because it isn't a bitmap, it's a collection of dynamic DOM elements.  Surely there is a way, I know there are others trying to do this.

Finally for bonus points, how can this all be done regardless of the screen resolution of the device it's running on (e.g. device display is less than 1080p, but it would be desirable to render the video + overlay @ 1080p resolution for the sake of the video encode, then scale it back to fit the display).

Good thing I'm not asking for much, eh?  Any, and I do many ANY help or direction, would be greatly appreciated.  I'm going in circles with this and only getting more and more confused as to how to approach it.

Answer 1

There isn't a good easy way to do this.

Narrowing this down you have two problems:

1) How to encode video from frame data. This can be done either with your MFT or WebM solutions. It sounds like you have a good handle on this one.

2) How to capture the DOM elements as bitmaps you can include in your frame. This is the tricky part. If they were in a Canvas you could use Canvas.getImageData to get the Canvas' pixels, but there is no way to directly capture arbitrary DOM elements on top of the Canvas. You could approximate this by enumerating the DOM and rendering the elements onto your canvas. I haven't tried it myself, but a search found an existing html2canvas solution which does this.

--Rob

Answer 2

On the one hand, it's nice to find that I'm not just missing something obvious, but on the other hand it's the answer I already feared.  I guess I have no choice but to use canvas exclusively for my overlay elements, then it's just a matter of making some choices about the encoding (JS/WebM or C++/MFT for a broader choice of encoding formats).

I wonder about the specialized world of broadcast video (which I know nothing about), how do they achieve all that sexy HD imagery?  Seemingly, any source can be blended/overlayed with anything else and wind up in every conceivable format by design.  Why should it be so much more difficult in computing?