locked
Media Compression support RRS feed

  • Question

  • By supporting the next wave of high performance media, won't there be a need for a new way to compress video/audio streams? Our current system is flawed, rendering entire videos into raw streams would require vast amounts of hard drive capacity. I propose a solution to the media storage problem.

    Given that Media currently is stored in a playable format, meaning you can click it and instantly start playing it in media player, or whatever player you use, why not develop a simple system for storing this content when not in use.

    Let's get technical, and talk a little bit of math.

    Let's say you transcode (hexdump) an entire video to a text file. This hex dump can be rebuilt by most hex dumping software, back into the video you dumped. Now, looking at the facts, a 1GB (1024 MB) video file would be  2048 MB in a hex dump, without any line breaks, or spaces (any character including the [return] function consumes an entire byte) that's a 200% inflation over the video file. Now, let's look at this hexdump a little more clearly. Let's say, we took each set of Hex Characters in the dump, (32 characters 4 bits per character representation)  00000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

    32^16 or 1,208,925,819,614,629,174,706,176 possibilities. let's say instead of using this system we switch to a more robust system.

    Let's say instead of 16 characters 0-F in hex, we where to custom build a hex dump that would use 0-9, A-Z, a-z and "," and "." at 6 bits per character, (meaning the  dump would only be 125% of the video files original size.)

    64 Characters representing 6 bits each, if we used all 64 characters we would represent 384 bits, unlike the 64 bits possible with hex (16 characters x 4 bits).

    using this system, once the 1GB movie file is converted to a 1.28GB Text File, that file can then be COMPRESSED using a standard compression routine (LMZA etc) to between 25-50% of it's original size. RECAP: 1 GB movie - 20 seconds - 1.28 GB text file - 10 mins later - 350-600 MB File.  NOW, before i get ahead of myself, let's say we tailor a special compression routine specifically for these 64 character text-dumps. let's say we give the character used the most the 0 bit, and the character used 2nd most the 1 bit (80% compression) that is usually where the most compression happens. NOW, let's say we seperate this file, into 64 shell files using space characters to represent the "missing" characters from the other files, making 32 special text files, with a different set of characters, but the same principle at hand.

    0-1 and a space character in file 1

    2-3 and a space character  in file 2 etc

    These smaller files, because they only use 3 different characters can be compressed up to 75-90%.

    ok, RECAP: 1GB Movie File ->1.28 GB text file processed one section at a time into special text files which are compressed and held in temp until all 64 are done, then merged into a special "holding" unit that can be used to automatically decompress -restore and reconfigure the movie file, allowing compression of up to 75-90% on a 1GB movie file.

    -William Dover

    Thursday, May 18, 2006 11:07 PM

Answers

  • Did you try that?

    If this works, what stops you from implementing the whole system?

    WMSDK tells you how to create your own IWMReader...

    I doubt it works though... There are entropy rules that are independent of representation base, that you are using as main argument of your proposal...

     

    Friday, May 19, 2006 6:43 AM

All replies

  • Did you try that?

    If this works, what stops you from implementing the whole system?

    WMSDK tells you how to create your own IWMReader...

    I doubt it works though... There are entropy rules that are independent of representation base, that you are using as main argument of your proposal...

     

    Friday, May 19, 2006 6:43 AM
  • EDIT: my first reply had alot of funky math in it, that's what i get for posting at 5:30 in the morning.

    Ok, the first part of this topic was assuming it was possible to break down the one giant text file, each character representing 6 bits from the bitstream, into 64 seperate files with only 3 characters in each file. Making it easier to compress, but the file would be just as big as the original file.

    Regardless, i have figured out a solution to the original format i envisioned above. Instead of using a normal Ansii text file, let's create a NEW type of format, let's call it Multi-Block-Cataloging or MBC Text. This format will seperate every character in the file into a "block" or "cell" Not unlike Excel, but instead of using the normal Row/Column combination, let's give every cell the usual line (i) number, found in most text files, first cell would be i=1, 2nd cell would be i=2, etc. this will eliminate the need for the [return] byte, and still allow a format easily read by a computer. Now let's deem the first row of cells, z= 1, etc., let's put 256 blocks in every z row.

    Z #1 would have blocks(i) 1-256

    Z #2 would have blocks(i) 257-512

    all the way down until we meet the count of lines needed for a 1,024,000,000 byte file

    which according to my  calculations at 256 characters per line, each character representing 6 bits, would be 8,192,000,000 bits / 6 bits = 1365333334 characters / 256 characters per line = 5,333,334 lines, or  Z's, at 256 cells per Z.

    Now, we fill in all the necessary characters into the file. 1,365,333,334 characters makes this file pretty heavy, but this format recognizes that each of the characters that are input only take up the specific binary pattern that the conversion table weighs them as (6 binary bits), so it sets them back down to their original binary pattern in each cell, meaning this text format only supports the 64 character set made in this particular formating proccess, and nothing else.

    Now we are back down to the file size that we had originally, and each character is back at it's normal binary weight, but now that it's in text, it can be coordinated and compressed using a special sequencing algorythm by use a special GUI program that locks each character into a special cell then physically (think of it like 3 desktop icons, when the middle icon is moved, the 2 other icons remain exactly where they where) moves each of the character sets to different areas of the file, and gives each of them a weight of 1 bit in refrence to the character they are stored under. Do this for all 64 patterns, compressing each square down to 1 bit and referncing that bit to the character it's stored under. The program will know exactly where each cell would sit natively, allowing for easy reformation of the original file, under the appropriate binary weights.

    z1  | cell 1|  |cell 2|  |cell 3|  |cell 4|  |cell 5|  |cell 6|

     

    Area 1 character 0  (000000 bit pattern)

           1 bit                      1 bit                            1 bit

          |cell 1 |                |cell 3|                           | cell 6|

    Area 2 character 1 (000001 bit pattern)

                        1 bit                 1 bit      1 bit 

                     |cell 2|               | cell 4|   |cell 5|

    Area 3 Charater 2 (000010 bit pattern)

                                                                                         1bit       1bit

                                                                                        |cell 7|   |cell 8|

     

    the blank spaces don't take up any more space, because each cell is built into the program with it's own coordinates on the cell map.

    Ok, so to recap, using a special "cell" technology to compress a 1 GB file down into 6 bit patterns (64 character set) and referncing each of those bits to a cell in a special program which recognizes their actual weight, and uses a special cell structure with "memory" technology and seperating each character from the rest, and giving it a 1 bit weight in refrence to the character it's stored under, we can compress a 1 GB file, 8,192,000,000 bits, down to 1365333334 characters then refrence them in a special program to 1 bit each, so 1,365,333,334 bits or 170,666,666.75 Bytes Before the file is actually compressed by a normal compression routine.

     

    -William Dover

     

    Friday, May 19, 2006 11:04 AM
  • Basically, without getting to technical, it's impossible for a Media format like DVD, or HDDVD to use every possible string of 1024 bytes. We use to many procedures that have to many defined outputs for it to be possible for any number of video files, regardless of quality to use every single possible occurance of 1024 bytes, because logically speaking we restrict ourselves on so many levels when we compress video and audio to digital media. 24 bit RGB, 32 bit RGB + Alpha, 384kbps audio at 48khz etc. There is a set number of different ocurrances of 1024 bytes, that we allow ourselves to use, by restricting ourselves to these patterns. So i predict, if you take 1000 HD-DVD formated movies, of any type and genre, and store every occurance of 1024 bytes from the first file, give them a simple address from a set system so they can be refrenced, and store a templete file of the first movie, of just addresses of 1024 bytes, compare it to the next movie, and store any occurances in the same way, and keep going, eventually, the seemingly random occruances of data will cease and you'll be left with the complete set of 1024 bytes that do occur in that particular format. Repeat this process as many times as possible, and eventually you would end up with a set that follows this simple forumla:

    Variable 1  x Variable 2 x Variable 3 x Variable 4 x Variable  5 etc.

    Variable 1 + Variable 2 + Variable 3 + Variable 4 + Variable 5 etc.

    all of that, x 1024 bits.

    We know that in video, resolution HxW = pixel area, and in pure RGB we have 3 bytes per pixel, 4 bytes with alpha.  lets take a resolution of 2800x2400 = 6,720,000 pixels, at a max of 4 bytes each = 26,880,000 bytes per frame, 1,000,000 frames =

    26,880,000,000,000 bytes

    Variable 1 in this instance is 6720000 pixel size

    Variable 2 is 4 bytes per pixel

    Variable 3 is 1000000 frames

    Variable 4 is 1024 bytes

    using the above formula we get

    (6720000 x 4 x 1000000 x 1024) / (6720000+4+1000000+1024) =

    27525120000000000 / 7721028 = 3,564,955,341~ possible occurances of 1024 bytes. That for the format itself. Any video of 32 Bit capacity, with 1 million frames, with a size of 2800x2400 will have approx. 3.565 billion Different occurances of 1024 bytes. Anyone willing to check my math on this is welcome to try. This conclusion doesn't however eliminate the needless waste, i'm sure every pixel in a frame won't have a different color at any one time.

    The math on a 24 bit, 720x480 dvd video with 250,000 frames would be

    (345600 pixels per frame x 3 bytes per pixel x 250,000 frames x 1024 bytes) =

    265420800000000 / 3707027 = 71,599,370 possible occurances of 1024 bytes. Again this doesn't include the fact that there is never going to be a frame with every possiblitty of a color on it.

    -Light2Dark

    Sunday, May 21, 2006 8:49 PM
  • Basically it works like this.
    Using only the odd set of characters from the 10000001-11111111 region of the ASCII 2 chart.
    The 6 Bits from the original file will be "shrouded" with the 1 bits on both sides, meaning 000000 becomes 10000001, etc. This limits the actual character values for each word and addresses the need for an order to chaos. The most commonly used 8 bit string, was given the 0 bit, the 2nd most commonly used 8 bit string was given the 1 bit, then the next 4 sets where given 00, 01, 10, 11. Followed by the next 8 sets of 000, 001, 010, 011, 100, 101, 110, 111. Then the next 16 sets of 0000, 0001, 0010, 0011, 0100, 0101, 0110, 0111, 1000, 1001, 1010, 1011, 1100, 1101, 1110, 1111. Then the next 32 sets where given the values from 00000-11111, and the remaining 2 least commonly used sets where given the 000000, and 000001 sets. Each sets refrence can be stored in about a 100 byte dictionary. 64 bytes + 2 bits +8bits +24bits +64bits+160bits +12bits = 270 bits or 34 bytes, so 34 + 64 = 98 byte dictionary. With the least commonly used strings being the only uncompressed area of the file. Keep in mind, in an evenly dispersed file, there would be a compression ratio of half of the chatacter set, at 83% compression (the 5 bits per 6 bits), 1/32 of the character set at 16% compression, and 1/32 of the character set at 0% compression. But the 2 most commonly used strings are given the best compression ratios.  I hypothosize, a 1 GB file, with evenly dispersed strings (which is worst case compression) accross the 64 character spectrum will compress down to:
    8,192,000,000 bits/6 = 1365333334 sets of 8 bit characters = 10,922,666,667 bits in a temp file then dropped down to 170666667 bits per character set, the first set being compressed 1/6th, the second set being compressed 1/6th, the next 4 sets compressed 1/3rd, the next 8 sets compressed 1/2, the next 16 sets compressed 2/3rds, the next 32 sets compressed 5/6ths, and the remaining 2 sets uncompressed, would equal
     (rounded for visual flow)
     
    170,700,000 * 17% = 29,019,000 bits
    170,700,000 * 17% = 29,019,000 bits
    170,700,000 * 4 * 33% = 225,324,000 bits
    170,700,000 * 8 * 50% = 682,800,000 bits
    170,700,000 *16* 66% = 1,802,592,000 bits
    170,700,000 *32* 83.3% = 4,550,179,200 bits
    170,700,000 *100%*2 = 341,400,000 bits
     
    Totalling = 6,011,883,300 bits or 751,485,413 bytes (734 MB), meaning a completely dispersed file, all parts equalling out, will still be compressed 734/1024MB = 71.7% This of course is phenominal considering the current compression ratios on video files of this size, without loss of quality.
     
     
     
    William Dover
    President
    Light2Dark Technologies
    Tuesday, May 23, 2006 10:51 PM