none
NUL byte in NRBF which I can't figure out the purpose of by reading the docs RRS feed

  • Question

  • I am attempting to write an NRBF parser, but I found a chunk of NRBF where it seems like I'm getting out the right values, yet I have a spare byte that I can't account for in the documentation. I was wondering whether anyone could shed light on this apparent anomaly.

    The binary:

    00000000  00 01 00 00 00 ff ff ff  ff 01 00 00 00 00 00 00  |................|
    00000010  00 0c 02 00 00 00 50 50  61 69 6e 74 44 6f 74 4e  |......PPaintDotN|
    00000020  65 74 2e 44 61 74 61 2c  20 56 65 72 73 69 6f 6e  |et.Data, Version|
    00000030  3d 34 2e 31 30 36 2e 37  30 31 36 2e 33 38 30 37  |=4.106.7016.3807|
    00000040  34 2c 20 43 75 6c 74 75  72 65 3d 6e 65 75 74 72  |4, Culture=neutr|
    00000050  61 6c 2c 20 50 75 62 6c  69 63 4b 65 79 54 6f 6b  |al, PublicKeyTok|
    00000060  65 6e 3d 6e 75 6c 6c 05  01 00 00 00 14 50 61 69  |en=null......Pai|
    00000070  6e 74 44 6f 74 4e 65 74  2e 44 6f 63 75 6d 65 6e  |ntDotNet.Documen|
    00000080  74 06 00 00 00 0a 69 73  44 69 73 70 6f 73 65 64  |t.....isDisposed|
    00000090  06 6c 61 79 65 72 73 05  77 69 64 74 68 06 68 65  |.layers.width.he|
    000000a0  69 67 68 74 09 73 61 76  65 64 57 69 74 68 11 75  |ight.savedWith.u|
    000000b0  73 65 72 4d 65 74 61 64  61 74 61 49 74 65 6d 73  |serMetadataItems|
    000000c0  00 04 00 00 03 03 01 15  50 61 69 6e 74 44 6f 74  |........PaintDot|
    000000d0  4e 65 74 2e 4c 61 79 65  72 4c 69 73 74 02 00 00  |Net.LayerList...|
    000000e0  00 08 08 0e 53 79 73 74  65 6d 2e 56 65 72 73 69  |....System.Versi|
    000000f0  6f 6e e6 01 53 79 73 74  65 6d 2e 43 6f 6c 6c 65  |on..System.Colle|
    00000100  63 74 69 6f 6e 73 2e 47  65 6e 65 72 69 63 2e 4b  |ctions.Generic.K|
    00000110  65 79 56 61 6c 75 65 50  61 69 72 60 32 5b 5b 53  |eyValuePair`2[[S|
    00000120  79 73 74 65 6d 2e 53 74  72 69 6e 67 2c 20 6d 73  |ystem.String, ms|
    00000130  63 6f 72 6c 69 62 2c 20  56 65 72 73 69 6f 6e 3d  |corlib, Version=|
    00000140  34 2e 30 2e 30 2e 30 2c  20 43 75 6c 74 75 72 65  |4.0.0.0, Culture|
    00000150  3d 6e 65 75 74 72 61 6c  2c 20 50 75 62 6c 69 63  |=neutral, Public|
    00000160  4b 65 79 54 6f 6b 65 6e  3d 62 37 37 61 35 63 35  |KeyToken=b77a5c5|
    00000170  36 31 39 33 34 65 30 38  39 5d 2c 5b 53 79 73 74  |61934e089],[Syst|
    00000180  65 6d 2e 53 74 72 69 6e  67 2c 20 6d 73 63 6f 72  |em.String, mscor|
    00000190  6c 69 62 2c 20 56 65 72  73 69 6f 6e 3d 34 2e 30  |lib, Version=4.0|
    000001a0  2e 30 2e 30 2c 20 43 75  6c 74 75 72 65 3d 6e 65  |.0.0, Culture=ne|
    000001b0  75 74 72 61 6c 2c 20 50  75 62 6c 69 63 4b 65 79  |utral, PublicKey|
    000001c0  54 6f 6b 65 6e 3d 62 37  37 61 35 63 35 36 31 39  |Token=b77a5c5619|
    000001d0  33 34 65 30 38 39 5d 5d  5b 5d 02 00 00 00 00 09  |34e089]][]......|
    000001e0  03 00 00 00 20 03 00 00  58 02 00 00 09 04 00     |.... ...X......|
    000001ef

    I'm building the description here: https://github.com/trejkaz/nrbf.ksy

    The interpretation I get from my current description:

    {
      "records": [
        {
          "recordTypeEnum": { "name": "SERIALIZED_STREAM_HEADER", "value": 0 },
          "payload": {
            "topId": 1,
            "headerId": 4294967295,
            "majorVersion": 1,
            "minorVersion": 0
          }
        },
        {
          "recordTypeEnum": { "name": "BINARY_LIBRARY", "value": 12 },
          "payload": {
            "libraryId": 2,
            "libraryName": {
              "length": {
                "groups": [
                  {
                    "b": 80,
                    "hasNext": false,
                    "value": 80
                  }
                ],
                "len": 1,
                "value": 80
              },
              "chars": "PaintDotNet.Data, Version=4.106.7016.38074, Culture=neutral, PublicKeyToken=null"
            }
          }
        },
        {
          "recordTypeEnum": { "name": "CLASS_WITH_MEMBERS_AND_TYPES", "value": 5 },
          "payload": {
            "classInfo": {
              "objectId": 1,
              "name": {
                "length": {
                  "groups": [
                    {
                      "b": 20,
                      "hasNext": false,
                      "value": 20
                    }
                  ],
                  "len": 1,
                  "value": 20
                },
                "chars": "PaintDotNet.Document"
              },
              "memberCount": 6,
              "memberNames": [
                {
                  "length": {
                    "groups": [
                      {
                        "b": 10,
                        "hasNext": false,
                        "value": 10
                      }
                    ],
                    "len": 1,
                    "value": 10
                  },
                  "chars": "isDisposed"
                },
                {
                  "length": {
                    "groups": [
                      {
                        "b": 6,
                        "hasNext": false,
                        "value": 6
                      }
                    ],
                    "len": 1,
                    "value": 6
                  },
                  "chars": "layers"
                },
                {
                  "length": {
                    "groups": [
                      {
                        "b": 5,
                        "hasNext": false,
                        "value": 5
                      }
                    ],
                    "len": 1,
                    "value": 5
                  },
                  "chars": "width"
                },
                {
                  "length": {
                    "groups": [
                      {
                        "b": 6,
                        "hasNext": false,
                        "value": 6
                      }
                    ],
                    "len": 1,
                    "value": 6
                  },
                  "chars": "height"
                },
                {
                  "length": {
                    "groups": [
                      {
                        "b": 9,
                        "hasNext": false,
                        "value": 9
                      }
                    ],
                    "len": 1,
                    "value": 9
                  },
                  "chars": "savedWith"
                },
                {
                  "length": {
                    "groups": [
                      {
                        "b": 17,
                        "hasNext": false,
                        "value": 17
                      }
                    ],
                    "len": 1,
                    "value": 17
                  },
                  "chars": "userMetadataItems"
                }
              ]
            },
            "memberTypeInfo": {
              "binaryTypeEnums": [
                { "name": "PRIMITIVE", "value": 0 }, { "name": "CLASS", "value": 4 }, { "name": "PRIMITIVE", "value": 0 }, { "name": "PRIMITIVE", "value": 0 }, { "name": "SYSTEM_CLASS", "value": 3 }, { "name": "SYSTEM_CLASS", "value": 3 }
              ],
              "additionalInfos": [
                {
                  "i": 0,
                  "primitiveType": { "name": "BOOLEAN", "value": 1 },
                  "binaryTypeEnum": { "name": "PRIMITIVE", "value": 0 }
                },
                {
                  "i": 1,
                  "classTypeInfo": {
                    "typeName": {
                      "length": {
                        "groups": [
                          {
                            "b": 21,
                            "hasNext": false,
                            "value": 21
                          }
                        ],
                        "len": 1,
                        "value": 21
                      },
                      "chars": "PaintDotNet.LayerList"
                    },
                    "libraryId": 2
                  },
                  "binaryTypeEnum": { "name": "CLASS", "value": 4 }
                },
                {
                  "i": 2,
                  "primitiveType": { "name": "INT32", "value": 8 },
                  "binaryTypeEnum": { "name": "PRIMITIVE", "value": 0 }
                },
                {
                  "i": 3,
                  "primitiveType": { "name": "INT32", "value": 8 },
                  "binaryTypeEnum": { "name": "PRIMITIVE", "value": 0 }
                },
                {
                  "i": 4,
                  "className": {
                    "length": {
                      "groups": [
                        {
                          "b": 14,
                          "hasNext": false,
                          "value": 14
                        }
                      ],
                      "len": 1,
                      "value": 14
                    },
                    "chars": "System.Version"
                  },
                  "binaryTypeEnum": { "name": "SYSTEM_CLASS", "value": 3 }
                },
                {
                  "i": 5,
                  "className": {
                    "length": {
                      "groups": [
                        {
                          "b": 230,
                          "hasNext": true,
                          "value": 102
                        },
                        {
                          "b": 1,
                          "hasNext": false,
                          "value": 1
                        }
                      ],
                      "len": 2,
                      "value": 230
                    },
                    "chars": "System.Collections.Generic.KeyValuePair`2[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]][]"
                  },
                  "binaryTypeEnum": { "name": "SYSTEM_CLASS", "value": 3 }
                }
              ]
            },
            "libraryId": 2
          }
        },
        {
          "recordTypeEnum": { "name": "SERIALIZED_STREAM_HEADER", "value": 0 },
          "payload": {
            "topId": 777,
            "headerId": 204800,
            "majorVersion": 153600,
            "minorVersion": 264448
          }
        }
      ]
    }


    So as you can see the library ID matches up 0x2 in both records, but then there is a NUL byte which looks like another stream header and I get garbage.

    Any idea what's going on here? Is this a byte of padding that I have missed in the docs?

    Wednesday, July 3, 2019 11:42 AM

Answers

  • Hi Trejkaz,

    I believe I've identified the bytes in question. 

    I noticed that some of the bytes in question appear to be MemberReference records (0x09 followed by a 4 byte ID). When I looked up MemberReference records, I noticed:
     
    2.5.3 MemberReference
    The MemberReference record contains a reference to another record that contains the actual value. The record is used to serialize values of a Class Member and Array items. The mechanism to serialize a Class instance is described in [MS-NRTP] section 3.1.5.1.6. The mechanism to serialize an Array instance is described in [MS-NRTP] section 3.1.5.1.7.
     
    Looking in [MS-NRTP], I find:
    3.1.5.1.6 Mapping Class Instances
    (near the end)
    The values of all the Members MUST be serialized following this record.
     
    I believe the mystery bytes are the values of the class members. Looking at what class members to expect:
    (Boolean) isDisposed
    (Class) layers
    (Int32) width
    (Int32) height
    (SystemClass) savedWith
    (SystemClass) userMetadataItems
     
    ...and comparing to the bytes:
     00: isDisposed 

     09 03 00 00 00: Reference to layers
     20 03 00 00: Width 
     58 02 00 00: Height
     09 04 00 00 00: Reference to savedWith
     09 05 00 00 00: userMetadataItems

    Hope that helps!


    Jeff McCashland | Microsoft Protocols Open Specifications Team

    Thursday, July 11, 2019 6:12 PM
    Moderator

All replies

  • Hi Trejkaz,

    Thank you for your question on Windows Protocols. One of our engineers will respond soon. 

    Thanks,


    Jeff McCashland | Microsoft Protocols Open Specifications Team

    Wednesday, July 3, 2019 4:29 PM
    Moderator
  • Hi Trejkaz,

    I am analyzing the above bytes. I'll let you know what I find. 

    Thanks,


    Jeff McCashland | Microsoft Protocols Open Specifications Team

    Wednesday, July 3, 2019 10:09 PM
    Moderator
  • I've skipped forwards and identified more records which seem to be well-formed, but there's a pattern going on with there being junk after each one. I thought some padding might jump out, but all the records start at different locations and the gaps have different sizes, so I have no idea what's going on. The first few records even have no gap at all.


    Friday, July 5, 2019 3:48 AM
  • Hi Trejkaz,

    Thank you for the additional information. I will let you know what I find. 

    JeffM


    Jeff McCashland | Microsoft Protocols Open Specifications Team

    Friday, July 5, 2019 4:17 AM
    Moderator
  • Hi Trejkaz,

    I see what you mean about the extra bytes. I am investigating where these might come from. 

    In the meantime, could you please email us at our DocHelp (at Microsoft dot com) alias? I would like to collect the network trace for analysis, as I cannot do a deeper analysis on the image in this thread. 

    Thanks!


    Jeff McCashland | Microsoft Protocols Open Specifications Team

    Tuesday, July 9, 2019 5:09 PM
    Moderator
  • Sure. I've actually been meaning to provide full files but keep finding other things to do.

    Actually, this particular stream didn't come from a network application, but rather a file format which chose to use the same serialisation for whatever reasons. So I'll send the full files, which gives some context for what's being put in the file. I'm 100% sure that it's NRBF still because I found at least one other library written for .NET using the same method for reading the same format.

    I've also dug up some alternative non-.NET implementations of the same format and have started trying to figure out what's going on from those.

    So I was reading this:

    https://github.com/addisonElliott/pypdn/blob/master/pypdn/nrbf.py#L421

    So I'm guessing this is the member data, not part of the ClassWithMembers record, but following it? Which lead to scraping the documentation for where this might be described, which is where I finally discovered this text in the preface of section 2.3:

    >>>

    The values of the Members of the Class (1) MUST be serialized as records that follow this record, as specified in section 2.7. The order of the records MUST match the order of MemberNames as specified in the ClassInfo (section 2.3.1.1) structure.

    <<<

    So obviously what I'm looking at is the member values.

    I think this is how I got mislead:

    • The record is called "class", but it's actually an instance of a class, or rather it's an object.
    • The individual structure sections don't mention the member values following the rest of the structure, and could probably have shown it in the diagrams for clarity.


    Thursday, July 11, 2019 6:48 AM
  • Well, I guess the NUL is still a mystery though, even taking all this into account. I'll send the files.

    Thursday, July 11, 2019 9:18 AM
  • Hi Trejkaz,

    I received the files you sent. Thank you for those and the additional information above. I'll continue investigating and let you know if I find anything. 

    Thanks,


    Jeff McCashland | Microsoft Protocols Open Specifications Team

    Thursday, July 11, 2019 3:18 PM
    Moderator
  • Hi Trejkaz,

    I believe I've identified the bytes in question. 

    I noticed that some of the bytes in question appear to be MemberReference records (0x09 followed by a 4 byte ID). When I looked up MemberReference records, I noticed:
     
    2.5.3 MemberReference
    The MemberReference record contains a reference to another record that contains the actual value. The record is used to serialize values of a Class Member and Array items. The mechanism to serialize a Class instance is described in [MS-NRTP] section 3.1.5.1.6. The mechanism to serialize an Array instance is described in [MS-NRTP] section 3.1.5.1.7.
     
    Looking in [MS-NRTP], I find:
    3.1.5.1.6 Mapping Class Instances
    (near the end)
    The values of all the Members MUST be serialized following this record.
     
    I believe the mystery bytes are the values of the class members. Looking at what class members to expect:
    (Boolean) isDisposed
    (Class) layers
    (Int32) width
    (Int32) height
    (SystemClass) savedWith
    (SystemClass) userMetadataItems
     
    ...and comparing to the bytes:
     00: isDisposed 

     09 03 00 00 00: Reference to layers
     20 03 00 00: Width 
     58 02 00 00: Height
     09 04 00 00 00: Reference to savedWith
     09 05 00 00 00: userMetadataItems

    Hope that helps!


    Jeff McCashland | Microsoft Protocols Open Specifications Team

    Thursday, July 11, 2019 6:12 PM
    Moderator
  • The missing piece of the puzzle for me was that the boolean and int values are stored as MemberPrimitiveUnTyped. With that sorted out, everything lines up again.

    So I just have to sort out a nice way to conditionally switch types between full records and the bare records for this one spot, which should hopefully not be too hard to get working.

    Thanks!


    Thursday, July 11, 2019 11:36 PM
  • Hi Trejkaz,

    Glad to hear that helped! I will see if we can improve the documentation. 

    Please do email us at DocHelp or start a new thread if any new questions on the protocol docs come up. 

    Thanks!


    Jeff McCashland | Microsoft Protocols Open Specifications Team

    Friday, July 12, 2019 3:49 PM
    Moderator