Circular Reference in the TNEF spec?

Question

Section 2.4 Encapsulated Message Properties

 

This looks like a couple of circular references to me:

 

MsgPropertyContent = PropertyScalarContent / PropertyMultiScalarContent / PropertyMultiVariableContent

PropertyScalarContent = MsgPropertyContent [PropertyPad]

PropertyMultiScalarContent = PropertyContentCount *MsgPropertyContent [PropertyPad]

 

I can't find an explanation for this type of encoding in the ABNF rfc and I can't think of how to implement it.  Any explanation would be appreciated.

 

Greg

Answer 1

Hi Greg,

Thanks for your question. I have moved your posting to the Windows Protocols forum which is the better forum for this question.

Answer 2

I should also like to point out that on page 22, the comment indicates that Boolean types are 16 bits but examining TNEF data leads me to believe they are actually 32-bit.

 

Answer 3

2.4 Encapsulated Message Properties

 

; Only present when MsgPropertyId is > 0x8000

NamedPropSpec = PropNameSpace PropIDType PropMap

 

Should be:

 

; Only present when MsgPropertyId is >= 0x8000

Answer 4

Page 12

 

2.2 ABNEF Description

 

;Attachment RendData

idAttachRendData = %x02.02.06.00

 

Should be:

idAttachRendData = %x02.09.06.00

Answer 5

Section 2.4 Encapsulated Message Properties

 

page 21

 

; Used if PropIDType is IDTypeString. Contains a length, then a

; UTF-16LE encoded UNICODE string ; and terminating 2-byte zero character, padded to UINT32 boundary

PropMapString = UINT32 *UINT16 %00.00 [PropMapPad]

 

Should be clarified as:

; Used if PropIDType is IDTypeString. Contains a length, then a

; UTF-16LE encoded UNICODE string.  The length includes the terminating 2-byte zero character.  Optional padding to UINT32 boundary

 

Answer 6

Section 2.4 Encapsulated Properties

 

page 24

 

This is completely wrong:

 

PropertyVariableContent = MsgPropertySize MsgPropertyValue [PropertyPad]

 

MsgPropertyValue is defined at the top of the section (page 21) as:

 

MsgPropertyValue = MsgPropertyTag MsgPropertyContent

MsgPropertyTag = MsgPropertyType MsgPropertyId [NamedPropSpec]

 

Variable length  types do not include the MsgPropertyTag again.  These just contain the data for the specified length.

Answer 7

I'm getting pretty excited.  My original post was almost a month ago which means it might be getting a response any week now!  Woohoo!  Just in time for me to ship my reader.

 

For anyone else out there building a TNEF reader, it is possible to do using the MS-OXTNEF spec.  Sort of. 

 

Use the errata listed above, of course, but there are little interesting things like nested messages that you'll just have to pick through the TNEF data using a binary editor to get right. 

 

Some of the property types (strings) always seem to have a multi-value count, other do only for MV types.

 

In short, the spec is an interesting and helpful starting point for building a TNEF reader/writer but don't go just by the written word.  Look at TNEF output from different versions of Exchange.

Answer 8

Greg,

 

Sorry this got missed as your orginal post got moved to the wrong forum and just plain fell under the radar.  I've moved it to the Exchange Protocol Forum so it will receive the proper attention.  I'm taking your feedback and filing bugs against the documentation. 

 

Thank you for your feedback.

Answer 9

Greg, Booleans and I2 are 16-bit and all values must be padded to 4 bytes which is what you are seeing in the stream.

Answer 10

Gregory K. Miskin wrote:

2.4 Encapsulated Message Properties   ; Only present when MsgPropertyId is > 0x8000 NamedPropSpec = PropNameSpace PropIDType PropMap   Should be:   ; Only present when MsgPropertyId is >= 0x8000

 

Bug has been filed against the documentation.

Answer 11

Gregory K. Miskin wrote:

Section 2.4 Encapsulated Message Properties   This looks like a couple of circular references to me:   MsgPropertyContent = PropertyScalarContent / PropertyMultiScalarContent / PropertyMultiVariableContent PropertyScalarContent = MsgPropertyContent [PropertyPad] PropertyMultiScalarContent = PropertyContentCount *MsgPropertyContent [PropertyPad]   I can't find an explanation for this type of encoding in the ABNF rfc and I can't think of how to implement it.  Any explanation would be appreciated.   Greg

 

Scalar content has a single value and a pad if needed, to reach a 4 byte boundary.  Multi-scaler has a count padded to 4 bytes, followed by that number of properties, each padded to 4 bytes.  Varaible content is a count, then that number of properties in the form size, value, pad.

 

Does this help?

Answer 12

Tom Devey - MSFT wrote:

Gregory K. Miskin wrote: Section 2.4 Encapsulated Message Properties   This looks like a couple of circular references to me:   MsgPropertyContent = PropertyScalarContent / PropertyMultiScalarContent / PropertyMultiVariableContent PropertyScalarContent = MsgPropertyContent [PropertyPad] PropertyMultiScalarContent = PropertyContentCount *MsgPropertyContent [PropertyPad]   I can't find an explanation for this type of encoding in the ABNF rfc and I can't think of how to implement it.  Any explanation would be appreciated.   Greg   Scalar content has a single value and a pad if needed, to reach a 4 byte boundary.  Multi-scaler has a count padded to 4 bytes, followed by that number of properties, each padded to 4 bytes.  Varaible content is a count, then that number of properties in the form size, value, pad.   Does this help?

 

Tom,

 

Thanks for the reply.

 

I grok the padding and count.  I was trying to point out that there is a circular reference for MsgPropertyContent, that is, you can't get to the end of its definition. 

 

In other words, in a simplified form, MsgPropertyContent == PropertyScalarContent == MsgPropertyContent....

 

There's no real definition of PropertyScalarContent and PropertyMultiScalarContent.  You have to, in my opinion, define the scalars and not assume anything.

 

Greg

Answer 13

Thanks, I will pass the feedback to the documentation group. 

Answer 14

Section 2.3.3 attFrom

There is no mention of padding here.  Clearly padding is taking place between the end of sender-email-address and the checksum.  The padding is the delta between structure-length and (8+ sender-name-length + sender-email-length) .  Maybe it isn't padding but there are NULL bytes there that cannot be accounted for with the string lengths.  In one particular case it was 8 bytes.

I can't account for these bytes in the spec and am assuming they are padding.

Answer 15

Greg,  I've emailed the document owner for clarification.

---

Developer Consultant

Answer 16

Greg,

The version 1.0 of the Exchange Server Protocol documents released today.  The top-level URL for the documents can be found here http://msdn.microsoft.com/en-us/library/cc307725(EXCHG.80).aspx.   You can also download the zip file that contains the PDF version of the Exchange Server Protocol documents http://go.microsoft.com/fwlink/?LinkId=115073

For your interest the updated [MS-OXTNEF] document can be downloaded at http://download.microsoft.com/download/5/D/D/5DD33FDF-91F5-496D-9884-0A0B0EE698BB/%5BMS-OXTNEF%5D.pdf

From the bugs you filed:

• "MsgPropertyId is > 0x8000" now correctly reads "MsgPropertyId is >= 0x8000"
  
• Boolean types are 16 bits but examining TNEF data leads me to believe they are actually 32-bit.

The documentation has been updated to added the follow:

PropertyScalarContent = MsgPropertyContent [PropertyPad]
 
 ; PropertyPad – between 0 and 3 zero-filled bytes, added to the streamed
 ; property values to achieve 4-byte boundary. Writers MUST use zero bytes
 ; and Readers MUST permit non-zero bytes. These pad bytes MUST be counted
 ; in the checksum of the containing attribute.
 
 PropertyPad=*3ZERO
 ZERO= %x00

• Section 2.2 ABNF has been updated
  
• Section 2.4 Encapsulated Message Properties updated
  
   ;Used if PropIDType is IDTypeString. Contains a length, then a
   ; UTF-16LE encoded UNICODE string
   ; The length includes the terminating 2-byte zero character. Optional padding ; to UINT32 boundary
  
• Section 2.4 Encapsulated Properties
   "PropertyVariableContent = MsgPropertySize MsgPropertyValue [PropertyPad]"
   now reads
   "PropertyVariableContent = MsgPropertySize MsgPropertyContent [PropertyPad]"
  

Note that the "Section 2.3.3 attFrom" and the "Cirular reference issue" will be updated in the next round of protocol document updates.

If you have any questions or find any discrepancies in the new documentation please don’t hesitate to post questions or feedback to this forum.

Again thank you for providing valueable feedback.
 

---

Developer Consultant

Answer 17

I talked to the document owner yesterday in regards to the Message Properties Circular Reference and he made some changes in the version 1.0 release that should address this issue.

Please let me know it this doesn't address your issue so I can follow-up with the document owner to make sure it does get fixed.

Answer 18

Page 23:

MsgPropertyContent = PropertyScalarContent / PropertyMultiScalarContent / PropertyMultiVariableContent

; Scalars – Types Int16, Int32, Flt32, Flt64, Currency, AppTime, ; Bool, Int64, Systime, CLSID ; The data for the particular property is written to the stream and if necessary, ;padded with bytes (which SHOULD be zero) to achieve a multiple of 4-bytes in length.

PropertyScalarContent = MsgPropertyContent [PropertyPad]

This is still a circular reference until you define PropertyScalarContent without reusing MsgPropertyContent.  So it should read:

PropertyScalarContent = TypeInt16 / TypeInt32 / TypeFlt32 / TypeFlt64 / Type Currency / TypeAppTime / TypeBoolean / TypeInt64 / TypeSystime / TypeCLSID [PropertyPad]

See the difference?  I can now get to the end of the definition of MsgPropertyContent.  The same thing needs to be done with PropertyVariableContent but I don't want to do all the work here. :)

Answer 19

"Section 2.3.3 attFrom

There is no mention of padding here.  Clearly padding is taking place between the end of sender-email-address and the checksum.  The padding is the delta between structure-length and (8+ sender-name-length + sender-email-length) .  Maybe it isn't padding but there are NULL bytes there that cannot be accounted for with the string lengths.  In one particular case it was 8 bytes.

I can't account for these bytes in the spec and am assuming they are padding."

The feedback from development is it doesn't matter since the decoding will work fine with or without the pad.   In reviewing the documentation  development is looking at putting verbage to clarify this even though it should work fine either way.

Thanks for the great feedback.

---

Developer Consultant

Answer 20

Hi Tom,

Even though this thread is a bit old - I found it relevant as the documentation still hasn't been corrected.

I'm using MS-OXTNEF v2.0 released April 10, 2009 - but the circular reference issue with PropertyScalarContent [page 24] still exist.

Also, the "Section 2.3.3 attFrom" issue may also not have been addressed(?) I haven't verified myself as of yet whether or not padding exists - but if it does as Gregory mentions, then the documentation doesn't appear to reflect this.


/Michael

Answer 21

A quick review of the orginal Technical Document Issue (TDI) that I filed it was supposed to have been fixed with the August documentation release of 2008.    Reviewing both Exhange 2007 and the Exchange 2010 [MS-OXTNEF] documentation it was never fixed.  I created a new TDI to get this fix.

Thank you for your feedback.

---

Developer Consultant

Answer 22

As for Section 2.3.3 attFrom there was an end note that was added to reflect the differences for the "pad".

Hope this helps.

---

Developer Consultant

Answer 23

Hi again Tom,

I just discovered a minor issue in the documentation page 11 - the property idParentID seems to have been removed between v1 and v2 of the document.

It should read:

idMessageAttr = idMessageClass / idFrom / idSubject / idDateSent / idDateRecd / idMessageStatus / idMessageID / idParentID /

idConversationID / idBody / idPriority / idDateModified / idRecipTable / idOriginalMessageClass / idOwner / idSentFor /

idDelegate / idDateStart / idDateEnd / idAidOwner / idRequestRes

; PidTagParentKey

idParentID = %x0A.80.01.00


I found the problem after running a forward rule (in Outlook2007 on an Exchange 2003)


/Michael

Answer 24

Michael,

I have filed a Technical Document Issue (TDI) to get this fixed.  Thank you for the continued feedback on this document.

---

Developer Consultant