none
[MS-PST] Undocumented steps necessary to create HeapOnNode / TableContext

    Question

  • Hello and thanks for publishing the [MS-PST] specifications,

    My company develops an application that writes to [MS-PST] files, and we suspect that there are a few notes missing from the MS-PST specifications:

    1. HeapOnNode (HN): we're getting the impression that padding must be added before the HNPAGEMAP, so that it would be aligned to 2 byte boundary. (In other words: ibHnpm must be divided by 2) (we had issues with Outlook 2003 when it was not the case)

    2. Table Context (TC): we're getting the impression that rgTCOLDESC must be sorted by (property) tag, AFAIK the document does not specify this.

    3. Table Context (TC): When adding columns to the Contents Table, we're getting the impression that the IBit / offset of columns from the 8- and 4-byte data value group that were originated from the NID_CONTENTS_TABLE_TEMPLATE must not be modified (even when there are no rows),

    we're not sure what's the rule here, but when we rearrange the column offsets / IBits within the 8- and 4-byte data value group, Outlook 2003 misbehaves.

    We would appreciate if the specification team will review those notes and update the document where necessary,

    Thanks,

    Tal Aloni,

    ROM Knowledgeware

    Thursday, May 03, 2012 6:02 PM

Answers

  • Hi Tal, I have a few answers for you. I am still looking into the others.

     

    Question #2

    You are correct. The array of TCOLDESC structures contained in the rgTCOLDESC property of the TCINFO structure must be sorted by the tag property. I have filed a request to have this information added to section 2.3.4.1 and that should be included in a future release of the document.

     

    Question #4

    You are correct. Actually, this goes beyond TC  blocks in that all XBLOCK and XXBLOCK structures should follow this behavior. Any block that is NOT the last data block should be the maximum block size of 8192 bytes. I have filed a request to have this information added to section 2.3.4.4 and that should be included in a future release of the document.

     

    Question #5

    You are correct. The action "Update the Fill Level Map that corresponds to the modified block (HNs with a data tree)." should be in the Required section. I have filed a request to have section 2.6.2.1.2 corrected and that should be fixed in a future release of the document.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    • Marked as answer by Tal Aloni Monday, June 11, 2012 9:30 PM
    Thursday, May 24, 2012 2:04 PM
  • Hi Tal, here's some information about question #1. I am still looking into question #3.

    Question #1
    The description of the HNPAGEMAP in section 2.3.1.5 is misleading. The HNPAGEMAP is actually the last item in the variable length data portion of the block. So, there could be anywhere from 0 to 63 bytes of padding between it and the block trailer. Also, you are right about the byte alingment of the HNPAGEMAP. We will add 1 byte of padding if necessary to ensure that the HNPAGEMAP starts on a 2-byte boundary. I have filed a request to have this information added to section 2.3.1.5 and that should be included in a future release of the document.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    • Marked as answer by Tal Aloni Monday, June 11, 2012 9:30 PM
    Friday, June 01, 2012 3:36 PM
  • Hi Tal, here is what I was able to find out about question #3.

     

    1.         Columns have to be sorted  by tag per question #2.
    2.         PidTagLtpRowId must be assigned iBit == 0
    3.         PidTagLtpRowId must be assigned ibData == 0
    4.         PidTagLtpRowVer must be assigned iBit == 1
    5.         PidTagLtpRowVer must be assigned ibData == 4
    6.         For any other columns iBit can change/be any valid value (other than 0 and 1)
    7.         For any other columns ibData can be any valid value

     

    It must look as if the PidTagLtpRowId was the very first column added to the TC and PidTagLtpRowVer was the second column added to the TC. I have filed a request to have this information added to section 2.3.1.5 and that should be included in a future release of the document.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, June 11, 2012 6:55 PM
  • Hi Tal, here are a few more answers for you.
     
    7a. If cbData is not a multiple of k the SMQ is definitely corrupted. The rest of the PST file may or may not be perfectly fine. It's not possible to make that determination based on the state of the SMQ. You may be able to use the scanpst.exe utility to repair the file. However, determining the overall 'health' of a PST file and /or repairing it is outside the scope of the protocol documentation.
     
    7c. Per section 2.4.8.4.1, "Failure to queue the appropriate SUD entries or queuing the SUD entries out-of-order results in search Folder objects going out of sync with the actual contents."
     
    7d. Per section 2.4.8.3 "a queue is implemented as a node" which means the data  could be contained in a single block or span multiple [X|XX]Blocks. The diagram is generic and is not meant to imply that a queue will be contained in a single data block.

    8. There is currently no way to determine which version of Outlook originally created the PST file.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Wednesday, June 13, 2012 3:00 PM
  • Hi Tal, in regards to question 7c, whether a folder is a search folder is independent of container class.  It sounds like you might be confusing the "Search Folders" you see in the Navigation Pane with MAPI search folders.  MAPI search folders are more or less a superset of the Navigation Pane search folders.  See http://msdn.microsoft.com/en-us/library/cc815479.aspx for a good definition of MAPI search folders. If you touch a message anywhere in the store, you must enqueue the appropriate SUD so that all search folders in the store can update themselves.

    Question 11
    Thanks for pointing that out. I will review the code and make changes as needed.

    Question 12
    I am looking into this and will let you know when I have more information.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Saturday, June 23, 2012 1:53 AM
  • Hi Tal, in regards to question 13, after looking into this some more, I have to agree with you that it is necessary to know when to queue the different SUD types. The following is the criteria used by Outlook when determining whether or not to enqueue the SUD. I have filed a request to have this information added to the documentation.

     

    1. If Windows Desktop Search indexing is enabled, enqueue the following types:
      • SUDT_MSG_ADD
      • SUDT_MSG_MOD
      • SUDT_MSG_DEL
      • SUDT_FLD_ADD
      • SUDT_FLD_MOD
      • SUDT_FLD_DEL
      • SUDT_IDX_MSG_DEL
      • SUDT_MSG_MOV
    1. Always enqueue the following types:
      • SUDT_SRCH_ADD
      • SUDT_SRCH_MOD
      • SUDT_SRCH_DEL
    1. For the following types, search the SDO for the NIDs of the old and new parent folders.  If either of the NIDs is contained in the SDO, enqueue the SUD:
      • SUDT_MSG_MOV
      • SUDT_FLD_MOV
      • SUDT_MSG_ADD
      • SUDT_MSG_MOD
      • SUDT_MSG_ROW_MOD
      • SUDT_MSG_DEL
      • SUDT_MSG_SPAM
      • SUDT_IDX_MSG_DEL
      • SUDT_FLD_ADD
      • SUDT_FLD_MOD
      • SUDT_FLD_DEL
    1. If none of the above conditions is satisfied, do not enqueue the SUD

    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, July 02, 2012 6:19 PM
  • Hi Tal, in regards to question 12, you are correct about the PidTagConversationIndex property. In fact, it's been in use since as far back as Outlook and Exchange 2003. I have filed a request to have that behavior note removed from the documentation.

     

    I am currently looking into question 14. However, scanpst.exe has been around since far before the MS-PST document was written and was not designed to be a file validation tool of any sort. Its purpose is to repair corrupted PST files to a state such that the e-mails can be recovered and moved into a new PST file. It does validate some structures, but only for this purpose.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Tuesday, July 03, 2012 2:42 PM
  • Hi Tal, here are a few more answers. I am still looking into your other questions and hope to have more information for you soon.

    Question 16:
    Thank you for pointing this out. I have filed a request to have section 2.4.8.2.3 corrected and that should be fixed in a future release of the document.

    Question 17:
    This property is not documented because it does not meet the criteria for being included in the Exchange Server Protocols documentation: it is not a critical Message Object property used for communication between clients and servers. It's considered internal and not necessary for interoperability.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, July 09, 2012 4:51 PM
  • Hi Tal, here are some more answers to your questions. I am still looking into question 18.


    Question 14:
    In general, any 32-bit value is valid for the dwRowID value. However, Outlook expects the following.

     - For the contents table, it is the message nid.
     - For the hierarchy table, it is the folder nid.
     - For all other tables, it’s a globally unique counter value > 0.


    Question 19:
    SLBlocks are required to have at least 1 entry in the rgentries array. If the entry count (cEnt) reaches 0 the block should be deleted. I have filed a request to have this information added to the documentation.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, July 16, 2012 8:34 PM
  • Hi Tal, here is what I was able to find out about the BID structure (question 18). I have filed multiple requests to have the documentation updated to make it easier to understand.

    Your observations are correct. When a BID represents a page block the value is a UInt64 and increases by 1 each time. When a BID represents a data block the 2 least significant bits are reserved and used as flags for internal purposes. The remaining 62 bits are used to represent a UInt62 and increases  by 1 each time. The value should be read as if it was a UInt64. However, because the 2 lowest bits are reserved and should be ignored, the value will appear to be increasing by 4 each time.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, July 23, 2012 5:03 PM
  • Hi Tal, after looking at question #13 some more I found out that Outlook 2003 does not support Windows Desktop Search in the same way that Outlook 2007/2010 does. So, clause #1 doesn't apply to Outlook 2003. I have added this information to the original request.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Thursday, July 26, 2012 9:37 PM

All replies

  • Hi Tal,

    Thank you for your question.  A colleague will contact you soon to investigate this issue.

    Regards,

    Mark Miller | Open Specifications Support Team

    Thursday, May 03, 2012 6:29 PM
  • Hi Tal, I am the engineer who will be working with you on this issue. I am currently researching the problem and will provide you with an update soon. Thank you for your patience.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Thursday, May 03, 2012 7:13 PM
  • Thanks Josh, looking forward to hear your findings,

    I have another note I would like to bring forward:

    4. When adding a row to a Table Context (TC), at the point where an additional Data Block is needed to be added in order to store the row, Outlook 2003 expects the first DataBlock to be completely filled.

    in other words, padding must be appended in order for the data in the first Data Block length to be 8196 - BlockTrailerLength.

    [MS-PST] does guide readers that padding may have been appended, but it does not instruct writers to append padding.

    Best regards,

    Tal Aloni,

    ROM Knowledgeware

    Monday, May 07, 2012 4:41 PM
  • Hi Tal, I will look into that for you as well. Thank you for letting me know.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, May 07, 2012 6:58 PM
  • Hi Tal, I will try to address each of your questions…

    1. After reviewing the documentation I believe that you are correct that the HNPAGEMAP will always start on a 2-byte boundary. I don't see how it's possible for the HNPAGEMAP to start on an odd byte number unless it has been modified by something that doesn't follow the specification. Blocks are aligned on 64 byte boundaries (2.2.2.8). The HNPAGEMAP, assuming that the rgibAlloc will contain a minimum of 1 entry, will be a minimum of 6 bytes and increase 2 bytes at a time for each new entry is added to rgibAlloc (2.3.1.5). The Block trailer will either be 16 or 12 bytes in length (Unicode vs. ANSI) (2.2.2.8.1). Based on that, I don't see how the HNPAGEMAP could NOT be aligned on a 2-byte boundary.

    2. I am not able to locate anything that specifies that the array of TCOLDESC structures in rgTCOLDESC MUST be sorted in any way. It's certainly possible that Outlook may store them sorted for some reason. What evidence do you have to suggest that it is required to be sorted by tag?

    3. This is going to take additional research. I will let you know when I have more information.

    4. It is mentioned in almost every section that describes the layout of various block types that padding will optionally exist between the data section of the block and the block trailer to ensure that blocks are aligned to a 64 byte boundary and/or the 8192 byte maximum size. Are you stating that you believe that the block must be exactly 8192 bytes in length and likely contain more than 63 bytes of padding if the row doesn't fit and requires a new data block? What evidence do you have to suggest that Outlook 2003 requires this?


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Wednesday, May 09, 2012 3:50 PM
  • Hi Josh, Thanks for the effort and the fast reply,

    1. The HNPAGEMAP follows the Heap Items, so if the total length of all the heap items is odd, Outlook 2003 will put a single byte of padding before the HNPAGEMAP to make sure it will start on 2-byte boundary. unless I'm missing something (and I apologize if I am), this padding requirement is not reflected in the specification.

    2. As I mentioned, we created a library that read / write PST files, when we create a Contents Table TC with rgTCOLDESC that is not sorted, Outlook 2003 SP3 misbehaves (not in a consistent manner), I have personally taken the time to verify that this behavior is not a result of a bug somewhere else in out code, and this issue is resolved as soon as we store rgTCOLDESC sorted by Tag.

    4. That's exactly what I'm stating.

    I'm well aware that it is mentioned that padding will exist between the data section of the block and the block trailer to ensure that blocks are aligned to a 64 byte boundary. and it is also mentioned that 8192 bytes is the maximum block length (including the block trailer).

    I'm stating that in some cases additional padding must be appended (i.e. more than 63) to Data Blocks.

    This is true for both the Data Tree that holds the TC Rows under a Subnode, and for the Data Tree that holds the heap items for that TC. Outlook 2003 SP3 will report the folder as corrupt unless the Data Blocks preceding the last (and excluding the last) Data Block have additional padding in addition to those mentioned by the specs, as soon as we imitated Outlook by completely filling those data blocks, the problem was solved. I personally verified that this issue does not originate from elsewhere in our code.

    Thanks,

    Tal Aloni,

    ROM Knowledgeware



    Wednesday, May 09, 2012 7:16 PM
  • Hi Tal, thanks for the additional information. I will continue to look into this and let you know when I have more information.

    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Wednesday, May 09, 2012 7:31 PM
  • Hi Tal, Unless Outlook 2003 isn't enforcing the 64-byte boundary rule there would be no reason to believe that the HNPAGEMAP would ever start on an odd byte and the actual number of bytes between the last heap item and the HNPAGEMAP isn't important. The start of the HNPAGEMAP is always relative to the end of the block, not the last heap item. Have you seen Outlook 2003 behave differently?

    I will continue to look into the other items.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Thursday, May 10, 2012 4:29 PM
  • Hi Josh,

    Yes, I have seen Outlook 2003 behaves differently,

    The specs mention that "The HNPAGEMAP record is located at the end of each HN block immediately before the block trailer".

    But Outlook 2003 behaves a bit differently:

    Outlook 2003 treats the HNPAGEMAP as part of the block's data,  the header and Heap Items comes first, then the HNPAGEMAP, then the block padding (necessary for 64-byte alignment) and then the Block Trailer. now, because the HNPAGEMAP is treated as part of the block's data, the 64-byte boundary rule does not come into play.

    [This actually makes more sense, because the block padding belongs to a lower level than the HNPAGEMAP]

    Emphasis: The word immediately is inaccurate.

    Thanks,

    Tal


    • Edited by Tal Aloni Thursday, May 10, 2012 9:02 PM
    Thursday, May 10, 2012 7:57 PM
  • Hi Tal, thank you for the clarification. Now I understand the behavior that you are describing.

    Also, it sounds like none of this is actually blocking you right now. But you would like to have behavior notes added to the document for these issues that you've found? Please let me know if that is true.

    Thanks.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Friday, May 11, 2012 6:40 PM
  • Hi Josh, Yes, That is true,

    One possible exception:

    in all issues except issue number 3, I was able to figure out exactly what Outlook does and imitate it. however, I was not able to come up with a set of rules that would explain why Outlook 2003 sets the iBit and offset of a new table column the way it does. I'm a little bit troubled by the fact that I'm not fully understand the missing requirement.


    I have an additional note:

    5. Heap-On-Node (HN) - rgbFillLevel

    Under 2.6.2.1.2, "Allocating from the HN", [MS-PST] specify that it's Recommended for the implementor to Update the Fill Level Map that corresponds to the modified block.

    My experience with Outlook 2003 suggest however that implementers MUST do so, failing to do so (for Contents Table TC HN) after filling out the first Data Block will cause Outlook 2003 to report an error while trying to add an additional message to the folder.

    Thanks for your kind assistance,

    Tal

    Friday, May 11, 2012 8:57 PM
  • Hi Tal, I am still looking into this issue. I hope to have more information for you soon. Your patience is greatly appreciated.

    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Tuesday, May 22, 2012 5:25 PM
  • Hi Tal, I have a few answers for you. I am still looking into the others.

     

    Question #2

    You are correct. The array of TCOLDESC structures contained in the rgTCOLDESC property of the TCINFO structure must be sorted by the tag property. I have filed a request to have this information added to section 2.3.4.1 and that should be included in a future release of the document.

     

    Question #4

    You are correct. Actually, this goes beyond TC  blocks in that all XBLOCK and XXBLOCK structures should follow this behavior. Any block that is NOT the last data block should be the maximum block size of 8192 bytes. I have filed a request to have this information added to section 2.3.4.4 and that should be included in a future release of the document.

     

    Question #5

    You are correct. The action "Update the Fill Level Map that corresponds to the modified block (HNs with a data tree)." should be in the Required section. I have filed a request to have section 2.6.2.1.2 corrected and that should be fixed in a future release of the document.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    • Marked as answer by Tal Aloni Monday, June 11, 2012 9:30 PM
    Thursday, May 24, 2012 2:04 PM
  • Thanks Josh, That's great!

    6. There is an older issue that was raised two years ago about the fact that the document has bidNextB shown twice in the unicode HEADER structure.

    ( see http://social.msdn.microsoft.com/Forums/nb-NO/os_binaryfile/thread/b50106b9-d1a0-4877-aaa5-8d23be1084fd for previous discussions )

    As far as I can tell, the bidNextB that appears immediately after bidNextP is a mistake, could you please look into it and file a request to have the redundant bidNextB removed from the the documentation?

    Thanks again for your efforts!

    Tal


    Thursday, May 24, 2012 6:01 PM
  • Hi Tal, thank you for bringing that to my attention. You are correct that the Unicode block diagram is wrong and that that instance of bidNextB that is immediately following bidNextP should be removed. I will follow-up on the original request.



    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Thursday, May 24, 2012 7:05 PM
  • Hi Tal, here's some information about question #1. I am still looking into question #3.

    Question #1
    The description of the HNPAGEMAP in section 2.3.1.5 is misleading. The HNPAGEMAP is actually the last item in the variable length data portion of the block. So, there could be anywhere from 0 to 63 bytes of padding between it and the block trailer. Also, you are right about the byte alingment of the HNPAGEMAP. We will add 1 byte of padding if necessary to ensure that the HNPAGEMAP starts on a 2-byte boundary. I have filed a request to have this information added to section 2.3.1.5 and that should be included in a future release of the document.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    • Marked as answer by Tal Aloni Monday, June 11, 2012 9:30 PM
    Friday, June 01, 2012 3:36 PM
  • Hi Josh, Thanks again for your determined assistance,

    If you'd be so kind, I will appreciate some clarification about the Search Management Queue:

    7. When creating a new folder in the PST file, section 2.6.3.2.1 specify that the implementor MUST "queue a properly-formatted SUD of type SUDT_FLD_ADD to the SMQ". however, section 2.4.8.3 specify that "implementations MUST NOT process the contents of a queue if cbData is not an integer multiple of k".

    a. Am I to understand from the latter that a PST is valid even if it has SMQ that has invalid length?

    b. If this is the case, then why queuing a SUD to the SMQ when creating a folder is a MUST instead of SHOULD?

    c. What are the consequences of not queuing a SUD to the SMQ when creating a folder / message?

    d. I'm confused by 2.4.8.3, My best guess is that the BBT entry that belongs to the Basic Queue Node points to a Data Tree (either Data Block, XBlock or XXBlock), however, only the data itself is shown in the diagram (without mentioning whether the BBT is pointing to a Data Tree or just a single Data Block [i.e. with Block Trailer] or just raw data), could you please clarify this?

    e. I'm not sure which SUD flags (wFlags) should be included with SUD of type SUDT_FLD_ADD (if any), could you clarify this please?

    Thank you very much,

    Tal


    Friday, June 01, 2012 11:05 PM
  • Hi Tal, I will look into this for you as well.

    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, June 04, 2012 1:47 PM
  • Hi Tal, here is what I was able to find out about question #3.

     

    1.         Columns have to be sorted  by tag per question #2.
    2.         PidTagLtpRowId must be assigned iBit == 0
    3.         PidTagLtpRowId must be assigned ibData == 0
    4.         PidTagLtpRowVer must be assigned iBit == 1
    5.         PidTagLtpRowVer must be assigned ibData == 4
    6.         For any other columns iBit can change/be any valid value (other than 0 and 1)
    7.         For any other columns ibData can be any valid value

     

    It must look as if the PidTagLtpRowId was the very first column added to the TC and PidTagLtpRowVer was the second column added to the TC. I have filed a request to have this information added to section 2.3.1.5 and that should be included in a future release of the document.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, June 11, 2012 6:55 PM
  • Hi Josh, Thanks for not giving up on question #3, must have been tough to pinpoint.

    (by the way, it might be beneficial to mention in MS-PST that PidTagLtpRowVer may be allocated from dwUnique in the header, I believe that's what outlook uses)

    I have another question, I hope this one is easy:

    8. Is there a field in the PST file that (Outlook use to) store the version of Outlook that created the PST file?

    (I know that AppointmentRecurrencePattern of recurring appointments store the writer version, but I was wondering if there is something that exists in every PST)

    Thanks,

    Tal

    Monday, June 11, 2012 7:36 PM
  • Hi Tal, here are a few more answers for you.
     
    7a. If cbData is not a multiple of k the SMQ is definitely corrupted. The rest of the PST file may or may not be perfectly fine. It's not possible to make that determination based on the state of the SMQ. You may be able to use the scanpst.exe utility to repair the file. However, determining the overall 'health' of a PST file and /or repairing it is outside the scope of the protocol documentation.
     
    7c. Per section 2.4.8.4.1, "Failure to queue the appropriate SUD entries or queuing the SUD entries out-of-order results in search Folder objects going out of sync with the actual contents."
     
    7d. Per section 2.4.8.3 "a queue is implemented as a node" which means the data  could be contained in a single block or span multiple [X|XX]Blocks. The diagram is generic and is not meant to imply that a queue will be contained in a single data block.

    8. There is currently no way to determine which version of Outlook originally created the PST file.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Wednesday, June 13, 2012 3:00 PM
  • Hi Josh,

    Thanks again,

    7c. What are the consequences of "search Folder objects going out of sync with the actual contents"? I'm trying to figure out what user operation won't work in Outlook without queuing a SUD to the SMQ when creating a folder.


    9. I've noticed that Outlook 2003 sets the fAMapValid in the root structure to 0x01 (when valid), while Outlook 2007 SP2 / Outlook 2010 set it to 0x02,

    Does it have any significance elsewhere in the file? are there any nodes / structures / fields / values that are expected to be in the PST when fAMapValid is set to 2 instead of 1? does it have anything to do with the Density List?


    10. It appears to me that Outlook 2003 and Outlook 2010 calculate the value of PidTagMessageSize for message objects using two different methods,

    It appears that Outlook 2003 uses the "sum of the sizes of all properties on the message object", while Outlook 2010 uses the length of the data in the Data Tree, could you clarify this please?

    Much appreciated,

    Tal

    Wednesday, June 13, 2012 9:00 PM
  • Hi Tal, here are the rest of the answers to your questions. Please let me know if you need anything else.

    Question 7c
    This basically means that the search folder won't populate.

    Question 7e
    In all cases where the wSUDType does not have any associated wFlags, the value of wFlags should be 0x0000.  I have filed a request to have this information added to section 2.4.8.1.1 and that should be included in a future release of the document.

    Question 9
    It's important to remember that the documentation is valid for the most current version of the product(s) that implement it. When the behavior differs for older versions there will be a link to a behavior note that looks like <#>. The description for VALID_AMAP1 states that the value has been deprecated, which means it's not valid for the current version of Outlook, and contains a link to behavior note <6> which states "Office Outlook 2003 uses VALID_AMAP1 to indicate that the AMaps are valid." So, the difference you are seeing is expected and documented correctly.

    Question 10
    The PidTagMessageSize property is calculated by the server, not the client. The actual calculation that is used is out of scope of the protocol documentation because it's considered to be product behavior. The following sections from MS-OXCFXICS explain how that value should only be treated as an estimate and that different implementations may calculate it differently.

    MS-OXCFXICS 3.2.5.4 Calculating the PidTagMessageSize Property Value
    The server SHOULD make the best effort to calculate this property, but because values for properties can change before the client downloads the message, and because the client specifies what data it does and does not require, it MUST be used only as an estimate by the client.

    MS-OXCFXICS 3.3.5.12 Using the PidTagMessageSize Property Value
    The value for the PidTagMessageSize property MUST be treated only as an estimate by the client. For more details, see section 3.2.5.4.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Tuesday, June 19, 2012 6:07 PM
  • Thanks Josh, I feel we are almost done here,

    Question 7c

    Isn't the search folder only relevant for folders with PidTagContainerClass of "IPF.Note" or folders without PidTagContainerClass at all (e.g. "Deleted Items")?

    Do I have to queue a SUD to the SMQ when creating a folder with PidTagContainerClass of "IPF.Appointment"?

    Tuesday, June 19, 2012 6:57 PM
  • Hi Josh,

    11. I found your blog post very helpful:

    http://blogs.msdn.com/b/openspecification/archive/2012/02/08/ms-pst-how-to-decode-data-pages-using-permutative-decoding.aspx

    You do have one mistake, the line should be

    int idx = (fEncrypt ? 0 : 512);

    [instead of "int idx = (fEncrypt ? 512 : 0);"]

    12. [MS-OXOCFG] Appendix A, note 24 specify that:

    "Exchange 2003, Exchange 2007, Office Outlook 2003, and Office Outlook 2007 do not set the PidTagConversationIndex property"

    This is not correct, Outlook 2007 do set PidTagConversationIndex, I think someone got confused with PidTagConversationId which is only set by Outlook 2010.


    Friday, June 22, 2012 3:42 PM
  • Hi Tal, in regards to question 7c, whether a folder is a search folder is independent of container class.  It sounds like you might be confusing the "Search Folders" you see in the Navigation Pane with MAPI search folders.  MAPI search folders are more or less a superset of the Navigation Pane search folders.  See http://msdn.microsoft.com/en-us/library/cc815479.aspx for a good definition of MAPI search folders. If you touch a message anywhere in the store, you must enqueue the appropriate SUD so that all search folders in the store can update themselves.

    Question 11
    Thanks for pointing that out. I will review the code and make changes as needed.

    Question 12
    I am looking into this and will let you know when I have more information.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Saturday, June 23, 2012 1:53 AM
  • Hi Josh,

    13. I have noticed that Outlook 2003 SP3 does not process (i.e. clears) the SMQ if there is no search folder defined under "Search Folders". If my observation is correct, then the SMQ will keep ballooning in this case.

    My question is, should I behave differently in this specific case? (i.e. not queue to the SMQ in some cases)

    Thanks,

    Tal

    Monday, June 25, 2012 9:31 AM
  • Hi Tal, in regards to question 13, under certain conditions, not every SUD type will be queued to the SMQ. The logic to determine this in out of scope of the PST documentation and is considered to be internal product behavior. However, the SUDT_SRCH_* types should always be queued.

    I am still looking into question 12.



    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Thursday, June 28, 2012 4:49 PM
  • Question 13

    Josh, We are dealing with SUDT_FLD_ADD, SUDT_FLD_DEL, SUDT_MSG_ADD and SUDT_MSG_DEL,

    We are planning to add and delete tens of thousands of messages to the end user's PST every morning,

    if the SMQ is not cleared by Outlook, then (admittedly after several years) the SMQ will be completely filled, and we're going to have a major problem. it doesn't make sense that the specifications lead to this situation.

    We must know when to queue SUDs to the SMQ, can you please shed some more light on this issue?

    Thanks!

    Tal


    Thursday, June 28, 2012 6:11 PM
  • Hi Tal, in regards to question 13, after looking into this some more, I have to agree with you that it is necessary to know when to queue the different SUD types. The following is the criteria used by Outlook when determining whether or not to enqueue the SUD. I have filed a request to have this information added to the documentation.

     

    1. If Windows Desktop Search indexing is enabled, enqueue the following types:
      • SUDT_MSG_ADD
      • SUDT_MSG_MOD
      • SUDT_MSG_DEL
      • SUDT_FLD_ADD
      • SUDT_FLD_MOD
      • SUDT_FLD_DEL
      • SUDT_IDX_MSG_DEL
      • SUDT_MSG_MOV
    1. Always enqueue the following types:
      • SUDT_SRCH_ADD
      • SUDT_SRCH_MOD
      • SUDT_SRCH_DEL
    1. For the following types, search the SDO for the NIDs of the old and new parent folders.  If either of the NIDs is contained in the SDO, enqueue the SUD:
      • SUDT_MSG_MOV
      • SUDT_FLD_MOV
      • SUDT_MSG_ADD
      • SUDT_MSG_MOD
      • SUDT_MSG_ROW_MOD
      • SUDT_MSG_DEL
      • SUDT_MSG_SPAM
      • SUDT_IDX_MSG_DEL
      • SUDT_FLD_ADD
      • SUDT_FLD_MOD
      • SUDT_FLD_DEL
    1. If none of the above conditions is satisfied, do not enqueue the SUD

    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, July 02, 2012 6:19 PM
  • Thanks Josh, this is very helpful!

    14. Back to Table Context: I've been creating Recipient Tables, and for dwRowID (part of RowIndex) I've decided to use auto-incremented value starting from 0, (2.3.4.3 suggest that "it is up to the implementation to decide what value to use for the primary key.")

    To the best of my knowledge, the specification does not mention that 0 cannot be used for RowID or BTH in general,

    Is there such limitation / recommendation?

    (FYI: scanpst.exe (all version) report the following error when the recipient table contains dwRowID of 0:

    "!!BTH node (nid=692, hid=80), keys overlap (dwkey=0, dwkeyMin=0)")

    Thanks!

    Tal

    Monday, July 02, 2012 6:45 PM
  • Hi Tal, in regards to question 12, you are correct about the PidTagConversationIndex property. In fact, it's been in use since as far back as Outlook and Exchange 2003. I have filed a request to have that behavior note removed from the documentation.

     

    I am currently looking into question 14. However, scanpst.exe has been around since far before the MS-PST document was written and was not designed to be a file validation tool of any sort. Its purpose is to repair corrupted PST files to a state such that the e-mails can be recovered and moved into a new PST file. It does validate some structures, but only for this purpose.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Tuesday, July 03, 2012 2:42 PM
  • Hi Josh,

    15. The SUD structure of SUDT_MSG_ROW_MOD does not appear in MS-PST.

    16. MS-PST: The SUD structure SUD_FLD_ADD contains a variable called "nidMsg" instead of "nidFld", the description is correct, but someone copy-pasted the wrong name.

    17. I know this is product specific, but I've noticed that PropertyID 0x6909 is added to every attachment table Outlook (all versions) creates,

    It's the only unknown property (i.e. not in MS-PST or MS-OXPROPS) in the attachment table, Can you give me a hint to its role, just for completeness sake?

    Thanks a lot,

    Tal

    Wednesday, July 04, 2012 11:16 AM
  • Sorry, Strike question 15, The SUD structure of SUDT_MSG_ROW_MOD is SUD_MSG_MOD.
    Wednesday, July 04, 2012 11:36 AM
  • Hi Josh,

    18. The specification do not detail this well, but I observed that bidNextP and bidNextB (in the header) have the following difference:

    bidNextP is not really a BID structure, and it's UInt64 value is incremented by 1 each increment,

    while bidNextB is a BID structure, and it's bidIndex is incremented by 1 (as a result, the UInt64 value of bidNextB is incremented by 4).

    I think you may want to clarify this in the document, as it took me a while to figure out that the I (Internal) bit in Block ID of BBT / NBT pages does not really denote anything, and is part of sequential number.

    Thanks,

    Tal

    Thursday, July 05, 2012 12:09 PM
  • Hi Josh,

    19. Let's say I have a Subnode-BTree with a single SLBlock as root, is it valid for that SLBlock to have 0 rgentries, or is 1 the minimum allowed?

    Thanks!

    Tal

    Thursday, July 05, 2012 8:26 PM
  • Hi Tal, here are a few more answers. I am still looking into your other questions and hope to have more information for you soon.

    Question 16:
    Thank you for pointing this out. I have filed a request to have section 2.4.8.2.3 corrected and that should be fixed in a future release of the document.

    Question 17:
    This property is not documented because it does not meet the criteria for being included in the Exchange Server Protocols documentation: it is not a critical Message Object property used for communication between clients and servers. It's considered internal and not necessary for interoperability.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, July 09, 2012 4:51 PM
  • Hi Tal, here are some more answers to your questions. I am still looking into question 18.


    Question 14:
    In general, any 32-bit value is valid for the dwRowID value. However, Outlook expects the following.

     - For the contents table, it is the message nid.
     - For the hierarchy table, it is the folder nid.
     - For all other tables, it’s a globally unique counter value > 0.


    Question 19:
    SLBlocks are required to have at least 1 entry in the rgentries array. If the entry count (cEnt) reaches 0 the block should be deleted. I have filed a request to have this information added to the documentation.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, July 16, 2012 8:34 PM
  • Hi Josh,

    In regard for question 13, in clause 1 of your answer you mentioned several SUDs that should always be queued if "Windows Desktop Search indexing is enabled":

    1. Outlook 2003 SP3 on Windows XP with Windows Search 4.0 will not dequque those SUDs that I'm queuing

    just because of clause 1.

    2. If I ignore clause 1 altogether, and not queue the required SUDs even when indexing is enabled, I cannot detect any discrepancy in the search results of Windows Search.

    3. Clause 1 specify when to queue SUDs, However, to prevent the SMQ from being clogged, what's more important to know is which SUDs will be dequeued by Outlook the next time the PST is attached.

    I would appreciate if you could take another look and see if you could find more details about this issue,

    Thanks!

    Tal

    Wednesday, July 18, 2012 10:56 AM
  • Hi Tal, here is what I was able to find out about the BID structure (question 18). I have filed multiple requests to have the documentation updated to make it easier to understand.

    Your observations are correct. When a BID represents a page block the value is a UInt64 and increases by 1 each time. When a BID represents a data block the 2 least significant bits are reserved and used as flags for internal purposes. The remaining 62 bits are used to represent a UInt62 and increases  by 1 each time. The value should be read as if it was a UInt64. However, because the 2 lowest bits are reserved and should be ignored, the value will appear to be increasing by 4 each time.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Monday, July 23, 2012 5:03 PM
  • Hi Tal, after looking at question #13 some more I found out that Outlook 2003 does not support Windows Desktop Search in the same way that Outlook 2007/2010 does. So, clause #1 doesn't apply to Outlook 2003. I have added this information to the original request.


    Josh Curry (jcurry) | Escalation Engineer | Open Specifications Support Team

    Thursday, July 26, 2012 9:37 PM