none
[MS-OXCTABL] Cleaning up after RopSetCollapseState RRS feed

  • Question

  • Hi,

    I'm working with MS-OXCTABL, v1.03.

    Section 2.2.2.20 "RopSetCollapseState Semantics" states:
    RopSetCollapseState rebuilds a table’s collapsed state, which is specified by the data
    returned from RopGetCollapseState.The RopSetCollapseState response contains a
    bookmark referencing the row that was identified by the RowId and RowInstanceNumber
    fields in the RopGetCollapseState request. This ROP is only valid on Table Objects.
    The collapsed state sent to the server need not have been retrieved from the same table to
    which it is being applied. The table MUST have the same sort and restriction for the ROP to
    succeed. If the table is not the same table from which the collapse state was retrieved, the
    bookmark specified in the RopSetCollapseState response will be invalid. The bookmark
    returned MUST be freed by using RopFreeBookmark.


    On my testing, the RopFreeBookmark call of the bookmark is failing, and I'm trying to diagnose why.
    (You can see the test code at http://trac.openchange.org/browser/trunk/utils/mapitest/modules/module_oxctable.c#L719, the test that is failing is the call to FreeBookmark on line 896, which you can see at http://trac.openchange.org/browser/trunk/utils/mapitest/modules/module_oxctable.c#L896)

    As I understand it, RopSetCollapseState should work on a Contents table (since that is the case where expanding / collapsing rows is most useful).

    I note from 3.2.5.15 "Processing RopFreeBookmark" that:
    RopFreeBookmark MUST be supported by hierarchy tables.
    RopFreeBookmark SHOULD be supported by contents tables. <2>

    and from Appendix A, that
    <2> Section 3.2.5.15: Exchange 2003 SP2 and Exchange 2007 do not conform to the
    specification and currently send the value ecNotSupported in the ReturnValue field of the
    ROP response buffer for contents tables.

    From that, I'm not sure how to clean up after the RopSetCollapseState.

    Questions:
    1. Is the bookmark returned by RopSetCollapseState even valid?
    2. Is there any way to clean up that bookmark on the server?

    and, of course, is there anything else you can tell me that might help?

    Brad
    Saturday, February 28, 2009 1:56 AM

Answers

  • Nope.  I also looked at Exchange Server 2010 beta and the current behavior is the same.


    Developer Consultant
    • Marked as answer by Brad Hards Friday, April 24, 2009 12:02 AM
    Thursday, April 23, 2009 11:38 PM
    Moderator

All replies

  • Hi Brad,

    I will look into this and get back to you. 
    Developer Consultant
    Saturday, February 28, 2009 5:16 AM
    Moderator
  • Brad,

    When you say the boolmark is failing what do you mean?    Is there an error code returned in the response?

    The reason I ask it there may be an issue  in the documentation with the RopFreeBookmark.  A hex dump of the Rop request might be helpful.

    Developer Consultant
    Saturday, February 28, 2009 6:55 PM
    Moderator
  • Tom,

    The error code for the transaction is 0xC0000023 (BUFFER_TOO_SMALL).

    I'll try to get you a proper packet capture.

    Brad
    Sunday, March 1, 2009 9:36 AM
  • How would you like me to provide the capture?

    Brad
    Wednesday, March 4, 2009 10:03 AM
  • A Microsoft Network Monitor 3.2 (download from here) capture and please upload it to the Microsoft Services File Transfer here.

    Password is: @CDltk{qLumO5[Wb
    Developer Consultant
    Wednesday, March 4, 2009 6:22 PM
    Moderator
  • Tom,

    Only have wireshark on my development machine (Netmon won't run on Linux). Hopefully will be OK.

    I tried the upload a couple of times - didn't get any feedback that it worked or not. Please advise if not received.

    Thanks

    Brad
    Wednesday, March 4, 2009 7:46 PM
  • I just checked an nothing has been uploaded.  Netmon 3.2 capture would make my life easier.


    Developer Consultant
    Wednesday, March 4, 2009 10:32 PM
    Moderator
  • Tom,

    I still can't make the file upload work (with Firefox or Konqueror). Have sent a private mail in lieu.

    I'll work on getting netmon onto my sbs2003 test server.

    Brad
    Thursday, March 5, 2009 6:10 AM
  • Looks like the transfer site test matrix was limited to IE.  I have contacted the program manager for the transfer site to see if this can be addressed in the future.

    Sorry about that.


    Developer Consultant
    Thursday, March 5, 2009 6:12 PM
    Moderator
  • Tom,

    Have hopefully managed to capture the required traffic and upload the netmon .cap file.

    Brad
    Friday, March 6, 2009 11:27 PM
  • Tom,

    One of our developers did some further investigation into this.

    He reported
    I have been investigating the SetCollapseState/FreeBookmark trying to
    solve this last issue and think the problem may come from MS
    specifications.

    If we compare GetCollapseState reply and SetCollapseState reply:

                            mapi_GetCollapseState: struct GetCollapseState_repl
                                CollapseState            : SBinary_short cb=217
    [0000] AC 00 21 7F 80 00 00 00   7F 07 3F 0E 51 0E 99 07   ..!..... ..?.Q...
    [0010] 42 07 02 0E 1A 0E 9F 0E   51 0E 51 0E A7 0C 03 01   B....... Q.Q.....
    [0020] 01 01 01 00 21 7F 80 00   00 00 7F 07 3F 0E 51 0E   ....!... ....?.Q.
    [0030] 99 07 42 07 02 0E 1A 0E   9F 0E 51 0E 51 0E A7 0C   ..B..... ..Q.Q...
    [0040] 21 01 01 01 01 00 21 7F   80 00 00 00 7F 07 3F 0E   !.....!. ......?.
    [0050] 51 0E 99 07 42 07 02 0E   1A 0E 9F 0E 51 0E 51 0E   Q...B... ....Q.Q.
    [0060] A7 0C 33 01 01 01 01 00   21 7F 80 00 00 00 7F 07   ..3..... !.......
    [0070] 3F 0E 51 0E 99 07 42 07   02 0E 1A 0E 9F 0E 51 0E   ?.Q...B. ......Q.
    [0080] 51 0E A7 0C 46 01 01 01   01 00 21 7F 80 00 00 00   Q...F... ..!.....
    [0090] 7F 07 3F 0E 51 0E 99 07   42 07 02 0E 1A 0E 9F 0E   ..?.Q... B.......
    [00A0] 51 0E 51 0E A7 0C 58 01   01 01 01 00 2B 7F 01 7F   Q.Q...X. ....+...
    [00B0] 07 3F 0E 51 0E 99 07 42   07 02 0E 1A 0E 9F 0E 51   .?.Q...B .......Q
    [00C0] 0E 51 0E A7 07 02 0E 23   0E 8A 0E 7C 0E 51 01 01   .Q.....# ...|.Q..
    [00D0] 01 01 00 7F 80 00 00 00   00                       ........ .
                    mapi_response            : (handles) number=1


                            mapi_SetCollapseState: struct SetCollapseState_repl
                                bookmark                 : SBinary_short cb=87
    [0000] 03 00 00 00 00 00 00 00   00 00 00 00 01 00 00 00   ........ ........
    [0010] 00 1B FC 14 01 00 00 00   00 1C 8D C5 00 00 00 00   ........ ........
    [0020] 06 00 00 00 00 00 00 00   00 00 2B 00 7F 01 7F 07   ........ ..+.....
    [0030] 3F 0E 51 0E 99 07 42 07   02 0E 1A 0E 9F 0E 51 0E   ?.Q...B. ......Q.
    [0040] 51 0E A7 07 02 0E 23 0E   8A 0E 7C 0E 51 01 01 01   Q.....#. ..|.Q...
    [0050] 01 00 7F 80 00 00 00                              .......
                    mapi_response            : (handles) number=1

            * The CollapseState blob returned by GetCollapseState specifies
            - accordingly to MS specifications - a collapse state for a
            categorized table.

            * The bookmark blob returned by SetCollapseState specifies the
            current cursor position.

    Both blobs sounds to have a slightly different meaning but however share
    a significant number of bytes starting at offset 0xAD for
    GetCollapseState blob and 0x2C for SetCollapsetState reply:

                                                 7F 01 7F 07   ........ ..+.....
    [0030] 3F 0E 51 0E 99 07 42 07   02 0E 1A 0E 9F 0E 51 0E   ?.Q...B. ......Q.
    [0040] 51 0E A7 07 02 0E 23 0E   8A 0E 7C 0E 51 01 01 01   Q.....#. ..|.Q...
    [0050] 01 00 7F 80 00 00 00

    This blob is prefixed with a 2 bytes integer which specifies its size.

    Furthermore GetCollapeState first 2 bytes (00 AC) specifies a 0xAC size
    which strangely matches with a blob ending where common bytes start.

    This made be conclude both the GetCollapseState and SetCollapseState
    blobs have more complex structures than what MS specifications say and
    the bookmark blob only be part of SetCollapseState returned one.

    Using this conclusion, I've been hacking around the potential "real
    bookmark" blob returned by SetCollapseState and noticed that
    FreeBookmark would return MAPI_E_CALL_FAILED whenever our blob size
    extends 0x4 bytes. Hum 0x4 bytes ... uint32_t?

    I've truncated the internal bookmark blob to be 4 bytes rather than the
    full SetCollapseState returned blob and FreeBookmark no longer return
    MAPI_E_CALL_FAILED but MAPI_E_NO_SUPPORT, which despite the error looks
    to me like real progress:

         EcDoRpc: struct EcDoRpc
            in: struct EcDoRpc
                handle                   : *
                    handle: struct policy_handle
                        handle_type              : 0x00000000 (0)
                        uuid                     : 4373d3a6-9b11-4ade-ad03-92fffe51071b
                size                     : 0x0000fff0 (65520)
                offset                   : 0x00000000 (0)
                mapi_request             : *
                    mapi_len                 : 0x0000000f (15)
                    length                   : 0x000b (11)
                        mapi_request: struct EcDoRpc_MAPI_REQ
                            opnum                    : 0x89 (137)
                            logon_id                 : 0x00 (0)
                            handle_idx               : 0x00 (0)
                            u                        : union EcDoRpc_MAPI_REQ_UNION(case 137)
                            mapi_FreeBookmark: struct FreeBookmark_req
                                bookmark                 : SBinary_short cb=4
    [0000] 03 00 00 00                                       ....
                        mapi_request             : (handles) number=1
                            handle                   : 0x00000033 (51)
                    length                   : *
                        length                   : 0x000f (15)
                    max_data                 : 0xfff0 (65520)
    rpc request data:
    [0000] 00 00 00 00 A6 D3 73 43   11 9B DE 4A AD 03 92 FF   ......sC ...J....
    [0010] FE 51 07 1B F0 FF 00 00   00 00 00 00 0F 00 00 00   .Q...... ........
    [0020] AE A5 2C A5 A5 A1 A5 A6   A5 A5 A5 96 A5 A5 A5 00   ..,..... ........
    [0030] 0F 00 F0 FF                                       ....
    rpc reply data:
    [0000] 00 00 00 00 A6 D3 73 43   11 9B DE 4A AD 03 92 FF   ......sC ...J....
    [0010] FE 51 07 1B F0 FF 00 00   00 00 00 00 0C 00 00 00   .Q...... ........
    [0020] AD A5 2C A5 A7 A4 A1 25   96 A5 A5 A5 0C 00 00 00   ..,....% ........
    [0030] 00 00 00 00                                       ....
         EcDoRpc: struct EcDoRpc
            out: struct EcDoRpc
                handle                   : *
                    handle: struct policy_handle
                        handle_type              : 0x00000000 (0)
                        uuid                     : 4373d3a6-9b11-4ade-ad03-92fffe51071b
                size                     : 0x0000fff0 (65520)
                offset                   : 0x00000000 (0)
                mapi_response            : *
                    mapi_response            : length=8
                    mapi_response: ARRAY(6)
                        mapi_repl: struct EcDoRpc_MAPI_REPL
                            opnum                    : 0x89 (137)
                            handle_idx               : 0x00 (0)
                            error_code               : MAPI_E_NO_SUPPORT (0x80040102)
                    mapi_response            : (handles) number=1
                        handle id                : 0x00000033 (51)
                length                   : *
                    length                   : 0x000c (12)
                result                   : MAPI_E_SUCCESS (0x0)
    * FreeBookmark                       : MAPI_E_NO_SUPPORT


    Finally, another thing that may point in this 0x4 sized bookmark's blob:
    CreateBookmark always - according to our tests - returns a blob which
    size is 0x4 bytes ...

    Friday, March 6, 2009 11:28 PM
  • Brad, 

    There is a check in the code that returns an error you are seeing if the number bytes is greater than 4 bytes.  I will file a TDI against the document to fix this.


    Developer Consultant
    Saturday, March 7, 2009 12:35 AM
    Moderator
  • Tom,

    Thanks for the info. I assume that "the code that returns an error" is part of the path for handling RopFreeBookmark.

    I'm still not sure how to proceed on this one though. Is it possible to clean up the Bookmark returned by RopSetCollapseState?

    Brad
    Saturday, March 7, 2009 3:09 AM
  • Yes it's part of the code path but when you get's parsing error like this Exchange stops processing the ROPs in the requests as it considers the request invalid. 

    In correcting the size of the bookmark should will address the parsing error and allow the ROP to be parsed and executed.  But having fixed that the operational error you are now getting I will research this today.


    Developer Consultant
    Monday, March 9, 2009 6:20 PM
    Moderator
  • Brad,

    I'm still waiting for a response from development to confirm my research about bookmarks.  I have asked specifically more details on the use cases around RopFreeBookmark against the ROPs that do return a Bookmark in a response.


    Developer Consultant
    Friday, March 13, 2009 5:41 PM
    Moderator
  • Tom,

    Thanks for the update - appreciated.

    Brad
    Friday, March 13, 2009 11:03 PM
  • RopFreeBookmark is used for custom bookmarks created by the RopCreateBookmark.

    RopFreeBookmark does not apply to pre-defined bookmarks (such as BOOKMARK_BEGINNING, BOOKMARK_CURRENT and BOOKMARK_END) used in the RopSeekRow request, the RopQueryRows response, and the RopFindRow request.

    RopCreateBookmark supports only contents table and hierarchy table. As you have discovered that if you do send a RopFreeBookmark request for contents views the server returns will return ecNotSupported.

    Note that if a client does use a custom bookmark and does not send RopFreeBookmark the Exchange server will release all bookmarks related to a table when that table is released (as a result of sending ROPRelease on the table) or when RopResetTable, RopSortTable, or RopRestrict is sent.

    Developer Consultant
    Tuesday, March 17, 2009 1:15 AM
    Moderator
  • Tom,

    Thanks for this, however it doesn't explicitly cover RopSetCollapseState. I guess you haven't proposed this as an answer for that reason.

    Can you confirm that the Bookmarks returned using RopSetCollapseState should not be free'd using RopFreeBookmark (i.e. MS-OXCTABL Section 2.2.2.20 "RopSetCollapseState Semantics" is incorrect when it says " The bookmark returned MUST be freed by using RopFreeBookmark.")

    Could we also get some additional data in MS-OXCTABL Section 3.2.5.15 "Processing RopFreeBookmark" on the other error codes that may be returned, and what they mean?

    Brad


    Tuesday, March 17, 2009 7:01 AM
  • Hi Brad,

    Yea I didn't mark this as proposed as there is more information needed to clarify the use cases.  In discussion with development today here is some additional information to help clarify your issue.

    • RopSetCollapseState only supports Contents Tables.
    • RopFreeBookmark only supports Hierarchy Tables. 

    RopFreeBookmark should in my opinion support Contents Tables as spelled out in the documentation but in the current implementation of Exchange it is not supported hence the error.  The documentation in it's current form as you pointed out is incorrect.



    Developer Consultant
    Wednesday, March 18, 2009 4:45 AM
    Moderator
  • I forgot to mention that I have filed a TDI to get some more detail on MS-OXCTABL Section 3.2.5.15 on other error codes that may be returned.  I will post these once I get feedback from development.


    Developer Consultant
    Sunday, March 22, 2009 4:04 AM
    Moderator
  • Hi Brad,

    There are various layers for errors returned that is the size of incremental parsing the stream which will return RpcFormat (0x000004B6) and ROP field checking with size validation.  As you stepped on the field for bookmark cannot be larger than 4 byte returns RpcFormat error.  The TDI that has been filed is asking that there should be additional error codes be added for protocol level field checking.

    After the incremental parsing you have checking to validate that you have a valid logon object.  If you do not have a valid logon object NullObject (0x000004B9) will be returned.   Then there is a check to make sure that the object you are referencing is still valid.  If the object is in an invalid state then InvalidObject (0x80040108) will be returned.  And finally there is a check to make sure you have access to the server object.  The if you do not have access then AccessDenied (0x80070005) will be returned. 

    RopFreeBookmark request then checks to verify that the FreeBookmark is on a FolderView object.  If not then you get the returned error code NotSupported (0x80040102). And finally there is a validation of the actual bookmark and if it is considered invalide then InvalidBookmark (0x80040405) is returned.

    The error codes above are documented in Section 2.4 Error Codes of [MS-OXCDATA].

    Does this help?


    Developer Consultant
    Thursday, April 9, 2009 7:20 PM
    Moderator
  • Tom,

    This is very valuable information, much appreciated.

    Since it doesn't answer the original question about cleanup after RopSetCollapseState, I'm leaving this one as unanswered.

    Brad
    Thursday, April 9, 2009 10:55 PM
  • I was reviewing my discussion with development I never got closure on the bolded text below.  Let me circle back with them so we can close this out and file the TDI.

    <snippet>
    When a higher layer or the user needs to reset the expanded/collapsed state of the categories to a previously cached state, it MUST send RopSetCollapseState with the BLOB returned from RopGetCollapseState. The bookmark sent in the response to RopSetCollapseState SHOULD be freed using RopFreeBookmark. The client MAY choose to not send RopFreeBookmark, however, this may degrade server performance until the table is released via RopRelease.
    </snippet>


    Developer Consultant
    Friday, April 10, 2009 5:15 AM
    Moderator
  • The above <snippet> is confusing because of the word "SHOULD".  The current implementation of Exchange 2007 RopFreeBook mark will fail if the Bookmark from RopSetCollapseState is passed in. 

    There should be a note in the documentation that calls this out that this has not been implemented for Exchange 2007.  I will file a TDI to add a note to this section calling out that this is not implemented in Exchange 2007.  I also confirmed with Outlook that they do not call RopFreeBook for content views.


    Developer Consultant
    Saturday, April 11, 2009 1:41 AM
    Moderator
  • Tom,

    So back to the original question: is there any way to clean up after RopSetCollapseState?

    Brad
    Wednesday, April 22, 2009 9:39 PM
  • Nope.  I also looked at Exchange Server 2010 beta and the current behavior is the same.


    Developer Consultant
    • Marked as answer by Brad Hards Friday, April 24, 2009 12:02 AM
    Thursday, April 23, 2009 11:38 PM
    Moderator
  • Tom,

    Thanks.

    Brad
    Friday, April 24, 2009 12:03 AM