locked
WSE / DIME File Transfer Performance

    Question

  • I'm working on an file transfer gateway using WSE with DIME for file attachments. Our goal is to replace our direct file repository access (via windows network folder sharre) with the Web Service gateway for security purposes.
     
    As it stands now, all workstations have direct read-write access to the file repository share. The Web Service gateway will secure this for us. While the web service gateway is more secure, it doesn't perform anywhere near up to the performance of a direct file copy. I'm working with a 27 MB file in my test environment; the client and the web service are on the same workstation.
     
    Using direct file access, I can copy this 27 MB file in under 2 seconds. Using the web service, it take between 5 and 6 seconds to transfer the same file. We understand that we might have to trade a bit of speed for the increased security, however a 400% increase is really more of a hit than we'd been anticipating. Is there any way that I can tune performance to optimize WSE / DIME for large file attachments?
     
    Thanks for your help!
    Wednesday, September 7, 2005 5:11 AM

Answers

  • Some points to note here

    1) WSE (2.0 and 3.0) is XML DOM based which means that for large documents these are loaded into memory. You may be doing this with you file transfer protocol, but it has a significant impact on the throughput. This cannot be avoided
    2) Unless you use HTTPS, DIME is not secure in that you cannot use message level security to protect the message. Do you use HTTPS and DIME?
    3) DIME as a protocol is being replaced by MTOM. See the WSE 3.0 Beta on MSDN. This is also true for WCF. MTOM is a better bet for a project that you are working on now. The downside is that you need to move to .NET V2.0 for this and this may not be possible for you.
    4) If you use WSE 3.0 and MTOM you can stream from the server to the client which improves the performance significantly. You cannot stream from the client to the server though.
    5) The maximum document size for DIME and WSE 2.0 is approx 100MB and this is pushing it. I strongly suggest that you consider some message chunking and reconsitution approach for large documents, thereby reducing the message size. Depending on your scenario and how you tune this, this may give a throughput performance

    Thanks, Mark Fussell, WSE Program Manager
    Tuesday, September 27, 2005 6:54 AM

All replies

  • Some points to note here

    1) WSE (2.0 and 3.0) is XML DOM based which means that for large documents these are loaded into memory. You may be doing this with you file transfer protocol, but it has a significant impact on the throughput. This cannot be avoided
    2) Unless you use HTTPS, DIME is not secure in that you cannot use message level security to protect the message. Do you use HTTPS and DIME?
    3) DIME as a protocol is being replaced by MTOM. See the WSE 3.0 Beta on MSDN. This is also true for WCF. MTOM is a better bet for a project that you are working on now. The downside is that you need to move to .NET V2.0 for this and this may not be possible for you.
    4) If you use WSE 3.0 and MTOM you can stream from the server to the client which improves the performance significantly. You cannot stream from the client to the server though.
    5) The maximum document size for DIME and WSE 2.0 is approx 100MB and this is pushing it. I strongly suggest that you consider some message chunking and reconsitution approach for large documents, thereby reducing the message size. Depending on your scenario and how you tune this, this may give a throughput performance

    Thanks, Mark Fussell, WSE Program Manager
    Tuesday, September 27, 2005 6:54 AM
  • Mark, read your reply with interest. I have a similar project, but the consideration is not filesize so much as bandwidth. What I want to avoid is adding any overhead, and SOAP may be that. But I note that with WSE 3.0 and MTOM, soap envelope stuff can be turned off.

    in my case I'm looking at MTOM as a way to stream binary files off our very slow VHF based network (think police cars now). Essentially I get a 9.2-18 kbps throughput. But the good news is that I want to send small files, mostly between 3 and 100kb.

    To summarize, the major solution points are:

    1) reduce bandwidth usage (use either http or tcp)

    2) ensure reliable transfer with minimal intervention (automation and scheduling behind scenes)

    3) return MD5 checksum and verify completion of file transfer

    4) clients (cars) all feed to central web service which smartly redirects completed files to their final destinations

    Any comments as to the applicability of WSE 3.0 here?

    TIA

     

    Thursday, May 18, 2006 8:45 PM