none
Big Data Transfer RRS feed

  • Question

  • Hi,

    Need some suggestions on coming with a fault-tolerant way of transferring big data over http.

    I have a big database (40GB) and I need to take parts of it and transfer them to a client through WCF services. Data to be transfer can go from 1MB to 10GB depending on the client selection.

    So, the steps are:

    1. Client makes a selection

    2. Data is pulled from the database and provided to the client (WCF service)

    3. Client process the data and add it in its local database

    What's the most appropriate way of sending this data? I want to be able to resume the process if it fails in the middle of the transfer. Should I be using a WCF streaming?

    Thanks.

    Thursday, December 12, 2013 5:58 PM

Answers

  • You can't use FTPS?  there isn't really much difference in security between https and ftps. Anyway have your look at the webpage below

    http://msdn.microsoft.com/en-us/library/ms733742(v=vs.110).aspx

    It says the following

    As mentioned earlier, enable streaming only for large messages (with text or binary content) if the data cannot be segmented, if the message must be delivered in a timely fashion, or if the data is not yet fully available when the transfer is initiated.

    The two key items are

    1) data cannot be segmented - You don't have any restriction about adding segmenting

    2)  if the message must be delivered in a timely fashion - real time data.  This doesn't apply in your case. 

    WCF streaming will add XML tags increasing the size of you transfer which is already very large.

    What you want to be able to do is really be able to resume a message if the connection terminates.  You can setup your own resume protocol using http.  Simply send the byte count a the start of the upload along with the file name.  Then break the file uploads into segments of fix sizes like 4096.  Only store into the file when you get valid segments that contain 4096 bytes.  To implement a resume simply get the byte count of the existing file on server.  Then when you open the upload file in client offset the stream by the number of bytes already in the server.


    jdweng

    Friday, December 13, 2013 11:39 AM

All replies

  • FTP allows a resume mode if the connection has been broken.  See webpage below.  Two links that you should look at.

    http://stackoverflow.com/questions/5678119/how-does-a-ftp-server-resume-a-download

    Then see microsoft webpage below

    http://social.msdn.microsoft.com/Forums/en-US/7b655de3-6744-4005-912b-c5a8e9ea6bd3/resume-ftp-file-upload


    jdweng

    Thursday, December 12, 2013 7:41 PM
  • Thank you for your answer. Unfortunately I can't use ftp because of the clients' restrictions. The only protocols  that are allowed are http and https.

    Thursday, December 12, 2013 8:37 PM
  • You can't use FTPS?  there isn't really much difference in security between https and ftps. Anyway have your look at the webpage below

    http://msdn.microsoft.com/en-us/library/ms733742(v=vs.110).aspx

    It says the following

    As mentioned earlier, enable streaming only for large messages (with text or binary content) if the data cannot be segmented, if the message must be delivered in a timely fashion, or if the data is not yet fully available when the transfer is initiated.

    The two key items are

    1) data cannot be segmented - You don't have any restriction about adding segmenting

    2)  if the message must be delivered in a timely fashion - real time data.  This doesn't apply in your case. 

    WCF streaming will add XML tags increasing the size of you transfer which is already very large.

    What you want to be able to do is really be able to resume a message if the connection terminates.  You can setup your own resume protocol using http.  Simply send the byte count a the start of the upload along with the file name.  Then break the file uploads into segments of fix sizes like 4096.  Only store into the file when you get valid segments that contain 4096 bytes.  To implement a resume simply get the byte count of the existing file on server.  Then when you open the upload file in client offset the stream by the number of bytes already in the server.


    jdweng

    Friday, December 13, 2013 11:39 AM