Uploading large files into a SharePoint document library. RRS feed

  • Question

  • I am using Visual studio 2008 with .net 3.5 sp1 and WSS 3.0 sp1:

    I wrote a client application that runs on user's desktop machines that automates file uploads to our WSS document libraries.  It works very well for small files, but chokes on large files because the .net webclient and httprequest classes will load an entire file into memory and I will get out of memory exceptions.

    I am using Front Page RPC calls inside of a httprequest.  Even when I load my files into a stream, the httprequest.write method will load the entire file into memory.  The memory used will actually be two to three or more times the actual file size which I think is odd.  I tried a 350mb file and watched performance monitor use up 600 mb of RAM and the page file increased by a similar amount before it crashed out; so about 1.2 gb at the minimum.

    After googling around I found that setting the "AllowWriteStreamBuffering" property to false on the request object will prevent this behavior, but will cause authentication issues.  Memory usage will be very low since I buffer only a small amount of data at a time, but then I get an error "This request requires buffering data to succeed".  Apparently the server is rejecting the data that isn't a part of the default writestream buffer.  I found a solution here: that instructs to pre-authenticate the request with a head request before sending the data.  It also instructs to modify a few IIS metabase settings.  I'm not sure if this article is compatible with SharePoint or not because if I, for example, set IIS to 'Negotiate', then IE 8 can no longer authenticate to the server. 

    In any case  I followed the example code anyway but I always get the "This request requires buffering data to succeed" error.

    Here is my test code.  It works if I set AllowWriteStreamBuffering to true and only upload smaller files, but If i set that property to false, I cannot get it to authenticate.

    Anyone know what I might be doing wrong or know any additional troubleshooting steps?


            public void StreamUpload()
                string serviceName = "http://devserver/subsite/_vti_bin/_vti_aut/author.dll";
                string documentName = "testlibrary/35481.txt";
                string metaInfo = "vti_title;SW|" + "TEST TITLE";
                string putOption = "overwrite";
                string comment = string.Empty;
                string returnStr = string.Empty;
                string returnError = string.Empty;
                string method = "method=put+document%3a12.0.4518.1016&service_name=%2f&document=[document_name={0};meta_info=[{1}]]&put_option={2}&comment={3}&keep_checked_out={4}\n";
                method = String.Format(method, documentName, metaInfo, putOption, HttpUtility.UrlEncode(comment), "false");
                putOption = HttpUtility.UrlEncode(putOption);
                byte[] data;
                    //add line feed character to delimit end of command 
                    byte[] fpRPCCall = System.Text.Encoding.UTF8.GetBytes(method + "\n");
                    data = new byte[fpRPCCall.Length];
                    fpRPCCall.CopyTo(data, 0);
                    Uri URL = new Uri(serviceName);
                    HttpWebRequest wReq;
                    HttpWebResponse response;
                    CredentialCache myCredCache = new CredentialCache();
                    myCredCache.Add(URL, "NTLM", (NetworkCredential)CredentialCache.DefaultCredentials);
                    wReq = (HttpWebRequest)HttpWebRequest.Create(serviceName);
                    wReq.PreAuthenticate = false;
                    wReq.Credentials = myCredCache;
                    wReq.UnsafeAuthenticatedConnectionSharing = true;
                    wReq.UserAgent = "Upload Test";
                    wReq.Method = "HEAD";
                    wReq.Timeout = 10000;
                    response = (HttpWebResponse)wReq.GetResponse();
                    Stream s = response.GetResponseStream();
                    StreamReader sr = new StreamReader(s);
                    String sReturnString = sr.ReadToEnd();
                    //Real Request
                    wReq = (HttpWebRequest)WebRequest.Create(serviceName);
                    wReq.PreAuthenticate = false;
                    wReq.Credentials = myCredCache;                
                    wReq.UnsafeAuthenticatedConnectionSharing = true;
                    wReq.UserAgent = "Upload Test";
                    wReq.Method = "PUT";
                    wReq.AllowWriteStreamBuffering = false;
                    wReq.KeepAlive = true;
                    wReq.Timeout = 1000000000;
                    wReq.ContentType = "application/x-vermeer-urlencoded";
                    wReq.Headers.Add("X-Vermeer-Content-Type", "application/x-vermeer-urlencoded");
                    FileStream fileBinary = new FileStream("C:\\temp\\doc\\35481.txt", FileMode.Open, FileAccess.Read);
                    wReq.ContentLength = fpRPCCall.Length + fileBinary.Length;
                    using (Stream requestStream = wReq.GetRequestStream())
                        requestStream.WriteTimeout = 100000000;
                        requestStream.Write(fpRPCCall, 0, fpRPCCall.Length);
                        int bytesRead = 0;
                        int chunkSize = 4096;
                        int tailSize;
                        int chunkNum = Math.DivRem((int)fileBinary.Length, chunkSize, out tailSize);
                        //chunk the binary directly from the stream to buffer. 
                        for (int i = 0; i < chunkNum; i++)
                            data = new byte[chunkSize];
                            bytesRead = fileBinary.Read(data, 0, chunkSize);
                            requestStream.Write(data, 0, chunkSize);
                        //send the remainder if any. 
                        if (tailSize > 0)
                            data = new byte[tailSize];
                            bytesRead = fileBinary.Read(data, 0, tailSize);
                            requestStream.Write(data, 0, tailSize);
                        //Now get the response from the server 
                        response = (HttpWebResponse)wReq.GetResponse();
                        s = response.GetResponseStream();
                        sr = new StreamReader(s);
                        sReturnString = sr.ReadToEnd();
                catch (Exception ex)
                    //error handling 
                    Console.WriteLine("Error during stream upload" + ex.Message);
    • Edited by Nate Gross Monday, February 1, 2010 9:29 PM Code block was duplicated
    Monday, February 1, 2010 9:27 PM