Large File Upload in SharePoint 2010

Posted on 3/24/2010 @ 11:44 AM in #SharePoint by | Feedback | 5214 views

Okay this is a big BIG B-I-G problem. And with SP2010 it’s going to be more prominent, because atleast at the server side, SharePoint can support large files much much better than SharePoint 2007 ever did.

The issue with very large files being uploaded through any browser based API are -

  1. Reliably transferring gigabyte or bigger files without breakages over a protocol like HTTP, which is better suited for tiny transfers like images and text.
  2. Not killing your browser because it has to load all that in memory
  3. Not killing your web server because
    1. All that you upload through HTTP post, first gets streamed into IIS Memory, w3wp.exe memory before the ENTIRE FILE finishes uploading .. before it is stored. Which means,
      1. You cannot show an accurate and live progress bar of the upload, IIS gives you no such accurate metric of an upload. All the counters it gives you are approximate.
      2. Your w3wp.exe eats up all server memory – 4GB of it, for a 4GB upload.
      3. A thread is kept busy for the entire duration of the upload, thereby greatly limiting your web server’s capability to serve newer requests.
      4. Kills effective load balancing.
  4. Not killing your content database because,
    1. As you are uploading a very large file, that large file gets written sequentially into the DB, and therefore for a very large file very severely impacts the database performance.

I had put together another video showing RBS usage in SharePoint 2010. I talked about many practical ramifications of using RBS in SharePoint in that video. Note that enabling large file support will never ever be a point and click job, simply because there are too many questions one needs to ask, and too many things one needs to plan for.

However, one part that will remain common across all large file upload scenarios, in SharePoint or outside of SharePoint is to do it efficiently while not killing the web server.

In this video, I describe using the Telerik Silverlight Upload control with SharePoint 2010 to enable efficient large file uploads in SharePoint. Presenting ..

The video


Sound off but keep it civil:

Older comments..


On 3/25/2010 3:15:48 AM Pritesh said ..
Nice Article.

How to download tetrik Control ... or Need to purchase


On 3/25/2010 6:40:10 AM Sahil Malik said ..
Pritesh, see www.telerik.com. You can download a fully functional trial, or purchase directly from the website.

Sahil


On 3/25/2010 10:43:04 PM Yang said ..
very nice!


On 4/3/2010 4:45:34 PM Jeremy Ragan said ..
"...Note that enabling large file support will never ever be a point and click job, simply because there are too many questions one needs to ask, and too many things one needs to plan for..." --> Evidently you've never seen a demo of StoragePoint 8^) . There's a reason it won MSFT's award for "Most Innovative Product" at the last SPC.


On 4/4/2010 2:37:50 AM Sahil Malik said ..
Jeremy, I've seen the demo. I still think large file upload will never be a point and click job. I am not dinging your company's product though, but I just don't buy it that the product replaces good planning and oversight.


On 4/12/2010 11:45:41 AM Daryl Gwyn said ..
Sahil,

Very interesting article - thanks!

You mention both in the article and in the video the example of 4GB files. SharePoint does not support files of over 2GB in size. Even when using RBS for external storage, files large than 2GB are not supported.


Are you suggesting that this Telerik control somehow overcomes the 2GB filesize limit?


On 4/12/2010 2:22:06 PM Sahil Malik said ..
Daryl,

I must be mistaken then. I thought RBS let you cross the 2GB limit because 2GB was driven by the DB. Is that not the case?

Sahil


On 6/2/2010 12:10:45 PM Mac said ..
Nice video. I am using this control for my asp.net web application.


Couple of questions though

1) Does this support webfarm environment?


2) How does it handles file upload if multiple users are uploading same file at same time. (Common in healthcare app)


On 11/24/2010 10:45:15 AM John said ..
This is great. Except that SPContext.Current.Web.Files.Add does not accept Filestream ?!?

Seriously, I feel like I'm taking crazy pills. Am I the only one who sees that?!

Sincerely,


John


On 11/24/2010 12:20:41 PM John said ..

On 11/23/2011 5:21:49 PM Paul said ..
I know this is an older post but still applies for anyone that stumbles upon it:


1) SharePoint in the API has a 2 GB file size limitation. Even with a add-on like StoragePoint (which operates under the API), it is constrained by this 2GB limit.


2) transferring files via the browser is a real bad idea as the author suggests but the common way to deal with this is to construct an HTTPHandler so that the complete file is not loaded into browser memory. This is actually the approach the Telerik upload control uses. If you want complete control over the process and be completely fault tolerant (if you cannot afford to lose track of file uploads) then I'd suggest you roll your own HTTPHandler.


On 1/27/2012 11:13:49 AM Pinaki said ..
Hi Sahil.


I need help on this please. I followed your video to the point, but I face a problem.


When I say "Start upload" on the control, all uploads become yello warning signs with "handler not found". I think many others are facing this problem too?


I have the ashx as "Build action : none" and so it gets copied ot the layouts folder.

I have BuildAction: compile on the .cs handler code.

What am I missing?


Please help.


On 3/13/2012 10:22:02 AM Purna said ..
Hi Sahil,

I am also facing the same problem.

When I say "Start upload" on the control, all uploads become yello warning signs with "handler not found".


Please help me on this.