Uploading Large Files to SharePoint 2013 from .NET Using CSOM and REST

This is a topic that seems to come up with some frequency and when I needed to do it recently I could not find a good working sample of doing this from server-side code.  The scenario here is imagine you want to upload some very large files to SharePoint via CSOM.  You have some code running "somewhere" – could be in your own Web API controller or something similar to it that can tolerate a long upload time.  The upload time could be long because you want to upload large files, i.e. larger than the 1.5MB support from CSOM alone when uploading files to SharePoint.  In that case you need to use the REST interface into SharePoint.  In pulling together the code to do this I didn't find a complete sample anywhere, but I managed to cobble together the code I wrote along with random pointers here and there from about four other TechNet articles.  Blech!  I feel like I should get some kind of finder's fee for figuring this out and pulling it together.  But I digress…  😉  Trust me when I say you're probably going to want to bookmark this posting because I think you will find it handy.

So let me start I suppose by hitting up some of the main challenges I hit when doing this and how I tackled each one, and then I'll finish with a fairly complete code sample (i.e. it will work for you when you plug in your own file you want to upload).  Here we go:

Challenge #1 – I Need an App Only Token and I'm using ACS

Virtually every sample I've ever seen for developing SharePoint apps using ACS (i.e. low trust) assume that there is some browser and user context present.  Well if you are really doing this via some code running "somewhere" then chances are it may not be triggered directly from a browser request.  Maybe it's a scheduled job, who knows, but for now let's just say that it was part of my scenario so I needed to solve it.  The problem gets more complex when you are doing this from something like a WCF or REST solution because they don't expose an HTTP Context object.  That means that you can't use the method in TokenHelper to get an access token because it requires the HTTP Context.  The good news is there is another method in TokenHelper that we can use to get the access token that we'll use to talk to SharePoint.  It looks like this:

//start out by getting a client context for o365

string o365ClientId = ConfigurationManager.AppSettings["o365ClientId"];

string o365ClientSecret = ConfigurationManager.AppSettings["o365ClientSecret"];

//get the Uri for the personal site

Uri siteUri = new Uri(siteUrl);

//this makes a connection to o365 using an App Only token since the user is not currently connected to o365

//NOTE:  The 5 parameter version of this method is something custom I wrote, but not needed for the general

//use case; just use the standard 3 parameter version of this method that comes out of the box.  Thanks!

var token = TokenHelper.GetAppOnlyAccessToken(TokenHelper.SharePointPrincipal, siteUri.Authority, TokenHelper.GetRealmFromTargetUrl(siteUri),

o365ClientId, o365ClientSecret).AccessToken;

So what I've done here is I had the ClientId and ClientSecret in my web.config for my SharePoint App anyways.  For reasons that aren't important to this scenario, I copied those values into two new settings in web.config – one called o356ClientId and the other called o365ClientSecret.  I take those values along with the Url to the site where I'm going to upload the file (which I put the siteUri variable) and let SharePoint and ACS go do their authentication oauth token thing.  When I'm done I have an access token I can use for an App Only call into the SharePoint site.

Challenge #2 – I need to get a Form Digest Value

Almost all of the example code you finds assumes that you are in a SharePoint hosted app and you are running client side code.  That's great because SharePoint emits a form digest value into every SharePoint page.  However in our scenario we're not working with a SharePoint page so how do we get this?  Well you can just take the siteUri you created above, and go make a POST to that site in the _api/contextinfo path and you will get a bunch of JSON back that includes a form digest value.  You don't include any content when you make your POST request however.  Here's an example of what that looks like:

HttpWebRequest restRqst = (HttpWebRequest)HttpWebRequest.Create(siteUrl + "_api/contextinfo");

restRqst.Method = "POST";

restRqst.Accept = "application/json;odata=verbose";

restRqst.Headers.Add("Authorization", "Bearer " + token);

restRqst.ContentLength = 0;

//get the response so we can read in the request digest value

HttpWebResponse restResponse = (HttpWebResponse)restRqst.GetResponse();

Stream postStream = restResponse.GetResponseStream();

StreamReader postReader = new StreamReader(postStream);

string results = postReader.ReadToEnd();

So you can see I use the access token I got in the previous chunk of code and use that in an authorization header to get access to the SharePoint site.  I then make my 0 byte POST request and I get back the JSON string, which in my code above has been stuck in the "results" variable.

Challenge #3 – How Do I Dig out the Form Request Digest

This of course is not a huge problem, but I did come across an interesting solution so I'm sharing here.  Typically when I work with JSON on the server side I'll design some classes into which I'll serialize the JSON so I have a nice object model that I can use to work with the results.  Well some very sharp folks that know ASP.NET much better than me turned me on to a much simpler way of digging out that data than trying to do some string parsing.  Here's what that code looks like:

JavaScriptSerializer jss = new JavaScriptSerializer();

var d = jss.Deserialize<dynamic>(results);

string xHeader = d["d"]["GetContextWebInformation"]["FormDigestValue"];

I really haven't deconstructed the "<dynamic>" feature enough to explain it well.  For now I will just say "hey, look at my cool code sample!"  The way I figured out the hierarchy to get down to FormDigestValue was just to look at the "d" variable in the debugger after it had been populated.  It essentially has a series of Dictionary objects that it builds on the fly to map a pseudo object model the JSON that was consumed.  So for example it had a Dictionary with a key of "d".  The value for that item was another Dictionary that had a key of "GetContextWebInformation".  And so on, and so on it goes, until it maps out the JSON that was returned.  It's pretty cool.

Challenge #4 – Where In the Blazes Do I Upload by File

This may have actually been the most challenging aspect of all.  Pretty much every single upload example I saw for using the REST endpoint tells you to upload it to "/_api/web/GetFolderByServerRelativeUrl('" + serverRelativeUrl + "')/Files/Add…blah…  No joy.  Never.  Ever.  To be clear, all I was trying to do was to upload the file into the Documents library in a user's OneDrive site.  Wasted LOTS of time trying different tweaks around that Url and got no where.  Finally found kind of a random sample in one of the TechNet docs that got me pointed to the correct location, so here is where I POST'ed my file upload:  siteUrl + "_api/web/lists/getByTitle('Documents')/RootFolder/Files/Add(url='" + fileName + "', overwrite=false)".  Yeah…get the list by title and then go into its RootFolder.  Yahooo, order is restored in my world.

So…assuming you have a stream of data from somewhere (like opening a local file, taking a file that's been uploaded, etc.) and you've stuck it in a byte array (I put mine in fileBytes), here is the complete chunk of code to do an upload for completeness; apologies in advance for the absurd formatting this blog site will put on the code:

string uploadUrl = siteUrl + "_api/web/lists/getByTitle('Documents')/RootFolder/Files/Add(url='" + fileName + "', overwrite=false)";

bool fileUploadError = false;


//now that we have the Url and byte array, we can make a POST request to push the data to SharePoint



//need to get the X-RequestDigest first by doing an

//empty post to http://<site url>/_api/contextinfo

HttpWebRequest restRqst = (HttpWebRequest)HttpWebRequest.Create(siteUrl + "_api/contextinfo");

restRqst.Method = "POST";

restRqst.Accept = "application/json;odata=verbose";

restRqst.Headers.Add("Authorization", "Bearer " + token);

restRqst.ContentLength = 0;


//get the response so we can read in the request digest value

HttpWebResponse restResponse = (HttpWebResponse)restRqst.GetResponse();

Stream postStream = restResponse.GetResponseStream();

StreamReader postReader = new StreamReader(postStream);

string results = postReader.ReadToEnd();


//get the FormDigestValue node

JavaScriptSerializer jss = new JavaScriptSerializer();

var d = jss.Deserialize<dynamic>(results);

string xHeader = d["d"]["GetContextWebInformation"]["FormDigestValue"];

//now create a new request to do the actual upload

restRqst = (HttpWebRequest)HttpWebRequest.Create(uploadUrl);

restRqst.Method = "POST";

restRqst.Accept = "application/json;odata=verbose";

restRqst.Headers.Add("Authorization", "Bearer " + token);

restRqst.Headers.Add("X-RequestDigest", xHeader);

restRqst.ContentLength = fileBytes.Length;


//take the document and get it ready to upload

postStream = restRqst.GetRequestStream();

postStream.Write(fileBytes, 0, fileBytes.Length);



//do the upload

restResponse = (HttpWebResponse)restRqst.GetResponse();


//assuming it all works then we'll get a chunk of JSON back

//with a bunch of metadata about the file we just uploaded

postStream = restResponse.GetResponseStream();

postReader = new StreamReader(postStream);

results = postReader.ReadToEnd();


There you go, hope you can find a good use for this.




Comments (9)

  1. alexandrad9x says:

    Tao http://dichvuketoanlongbien.com/
    Thằng http://dichvuketoanlongbien.com/a2-98-dich-vu-ke-toan-thue.html

    Tao. http://www.trungtamketoan.com.vn/


    Như http://www.tosvn.com
    Thế. http://iketoan247.blogspot.com
    Loại http://tailieuveketoan.blogspot.com
    Chó http://mauhinhnendep.blogspot.com
    Má. http://www.tosvn.com/search/label/Hack%20CF
    Tao http://www.tosvn.com/search/label/Hack%20AvatarStar
    Rủa http://www.tosvn.com/search/label/Hack%20Warcraft-Dota2
    Những http://hocketoan360.com/category/tai-lieu-ke-toan/
    Thằng http://iketoan247.blogspot.com/search/label/thong-tin-kinh-te
    Soi http://iketoan247.blogspot.com/search/label/tin-bai-ve-thue
    Tao http://hoclamketoan.edu.vn/
    Sẽ http://hoclamketoan.edu.vn/category/khoa-hoc-ke-toan
    Tan http://hoclamketoan.edu.vn/category/dich-vu-ke-toan
    Cửa http://hoclamketoan.edu.vn/category/hoc-lam-ke-toan
    Nát http://hoclamketoan.edu.vn/category/tai-lieu-ke-toan
    Nhà http://hocketoan360.com/
    Haha http://hocketoan360.com/category/khoa-hoc-ke-toan/

  2. Ed (DareDevil57) says:

    thanks for sharing.

  3. SG says:

    Great explanation. Thanks for sharing!

  4. m88 says:

    Statistics on recent Bleacherreport again between Ronaldo and Messi showed no great distance appear to form M88 :
    , so that may be made ​​the comment one more excellent player.
    In 10 years of playing football (since 2004), Messi played a total of 284 games, scored 249 goals and 100 of assists.

    Meanwhile, superstar M88 Portugal played 331 games, scored 270 goals, but applicants who are modest compared to the "rivalry" with the number 79 overall properties of the sliding shoe caps to professional soccer, Ronaldo and Messi had a goal and create total

    Some people said that Messi is not as comprehensive as Ronaldo, but getting 349 goals and tectonics, the owner of three golden balls M88 is actually a brilliant player. While Ronaldo’s performance this season that many people go from surprise to surprise the
    other. 13 goals in seven games in La Liga, CR7 is challenging the whole world, not only La Liga.

  5. ash says:

    Did you have any luck uploading files great than 300mb? I can get a file to upload but I cant get a response? In my test I uploaded a 327 mb file, it takes around 17 minutes to upload but on the restRqst.GetResponse(); it doesnt come back with anything, I’ve
    left it running for around 4 hours but nothing :s

    Any ideas?

  6. m88 says:

    m88 : http://m88en.com
    M88.com offer online sports games Asia, Sports Betting Asia, Sports Betting Sites Asia.

    m88asia : http://m88en.net
    Link to M88BET phone: m88en.com. – Register and Open Betting Account and Membership M88BET.

    m88bet : http://www.linkm88vip.com
    MANSION88 the house is one of the largest and most prestigious. Appeared quite early in the Asian market, the so-MANSION88 currently attracts more players.

    link m88 : http://m88wiki.com
    Home the M88 is the official sponsor of the football club in the Premier League
    Wish you happy with the new M88
    m88 casino online : http://m88free.com

  7. Free Online Training Videos, Courses and Tutorials says:

    I absolutely love your blog and find nearly all of your post’s to be precisely what I’m looking for.
    http://www.hungrymind.in">Free Online Training,
    http://www.hungrymind.in">Online Training Courses, http://www.hungrymind.in">Free Online Training Videos

  8. SharePoint 2013 Online Training in Hyderabad says:

    Great article …Thanks for your great information, the contents are quiet interesting. I will be waiting for your next post.
    http://staygreenacdemy.com">SharePoint 2013 Online Training in Hyderabad

  9. Tom Baxter says:

    I was having huge problems with this, I had to give a presentation on share point and I couldn’t get any of the big files to work!