Raw HTTP Access -- Adding a file to a folder in the mesh without using .NET or microsoft libraries

Question

My Boss and I have been really beating our heads in, looking for the right way to do this. It doesn't seem to be well documented at this time, and the little bits that I have found seem to be contradictory or outdated.  Basically what I'm trying to do is add a file to the mesh by posting it to the HTTP webserver, and then give it a data entry with meta-info, and I need to do all this without the use of .NET or microsoft libraries, as I'm am not on a windows platform. I would prefer to do this in raw-xml, or ATOM-xml, but if anyone can describe it using anything else, that would still be helpful.  I figured out how to do a lot of different GET,POST,PUT,OPTIONS, and DELETE operations, like creating and modifying a Mesh Object (custom and liveDesktopFolder), creating and changing a Data Feed, creating and changing Data Entries. I've been able to do some feed sync operations as well.

Now I'm trying to find the correct way to upload a file to an existing LiveDesktop Folder, as well as post any associated meta-data and File Name info.  I'm confident that I've read ALL the documentation that Microsoft has on the topic, as well as blogs and other postings, so I'd rather not be pointed to another msdn document that I've already read 10 times over.  But I will describe how far I've gotten (Mostly lucky guesses), and maybe if someone out there has some answers or insite, please help me out.

So far I was able to post a file directly to the /Mesh/MeshObjects/{ID}/DataFeeds/{ID}/MediaResources location, and the server creates a default data entry for the file (linked to the resource), as well as a default title; and I can also see it on the Live Desktop in the Folder that I posted it to! However I have troubles modifying the Data Entry and Naming it what I want (error --> "Optimistic Concurrency...").  So I have a feeling that this is not the correct way to upload a file.

Second time around I read a couple msdn docs explaining how the MediaStagingArea works, however they explain it using C#,and VB objects, so it's pretty ambiguous what is going on under the sheets. So far going that route I've been able to POST to /Mesh/MeshObjects/{ID}/DataFeeds/{ID}/MediaStagingArea successfully. However the server returns a 204 status (success,no-content), and I noticed a very strange looking HTTP header in the response with the following name-value-pair:

"Staged-Media-Resource-Uri" = "x-custom://stagedMediaResource.uri/<MediaResource xmlns:i=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns=\"http://user.windows.net\"><Hash>UiyQzzIChxlkgFtCbCg1mQ==</Hash><Id>29ada81b-ea14-4a40-9610-4d8afa86a5ba</Id><LastUpdatedTime>2009-01-22T20:51:14.4208808Z</LastUpdatedTime><Length>399396</Length><MediaType>audio/wav</MediaType><Name>MediaLinkEntry</Name><PublishDate>2009-01-22T20:51:14.4208808Z</PublishDate></MediaResource>";

In the past I've never really seen XML come back in the header --that's pretty curious first of all-- and it's just strange seeing the XML mixed in with the weird looking url (stubbed maybe?) in front of it "x-custom://stagedMediaResource.uri/". So in the documentation that I saw it looked like the .NET objects in the example knew what to do with this header, and where to go next, but I couldn't really tell what that was; something to do with FeedSync maybe? So I found another msdn document (http://msdn.microsoft.com/en-us/library/dd199343.aspx); which I'm guessing is outdated because it kept referring to a /SyncEntries location, but I couldn't find a /SyncEntries, so maybe it's talking about the /Sync feed? -- That's where I'm trying to post the rest of the info to. Anyway it describes (also very ambiguously) the steps below:

1. The client posts the media to /MediaStagingArea.
   
2. The server returns an ID corresponding to the media entry to the client in the form of a response header called Staged-Media-Resource-Url.
   
3. The client creates a new media link entry in the local feed in this format: <link rel="StagedMediaResource" href="ID">, where ID
   is the ID from the server.
4. The client posts this feed to the /SyncEntries feed. The POST operation on /SyncEntries must happen within eight hours of the post of the media resource to /MediaStagingArea. Otherwise, the media resource may get garbage collected.
   
5. The server creates a media resource from the ID obtained from <link rel="StagedMediaResource"> for the new media link entry and updates it with a link to the media resource.
   
6. The server performs the MERGE operation.

 Step 1 done; Step 2 still good (the header was actually Uri not Url, but whatever); now Step 3 I didn't really know how to approach.  I know where to find the "ID", but I don't really know where the new <link> element is supposed to go. So I tried a couple different things: First I added it to the toplevel, just inside and under the <feed> element (the outermost root element from the copied current ATOM-XML from the /Sync feed), I posted that but just got back the current feed again with no new changes. Second, I tried making a new <entry> element and put the <link> element inside that, still no go, same result as before. So then I tried progressively adding in other elements inside entry along with the <link> element, things like <title>, <summary>, <type>file<type>, and other things that the other existing <entry> elements had contained to make a 'real' file entry; but still get the same existing feed back as the response.

Can anybody explain how to POST an Entry for this new MediaResource?

Or does anybody know if there is another/better way to do what I'm trying to do here?

 

Thanks,

 Jake

 

 

 

---

Aaron: "You know that story, about how NASA spent millions of dollars developing this pen that writes in Zero G? Did you ever read that?" Abe: "Yeah." Aaron: "You know how the Russians solved the problem?" Abe: "Yeah, they used a pencil." Aaron: "Right. A normal wooden pencil. It just seems like Philip takes the NASA route almost every time." --Primer (2004)

Answer 1

I ran this file upload sample code with Fiddler intercepting the traffic.  It appears that the .NET SDK client gets the job done with a single POST to MediaResources rather than the multi-step MediaStagingArea process described in the documentation you found.  Here's what the POST looks like:

POST /V0.1/Mesh/MeshObjects/YB4YTGST7CCUFLEXJZLIO32SYQ/DataFeeds/TBLHYJFYGY4ULEJYV2YJWE4TLI/MediaResources HTTP/1.1

Slug: =?utf-8?B?TWVzaFRlc3Qud2F2?=

LiveFX-Safe-Agent: True

LiveFX-Parent-Id: urn:uuid:00000000-0000-0000-0000-000000000000

LiveFX-Default-Category: File

Accept: application/atom+xml;type=entry;application/atom+xml

Authorization: asdfasdfasdfasdfasdfasdfasdfasdfasdf

Content-Type: audio/wav

Host: user-ctp.windows.net

Cache-Control: no-store,no-cache

Pragma: no-cache

Transfer-Encoding: chunked

Expect: 100-continue

Hopefully you can simply imitate this.  As you continue to explore, it would probably be most efficient to use Fiddler plus .NET SDK test code to figure out what successful requests need to look like.

---

http://orand.blogspot.com

Answer 2

By the way, please keep us all posted (pun intended) on your progress with raw HTTP access.  Live Framework is intended to be easily accessible from any platform that can speak HTTP, so any programming model weirdness (XML in headers) or difficulties are quite valuable to bring to people's attention at this early stage while they can still change it.

---

http://orand.blogspot.com

Answer 3

Using this blog post as a starting point, here's some raw HTTP code in C# that works on my box:

string ticket = WindowsLiveIdentity.GetTicket(

new Uri("https://user-ctp.windows.net"),

new NetworkCredential("[e-mail]", "password"));

var request = (HttpWebRequest)WebRequest.Create(

"https://user-ctp.windows.net/V0.1/Mesh/MeshObjects/ASDF/DataFeeds/QWERT/MediaResources");

request.Method = "POST";

request.Accept = "application/atom+xml;type=entry;application/atom+xml";

request.Headers.Add("Slug", "MeshTest2.wav");

request.Headers.Add("LiveFX-Default-Category", "File");

request.Headers.Add(HttpRequestHeader.Authorization, ticket);

request.ContentType = "audio/wav";

var file = File.OpenRead(@"C:\temp\MeshTest.wav");

byte[] fileBytes = new byte[file.Length];

file.Read(fileBytes, 0, (int)file.Length);

file.Close();

var reqStream = request.GetRequestStream();

reqStream.Write(fileBytes, 0, fileBytes.Length);

reqStream.Flush();

reqStream.Close();

var response = request.GetResponse();

If you don't supply the "Slug" header, the data entry's name (and therefore the displayed filename) will end up as "MediaLinkEntry".  If you leave out "LiveFX-Default-Category" of "File", the category ends up as "DataEntry" but still appears to work fine from the Live Desktop.  The appropriate default LiveFX-Parent-Id gets created for us automatically so I didn't add it, but if you're doing nested folders then you will probably need to add it.  I'm not quite sure what LiveFX-Safe-Agent is all about, but it doesn't appear to be necessary.

---

http://orand.blogspot.com

Answer 4

Hey Oran -- Thanks for your swift and understandable response! Just what I was looking for!

It's funny you mentioned using fiddler to watch how .net handles the process, because just before I read your response I tried the same exact thing,  And was wondering what that slug header was all about, seeing how it was encoded, I had no idea. -- This makes a lot more since now.  I went back into my application and was able to get the file named correctly. Thank you very much!

I will definately keep everyone "POST"ed on what I find out, and will more than likely have more questions.

By the way -- do you know how the .NET libraries are encoding that 'slug' header, and for what reason? -- I'm just trying to determine if it would be a good idea for my application to follow suit.

 Thanks,

Jake

---

Aaron: "You know that story, about how NASA spent millions of dollars developing this pen that writes in Zero G? Did you ever read that?" Abe: "Yeah." Aaron: "You know how the Russians solved the problem?" Abe: "Yeah, they used a pencil." Aaron: "Right. A normal wooden pencil. It just seems like Philip takes the NASA route almost every time." --Primer (2004)

Answer 5

Actually -- I just found the answer to my last question about the Slug encoding (so I'll answer my own question)....

I relize that there would be some weird characters in this field, and apparently Slug being part of the Atom specification, allows for MIME:part3 (of the rfc variety http://www.ietf.org/rfc/rfc2047.txt) encoding notation (http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-13.html#rfc.section.9.6)

The notation is as follows:

=?char set?encoding?content?=

The actual accepted values for these are contained in the rfc document.

---

Aaron: "You know that story, about how NASA spent millions of dollars developing this pen that writes in Zero G? Did you ever read that?" Abe: "Yeah." Aaron: "You know how the Russians solved the problem?" Abe: "Yeah, they used a pencil." Aaron: "Right. A normal wooden pencil. It just seems like Philip takes the NASA route almost every time." --Primer (2004)

Answer 6

Great detective work!  The C# code that generates the "encoded-word" looks like this:

string.Format("=?{0}?B?{1}?=", Encoding.UTF8.BodyName, Convert.ToBase64String(Encoding.UTF8.GetBytes(slug)))

---

http://orand.blogspot.com

Answer 7

Thanks Oran for your great contribution to the Live Framework Community and quickly resolving request from Jake.

Jake - would love to hear more from your and participate in this forum. Let us know if you have a blog / space / feed we can follow.

thanks a lot! - great thread!

---

This posting is provided "AS IS" with no warranties, and confers no rights.

Answer 8

Jake,

Thanks for sharing your research. Here is my understanding based on an experiment in the recent past. Hopefully, it may help you as enclosures are the way to load big files as part of data entries and doanload them on demand as enclosures are not part of feedsync automatically during meshobject or datafeed syncs. This helps in perf and saving bandwidth.

Media Resources (also called enclosures) can be associated with DataEntries in two ways:

1) Per Atom Publishing Protocol: Here the basic interaction model is the following: the user issues a POST of a media resource against LOE, LOE in-turn manufactures a DataEntry with a <link rel=edit-media …/>, which is called Media Link Entry or MLE. Please refer to the section on Media Resources and Media Link Entries in Atom Publishing Protocol spec.

2) Per FeedSync binding for HTTP: The user uploads 1 or more enclosures at the /StagingMediaLocation (sp?) and LOE returns a temporary token (link with rel=stagingarea) back corresponding to each of the enclosures uploaded. The user later creates a set of DataEntries locally (this represents his local state) at his end and then associates the tokens returned from LOE for some set of DataEntries and uploads the DataEntries (using an HTTP POST). LOE converts each DataEntry containing the <link rel=stagingarea (again, sp?), to a valid MLE with rel=edit-media link.

In both cases, the result is an MLE.

Best regards,

Rajan

---

(This post is provided "AS IS" with no warranties, and confers no rights.)