UAG and Office integration

***Updated on 23 Jan 2013 with information about Forms-based authentication***

When publishing SharePoint servers, one key feature is office integration – the ability of Office applications to open documents from the UAG site. This might sound like something we should take for granted, but it’s actually a complex mechanism, and understanding it can be a real eye-opener when it comes time to troubleshoot issues with it.

The challenge with office integration stems from the fact that UAG proxies the connection to the SharePoint server, and so an application such as Word or Excel can’t just fetch a document, because it needs to authenticate to UAG and have a valid session with the UAG server. When you use UAG to access web applications, you normally login to the UAG server first using the login form, which establishes a session and allows you to complete subsequent requests to other URLs of the various applications UAG is publishing. However, when you attempt to open an office document, it has to be open with the office application…not the browser. When you click on a document, the browser triggers the office application to open, and gives it the URL of the document to download. In such a situation, when the office application asks UAG for the document, how would it know that it’s OK to give it out? How would it know that the office application is to be allowed access?

Classic MS-Office integration

When UAG was introduced originally, it was equipped with a special mechanism for SharePoint and Office integration to allow such actions. This mechanism is based on a special cookie that UAG creates for this purpose. This cookie is read by the office application, and it allows it to hook into the existing UAG session and use it to read the file. The cookie’s value is identical to the UAG session cookie, allowing UAG to recognize the request and service it. This is how this cookie looks:

image

The cookie’s name is derived from the trunk name, which is portal2adfs above, and appended with “PersistForOffice”.

An important property of this cookie is that it is a PERSISTENT cookie (hence the name PersistForOffice). A persistent cookie is a cookie that has a specific expiration time and date, as opposed to a SESSION cookie, which does not. A session cookie expires when you close the browser, but a persistent cookie may live on until its expiration time is reached. Another important property of a persistent cookie is that it’s accessible outside the browser, which is why the Office applications can read it. For this technology to work, we have to use a persistent cookie, because if it was a session cookie, the office apps wouldn’t have access to it. However, since the cookie is accessible outside the browser, we also need to be sure that it won’t be reused later, after the UAG session has concluded, so we set a very short expiration time on the cookie – 15 seconds, to be precise. This also means that we need to re-create the cookie every 15 seconds, as long as the UAG session is active. This is where the tricky part is…

This cookie is created by UAG when you launch a SharePoint application through it, and UAG has two mechanisms that complement each other for this purpose.

The first mechanism is a special JavaScript that UAG sends to the client browser. When the browser runs the script, the cookie is created with an expiration time of 15 seconds. This is the function:

function whlFunction() { var g_cookieName = whlGetCookieName(); var today = new Date(); var expiry = new Date(today.getTime() + 15 * 1000); // 15 seconds var g_cookie = Get_Cookie(g_cookieName); var g_domainName = "<%=GetCookieDomainAttr()%>"; var isSecureProtocol = null; if ("<%=g_secure%>" == "1") isSecureProtocol = "secure"; if (g_cookieName.indexOf("PersistForOffice") == -1) g_cookieName = g_cookieName + "PersistForOffice" ; Set_Cookie(g_cookieName,g_cookie,expiry,"/",g_domainName,isSecureProtocol); return true; }

Since the cookie would expire in 15 seconds, we would need to have it re-created automatically to provide continuous accessibility to Office, and so in another function, UAG sets the cookie-creation function whlFunction to run every 10 seconds (shorter than 15 seconds, so we can be sure it’s always there even if the JavaScript is a little delayed). To do so, the JavaScript uses the JavaScript setInterval command, with an internal of 10 seconds. This is the function that performs the rescheduling:

function isPersNeeded() { var coo = Get_Cookie("NeedPersCookie"); if (coo == "1") { if (g_inter != null) clearInterval(g_inter); g_inter = null; whlFunction(); setInterval("whlFunction()",10 * 1000); } }

As I said, these functions are unique to UAG, and UAG has to send them to the client browser somehow. UAG comes with a special ASP page called SharePoint.asp, which is part of UAG’s InternalSite, and contains these functions (as well as others). For the client to have these functions, it has to call the SharePoint.asp page, which it wouldn’t normally do by itself. To have this done, UAG injects a special command into one of the files that SharePoint would normally call for. This file is Init.js for SharePoint 2010, and Core.js for SharePoint 2007. UAG inserts the following commands into it:

image

To insert these commands, UAG uses AppWrap , which performs a search-and-replace on the default content of the Init.js or core.js files. UAG’s default AppWrap configuration file has this search-and-replace function designed specifically for this by the UAG product team. The function searches for a piece of text within the default SharePoint script file, and replaces it with “our” function. This is the original script before and after the insertion:

image

In AppWrap, this is how the Search-and-replace look:

<DATA_CHANGE> <URL case_sensitive="false">.*init.js.*</URL> <!-- sps14 insert the whlsp14.js into core.js –> <SAR> <SEARCH encoding="base64">function ULSA13()</SEARCH> <REPLACE encoding="base64"> dhtmlLoadScript("WhlOwnURLscripts/applicationScripts/whlsp14.js"); dhtmlLoadScript("WhlOwnURLsharepoint.asp? site_name=WhlSiteName&secure=WhlSecure"); function dhtmlLoadScript(url) { var e = document.createElement("script"); e.src = url; e.type="text/javascript"; document.getElementsByTagName("head")[0].appendChild(e); } function ULSA13() </REPLACE> </SAR> </DATA_CHANGE>

In the default AppWrap file, the content of the search and replace strings are Base64 encoded, so if you open the file on your own server, the strings would appear to be garbled. This is perfectly normal, and if you want, you can use a Base64 decoder to decode and see the text.

So, when a SharePoint page is called by the client, SharePoint will call the init.js file, and UAG will recognize the call to that file and perform the S&R on it, inserting the call to Sharepoint.asp. Then, the UAG functions in SharePoint.asp will run and start creating the cookie, as well as re-running and re-creating it repeatedly to keep the cookie there as long as the session is active.

Sharepoint 2007 differences.

The above procedure differs a little for SharePoint 2007. With 2007, the cookie is the same, but the injection with AppWrap is a little more complicated. With 2007, instead of injecting the code into init.js, we inject it into core.js, which is a different SharePoint script. To do this, we perform this action in 2 parts. First, we use UAG’s SRA mechanism (also known as AAP), which executes before AppWrap, to insert our keyword into Core.js. This is the SRA code:

<URL> <!-- When clicking on a link in site Hierarchy , the link should be opened in the same frame—> <NAME>.*core\.js.*</NAME> <SEARCH encoding="base64">dG9wLmxvY2F0aW9uPXVybDs=</SEARCH> <REPLACE encoding="base64">d2luZG93LmxvY2F0aW9uID0gdXJsOw==</REPLACE> <!-- insert a line into core.js so the AppWrap can hook into it and place sharepoint.asp –> <SEARCH encoding="base64">dmFyIGl0ZW1UYWJsZURlZmVycmVkPW51bGw7</SEARCH> <REPLACE encoding="base64">dmFyIGl0ZW1UYWJsZURlZmVycmVkPW51bGw7DQovL2luc2VydHNw</REPLACE> </URL>

Decoded back from Base64, it looks like this. I’ve highlighted the relevant portion, as the other one is unrelated.

<URL> <!-- When clicking on a link in site Hierarchy , the link should be opened in the same frame—> <NAME>.*core\.js.*</NAME> <SEARCH encoding="base64"> top.location=url;</SEARCH> <REPLACE encoding="base64"> window.location = url;</REPLACE> <!-- insert a line into core.js so the AppWrap can hook into it and place sharepoint.asp –> <SEARCH encoding="base64"> var itemTableDeferred=null;</SEARCH> <REPLACE encoding="base64"> var itemTableDeferred=null; //insertsp </REPLACE> </URL>

Now that core.js has the //insertsp keyword in it, AppWrap, which runs a tad later, will replace it with the actual function to call for SharePoint.asp:

<!-- insert sharepoint.js into core.js—> <SAR> <SEARCH encoding="base64">//insertsp</SEARCH> <REPLACE encoding="base64" >dhtmlLoadScript("WhlOwnURLsharepoint.asp?site_name=WhlSiteName&secure=WhlSecure"); function dhtmlLoadScript(url) { var e = document.createElement("script"); e.src = url; e.type="text/javascript"; document.getElementsByTagName("head")[0].appendChild(e); } </REPLACE> </SAR>

Originally, in base64, it looks like this:

<!-- insert sharepoint.js into core.js—> <SAR> <SEARCH encoding="base64">Ly9pbnNlcnRzcA==</SEARCH> <REPLACE encoding="base64" >ZGh0bWxMb2FkU2NyaXB0KCJXaGxPd25VUkxzaGFyZXBvaW50LmFzcD9zaXRlX25h bWU9V2hsU2l0ZU5hbWUmc2VjdXJlPVdobFNlY3VyZSIpOw0KZnVuY3Rpb24gZGh0bWxMb2FkU2NyaXB0KHVybCkNC nsNCiAgIHZhciBlID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgic2NyaXB0Iik7DQogICBlLnNyYyA9IHVybDsNCiAgIGUudHl wZT0idGV4dC9qYXZhc2NyaXB0IjsNCiAgIGRvY3VtZW50LmdldEVsZW1lbnRzQnlUYWdOYW1lKCJoZWFkIilbMF0uYXBw ZW5kQ2hpbGQoZSk7IA0KfQ0K</REPLACE> </SAR>

What could go wrong?

Sometimes, the client side integration fails, because it’s dependent on AppWrap and SRA. The most common cause is if the UAG administrator has decided to configure UAG to not run AppWrap and SRA. Normally, UAG would process every file, but you can tell it not to using the Skip Body Parsing option in the advanced trunk configuration:

Above you can see how the URL “.*” was added to the skip body parsing setting for SharePoint. Some organizations do this because they don’t want to have UAG process links in documents, but don’t realize that by doing this, they are killing a fundamental building block of UAG.

Another possible cause is if SharePoint is published not using the SharePoint template. The AppWrap is keyed to the SharePoint application type, so if the URL is not recognized as belonging to SharePoint, UAG won’t apply the appropriate search-and-replace, and so the functions won’t be there to create the cookie.

Secondary mechanism introduced in SP2

As part of working on UAG SP2, the product group have decided to add a secondary mechanism to this. As part of that change, if UAG identifies a request as pertaining to a SharePoint site, the filter will generate the cookie (on the server!) and attach it to the response. If you inspect such a request using HTTPWatch or Fiddler, you would see that each response to a request for a SharePoint page has the cookie as “received” or “set-cookie”, indicating the server has sent it. For example:

image

This behavior can also be seen in a UAG trace, where it would log a line like this:

[1]11a0.de4 09/27/2012-11:51:28.639 [whlfilter CExtECB::ProcessHeaderFromRWS WhlExt2IWS.cpp@7308] Noise:Try to AddHeader for PersistForOffice session cookie (PFC=000000000C81A318)

Cookie challenges

Even when either or both of these cookie-generation mechanisms work perfectly, there are still other challenges to this. One challenge comes from the fact that the Office applications don’t normally have access to the Persistent cookie. With Windows Vista and IE7, Microsoft introduced two new security features called Integrity Level (IL) and Protected Mode (PM). With Vista and IE7, the browser runs in Protected Mode, which is a virtual sandbox that protects the integrity of the operating system. Running in protected mode puts IE at Integrity Level “low”, while normal applications like Office have an integrity level of “medium”. Because of the different Integrity Level, the cookies created by IE cannot be read by Office…which would be bad news for our integration.

To address this, UAG’s Endpoint Cleanup component (a.k.a. The Attachment Wiper) has a special function that can move the cookie from Low IL to Medium IL, making it available to Office. For administrators who intentionally choose to NOT use the UAG Client components, another option is to add the SharePoint’s URL into the user’s Trusted Sites list in IE. Once the site is in Trusted Sites, IE operates at a higher Integrity Level for it, which allows it to put the cookie in the Medium storage and make it accessible to the office applications. If an administrator chooses to both disable the client components and is unable to add the site to the trusted sites list…then another option is to enable rich client 401, which is our next topic.

Another variation of cookie transfer/access issues is when IE’s protected mode is on. This mode is disabled by default for Trusted sites, and enabled for Intranet, Internet and Restricted Sites. Some organizations turn it on for Trusted sites as well as a security hardening measure. Once on, it prevents Office (or other applications) from accessing the cookies stored by IE. Some organizations don’t change the settings for protected mode, but might add the UAG and SharePoint URLs in the Intranet zone instead of Trusted Sites. Since protected mode is on by default for Intranet, putting the URLs in that zone effectively kills Office integration, and that’s bad news, of course. This type of settings, by the way, has a negative impact on ADFS authentication as well, as it also depends on cookie sharing between IE and apps. This type of situations can be confirmed if you can see the office integration successfully creating the PersistForOffice cookie mentioned earlier, but a UAG trace shows that UAG isn’t receiving the cookie. Naturally, before starting deep tracing, start by checking the zone settings, and observe which zone the browser shows when accessing the UAG URL.

image

Another thing to keep in mind regarding ADFS and cookies is that when using ADFS authentication, cookies become much more important. ADFS authentication relies on cookies to perform SSO, so anything that interrupts the flow of cookies can cause a problem. In addition to the zone settings being a possible challenge with ADFS cookies, another thing that affects it are the cookie’s persistent setting. If your ADFS is setup to use session cookies, then these cookies are not available to the Office applications, so when opening a document, the user will be prompted for authentication. The cookie settings are described in this whitepaper, and to achieve a transparent single-sign-on, I recommend setting the ADFS cookie to be persistent.

Yet another issue that could happen sometimes is a client/server time sync problem. As I noted above, with UAG SP2 and onward, the UAG server itself generates the persistent cookie and sends it to the client. If, however, the client’s clock is set to a different time than the UAG server, the cookie might expire before it arrives. Since the default expiration time for the cookie is within 15 seconds, even a few seconds difference between the UAG and the client’s clock could be a big problem. With the cookie being expired, the browser will clear it, and then the Office application won’t be able to use it to hook into the session. Note that your clients, just like UAG, are typically using the Windows Time Service to set their clock, and the user has little control over it other than turning it off and setting his clock manually (which would be tricky, as he would have to work over the phone with the UAG administrator to know the right time to set). If you suspect this is happening, you can confirm it with a Fiddler or HTTPWatch recording. In the recording, check the Set-Cookie from the server, and compare the cookie’s expiration time to the request time. For example:

image

image

In the above, you can see that a connection made at 16:00:00 receives a cookie that expires at 15:59:35 (the cookie response time is adjusted for GMT, so it appears as if it’s for 10PM). This is because the client time is ahead of the server by 25 seconds.

Rich client 401 authentication

Another integration mechanism that is available with UAG is Rich Client 401 authentication. The purpose of this is two-fold. First, it is a backup in case the basic integration (cookie-based) fails to work for some reason. Secondly, it allows the user to open a document directly from within the office application, even if he didn’t log in to UAG at all. Two scenarios where the latter is particularly useful is if the user wants to re-open a document that he already used earlier, and is in the MRU (Most Recently Used) list, or if a user wants to send a link to a document to someone else via Email, without the 2nd user having to open his browser and login to UAG first.

To enable this option, select it in the Portal Link tab of the SharePoint application:

image

If the administrator enables this, UAG would perform a 401 authentication conversation with the office application (instead of rejecting a request), create a session for it, and once that completes, deliver the requested document. In this situation, the office application will send the request for the document to UAG, which will respond to it with a 401 unauthorized http code. The Office app will display an authentication prompt to the user, and if the user feeds in correct credentials, the document will open:

image

I should note that a common misconception is that enabling this option sets UAG to do a pass-thru authentication…but that’s NOT true. Pass-through would mean that the office application authenticates directly against SharePoint, but as I said, that’s not the case. What this really means is that this eliminates the need to login to the trunk. Also, the authentication repository used is the one that’s selected on the SharePoint application’s authentication page (which could be different than the repository assigned to the trunk itself).

This option may be less savory for some organizations, which do not like the vague appearance of a 401, and so another option is available – MSOFBA (Microsoft Office Forms Based Authentication). This option has a mini-browser inside the office application, which allows it to login to UAG using UAG’s original login page. In such a case, a pop-up will open inside the office application, showing UAG’s login page. Once the user completes the login, the office application will be able to download the document.

Non-IE platforms

With non-IE platforms, the behavior is different. One reason for this is the fact that other browsers don’t use the same cookie mechanism that IE does. IE stores its cookies in the user’s cookie folder (%userprofile%\cookies), while Firefox, for example, stores them in a file called cookies.sqlite inside the user’s profile folder. This means that Office won’t be able to get the persistent cookie even though the classic integration works perfectly. However, normally, Firefox behaves differently in other ways, so this does not actually represent a problem, and here’s why.

As part of its design, SharePoint has a special function that overrides the browsers default action. Normally, when working with IE, clicking on an office document (as well as most other file links) would trigger the open/save dialog box. SharePoint’s special function (DispEx) configures the browser to behave differently, and send the link to the document to the office application, so the office application can download it off the site directly. This is good, because office has built-in functions that allow various workflows with SharePoint.

With Firefox, the behavior differs based on the platform. In Firefox running on 32-bit systems, the SharePoint DispEx integration function doesn’t work. Thus, when an office document is clicked, Firefox works using the classic “open/save” dialog box. Firefox then downloads the document with its own Download Manager, and then sends the document itself to the Office application:

image

Since DispEx doesn’t work with Firefox, this essentially bypasses the UAG integration as well, and Firefox is able to successfully open the document. Normally, “doesn’t work” is not good, but in our case…it is. Since the download is done in the browser, Office does not need to work with UAG itself, and it all goes smoothly. One limitation of this scenario is that it’s really only a partial integration. Since the document is downloaded locally by the browser, it doesn’t allow the various work-flow options that SharePoint usually offers. For example, if you make changes to the document, you can’t simply save it back, like you could with the IE-based integration (If you tried, it would simply save the document locally on the client).

On 64-bit systems, though, the DispEx function does work correctly in Firefox, and so Firefox behaves just like IE, and sends the link to the office application to open. Again, even though “works” usually is good…in our case, this is a problem, because the office app cannot pick up the cookie that it would normally get with IE. When the office app sends the request to UAG, UAG cannot hook it into the existing session and things don’t work out. If the option for allow rich clients to bypass trunk authentication is not enabled, UAG would either redirect the request to the login page, and the office app would open the login page as if it is the requested doc, or it would simply error out:

image

In such a situation, enabling the rich-client authentication option would allow the Office application to open the document, because with it, the office application can authenticate directly to UAG. The user will be presented with a 401 pop-up when the office app opens, and if he provides correct credentials, the document should open. Naturally, this is not really SSO…but that’s how things are with Firefox on 64 bit systems.

Office integration with Forms-Based authentication (FBA)

Some organizations have challenges using Windows authentication with their SharePoint servers, such as in situations where the target audience for the site are not employees of the company and don’t have accounts in the company’s domain. The recommended configuration for such a situation is to Federate with the partner organization, and configure ADFS to allow people from the other organization to log in. Another alternative is to configure SharePoint for Forms Based Authentication (FBA). This configuration allows the organization to define various data sources for authentication information, such as an SQL database. This, in itself, is far from simple to do, so before embarking on this journey, it’s important to be aware of the limitations. UAG comes pre-configured with SSO capabilities to SharePoint’s FBA, so when the users launch the SharePoint application from the UAG portal, UAG will detect the SharePoint login form, fill it, and log the user in. However, this SSO is limited to the browser only, so if the user attempts to open an office document using the client integration, things will fail.

The way this fails is that once the user clicks on the document in SharePoint, the client-integration ActiveX would send the link to the office app. The Office app will contact UAG to get the file, and to this, SharePoint will respond with a redirect to the forms-based login page. However, Office apps are incapable of handling this form, and so the user will be faced with a 401 authentication prompt. Even if he tries to fill and submit it, it will bounce back repeatedly until eventually, the SharePoint login form itself will open within the office application.

As I said above, this is the expected behavior, and cannot be solved. The SharePoint configuration page even states this explicitly:

image

The note above suggests turning OFF client integration. This disables the integration ActiveX, which will cause the document to be downloaded inside the browser itself, and then sent to the Office application. This will enable the document to be accessed, though it won’t allow the document to be uploaded back to the SharePoint site, or take advantage of various work-flow functions that office normally enables. If the user needs to update the document on SharePoint, he would have to save a local copy, and then use SharePoint’s upload page to update the document. This applies to all current SharePoint and office versions (2007, 2010 and 2013).

How to diagnose and resolve issues pertaining to office integration?

The 1st step would be to establish if the problem is with the classic integration, or somewhere else. For this, we must collect an HTTPWatch or Fiddler recording of a connection to SharePoint, and observe the content of init.js (or core.js in case of SharePoint 2007). If the UAG call to SharePoint.asp is not there…that’s trouble. You can also look for the URL SharePoint.asp itself being called. If UAG is running SP2, it will have the persistent cookie generated by the server, but it may appear to work intermittently. That could happen if the user opens a page (and then UAG would send the cookie) and then doesn’t do anything for over 15 seconds. The cookie will expire, and if then the user would try to open a document, the office application won’t be able to attach to the session and get an error instead.

To resolve this, make sure skip body parsing is not configured for anything other than the vti_bin items you can see above (these are fine and are part of the UAG configuration normally). If this is OK and still not working, verify the application type was selected correctly (for example, make sure that you are not publishing SharePoint 2007 with the 2010 template, or vice versa). You can also inspect a UAG trace of the request to init.js, and make sure the application type is recognized correctly. If the application type is incorrect, UAG would not apply the right S&R to the content, and the integration may fail.

image

Another situation you might see is if SharePoint has been configured for Debug mode. In such a case, SharePoint calls or different files – init.debug.js instead of init.js, and core.debug.js instead of core.js. Since UAG’s AppWrap is keyed to the file name, the difference in the files kills the AppWrap functions. In addition, the content of the files is different, so the S&R will fail anyway. Essentially, this mode is not supported, so you should put SharePoint back into regular mode. If this is not possible, you can create a custom AppWrap that will be keyed to the debug file, and have the appropriate S&R. With debug mode, the function that we need to replace is not ULSA13, but ULSxSy. For example, for SharePoint 2010, this would be the AppWrap code:

<APP_WRAP ver="3.0" id="RemoteAccess_HTTPS.xml"> <MANIPULATION> <MANIPULATION_PER_APPLICATION> <APPLICATION_TYPE>SharePoint14AAM</APPLICATION_TYPE> <DATA_CHANGE> <URL case_sensitive="false">.*init.debug.js.*</URL> <SAR> <SEARCH encoding="base64">ZnVuY3Rpb24gVUxTeFN5KCk=</SEARCH> <REPLACE encoding="base64">ZGh0bWxMb2FkU2NyaXB0KCJXaGxPd25VUkxzY3JpcHRz L2FwcGxpY2F0aW9uU2NyaXB0cy93aGxzcDE0LmpzIik7DQoJCQkJZGh0bWxMb2FkU2NyaXB 0KCJXaGxPd25VUkxzaGFyZXBvaW50LmFzcD9zaXRlX25hbWU9V2hsU2l0ZU5hbWUmc2Vjd XJlPVdobFNlY3VyZSIpOw0KCQkJCWZ1bmN0aW9uIGRodG1sTG9hZFNjcmlwdCh1cmwpDQoJ CQkJew0KCQkJCXZhciBlID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgic2NyaXB0Iik7DQoJCQ kJZS5zcmMgPSB1cmw7DQoJCQkJZS50eXBlPSJ0ZXh0L2phdmFzY3JpcHQiOw0KCQkJCWRvY 3VtZW50LmdldEVsZW1lbnRzQnlUYWdOYW1lKCJoZWFkIilbMF0uYXBwZW5kQ2hpbGQoZSk7 DQoJCQkJfQ0KDQoJCQkJZnVuY3Rpb24gVUxTeFN5KCk=</REPLACE> </SAR> </DATA_CHANGE> </MANIPULATION> </APP_WRAP>

If the integration does appear to behave properly (you can see the call to SharePoint.asp and the persistent cookie), you can confirm that UAG is receiving the request with the cookie correctly with a trace. Gather a trace of the request, and isolate the request for the document itself. Then, inspect the cookies that came with the request. This is how it should look:

image

Note the user agent, which shows that an office application is trying to read the document, and the PersistForOffice cookie that’s coming in. If the request does NOT contain the PersistForOffice cookie, then it means that office was unable to read the cookie:

image

Above, you can see the request is very similar – it has the same method (head) and agent (office), but the cookie is the regular trunk cookie, and not the PersistForOffice cookie.