Offlining Web Application Gallery Feed for Windows Azure Pack

[UPDATE]:   Based on feedback received on the contents of this post,  I have modified it to provide a more detailed explanation about hosting the Web Application Gallery in an environment with zero internet connectivity.

Introduction

The Windows Web Application Gallery (WAG) provides instant access to the most popular free web applications available today. With just a few clicks you can install any of these applications and be up and running within minutes. This is integrated into the Microsoft Web Platform Installer© and is available for users to download and host in their environments. More information about the Web Platform Installer can be found in the blog post by Shawn Gibbs,

Application Management - System Center and the Web Platform Installer (WebPI)

The WAG is also integrated within Windows Azure Pack for tenants to view and easily deploy the gallery items to the Windows Azure Web Sites Resource Provider in a few clicks.

Problem Statement

Organizations have firewalls set up to restrict internet access, and would like to host a copy of the feed and the installers locally within their infrastructure.

Organizations would like to host their own applications and expose them through the Web Application Gallery.

Scope

This guide will provide you information about offlining the applications in Web Application Gallery feed and their dependencies, hosting them and exposing it through the WAP Portal with no internet connectivity.

This will not provide you any information about adding your own custom applications to the gallery. You can find more information about adding custom applications to the Gallery at

https://blogs.msdn.com/b/azureappgallery/archive/2013/04/16/deconstructing-the-custom-feed-to-test-your-web-application-on-web-platform-installer.aspx 

Solution

Fortunately, there is a way to offline a WAG feed and host it locally using the Web Platform Installer command line tool (WebPICmd.exe). WebPICmd.exe has a /Offline parameter which, when provided a list of products to download and a target feed to download the products from, will download the products and their dependencies as mentioned in the feed.

You can host the offlined feed and the packages on an intranet location and modify the feed location in the WAP Admin Portal to point to the intranet location. This guide will provide you with instructions and scripts on how to do this.

Note: Once you offline the feed, you will need to maintain it on your own. You won't be notified automatically when there is a new version of an app, You might want to establish a schedule for updating your off-lined feed.

Click the download button to download the scripts used for this blog 

How the Web App Gallery Feature works in WAP

The Web Application Gallery in the portal is enabled by the ‘MgmtSvc-WebAppGallery’ service. The Web Application Feed which contains a list of Gallery Applications is specified in the Admin Portal under the Web Site Clouds –> Settings tab. The Web Products List is specified in the web.config of the Gallery service.

This ‘MgmtSvc-WebAppGallery’ service is responsible for processing the Web Application Feed to identify the products and use the Web Product List file to map the products to their dependencies and to store the location of the installation files for these, in the WAP database. The Gallery service updates this list roughly every 10 minutes.

When the Gallery is rendered in the Tenant Portal, the items are loaded from this table. When the users click on a certain item to install, the service will download the necessary installers and deploy the application in the Web Sites Resource Provider. The following diagram should give you an idea of the interactions involved.

image

Post offlining, these installers will be loaded from an offline server, as opposed to getting them from the internet.

Offlining the feeds and installers

Things you will need

  1. An Internet connected Machine
    You have to run the scripts on this machine to offline the feed. Once the packages are offline, you can then host them on a non-internet joined server and provide access to your users

  2. Web Platform Installer
    The WebPICmd.exe utility installs as a part of the Web Platform installer. You can installer Web Platform Installer from https://www.microsoft.com/web/downloads/platform.aspx

  3. Latest Web PI feed
    The latest feed at the time of writing this blog was for Web PI 4.6 at https://www.microsoft.com/web/webpi/4.6/webproductlist.xml. This is refreshed quite frequently so you have to identify the latest feed

  4. Web Application Gallery feed from the WAP portal
    Open the WAP Admin Portal and navigate to Web Site Clouds -> Settings. Note the FWD link in the FEED URL field under the web gallery feed. Paste the FWD link into the browser which will redirect you to the location of the WAP Gallery feed. This location is typically https://www.microsoft.com/web/wap/webapplicationlist.xml 

  5. Web Application Gallery feed in Web Platform installer: Open the Web PI feed and find the feed location of the Web Application Gallery referenced in the feed

        1: <?xml version="1.0" encoding="utf-8"?>
        2: <feed xmlns="https://www.w3.org/2005/Atom">
        3:   <webpiFile version="4.2.0.0" />
        4:   <title>Web Platform Installer 4.6 Feed</title>
        6:   <link rel="enclosure" href="https://www.microsoft.com/web/webpi/4.6/mediaproductlist.xml" />
        7:   <link rel="enclosure" href="https://www.microsoft.com/web/webpi/4.6/ToolsProductList.xml" />
        8:   <link rel="enclosure" href="https://www.microsoft.com/web/webpi/4.6/enterpriseproductlist.xml" />
        9:   <link href="https://www.microsoft.com/web/webpi/4.6/WebProductList.xml" />
       10:   <updated>2013-10-22T12:00:00.000Z</updated>
       11:   <author>
    

    Armed with these values, you can execute the following steps to offline the Gallery Feed

    Note: Instructions in Steps 6 to 11 are performed by executing the OfflineWebAppGallery.ps1 script attached to this blog. This script should be run from an elevated PowerShell console.

        1: .\offlinewebappgalleryv2.ps1 -webPiFeedUri "https://www.microsoft.com/web/webpi/4.6/webproductlist.xml" -webAppGalleryFeedUri "https://www.microsoft.com/web/wap/webapplicationlist.xml" -offlineLocation "E:\offlinewebpi"
    

    If you are curious to se what this script does, read on. Otherwise you can skip the following steps and go straight to the ‘Modifying the Offlined Feeds’ section

  6. Read the feeds from Steps 3 and 4

        1: [xml]$webPiFeedUriXml = (New-Object System.Net.WebClient).DownloadString($webPiFeedUri)
        2: [xml]$webAppGalleryFeedUriXml = (New-Object System.Net.WebClient).DownloadString($webAppGalleryFeedUri)
    

               

  7. The WAG feed is also a part of the standard Web PI feed. However, WAP supports a subset of the items that are present in the standard WAG feed and hence has a different feed from the one referenced in Web PI. Therefore, before we offline the feed, we have to modify the reference of the standard WAG feed from the Web PI feed to point to the Windows Azure Pack version of the Gallery feed. This can be done manually or by parsing the HTML. Use the value obtained in Step 5 for this

        1: $links = $webPiFeedUriXml.feed.link | Where-Object {$_.href -eq "https://www.microsoft.com/web/webpi/4.6/webapplicationlist.xml"} `
        2:     | %{$_.href = [string]$webAppGalleryFeedUri}
    
  8. Next would be to save the modified Web PI feed to a temporary location so that it can be provided as an input to WebPICmd.exe.

        1: $tempDir = "$env:temp\webPIoffline-"+(Get-Date).ToString("yyyy-MM-dd-HH-mm-ss-ffff")
        2: New-Item $tempDir -type directory -Force |Out-Null
        3: $tempWebPiFile = "$tempdir\webproductlist.xml"
        4: $webPiFeedUriXml.Save($tempWebPiFile)
    
  9. We now have to get a list of products for WebPiCmd.exe to offline. We can get this by parsing the WAG feed

        1: $productids = $webAppGalleryFeedUriXml.feed.entry.productid
        2: $list = [string]::Join(',',$productids)
    
  10. With this list of products, and a copy of the modified Web PI feed to point to the locations of the applications and their dependencies, we now have all the information required to pass on to WebPICmd.exe

        1: & "C:\Program Files\Microsoft\Web Platform Installer\WebpiCmd.exe" /Offline /XML:$tempWebPiFile /Path:$offlineLocation /Products:$list #/UpdateAll
    

    This will take each item in the $list and offline the application and its installers mentioned in the feeds, to the location specified by the $offlineLocation variable

  11. This will take a while depending on your internet connectivity. Once the process completes, you can use the feeds that are generated at $offlineLocation\feeds\latest\

As mentioned at the beginning, you can perform the above steps by executing the OfflineWebAppGallery.ps1 script that has been attached to this post from an elevated PowerShell Console

    1: .\offlinewebappgalleryv2.ps1 -webPiFeedUri "https://www.microsoft.com/web/webpi/4.6/webproductlist.xml" -webAppGalleryFeedUri "https://www.microsoft.com/web/wap/webapplicationlist.xml" -offlineLocation "E:\offlinewebpi"

Modify the –offlineLocation to point to your offline folder       

Modifying the offlined feeds

At this point, we have all the installers and an offlined version of the feeds pointing to these installers. We need to make a few tweaks to these feeds so that the Web Application Gallery Service can read them  and render the gallery on the tenant portal.

Warning: I will walkthrough the modifications that must be made to the feeds, but I sincerely suggest you run the ModifyOfflinedFeeds.ps1 script as it is *very* easy to make mistakes with these modifications. The Gallery Gods don’t take too lightly to erroneous feeds.

Before you run the script, you should know the location where the offline feed is going to be hosted. for instance, if I were to host the feed in a server called myserver  and in a folder called offlinewebpi , then the value of the –server parameter is “https://myservername/offlinewebpi/” - Note the '/’ at the end of the URL

  1. Make a backup of your offlined webproductslist.xml and webapplicationlist.xml
  2. Run the ModifyOfflinedFeeds.ps1 from an elevated PowerShell window and point I to the offlined feeds (per Step 11) as shown below
    1: .\ModifyOfflinedFeeds.ps1 -offlineWebPIFeed "E:\offlinewebpi\feeds\latest\webproductlist.xml" -offlineWebApplicationFeed "E:\offlinewebpi\feeds\latest\supplementalfeeds\webapplicationlist.xml" -server https://myservername/offlinewebpi/

Here’s what the script does (what you have to do manually if you’re not using the script Smile )

webProductlist.xml

  1. Modify the relative addressing for the webapplicationlist.xml to absolute addressing and in a URI format
  2. Modify ALL relative references to installers to absolute ones
  3. Assign all relativeHref values to href  and remove all relativeHref
  4. Assign all relativeIconUrl values to icon and remove all relativeIconUrl
  5. Assign all relativeInstallerURL values to installerURL and remove all relativeInstallerURL

webApplicationlist.xml

  1. Modify ALL relative references to installers to absolute ones
  2. Assign all relativeIconUrl values to icon and remove all relativeIconUrl
  3. Assign all relativeInstallerURL values to installerURL and remove all relativeInstallerURL

But, as I mentioned before, using the script will be faster.

Hosting the offlined feed

  1. At this point, you can now take the offlined folder and host it on the server and under the folder mentioned in the previous step

  2. This machine does not need internet connection and should be accessible by all the machines in that network. It does not have to be the same machine  hosting the Windows Azure Pack services

  3. We have to modify our environment to point to these offlined feeds. First, Open the WAP Admin Portal and navigate to Web Site Clouds -> Settings. Replace the value in the FEED URL field under the web gallery feed section with the offlined gallery feed and hit Save         
    image

  4. Now that the Application list in place, we should update the Products list location to point to the offline feed. This is in the web.config of the WebAppGallery service. Go to the machine where the WebAppGallery Service is installed and in a PowerShell window, decrypt the web.config of the service

        1: Unprotect-MgmtSvcConfiguration webappgallery
    
  5. Navigate to inetpub\MgmtSvc-WebAppGallery and open the web.config file

  6. Replace the value of the OfficialWebPIFeedUrl key to the offlined webproductlist.xml location and save it.
    image

  7. Open IIS Manager and find the WebAppGallery service. Stop the service and Start it again.  and Browse to the endpoint which is typically at port 30018. This is to kick off a collector process which will read the updated feed and add the Gallery items to the database.
    Note: Hitting ‘Restart’ does not have the same effect for some weird reason.

    image

  8. Now when a tenant user clicks on create Web Site from Gallery on the tenant Portal, they will be shown a list of Gallery items from the off lined feed

      

Troubleshooting

During the course of this, I ran into a couple of issues, mostly user errors, that I’d like to call out here:

  1. Ensure the machine that runs hosts the ‘MgmtSvc-WebAppGallery’ service has Web Platform Installer installed

  2. Always backup the feeds before running the ModifyOfflinedFeeds.ps1 script

  3. While running the ModifyOfflinedFeeds.ps1 take absolute care to get the –server parameter correct.

  4. Once the modification is done, copy one of the URLs that are displayed, and once you host the folder on a server, use this URL to ensure that the packages are reachable.
    image

    If they are not reachable, chances are that you’ve made a mistake in hosting the packages correctly, or the server does not have the right permissions for applications to reach the installers

  5. Use Debug Logs in Event Viewer. Enable the Debug logs in Event Viewer for the WebAppGallery service. this will give you valuable information about when the gallery service starts its sync-ing process and if the feeds are valid or not. If everything goes according to plan and you have successfully hosted the feeds, you should get a list of ‘Information’ level entries in the Event viewer. Each entry will indicate the application that has been extracted and stored in the gallery
    image

  6. Images don’t show up in the Gallery: This is a byproduct of the offlining process using the WebPICmd.exe. Some of the images do not get downloaded or get downloaded as tmp files. To use the ones that did successfully download, you should add an SSL binding to the server where you’ve hosted these feeds and use a trusted certificate for it. this will ensure that the icons that are downloaded will show up in the portal

Summary

Hope that gives you enough information about offlining the Web Application Gallery Feed and hosting it in an environment with no internet connectivity. The steps we followed are

  1. Downloaded the Web PI feed
  2. Replaced reference of the web app gallery feed to point to the WAP feed
  3. Extracted a list of product Ids from the WAP feed
  4. Executed the WebPICmd.exe utility with all the above information and offlined the feed to an offline location
  5. Modified the feeds so that the URLs are absolute URLs
  6. Hosted the feed on a server and pointed the WAP Admin portal to this hosted location for the Gallery Feed
  7. Modified web.config of the WebAppGallery service to point to the offlined Web Products feed.
  8. Restarted the services to force the collector to populate the gallery from the Offline feed