Modifying SharePoint RSS Feeds using a custom control adapter – Implementing Delta Encoding

We frequently get questions from customers if it is possible to modify the data sent in the RSS feed to the client. SharePoint itself allows to specify the columns which can be included in the Description field, the number of items to return as a maximum and the maximum days to include in the RSS feed.

A feature which is not supported by SharePoint is Delta Encoding for the feeds.

In short words: Delta Encoding would allow to reduce the amount of data sent in an RSS feed to the client to the exact information required by the client to get him up-to-date to the current point in time. That means that items that have been sent to the client earlier would not be sent a second time.

Delta Encoding requires that the client sends an If-Modified-Since http header to the server which allows the server to determine at which time this specific client received the last update.

SharePoint actually implements a check of If-Modified-Since and returns a 304 – not modified in case that the content of RSS feed has not changed. But in case that the content has changed it will return the whole RSS feed.

This implementation is correct as otherwise scenarios with caching proxy servers would fail as content for the exactly same URL would be different based on the If-Modified-Since header sent.

E.g. like this:

  • User 1 requests the RSS feed and receives content with items A, B and C which is then also cached by the proxy
  • Item D is added
  • Later User 1 requests the RSS feed after the cache is expired and the cached content is replaced with content D as this would be the delta for this user
  • Now User 2 requests the RSS feed and retrieves content D from the cache and never receives A, B and C

So Delta Encoding is only of useful in case that no caching proxy servers are between the clients and the SharePoint server.

Still some customers have requested this feature and I will now explain how this can be implemented using an ASP.NET ControlAdapter.

Control Adapters allow to modify the behavior of existing controls without subclassing by adding a simple configuration file to the App_Browser directory. To modify the behavior of the SharePoint RSS feeds it will be required to implement a custom PageAdapter which allows to consume the XML response generated by the RSS feed and filter it against the If-Modified-Since header.

Usually a control adapter that needs to modify the content rendered by a control would create a custom HtmlTextWriter pass this to the Render method of the control, consume the content generated by the control and modify it before sending it to the original HtmlTextWriter which will then send the content to the client:

protected override void Render(System.Web.UI.HtmlTextWriter output) 

    // catch the output of the original Control
    TextWriter tempWriter = new StringWriter();
    base.Render(new System.Web.UI.HtmlTextWriter(tempWriter)); 
    string origHtml = tempWriter.ToString(); 

    // adjust the content as required
    string newHtml = origHtml.Replace(...); 

    // send the adjusted output to the client
    output.Write(newhtml); 
}

Unfortunatelly this method does not work with the SharePoint RSS feeds as the RSS code does not use the passed in HtmlTextWriter to send the RSS response but uses Response.Write to send the content. This method bypasses the HtmlTextWriter.

So a different method is required to consume and modify the RSS response. ASP.NET allows to do this using a custom stream filter.

In .NET a stream filter is actually nothing but a Stream object which chains itself between the filtered stream and the receiver.

After the custom stream filter has been configured as stream for the http response object it will receive all content sent from the control and can modify it in any way it likes. We will use this method to implement Delta Encoding for the RSS feed.

The Stream filter has to implement all the method of a standard Stream object. Here is the basic stream filter without the custom logic to modify the content:

public class RSS_Filter : Stream

        private Encoding enc = null;
        private bool closed;
        Stream BaseStream; 

        public RSS_Filter(Stream baseStream, Encoding encoding)
        {
            // here we keep track of the underlaying base stream and the encoding.
            BaseStream = baseStream;
            enc = encoding;

            // stream is not closed when we create it.
            closed = false;
        }

        public override void Write(byte[] buffer, int offset, int count)
        { 
            // ensure that the stream has not been closed before 
            if (Closed) throw new ObjectDisposedException("RSS-Stream-Filter");

            // -- our logic to consume the written data needs to be added here --
        }

        public override bool CanRead
        {
            get { return false; }
        }

        public override bool CanWrite
        {
            get { return !closed; }
        }

        public override bool CanSeek
        {
            get { return false; }
        }

        public override void Close()
        {
            closed = true;
            BaseStream.Close();
        }

        protected bool Closed
        {
            get { return closed; }
        }

        public override void Flush()
        {
            // -- our logic to modify the written data before sending it on the wire needs to be added here --

            BaseStream.Flush();
        }

        public override int Read(byte[] buffer, int offset, int count)
        {
            throw new NotSupportedException();
        }

        public override long Length
        {
            get { throw new NotSupportedException(); }
        }

        public override long Seek(long offset, SeekOrigin origin)
        {
            throw new NotSupportedException();
        }

        public override void SetLength(long value)
        {
            throw new NotSupportedException();
        }

        public override long Position
        {
            get { throw new NotSupportedException(); }
            set { throw new NotSupportedException(); }
        }
}

As you can see in the above code there are two methods where we have to add custom logic: in the Write method and in the Flush method. This is required as the RSS response can be larger than a single write buffer. So it is not guaranteed that the whole RSS response is in the first buffer we get. As the write method receives a byte buffer it is also not guaranteed that UTF-8 characters (which can be larger than a single byte) do not spread between two buffers.

So in order to allow us to correctly process the RSS feed we have to buffer all the received buffers in the Write method till the Flush method is called which indicates the end of the stream.

In the Flush method we can then add the logic to post-process the RSS content.

The following code fragment demonstrates how the buffers are consumed:

        Queue byteQueue = new Queue();
        int responseLength=0;

        public override void Write(byte[] buffer, int offset, int count)
        { 
            // ensure that the stream has not been closed before
            if (Closed) throw new ObjectDisposedException("RSS-Stream-Filter");

            // handle the special case that only part of the given buffer 
            // should be sent to the wire. We will then copy the required part
            // to a second buffer and buffer this
            if (offset > 0)
            {
                byte[] newBuf = new byte[count];
                for (int i = 0; i < count; i++)
                {
                    newBuf[i] = buffer[i + offset];
                }
                // we store the buffer in a queue which ensures that we can easily 
                // process the content in the same sequence
                byteQueue.Enqueue(newBuf);
            }
            else
            {
                // we store the buffer in a queue which ensures that we can easily 
                // process the content in the same sequence
                byteQueue.Enqueue(buffer);
            }
            // current buffered response
            responseLength += count;
        }

In the Flush method we now add the logic that handles the delta encoding. To allow easy modification of the RSS feed we first have to convert the consumed byte buffer into a object which we can then easily filter using Xml methods. After the modification is done we have to convert the generated XML back into a byte buffer to allow us to send it to the client. To ensure that even if an exception occurs the original content is sent to the client we will add the relevant logic in a try/finally block:

        public override void Flush()
        { 
            // retrieve the reference time from the If-Modified-Since header
            DateTime ifModSince = Convert.ToDateTime(HttpContext.Current.Request.Headers["If-Modified-Since"]);

            // here we copy the different buffers into one large buffer
            // we need this in a single buffer for the conversion to string
            byte[] fullBuffer = new byte[responseLength];
            int index = 0;
            while (byteQueue.Count > 0)
            {
                byte[] buf = byteQueue.Dequeue() as byte[];
                buf.CopyTo(fullBuffer, index);
                index += buf.Length;
            }

            // now we convert the byte buffer into a string object
            string content = enc.GetString(fullBuffer, 0, responseLength);

            // before we convert the content we have to remove all characters 
            // which are not part of the Xml content
            string nonXml = content.Substring(0, content.IndexOf("<"));
            if (nonXml.Length > 0)
                content = content.Substring(nonXml.Length); 

            try
            {
                // -- here we add our code to manipulate the XML --
            }
            catch (Exception e)
            {
                // our code raised an exception - there is not much we can do about it
            }
            finally
            {
                // ensure to write the data we have to the wire 
                byte[] newBuffer = enc.GetBytes(nonXml + content);

                responseLength = newBuffer.Length;
                BaseStream.Write(newBuffer, 0, responseLength);
            }

            // now we flush the base stream
            BaseStream.Flush();
        }

The next step is to implement the logic that allows us to filter the RSS feed using Xml methods:

        // now we create a new XmlDocument and pass in the rss content as InnerXml 
        XmlDocument doc = new XmlDocument(); 
        doc.InnerXml = content; 

        // to get the Urls to the different items in the RSS feed we filter the Xml  
        // for the link nodes 
        XmlNodeList itemNodes = doc.GetElementsByTagName("link"); 
        Stack nodesToRemove = new Stack(); 

        // loop over each link in the RSS 
        foreach (XmlNode node in itemNodes) 
        { 
            // we are only interested in links that reside in item nodes of the RSS feed 
            if (node.ParentNode.Name == "item"
            { 
                // the link to the item is in the InnerText of the current node 
                string link = node.InnerText; 
                if (!string.IsNullOrEmpty(link)) 
                { 
                    // lets retrieve the sharepoint object identified by the link 
                    Uri uri = new Uri(link); 

                    // -- here we need to add the code to verify the last modified date        --
                    // -- of the item identified by the url against the If-Modified-Since date --
                } 
            } 
        } 
        while (nodesToRemove.Count > 0
        { 
            // get the item node we need to remove 
            XmlNode node = (nodesToRemove.Pop() as XmlNode).ParentNode; 
            node.ParentNode.RemoveChild(node); 
        } 
        content = doc.InnerXml;

After this code is executed we have only the items in the RSS feed which were modified after the date in the If-Modified-Since header. The missing piece here is only the code that verifies the last modified date of the item against the If-Modified-Since header:

        using (SPSite site = new SPSite(uri.AbsoluteUri.ToString())) 
        { 
            // to avoid exceptions we need to cut the URL if we find a reserved name. Reserved names start with an "_" char. 
            string modUrlWithoutReservedNames = uri.AbsolutePath.Substring(0, (uri.AbsolutePath + "/_").IndexOf("/_")); 
            using (SPWeb web = site.OpenWeb(modUrlWithoutReservedNames, false)) 
            { 
                // there are two possible way to encode the URL in SharePoint RSS feeds depending on how it is configured: 
                // using the ID and using direct item URL 
                if (HttpUtility.ParseQueryString(uri.Query)["ID"] == null
                { 
                    // here we handle the items identified using a direct Url 
                    // check if we can retrieve the object and verify if it is a List Item 
                    SPListItem item = web.GetObject(uri.GetComponents(UriComponents.PathAndQuery, 
                                                    UriFormat.UriEscaped)) as SPListItem; 

                    if (item != null
                    { 
                        DateTime modTime = (DateTime)item["Modified"]; 
                        if (DateTime.Compare(modTime, DateTime.Now.AddHours(-1)) < 0
                        { 
                            nodesToRemove.Push(node); 
                        } 
                    } 
                } 
                else 
                { 
                    // here we handle the items identified by ID 

                    // first we try to get the list 
                    SPList list = web.GetList(uri.AbsolutePath); 
                    if (list != null
                    { 
                        // to retrieve the list item we need to take the item id from the query string in the rss 
                        int itemId = int.Parse(HttpUtility.ParseQueryString(uri.Query)["ID"]); 
                        SPListItem item = list.GetItemById(itemId); 

                        // remove the item if it's last modified date is older than "If-Modified-Since" 
                        DateTime modTime = (DateTime)item["Modified"]; 
                        if (DateTime.Compare(modTime, ifModSince) <= 0
                        { 
                            // we cannot remove it directly as this would modify the collection foreach is running over 
                            // so we buffer it 
                            nodesToRemove.Push(node); 
                        } 
                    } 
                } 
            } 
        }

So we have now all code pieces together for the stream filter. To register this we have to register it as follows from within our control adapter:

        protected override void Render(System.Web.UI.HtmlTextWriter writer)
        {
            if (HttpContext.Current.Request.Headers.AllKeys.Contains("If-Modified-Since"))
            {
                // add our custom stream filter if we receive an If-Modified-Since header
                base.Page.Response.Filter = new RSS_Filter(base.Page.Response.Filter, base.Page.Response.ContentEncoding);
            }
            base.Render(writer);
        }

For your convenience here is now all the code together:

using System;
using System.Xml;
using System.IO;
using System.Collections.Generic;
using System.Collections;
using System.Linq;
using System.Text;
using System.Web;
using System.Web.UI.Adapters;
using Microsoft.SharePoint;

namespace StefanG.ControlAdapters
{
    public class RSS_Filter : Stream
    {
        private Encoding enc = null;
        private bool closed;
        Stream BaseStream;
        Queue byteQueue = new Queue();
        int responseLength=0;

        public RSS_Filter(Stream baseStream, Encoding encoding)
        {
            // here we keep track of the underlaying base stream and the encoding. 
            BaseStream = baseStream;
            enc = encoding;

            // stream is not closed when we create it.
            closed = false;
        }

        public override void Write(byte[] buffer, int offset, int count)
        {
            // ensure that the stream has not been closed before 
            if (Closed) throw new ObjectDisposedException("RSS-Stream-Filter");

            // handle the special case that only part of the given buffer  
            // should be sent to the wire. We will then copy the required part 
            // to a second buffer and buffer this 
            if (offset > 0)
            {
                byte[] newBuf = new byte[count];
                for (int i = 0; i < count; i++)
                {
                    newBuf[i] = buffer[i + offset];
                }
                // we store the buffer in a queue which ensures that we can easily  
                // process the content in the same sequence 
                byteQueue.Enqueue(newBuf);
            }
            else
            {
                // we store the buffer in a queue which ensures that we can easily  
                // process the content in the same sequence 
                byteQueue.Enqueue(buffer);
            }
            // current buffered response 
            responseLength += count;
        }

        public override bool CanRead
        {
            get { return false; }
        }

        public override bool CanWrite
        {
            g
et
 { return !closed; }
        }

        public override bool CanSeek
        {
            get { return false; }
        }

        public override void Close()
        {
            closed = true;
            BaseStream.Close();
        }

        protected bool Closed
        {
            get { return closed; }
        }

        public override void Flush()
        {
            // we retrieve the reference time from the If-Modified-Since header
            DateTime ifModSince = Convert.ToDateTime(HttpContext.Current.Request.Headers["If-Modified-Since"]);

            // here we copy the different buffers into one large buffer
            // we need this in a single buffer for the conversion to string
            byte[] fullBuffer = new byte[responseLength];
            int index = 0;
            while (byteQueue.Count > 0)
            {
                byte[] buf = byteQueue.Dequeue() as byte[];
                buf.CopyTo(fullBuffer, index);
                index += buf.Length;
            }

            // now we convert the byte buffer into a string object
            string content = enc.GetString(fullBuffer, 0, responseLength);

            // before we convert the content we have to remove all characters 
            // which are not part of the Xml content
            string nonXml = content.Substring(0, content.IndexOf("<"));
            if (nonXml.Length > 0)
                content = content.Substring(nonXml.Length);

            try
            {
                // now we create a new XmlDocument and pass in the rss content as InnerXml
                XmlDocument doc = new XmlDocument();
                doc.InnerXml = content;

                // to get the Urls to the different items in the RSS feed we filter the Xml 
                // for the link nodes
                XmlNodeList itemNodes = doc.GetElementsByTagName("link");
                Stack nodesToRemove = new Stack();

                // loop over each link in the RSS
                foreach (XmlNode node in itemNodes)
                {
                    // we are only interested in links that reside in item nodes of the RSS feed
              &nb
sp;     if (node.ParentNode.Name == "item")
                    {
                        // the link to the item is in the InnerText of the current node
                        string link = node.InnerText;
                        if (!string.IsNullOrEmpty(link))
                        {
                            // lets retrieve the sharepoint object identified by the link
                            Uri uri = new Uri(link);
                            using (SPSite site = new SPSite(uri.AbsoluteUri.ToString()))
                            {
                                // to avoid exceptions we need to cut the URL if we find a reserved name. Reserved names start with an "_" char.
                                string modUrlWithoutReservedNames = uri.AbsolutePath.Substring(0, (uri.AbsolutePath + "/_").IndexOf("/_"));
                                using (SPWeb web = site.OpenWeb(modUrlWithoutReservedNames, false))
                                {
                                    // there are two possible way to encode the URL in SharePoint RSS feeds depending on how it is configured:
                                    // using the ID and using direct item URL
                                    if (HttpUtility.ParseQueryString(uri.Query)["ID"] == null)
                                    { 
                                        // here we handle the items identified using a direct Url 
 
                                        // check if we can retrieve the object and verify if it is a List Item
                                        SPListItem item = web.GetObject(uri.GetComponents(UriComponents.PathAndQuery, UriFormat.UriEscaped)) as SPListItem;

                                        if (item != null)
                                        {
                                            DateTime modTime = (DateTime)item["Modified"];
                                            if (DateTime.Compare(modTime, DateTime.Now.AddHours(-1)) < 0)
                                            {
              
;                                  nodesToRemove.Push(node);
                                            }
                                        }
                                    }
                                    else
                                    { 
                                        // here we handle the items identified by ID 

                                        // first we try to get the list
                                        SPList list = web.GetList(uri.AbsolutePath);
                                        if (list != null)
                                        {
                                            // to retrieve the list item we need to take the item id from the query string in the rss
                                            int itemId = int.Parse(HttpUtility.ParseQueryString(uri.Query)["ID"]);
                                            SPListItem item = list.GetItemById(itemId);

                                            // remove the item if it's last modified date is older than "If-Modified-Since"
                                            DateTime modTime = (DateTime)item["Modified"];
                                            if (DateTime.Compare(modTime, ifModSince) <= 0)
                                            {
                                                // we cannot remove it directly as this would modify the collection foreach is running over
                                                // so we buffer it
                                                nodesToRemove.Push(node);
                                            }
                                        }
                                    }
                                }
                            }
             &n
bsp;          }
                    }
                }
                while (nodesToRemove.Count > 0)
                {
                    // get the item node we need to remove
                    XmlNode node = (nodesToRemove.Pop() as XmlNode).ParentNode;
                    node.ParentNode.RemoveChild(node);
                }
                content = doc.InnerXml;
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
            finally
            {
                // ensure to write the data we have to the wire 
                byte[] newBuffer = enc.GetBytes(nonXml + content);

                responseLength = newBuffer.Length;
                BaseStream.Write(newBuffer, 0, responseLength);
            }

            BaseStream.Flush();
        }

        public override int Read(byte[] buffer, int offset, int count)
        {
            throw new NotSupportedException();
        }

        public override long Length
        {
            get { throw new NotSupportedException(); }
        }

        public override long Seek(long offset, SeekOrigin origin)
        {
            throw new NotSupportedException();
        }

        public override void SetLength(long value)
        {
            throw new NotSupportedException();
        }

        public override long Position
        {
            get { throw new NotSupportedException(); }
            set { throw new NotSupportedException(); }
        }
    }

    public class RssControlAdapter : PageAdapter
    {
        protected override void Render(System.Web.UI.HtmlTextWriter writer)
        {
            if (HttpContext.Current.Request.Headers.AllKeys.Contains("If-Modified-Since"))
            {
                // add our custom stream filter if we receive an If-Modified-Since header
                base.Page.Response.Filter = new RSS_Filter(base.Page.Response.Filter, base.Page.Response.ContentEncoding);
            }
            base.Render(writer);
        }
    }
}

To implement this control adapter you need to do the following steps:

  1. create a new class library project in Visual Studio 2008
  2. replace the code in class.cs with the code listed above
  3. add a key file to allow signing of the generated assembly
  4. build the dll and add it to the Global Assembly Cache (GAC)

Afterwards you need to register this control adapter with your existing SharePoint web application. To do this you need to create a new file with the file extension “.browser” inside the App_Browsers directory (e.g. RssControlAdapter.browser) and add the following XML:

<browsers> 
  <browser refID=”Default> 
    <controlAdapters> 
      <adapter  
        controlType=”Microsoft.SharePoint.ApplicationPages.ListFeed, Microsoft.SharePoint.ApplicationPages” 
        adapterType=”StefanG.ControlAdapters.RssControlAdapter, AssemblyName, Version=1.0.0.0, Culture=neutral, PublicKeyToken=f180aea0269836ba” 
      /> 
    </controlAdapters> 
  </browser> 
</browsers> 

Be sure to adjust the Public Key Token f180aea0269836ba and the AssemblyName with the public key token of and name of your specific assembly

To ensure that the new .browser file is being used after the next application domain recycle you also need to delete the content of the following directory:

  • C:\windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root

If this directory is not purged the additional .browser file is usually not used.

4 Comments


  1. Thanks for the good post! I guess, one little misprint exists in the code. In Write() method, a ‘count’ should specify number of bytes to write and not the size of the buffer as it looks like. So instead:

    if (offset > 0)

    {

     byte[] newBuf = new byte[count – offset];

     for (int i = offset; i < count; i++)

     {

       newBuf[i – offset] = buffer[i];

     }

     // we store the buffer in a queue which ensures that we can easily  

     // process the content in the same sequence  

     byteQueue.Enqueue(newBuf);

    }

    there should be something like that:

    if (offset > 0)

    {

     byte[] newBuf = new byte[count];

     for (int i = 0; i < count; i++)

     {

       newBuf[i] = buffer[i + offset];

     }

     // we store the buffer in a queue which ensures that we can easily  

     // process the content in the same sequence  

     byteQueue.Enqueue(newBuf);

    }

    Best Regards,

    Sergio

    Reply

  2. Hi Sergio,

    thanks for the hint! You are right!

    I have adjusted my code.

    Cheers,

    Stefan

    Reply

  3. Hi Stefan

      This post is very useful. I am thinking of similar approach to filter appropriate entries based on certain key words in the feed. For example, I will display only Business news related to Microsoft from NYTimes feed. Do you see any better alternative approach to achieve this, compared to reusing the above approach that you posted?

    Thanks & Regards

    Srini

    Reply

  4. Hi Srini,

    if there is no way to configure the original feed (e.g. by providing specific parameters for the filtering) then I don't think there is a simpler solution for this.

    Cheers,

    Stefan

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.