Content Deployment – The complete Guide – Part 7 – Change Token Basics

Incremental deployment allows to deploy all content changes in the source database since the last successful deployment which are in the configured scope of the content deployment job.

To implement this functionality content deployment makes use of so called “change tokens”.

Understanding Change Tokens

Introduction

A very common issue with content deployment is that incremental content deployment fails with the following error message:

The changeToken refers to a time before the start of the current change log

But what does this actually mean?

What we have seen in the earlier article series about the content deployment and migration API to perform an incremental deployment we have to pass in a change token that will be used as a reference to identify which objects need to be exported.

Using a change token, an application can determine “what content has changed” within a given scope since the change token has been generated.

To do this the information included in the change token is compared against Change log.

About the Change Log

The Change log is stored in the EventCache table of the content database. This table stores the information about all content changes in the content database. Each entry in the Change log contains among other things the following information:

  • A sequential number (Id) identifying the row in the event cache table
  • The timestamp (EventTime) when the change has happened
  • The type of the object (ObjectType) that has changed
  • Identification (one or multiple IDs) of the object that was changed
  • URL to the object that was changed (if the object has a URL)
  • The type of modification (EventType) such as add, delete, … (see table below)

The following article contains some more details about these entries:
http://msdn.microsoft.com/en-us/library/bb861918.aspx

The change log stores information for a timeframe that can be configured in the Web application general settings:

About the Change Token

Each time the change log is queried for changes, it returns the changes together with a Change Token which corresponds to a database entry up to which the querying application has been synchronized. This change token is of the format Int1; Int2; Guid; Time; Int3.

Example: 1;1;df6828cf-c303-453e-9eb6-e733f85611ac;633782543234470000;1510

Where:

  • Int1: Contains version information for the change token. Currently this value is always 1.
  • Int2: The scope the change token has been requested for. For example, if you ask for changes in the site collection this value will have the scope “Site”. If you request changes for a sub site, it will contain the scope “Web”. See the following table for possible values. Related to content deployment and the underlying API, this value is always 1 as content deployment always generates a change token with a scope of the whole site collection.
  • Guid: Guid of the object the change token belongs to. If the change token was requested for a site collection this will be the Id of the SPSite object.
  • Time: Timestamp (in ticks) of the change the token belongs to.
  • Int3: change number in the event cache table the change belongs to.

The following scopes are possible for a change token:

Change Token Scope 

Int value 

Content DB 

0 

Site 

1 

Web 

2 

List 

3 

When it comes to troubleshooting change token problems, it is also important to be able to analyze the change token itself – e.g. to see the date/time of the change it represents.

The following method allows to decrypt the change token into a better readable format:

static void DecryptChangeToken(SPChangeToken cToken)
{
   string[] elements = cToken.ToString().Split(‘;’);

   Console.WriteLine(cToken);
   Console.WriteLine(“Version: “ + elements[0]);
   Console.WriteLine(“Scope: “ + (SPChangeCollection.CollectionScope)Enum.ToObject(typeof(SPChangeCollection.CollectionScope), int.Parse(elements[1])));
   Console.WriteLine(“Guid: “ + elements[2]);
   Console.WriteLine(“ChangeTime: “ + new DateTime(long.Parse(elements[3])));
   Console.WriteLine(“ChangeNumber: “ + elements[4]);
   Console.WriteLine();
}

This code will create output like the following:

1;1;a7e2f199-9176-42fc-9917-dabb913c66ee;633788601321030000;1863
Version: 1
Scope: Site
Guid: a7e2f199-9176-42fc-9917-dabb913c66ee
ChangeTime: 25.05.2009 14:55:32
ChangeNumber: 1863
 
1;2;dc4ffb77-6b1d-4437-88fb-6e3077128864;633788352696730000;1832
Version: 1
Scope: Web
Guid: dc4ffb77-6b1d-4437-88fb-6e3077128864
ChangeTime: 25.05.2009 08:01:09
ChangeNumber: 1832
 
1;3;ace2c0f6-ca8c-4765-93df-aa580d163b3e;633788367678070000;1842
Version: 1
Scope: List
Guid: ace2c0f6-ca8c-4765-93df-aa580d163b3e
ChangeTime: 25.05.2009 08:26:07
ChangeNumber: 1842

Change Log, Change Token and SharePoint Object Model

The Change log can be queried for changes at four different scopes: Content Database, site collection, site or list scope. Depending on the application and what scope the application is interested in defines which scope needs to be used. For example, if an application is only interested in changes in a specific list, it can query for changes at the list scope. On the other hand, if changes across the entire database need to be monitored, then the change log can be queried at the content database scope.

The GetChanges method can be used to query for changes:

// get reference to site collection 
SPSite mySite = new SPSite(“http://localhost”); 
SPList myList = mySite.RootWeb.Lists[“MyList”]; 

// retrieve a change token that identifies the current point in time for 
// this SPSite object. Result will be similar to this: 
// 1;1;a7e2f199-9176-42fc-9917-dabb913c66ee;633788601321030000;1863 
// the second “1” identifies the scope as Site Collection scope 
SPChangeToken siteToken = mySite.CurrentChangeToken; 

// retrieve a change token that identifies the current point in time for 
// this SPList object. Result will be similar to this: 
// 1;3;ace2c0f6-ca8c-4765-93df-aa580d163b3e;633788601321030000;1863 
// the “3” identifies the scope as list scope 
SPChangeToken listToken = myList.CurrentChangeToken; 

// Get all changes available in the change log of the site collection 
// be aware that only the first 1000 changes are returned. 
SPChangeCollection changes = mySite.GetChanges(); 

// Get the first 1000 changes in the site collection since siteToken 
SPChangeCollection changes = mySite.GetChanges(siteToken); 

// Get the first 1000 changes in the list since listToken 
SPChangeCollection changes = myList.GetChanges(listToken); 

// this will not work as the scope of siteToken is for a site and cannot be 
// used for a list 
SPChangeCollection changes = myList.GetChanges(siteToken);

When you have a SPChangeCollection you can evaluate each change individually. Each SPChange object exposes the properties:

  • ChangeToken – change token for this change
  • ChangeType – see here
  • SiteId – the unique ID identifying the site collection
  • Time – the timestamp when this change happened

You can cast the individual SPChange objects to the exact change to get further information. For example, SPChangeWeb exposes the ID of the changed site. SPChangeFolder exposes the unique ID of the folder and the unique id of the site.

foreach (SPChange change in changes)
{
    Console.WriteLine(change.ChangeToken);
    DecryptChangeToken(change.ChangeToken);
    Console.WriteLine(change.Time + ” – “ + change.GetType() + ” – “ +
        change.ChangeType);

    // get unique ID of the site collection the change was in
    Guid changeSiteId = change.SiteId;    

    // handle changes to sites
    if (change is SPChangeWeb)
    { 
        SPChangeWeb changeWeb = change as SPChangeWeb;

        // get unique ID of the changed site
        Guid changeWebGuid = changeWeb.Id;       
    }

    // handle changes to folders
    if (change is SPChangeFolder)
    { 
        …
    }
    …
}

12 Comments


  1. Hi Stefan,

    I have been investigating the Change Logs in one of our Content Database due to performance problem.  Regarding the "ChangeType",  I did check all the ChangeType you point in this blog, but I need to map the EventType Database Value with the ChangeType Enumeration value.  Most of the values I can find in the Internet for Add, Delete, Update, etc.  But when I querry the Content DB in the EventCache table, (I don't work directly in the Content DB, just a copy of the Backup Database), I have a EventType "134217728", which I cannot find the enumeration mapping anywhere, could you please help me to find an exhaustive mapping between the Database EventType Value (from the EventCache) table, and the ChangeType enumeration?

    Thank you in advance for your help.

    Kind regards,

    Thanh-Nu

    Reply

  2. Here is the mapping:

    Event Type Int Value (dec)

    Add 4096

    Modify 8192

    Delete 16384

    Rename 32768

    Move Into 65536

    Restore 131072

    RoleAdd 262144

    AssignmentAdd 524288

    SystemModify 1048576

    MemberAdd 2097152

    MemberDelete 4194304

    RoleDelete 8388608

    RoleUpdate 16777216

    AssignmentDelete 33554432

    MoveAway 67108864

    Navigation 134217728

    Reply

  3. Hi Stefan,

    Thanks for your quick reply; it helps us to interpret the peaks of our curve from the EventCache data.  Strangely, during the last 3 months, now and then we have performance issues in our publishing site, each time the problem is related with SQL Server, and the work around consists of watching closely the CPU usage of the SQL Server, when it is too high, we run a SQL job that recalculate the statistic which helps SQL to optimize the use of caching, and this solve the performance issue.  We apply this work-around, but still cannot understand what cause the performance issue, and why rerun the statistic optimization job in SQL helps to solve the problem.  I am analyzing the EventCache table to find a correlation, and all peaks correspond with "Navigation" events, and those peaks matched with the periods where the performance issues happened.  First I guess that when the content manager delete lots of sub sites in the content, the performances issues occur.  (This is a direct editing publishing site, because the staging, and production environments strategies cannot survive due to content deployment problems together with variations 🙂

    Now according to your mapping above, the peaks are really "Navigation" events and not "Delete" events.  At this point, I get lost, and don't know what can cause the update of "navigation" to create such peaks recorded in the EventCache?

    Have you ever run into such a case? With your experiences, I would hope you can enlighten us, because we are really sinking in the darkness with this recurrent performance issues.

    Thank you so much for your time, and for sharing many precious information.

    Reply

  4. Hi Thanh-Nu,

    unfortuntelly I haven't looked into the details why Navigation events would be created as I never had a need to look into this. I would need to do some research on this topic. Best would be to open a support case with Microsoft to get this analyzed.

    Cheers,

    Stefan

    Reply

  5. Hi Stefan,

    I want to extract values from SP Change Log for audit purpose. I can easily get the changed values but didn't find any way where I can get the 'Before Change' value. Can you pls throw some light on this approach..Pls also let me know if considering SP Change log for auditing purpose is correct approach.

    Reply

  6. Need your help in resolving below issue:

    We have share point authoring and production Global portal setup. We have created below jobs to deploy content from authoring to production

    1) Quick deploy Job- Runs every 15 min

    2) Full deploy Job – We ran this job initially to move entire content from Authoring to Production.

    3) Incremental deploy Job – Runs Every 1 hour

    Incremental job is working on schedule basis (every 1hr) without any issues since 1 year (may be some minor issues which got fixed)…Now we are facing below issue for this job when it’s trying to import content into production

    (Export is working fine, its able to export the content. But Import is throwing below error)

    "Minor Version limit exceeded, publish to major version…"

    Below steps we followed

    1) We have checked both in authoring and production for minor version limit. We haven’t found any issues.

    2) We have created new individual jobs, but its not successful

    3) Finally, we have run Full Deploy Job we thought it may fix the issue. It exported all the content but import

    getting below error

    " FatalError: The URL 'Pages/xxxx.aspx' is invalid.  It may refer to a nonexistent file or folder, or refer to a valid file or folder that is not in the current Web."

    At this stage i have steped in to fix all the issues. Now we are facing new issue with export, when we run incremental job. Its throwing below

    error:

    [7/26/2013 9:00:08 AM]: Error: The changeToken refers to a time before the start of the current change log.

    [7/26/2013 9:00:08 AM]: FatalError: The changeToken refers to a time before the start of the current change log.

      at Microsoft.SharePoint.Deployment.SPExport.ThrowInvalidChangeTokenError(DateTime minChangeTime, Int32 minChangeNumber)

      at Microsoft.SharePoint.Deployment.SPExport.GetIncrementalChanges()

      at Microsoft.SharePoint.Deployment.SPExport.CalculateObjectsToExport()

      at Microsoft.SharePoint.Deployment.SPExport.Run()

    I have checked the setting in CA for Change log. It set to "Never" option for "Delete entries in change log". As per my analysis, until change token date refers to any date in change log this issue wont get fixed.

    Running Full deploy job will reset change token date. Is there any risk running Full deploy job.

    Need your help to solve the change token issue.

    Sorry for such a lengthy description!!!

    ——————————————————————————–

    Srini

    Reply

  7. Hi Srini,

    a couple of things:

    1) never mix full and incremental deployment. You will end up with load of inconsistencies as full deploys less than incremental.

    2) if you have (e.g.) a picture which exists as minor version 1.1 in the source and deploy this to the target it will be imported there with a minor version of 1.1 or similar.

    Later when the item is deployed again – e.b. because referenced by a modified page – it will get imported as version 1.2 or similar although the version in the source has not been increased.

    That can go on 254 times – then the maximum minor version limit has been reached and you run into this issue.

    To avoid such problems you should ensure that referenced documents and items always exist as major version.

    If you need additional help, please open a support case with Microsoft.

    Cheers,

    Stefan

    Reply

  8. Hi Stefan,

    This is just a great post that you have posted here as well as the one at this link (http://blogs.technet.com/b/stefan_gossner/archive/2008/03/13/moss-common-issue-incremental-deployment-fails-with-the-changetoken-refers-to-a-time-before-the-start-of-the-current-change-log.aspx).

    However I am unable to figure out my issue. I have a content deployment job incremental and Full. Full deployment job is working fine, however incremental is failing for the same issue. I wanted to know the steps to solve the issue.

    Please let me know is there a way to reset the ChangeToken time hold by the job. How do I make it work again

    Reply

  9. Hi Stefan,
    Is this a known issue? Our target content DB is bigger than our source content DB. We have daily scheduled content deployment job. What we found was that on the target site collection, random pages have higher version history when compared to same source page.
    Why would the version on the target page automatically increase when source page version does not on deployment?

    Appreciate your thoughts.
    Thank you
    Nishan

    Reply

  10. Hi Nishan,

    this is normal: content deployment deploys referenced items when deploying a page. E.g. if you create a new version of a page in the source by changing some text, then all the images referenced in the page will be deployed as well – and you will get a new version
    of these images in the target.
    So you might have many more versions of the referenced resources in the target than in the source.

    Cheers,
    Stefan

    Reply

  11. Just to add: the version number on source and target are not related. You cannot expect to have the same versions on source and target.
    If a page is referenced from another page, then it is deployed together with the page to ensure that no broken links exist in the target. That is required as content deployment cannot spy into the target to see if the page is there already or not. So to prevent
    broken links content deployment will deploy pages where a hyperlink from another page points to as well.
    If a page is linked from various different pages, then the number of versions of this page will be significantly higher than on the source. E.g. there might only be one version on the source if this is (e.g.) a legal disclaimer which got never changed. but
    might have hundreds of versions on the target.
    Its not that the page gets a new version for every reference – but for every deployment which includes at least one page referencing this specific page.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.