Content Deployment – The complete Guide – Part 8 – Job and Change Token

Scope, Change Tokens and Export Objects

Each content deployment job maintains the change tokens for each item configured as ExportObject in the specific content deployment job.

Let’s assume we have the following site structure:

Now let’s assume we have 3 incremental content deployment jobs with a scope as listed below:

Job Inc 1 – here we select the branch starting at the root of the site collection:

In this situation the only item configured as ExportObject is the root of the site collection. The other items are deployed as child items.

What happens here is that the content deployment job configures the content deployment and migration API to export the root web including all child elements:

SPExportObject exportObject = new SPExportObject(); 
exportObject.Id = rootweb.ID
exportObject.IncludeDescendants = SPIncludeDescendants.All;
exportObject.Type = SPDeploymentObjectType.Web;
exportObject.ExportChangeToken = change-token-saved-for-rootweb;

After a successful export the Content deployment and migration API returns the change token to be used with the next run through the parameter SPExportSettings.CurrentChangeToken.

Job Inc 2 – here we only select the branch starting at Site-A level:

In this situation the only item configured as ExportObject is the site Site-A. Site-A-1 is deployed as child item.

What happens here is that the content deployment job configures the content deployment and migration API to export the Site-A including all child elements:

SPExportObject exportObject = new SPExportObject(); 
exportObject.Id = siteA.ID
exportObject.IncludeDescendants = SPIncludeDescendants.All;
exportObject.Type = SPDeploymentObjectType.Web;
exportObject.ExportChangeToken = change-token-saved-for-SiteA;

Job Inc 3 – here we select branch starting at Site-A level and branch starting at Site-B level:

In this situation the we have two item which are configured as ExportObjects: Site-A and Site-B. Site-A-1 is deployed as child object.

Content Deployment Change Token Management

The change token of the last successful deployment is stored in the DeploymentStatus field of the list item related to the content deployment path which we have covered in Part 3.

Content Deployment updates the ChangeToken of all configured ExportObjects configured in the current job after the target farm confirms that the import has succeeded.

After running Job Inc 1 the DeploymentStatus field will contain content similar to this:

<?xml version=”1.0” encoding=”utf-16?>
<ArrayOfDeploymentStatus 
xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” 
xmlns:xsd=”http://www.w3.org/2001/XMLSchema>
   <DeploymentStatus>
      <ObjectType>Web</ObjectType>
      <Url>/</Url>
      <Id>655e3b27-987e-46af-bbc1-3826c0ec9107</Id>
      <LastExportChangeToken>1;1;ed2662e4-8fde-423f-ad7d-52cac634ab8c;633952769446300000;2004</LastExportChangeToken>
      <IncludeDescendants>All</IncludeDescendants>
   </DeploymentStatus>
</ArrayOfDeploymentStatus>

If you later (after some changes have been done) execute Job Inc 2 the content of the DeploymentStatus field will look like this:

<?xml version=”1.0” encoding=”utf-16?>
<ArrayOfDeploymentStatus 
     xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” 
     xmlns:xsd=”http://www.w3.org/2001/XMLSchema>
   <DeploymentStatus>
      <ObjectType>Web</ObjectType>
      <Url>/</Url>
      <Id>655e3b27-987e-46af-bbc1-3826c0ec9107</Id>
      <LastExportChangeToken>1;1;ed2662e4-8fde-423f-ad7d-52cac634ab8c;633952769446300000;2004</LastExportChangeToken>
      <IncludeDescendants>All</IncludeDescendants>
   </DeploymentStatus>
   <DeploymentStatus>
      <ObjectType>Web</ObjectType>
      <Url>/Site-A</Url><Id>3c13b87a-d1fd-4c71-a528-0327025e0469</Id>
      <LastExportChangeToken>1;1;ed2662e4-8fde-423f-ad7d-52cac634ab8c;633952819295200000;2008</LastExportChangeToken>

      <IncludeDescendants>All</IncludeDescendants>
   </DeploymentStatus>
</ArrayOfDeploymentStatus>

You can see that the DeploymentStatus field contains two DeploymentStatus entries reflecting the change token for the root site – which was selected in Job Inc 1 – and for the Site-A – which was selected in Job Inc 2.

Both entries reflect a different change tokens (see highlighted part above) as the source database was in a different state when they were last executed successfully.

The Guid in the 3rd part of the change token will always be the same as all these jobs originate in the same source site collection. See Part 7 for more details about the structure of the change token.

If you later (after some additional changes have been done) execute Job Inc 3 the content will look like this:

<?xml version=”1.0” encoding=”utf-16?>
<ArrayOfDeploymentStatus xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xmlns:xsd=”http://www.w3.org/2001/XMLSchema>
   <DeploymentStatus>
      <ObjectType>Web</ObjectType>
      <Url>/</Url>
      <Id>655e3b27-987e-46af-bbc1-3826c0ec9107</Id>
      <LastExportChangeToken>1;1;ed2662e4-8fde-423f-ad7d-52cac634ab8c;633952769446300000;2004</LastExportChangeToken>
      <IncludeDescendants>All</IncludeDescendants>
   </DeploymentStatus>
   <DeploymentStatus>
      <ObjectType>Web</ObjectType>
      <Url>/Site-A</Url>
      <Id>3c13b87a-d1fd-4c71-a528-0327025e0469</Id>
      <LastExportChangeToken>1;1;ed2662e4-8fde-423f-ad7d-52cac634ab8c;633957827740670000;2012</LastExportChangeToken>
      <IncludeDescendants>All</IncludeDescendants>
   </DeploymentStatus>
   <DeploymentStatus>
      <ObjectType>Web</ObjectType>
      <Url>/Site-B</Url>
      <Id>136c83a5-5368-4860-9171-1faa2e227ff1</Id>
      <LastExportChangeToken>1;1;ed2662e4-8fde-423f-ad7d-52cac634ab8c;633957827740670000;2012</LastExportChangeToken>
      <IncludeDescendants>All</IncludeDescendants>
   </DeploymentStatus>
</ArrayOfDeploymentStatus>

You can see that the DeploymentStatus field now contains three DeploymentStatus entries reflecting the change token for the root site (which is selected in Job Inc 1), for the Site-A (which is selected in Job Inc 2 and in Job Inc 3) and for Site-B (which was selected in Job Inc 3).

You will notice that now as well the change token for Site-A and for Site-B has been updated when running Job Inc 3. That means: when Job Inc 2 is run later it will now only deploy changes since Job Inc 3 was run.

To complete the picture let’s assume we do exactly that: we run Job Inc 2 again after some additional changes have been done in the site collection.

The DeploymentStatus field will then show the content like this:

<?xml version=”1.0” encoding=”utf-16?>
<ArrayOfDeploymentStatus xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xmlns:xsd=”http://www.w3.org/2001/XMLSchema>
   <DeploymentStatus>
      <ObjectType>Web</ObjectType>
      <Url>/</Url>
      <Id>655e3b27-987e-46af-bbc1-3826c0ec9107</Id>
      <LastExportChangeToken>1;1;ed2662e4-8fde-423f-ad7d-52cac634ab8c;633952769446300000;2004</LastExportChangeToken>
      <IncludeDescendants>All</IncludeDescendants>
   </DeploymentStatus>
   <DeploymentStatus>
      <ObjectType>Web</ObjectType>
      <Url>/Site-A</Url>
      <Id>3c13b87a-d1fd-4c71-a528-0327025e0469</Id>
      <LastExportChangeToken>1;1;ed2662e4-8fde-423f-ad7d-52cac634ab8c;633957835007900000;2016</LastExportChangeToken>
      <IncludeDescendants>All</IncludeDescendants>
   </DeploymentStatus>
   <DeploymentStatus>
      <ObjectType>Web</ObjectType>
      <Url>/Site-B</Url>
      <Id>136c83a5-5368-4860-9171-1faa2e227ff1</Id>
      <LastExportChangeToken>1;1;ed2662e4-8fde-423f-ad7d-52cac634ab8c;633957827740670000;2012</LastExportChangeToken>
      <IncludeDescendants>All</IncludeDescendants>
   </DeploymentStatus>
</ArrayOfDeploymentStatus>

As expected there are now no additional DeploymentStatus entries in field. And as expected the DeploymentStatus entry for Site-A has been updated with the new change token.

That means on next run of Job Inc 3 Site-A will be exported with the change token ending in 2016 while Site-B will be exported with the change token ending in 2012. That will guarantee that no duplicate content is exported for the different subsites.

Implicit mix of full and incremental deployment

Let’s look back to the previous section. What would you assume was the change token used for Site-A in the scenario above when running Job Inc 2 the first time?

The answer is: there is no change token available for this object. That means that the first run of Job Inc 2 will have been a full deployment and not an incremental. That means: although all the content in Site-A and below has been deployed already with Job Inc 1 it will get deployed again on first run of Job Inc 2.

You can easily proof this by looking at the following scenario. We will use Site-C which has not yet been added as an ExportObject to any of the existing content deployment jobs.

As a first step, please upload a document to a document library in Site-C and ensure that it is checked-in correctly.

Now start Job Inc 1 which is the incremental deployment job for the whole site collection. After this deployment is finished the document added before will get deployed to the target site collection.

After the item has been deployed to the target delete the document on the source system.

Now create a new incremental deployment job that includes only Site-C

Job Inc 4 – here we select the branch starting at Site-C level:

Now execute this incremental deployment job. Check the number of objects exported: you will notice that a huge number of items has been exported although you recently ran Job Inc 1 which deployed the content. After the job has completed double check if the item you deleted on the source has been deleted on the target as well: You will notice that the item has not been deleted although the job is configured as incremental deployment job!

You can run this incremental deployment job a second time and you will notice that no further items get deployed. You will also notice that the document you deleted earlier will remain in the target site collection.

So how can this be fixed? That answer is simple: with the next run of Job Inc 1 all the changes since the last run of Job Inc 1 will be deployed to the target. Including the document deletion action that was missed by the full deployment done by the incremental deployment job Job Inc 4.

Now let’s get things even more complicated:

What will happen if you create another incremental content deployment job that deploys the branch Site-B and below?

Will the first run of this content deployment job be a full deployment as well as with Site-C or will it be an incremental deployment job?

Considering the scenarios covered in this article it will be an incremental deployment because there is already a Change Token available in the DeploymentStatus field of the content deployment path for Site-B. This token will now be used for the deployment.

So we see a huge benefit in storing the change token in the content deployment path as it allows us to use the same change token within different jobs.

But there are also caveats when using such an approach. Just consider the following scenario:

Let’s delete Job Inc 4 which we created earlier. What does this mean for the change token for Site-C stored in the content deployment path? It means that this change token will now no longer be updated as there is no content deployment job that has this specific site configured as an ExportObject.

All the items in Site-C will still be deployed in context of the content deployment job Job Inc 1 which deploys the whole website.

Now think about a situation where – maybe a year or so later – someone decides to create a new incremental content deployment job (e.g. Job Inc 17) that should deploy Site-A and Site-C.

What do you expect to happen on the first run of this deployment job?

The answer is: the deployment will fail with the famous error message “The changeToken refers to a time before the start of the current change log”.

The reason is that the newly created content deployment job tries to use the change token stored when Job Inc 4 was last run before it got deleted. And the only way to fix this would be to perform a manual full deployment of this timer job followed by an incremental deployment of Job Inc 1 to ensure that the delete actions missed by the full deployment gets deployed as well.

Possible problems with incremental export

As outlined earlier, the change log is preserved for a configurable timeframe. Problems will occur when the change token provided to search for changes is older than the configurable timeframe. In this case it is not possible to reliably return all changes that have happened since the change. The reason is obvious: If the change token is older than the oldest entry in the change log, then it could have happened that there were changes in the timeframe between the time when the change token was generated and the first entry in the change log. These changes are no longer available.

To ensure that incremental deployment will always produce a result that is consistent with the source, it is required that ALL changes get deployed to the target. This can no longer be guaranteed if the change token is older than the first item in the change log.

Due to this limitation, incremental content deployment will not work if the last successful execution of a content deployment job is older than the oldest entry in the change log.

Be aware that if you have multiple jobs for the same path each job keeps track of its own change token. That means you need to ensure that each job is successfully executed at least once within the configured timeframe for the change log.

Here is a list of all known reasons that can lead to a change token that is older than the oldest entry of the change log:

  • The last successful execution of the content deployment job is older than the configured timeframe for the change log
  • The source database has been restored using STSADM -o restore (this clears the event cache table)
  • The source database has been merged with a different db using STSADM -o mergecontentdbs
  • No updates have happened on the source DB within the configured timeframe for the change log
  • A change token was stored in the DeploymentStatus field of the content deployment path for an ExportObject used only in a historical deployment job which was deleted later. When creating a new content deployment job that selects this ExportObject will try to use the Change Token from the DeploymentStatus field. As this might have been created months or years ago it means it is most likely older than the beginning of the change log.
  • For a given path there is one job that deploys the whole site collection and another job which deploys the root site plus some subsites

A first and very simple check of the change token against the event cache table would be to compare the Int3 which represents the change number against the Id column.

ATTENTION: issuing SQL commands against a production database is not supported! Please perform the following steps against an offline copy of the database and not against the live database. More details about the supportability see here: KB841057

Example: change token: “1;3;ace2c0f6-ca8c-4765-93df-aa580d163b3e;633788367678070000;1842

Example: using the following query:

select * from eventcache WITH (NOLOCK) where id = 1842

If this query returns no result then the change related to the change token is no longer in the database.

To see at which date and with which change number the change log starts you can use the following query:

select top 1 eventtime,id from eventcache WITH (NOLOCK) order by id

The DecryptChangeToken code listed earlier allows you to evaluate a given change token easily and in more detail to verify how old it is and to compare it against the entries in the event cache table.

12 Comments


  1. In my content deployment scenario’s, I see a similar issue surfacing like in the "Implicit mix of full and incremental deployment" section of this blogpost.

    I’d like to have several incremental jobs which JUST copy changes to the scoped sites for the job. Unfortunately, I see a similar behavior as described in your post. Any other workarounds other than running a job like "master" "Job inc 1".

    Reply

  2. Hi Victor,

    no there is no supported other way.

    Cheers,

    Stefan

    Reply

  3. I am trying to figure out the functionality of the site deployment "scope" of a content deployment job. In the SharePoint user interface, I can either select "Entire Site Collection" or select specific sites.

    SharePoint DEV environment: MOSS, patched up to october commulative update.

    In my tests, even selecting specific sites in scope to deploy, I see the root site deployed as well! Is this because of dependent objects like content types?

    Reply

  4. Hi Vogelpoel,

    your impression is correct. The root is always deployed as it holds vital information about your site like the master pages used in sub sites, page layouts and style sheets.

    Not deploying these items would lead to a non-functional subsite, that’s why this site is always deployed.

    Cheers,

    Stefan

    Reply

  5. Hi Stefan,

    I have to migrate a number of sharepoint Excel Services based KPI’s from one environment to another.

    Using STSADM I find that the KPI’s remain wired to the pathe of the environment that they came from, is there a way of exporting them, and then importing them so that they point at the same location, but no on the new server?

    Cheers,

    Phil

    Reply

  6. Hi Phil,

    sorry this is not possible. Export/Import is a WSS feature which is not aware of Excel Services.

    You would need to fix these references afterwards.

    Cheers,

    Stefan

    Reply

  7. Hello, Stefan,

    Is there a way of knowing the root of the destination site through the generated .cab file?

    Great post!

    TIA,

    Goly

    Reply

  8. Hi Goly,

    no this is not possible. The Cab file only contains information about the source site.

    Cheers,

    Stefan

    Reply

  9. First of all, thanks for your fast answer.

    Isn’t ANY way of knowing the destination site root?

    I have a Web Application in a source farm and, when a Content Deployment Job is executed [this is outside my control] I need to delete the generated files from a database. The problem is that the content can be deployed in three [for now] Web Applications and despite that I know [thanks to the Manifest.xml] the the relative url of the file, I don’t know the root of the destination site.

    Any ideas would be really appreciated.

    TIA,

    Goly

    Reply

  10. Hi Goly,

    not from the files.

    The information about the target site is only available in the content deployment path on the source system.

    Cheers,

    Stefan

    Reply

  11. Stefan

    I have an issue where content deployment is changing urls in anchor tags within placeholder content. In our external environment we have a web app
    http://www.abc.com that has anonymous access enabled. We extended this web app and enabled SSO with ADFS and url is
    https://secure.abc.com. What is happening is on publishing box we have pages that have url to secured site that full qualified within placeholder content so
    https://secure.abc.com/sites/pages/default.aspx. These pages are available anonymously so we want the links to go to secured web app so we fully qualify the url. When the content is deployed,
    it gets changed to a relative path of/sites/pages/default.aspx. Any ideas that woudl cause this issue. We are running Moss with SP2 and the site does have variations enabled.

    Thanks

    Derek

    Reply

  12. Hi Derek,

    this might be by design. Links which point to the current site collection are adjusted during content deployment.

    You might want to raise a support case to verify this and to see if a solution can be provided.

    Cheers,

    Stefan

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.