Sometimes, working in support, you come across a best practice or a bit of knowledge that is well-known to some people...but that bit of knowledge has never actually been documented. Today was one of those days.
While working in an environment with multiple Exchange Server 2003 servers running Antigen 9.1 Hotfix Rollup 3, we had to reinstall Antigen on one of the servers. We installed Antigen 9.1 and tested to make sure that mail was flowing after the install (it was). We then configured Antigen, including re-installing the FSSMC agent, redeploying the template for this server, and disabling Antigen performance counters.
At that point, things went off the rails. When we opened the Antigen admin console, it told us "You have 4 days left on your evaluation". Confused, we tried various things, including rebooting the box; however, every time, the console mocked us with it's eval message.
Since we were working in a maintenance window and we had run out of time, we made a decision to disable Antigen temporarily and investigate further. We took the template to a non-production Antigen 9.1 server and applied it. After applying it, we opened the admin console and we were greeted with "You have 4 days left on your evaluation".
At this point, we knew we were onto something. After working with our sustained engineering team to investigate further, we found out the root cause was something that is an FSSMC best practice even though I don't know that it's been written down before:
You should never apply a template created with a later version of Antigen or Forefront Server to an earlier version.
In this case, the template had been created in Antigen 9.1 Hotfix Rollup 3. As long as we applied it to servers running rollup 3 (or later), everything was A-OK; however, when we applied it to an Antigen 9.1 with no hotfix rollup on it, we ran into trouble.
The trouble, in this case, is that the schema for scanjobs was changed to add some additional options into the scanjobs. The template includes this new information but, once it's applied, the older code doesn't know how to handle it. This resulted in memory corruption which caused the false eval notice.
The takeaway here is that, if you're running a mixture of patch levels for your Antigen or Forefront Server servers, you have to be sure that the templates you are deploying with FSSMC were created in the earliest patch level you have in production. This will mean that you can't take advantage of any settings that are added in later patch levels but it also means that you won't run into issues like the one we wrestled with today.
Alternately, you could create templates for each patch level but I think that would end up being more difficult to manage.