We need our directory to replicate immediately!

  • Article

A customer of ours is in the process of evaluating Active Directory and ADAM for use as their primary enterprise application directory. I was just informed that they've dropped Active Directory from the running (it's not in their "top three" directory servers) because "AD replication is scheduled and not event driven". When pressed for more info, they gave the following points:

  • Event driven replication is perceived to be faster than scheduled replication
  • AD replication is “pull” vs “push”
  • AD can't do sub-second replication and they need it
  • AD performance was great but running everything on one box means no redundancy

It sounds like their app doesn't handle replication latency very well. It expects to be able to read the same data it has written... and a multi-master replication system can cause problems with this. Perhaps their technique for dealing with latency is to wait until replication is complete? Some of the points above are easily refuted, but it leads to a much more interesting question. Let's start with the refuting part.

Event driven replication - Change notification is on by default within a site and is effectively "event driven". The same behaviour can be turned on between sites as well.

Pull vs push - Instead of pushing changes, we push a notification and the destination pulls the changes. Not much difference there.

Replication time - We can reduce replication intervals to about 1 second, but this shouldn't be required if the app handles latency intelligently. Even with other directories, replication is best effort and isn't guaranteed... thus the app needs to handle latency or it will experience intermittent errors.

Perf, single server - Running everything on one box solves the replication latency problem. Moreover, having a second server for redundancy won't necessarily cause a problem with latency, provided the app talks to the second box only when the first box has failed. Other customers have used this model because their third-party middleware solution doesn't handle latency.

What became immediately apparent is that we need to do a better job of explaining how to deal with replication latency in a multi-master environment. I suspect that a lot of LDAP apps have been written without this in mind... some work, some don't, and many will exhibit intermittent errors given the right conditions.

Here's a quote from Don Hacherl on a closely related topic. It sheds some light on why being latency-aware is important:

You'll never see a meaningful urgent replication flag because AD exposes the X.500 data model in which reads from different servers can have different results. If apps aren't prepared to deal with that then all you'll get is an infinite regression of "Well, Urgent wasn't fast enough and my app crashed again. Can I have a Really Truly Super Duper Urgent setting?"

I can imagine an unreliable urgent mechanism (say, urgent changes are multicast out by the source), but even with that you'd still end up with temporary inconsistencies and an inability to offer QOS-like guaranties. If a source server accepts an originating change while a destination server is flat out 100% CPU and I/O working to catch up on other replication (heck, say other urgent replication), what should happen? You'll still end up with latency, and apps will still have to deal with it.

Now what I consider the interesting question. What mechanisms are available to deal with replication latency?

One approach is designating a master server for write operations and critical readbacks. Regular read operations would still hit any of the replicas. In the case of failure, one of the replicas can be quickly designated as the master. This model may require applications to handle reads, writes, and critical reads differently, because they may need to be routed to different replicas. A variation of this approach is to route writes/readbacks in a predictive way to multiple masters, effectively dividing ownership over writes accross multiple masters.

Another approach is to ensure client affinity. There are variations on this model as well - affinity may be handled on a per-session or per-user basis by the application, or perhaps by a hardware or software load balancer. Multiple masters are still used, but each application session or client will communicate with a single master at a time. The decision on how to balance transactions safely across the available masters may need to be handled by the application.

A third approach is to cache the directory data at the application level. This might get tricky, especially when dealing with web clients and application state... but it's certainly possible. The complexity of the caching layer may end up being too high to be practical.

Personally, I think the first approach is the simplest to implement. The application logic is least complex and failover is relatively easy. There is still the potential for the loss of small number committed transactions, but this is a common problem in any multi-master LDAP system. I'm going to dig around a bit and find out how others handle the latency issue. Public and private comments are definitely welcome on this one.