Finding Duplicate Objects in Active Directory


For those of you that have embarked upon the trek to Office 365, you've undoubtedly run (or at least heard of) IDFix.  It detects and fixes a number of conditions that will cause the directory sync to report errors.

Today, I want to focus on a tool I wrote for a customer almost 2 years ago that addresses conditions not yet identified or remedied by IDFix.  I just updated the tool today, which reminded me of why I wrote it in the first place:

  1. When two objects of the same class have overlapping values in different attributes (ie, User 1 has a UPN of user1@domain.com and User 2 has a UPN of user2@domain.com but a proxy address of user1@domain.com)
  2. When two or more objects of different classes have overlapping values (ie, User 1 has a UPN of user1@domain.com and Contact 1 has a mail or proxy address of user1@domain.com).
  3. When an object has a value in msRTCSIP-PrimaryUserAddress that conflicts with the value of another attribute on another user object.

Updates:

  • -Auto switch (which is, shockingly, engaged automatically) to try to determine if you have the Active Directory schema extended for either Exchange (to check proxyAddresses) or LCS/OCS/Skype (to check msRTCSIP-PrimaryUserAddress).
  • Color highlighting for matches so you can see exactly where the pesky values are hiding.
  • Apparently, the original version of the script did substring matching (using the -LDAPFilter parameter of Get-ADObject), which can add extra objects to the output (user1@domain.com will match things like user1@domain.com1 and 1user1@domain.com).  For those of you who want to use it to find partial matches, I've put that functionality in the -LDAPStyle parameter and use a better syntax for finding exact matches.

Case 1: Overlapping attribute values in different attributes

In the first instance, we're going to look for users where they have an overlapping attribute value in different attributes.  IDFix is good at locating duplicate values when they appear in the same object class and the same attribute--like finding two users with user@domain.com set as the UPN or mail address.  What IDFix doesn't do is to identify users where these attributes may have been jumbled--User 1 may have an email address of user1@domain.com and a UPN of user@domain.com, while User 2 may have an email address of user@domain.com and a UPN of user2@domain.com.  From AAD's perspective, the object that gets written to the directory first wins, and all other objects with conflicting attributes result in an error condition.  From the user perspective, I typically err on the side of letting the user who has the value as an email address win and then fixing everything else.  AADConnect will let you know you have an error value, but it only shows you the value for the user in error--not the user that got there first.

Case 2: Overlapping attribute values in dissimilar object classes

This one also happens frequently: someone on the sales team thinks they need a shared mailbox (like sales@domain.com).  Someone on the sales team thinks they need a security group for the sales team home directory, and the service desk creates a security group.  The service desk has a procedure to put the email address of the person or group responsible for the security group in the mail attribute of the group.  Sounds crazy? I've had several enterprise customers that do exactly that. IDFix won't pick up on this, but Azure AD will let you know in no uncertain terms that you've gone out of bounds.

Case 3: Overlap in the msRTCSIP-PrimaryUserAddress attribute

Back in the olden days of Live Communications Server and Office Communications Server, this attribute was used to store the SIP user address.  Chances are, you no longer have these servers around.  Chances are that you also just shut them off like a lot of people did when you started using a new IM/conferencing service.  And, chances are you still have data in this attribute.  I had a customer last year that had legacy data in that attribute that was conflicting with new users being created (for example, a user named John Smith had a UPN and email address value of john.smith@domain.com, but jsmith@domain.com in the msRTCSIP-PrimaryUserAddress attribute when OCS 2007 had been deployed.  Years later, a new user named Jason Smith had somehow been provisioned with jsmith@domain.com as his primary SMTP address, and Azure AD alerted them that there was a conflict.

Why these edge cases are important

So, now that I've identified some things that don't work, let's talk about why they are important.

In all of these cases, it means that the first object to get synchronized to AAD wins, and all other objects that have conflicting attributes will fail to synchronize.  Undoubtedly, some of these objects are important (in the case of users with mailboxes that have attributes that were updated directly instead of via the Exchange interface or cmdlets).  Additionally, if users have incorrect values, it could mean non-delivery of email, inability to access services, or email gets delivered to the wrong user.  All of those seem not so good.

With that in mind, here's the abridged version of how to use the tool.

In my lab, I created a similar scenario:

  1. I created a test user (testuser1@aaronoffice365lab.com) and synchronized to AAD.
  2. I created a second test user (testuser1-1@aaronoffice365lab.com), but manually added the mail attribute testuser1@aaronoffice365lab.com, which will cause a collision, since the data in that indexed attribute can’t exist on any other object (the indexed attributes are UserPrincipalName, Mail, ProxyAddresses, and msRTCSIP-PrimaryUserAddress).

After synchronizing, I received the expected error:

1

To locate the offending user:

  1. Click on the data in the Error column (InvalidSoftMatch).
    2
  2. Click on the Detail button.
    3
  3. Review the error. In this case, it tells you that the attribute Mail with a value of testuser1@aaronoffice365lab.com is conflicting with something else.  Unfortunately, it doesn’t tell you what object that attribute is on (in this window) nor does it tell you the other object holding that value.
  4. Copy the value immediately following the identified attribute (in this case, testuser1@aaronoffice365lab.com).
  5. Open a PowerShell prompt and navigate to the directory containing the script you downloaded.
  6. Run .\Find-DuplicateValues.ps1 -IncludeExchange -Address user@domain.com
  7. Review the output.  In this case, you can see two user objects match:
    4

The first object is in OU=Test,DC=forest,DC=com and the conflicting object is in OU=Users,OU=Test,DC=forest,DC=com.  The attribute testuser1@aaronoffice365lab.com is both the UPN and primary SMTP address for the object in OU=Test,DC=forestc,DC=com, and is only in the mail attribute for the user in OU=Users,OU=Test,DC=forest,DC=com.  It will take a human to determine which one is “correct.”

You can find the full tool here: Find Objects with Duplicate UPN or SMTP Values in Active Directory - https://gallery.technet.microsoft.com/Find-Duplicate-Values-in-6b012059.

Comments (7)

  1. turbomcp says:

    Thanks for sharing this
    this is exactly what i asked in previous post relating to similar “issues”
    awesome stuff(in a geek kind of way:))

  2. turbomcp says:

    now if only we had a tool to find these in advance:)
    but that probably requires some crazy comparisons

    1. Hey, I’m working on it. 🙂

      1. turbomcp says:

        im sure:)
        you the man im telling you
        Thanks again

        1. Nah, but thanks. I’m just a guy trying to make it easier to use our products.

  3. Róbert Formódi says:

    A really useful tool!

Skip to main content