Friday Mail Sack: Newfie from the Grave Edition

  • Article

Heya, Ned here again. Since this another of those catch up mail sacks, there’s plenty of interesting stuff to discuss. Today we talk NSPI, DFSR, USMT, NT 4.0 (!!!), Win2008/R2 AD upgrades, Black Hat 2010, and Irish people who live on icebergs.

Faith and Begorrah!

Question

A vendor told me that I need to follow KB2019948 to raise the number of “NSPI max sessions per user” from 50 to 10,000 for their product to work. Am I setting myself up for failure?

Answer

Starting in Windows Server 2008 global catalogs are limited to 50 concurrent NSPI connections per user from messaging applications. That is because previous experience with letting apps use unlimited connections has been unpleasant. :) So when your vendor tells you to do this, they are putting you in the position where your DC’s will be allocating a huge number of memory pages to handle what amounts to a denial of service attack caused by a poorly written app that does not know how to re-use sessions correctly.

We wrote an article you can use to confirm this is your issue (BlackBerry Enterprise Server currently does this and yikes, Outlook 2007 did at some point too! There are probably others):

949469    NSPI connections to a Windows 2008-based domain controller may cause MAPI client applications to fail with an error code: "MAPI_E_LOGON_FAILED" https://support.microsoft.com/default.aspx?scid=kb;EN-US;949469

The real answer is to fix the calling application so that it doesn’t behave this way. As a grotesque bandage, you can use the registry change on your GC’s. Make sure these DC’s are x64 OS and not memory bound before you start, as it’s likely to hurt. Try raising the value in increments before going to something astronomical like 10,000 – it may be that significantly fewer are needed per user and the vendor was pulling that number out of their butt. It’s not like they will be the ones on the phone with you all night when the DC tanks, right?

Question

I have recently started deploying Windows Server 2008 R2 as part of a large DFSR infrastructure. When I use the DFS Management (DFSMGMT.MSC) snap-in on the old Win2008 and Win2003 servers to examine my RG’s, the new RG’s don’t show up. Even when I select “Add replication groups to display” and hit the “Show replication groups” button I don’t see the new RG’s. What’s up?

Answer

We have had some changes in the DFSMGMT snap-in that intentionally lead to behaviors like these. For example:

Here’s Win2008 R2:

clip_image002

and here’s Win2003 R2:

See the difference? The missing RG names gives a clue. :)

This is because the msDFSR-Version attribute on the RG gets set to “3.0” when creating an RG with clustered memberships or an RG containing read-only memberships. Since a Win2003 or Win2008 server cannot correctly manage those new model RG’s, their snap-in is not allowed to see it.

In both cases this is only at creation time; if you go back later and do stuff with cluster or RO, then the version may not necessarily be updated and you can end up with 2003/2008 seeing stuff they cannot manage. For that reason I recommend you avoid managing DFSR with anything but the latest DFSMGMT.MSC. The snap-ins just can’t really coexist effectively. There’s never likely to be a backport because – why bother? The only way to have the problem is to already have the solution.

Question

Is there a way with USMT 4.0 to take a bunch of files scattered around the computer and put them into one central destination folder during loadstate? For example, PST files?

Answer

Sure thing, USMT supports a concept called “rerouting” that relies on an XML element called “locationModify”. Here’s an example:

<migration urlid="<a href="https://www.microsoft.com/migration/1.0/migxmlext/pstconsolidate">
<component type="Documents" context="System">
<displayName>All .pst files to a single folder</displayName>
<role role="Data">
<rules>
<include>
<objectSet>
<script>MigXmlHelper.GenerateDrivePatterns ("* [*.pst]", "Fixed")</script>
</objectSet>
</include>
<!-- Migrates all the .pst files in the store to the C:\PSTFiles folder during LoadState -->         <locationModify script="MigXmlHelper.Move('C:\PSTFiles')">
<objectSet>
<script>MigXmlHelper.GenerateDrivePatterns ("* [*.pst]", "Fixed")</script>
</objectSet>
</locationModify>
</rules>
</role>
</component>
</migration>

The <locationModify> element allows you to choose from the MigXmlHelpers of RelativeMove, Move, and ExactMove. Move is typically the best option as it just preserves the old source folder structure under the new parent folder to which you redirected . ExactMove is less desirable as it will flatten out the source directory structure, which means you then need to explore the <merge> element and decide how you want to handle conflicts. Those could involve various levels of precedence (where some files will be overwritten permanently) or simply renaming files with (1), (2), etc added to the tail. Pretty gross. I don’t recommend it and your users will not appreciate it. RelativeMove allows you to take from one known spot in the scanstate and move to another new known spot in the loadstate.

Question

I’m running into some weird issues with pre-seeding DFSR using robocopy with Win2008 and Win2008 R2, even when following your instructions from an old post. It looks like my hashes are not matching as I’m seeing a lot of conflicts. I also remember you saying that there will be a new article on pre-seeding coming?

Answer

1. Make sure you install these QFE version that fixes several problems with ACL’s and other elements not correctly copying in 2008/2008R2 – all file elements are used by DFSR to calculate the SHA-1 hash, so anything being different (including security) will conflict the file:

973776  The security configuration information, such as the ACL, is not copied if a backup operator uses the Robocopy.exe utility together with the /B option to copy a file on a computer that is running Windows Vista or Windows Server 2008 https://support.microsoft.com/default.aspx?scid=kb;EN-US;973776

979808    "Robocopy /B" does not copy the security information such as ACL in Windows 7 and in Windows Server 2008 R2 https://support.microsoft.com/default.aspx?scid=kb;EN-US;979808

2. Here’s my recommended robocopy syntax.  You will want to ensure that the base folder (where copying from and to) have the same security and inheritance settings prior to copying, of course.

3. If you are using Windows Server 2008 R2 (or have a Win7 computer lying around), you can use the updated version of DFSRDIAG.EXE that supports the FILEHASH command. It will allow you to test and see if your pre-seeding was done correctly before continuing:

C:\>dfsrdiag.exe filehash
Command "FileHash" or "Hash" Help:
Displays a hash value identical to that computed by the DFS Replication
service for the specified file or folder
Usage: DFSRDIAG FileHash </FilePath:filepath>

   </FilePath> or </Path>
File full path name
Example: /FilePath:d:\directory\filename.ext

It only works on a per-file basis, so it’s either for “spot checking” or you’d have to script it to crawl everything (probably overkill). So you could do your pre-seeding test, then use this to check how it went on some files:

dfsrdiag filehash /path:\\srv1\rf\somefile.txt
dfsrdiag filehash /path:\\srv2\rf\somefile.txt

If the hashes fit, you must acquit!

Still working on the full blog post, sorry. It’s big and requires a lot of repro and validation, just needs more time – but it had that nice screenshot for you. :)

Question

  1. Can a Windows NT 4.0 member join a Windows Server 2008 R2 domain?
  2. Can Windows7/2008 R2 join an NT 4.0 domain?
  3. Can I create a two-way or outbound trust between an NT 4.0 PDC and Windows Server 2008 R2 PDCE?

Short Snarky Answer

  1. Yes, but good grief, really!?!
  2. No.
  3. Heck no.

Long Helpful Answer

  1. If you enable the AllowNt4Crypto Netlogon setting and all the other ridiculously insecure settings required for NT 4.0 below you will be good to go. At least until you get hacked due to using a 15 year old OS that has not gotten a security hotfix in half a decade.823659    Client, service, and program incompatibilities that may occur when you modify security settings and user rights assignments https://support.microsoft.com/default.aspx?scid=kb;EN-US;823659

    942564    The Net Logon service on Windows Server 2008 and on Windows Server 2008 R2 domain controllers does not allow the use of older cryptography algorithms that are compatible with Windows NT 4.0 by default
    https://support.microsoft.com/default.aspx?scid=kb;EN-US;942564

  2. Windows 7 and 2008 R2 computers cannot join NT 4.0 domains due to fundamental security changes. No, this will not change. No, there is no workaround.940268    Error message when you try to join a Windows Vista, Windows Server 2008, Windows 7, or Windows Server 2008 R2-based computer to a Windows NT 4.0 domain: "Logon failure: unknown user name or bad password" https://support.microsoft.com/default.aspx?scid=kb;EN-US;940268

  3. Windows Server 2008 R2 PDCE’s cannot create an outbound or two-way trusts to NT 4.0 due to fundamental security changes . We have a specific article in mind for this right now, but the KB942564 was updated to reflect this also. No, this will not change. No, there is no workaround.

The real solution here is to stop expending all this energy to be insecure and keep ancient systems running. You obviously have newer model OS’s in the environment, just go whole hog. Upgrade, migrate or toss your NT 4.0 environments. Windows 2000 support just ended, for goodness sake, and it was 5 years younger than NT 4.0! For every one customer that tells me they need an NT 4.0 domain for some application to run (which no one ever actually checks to see if that’s true, because they secretly know it is not true), the other nineteen admit that they just haven’t bothered out of sheer inertia.

Let me try this another way – go here: https://www.microsoft.com/technet/security/bulletin/summary.mspx. This is the list of all Microsoft security bulletins in the past seven years. For five of those years, NT 4.0 has not gotten a single hotfix. Windows 2000 – remember, not supported now either – has gotten 174 security updates in the past four years alone. If you think your NT 4.0 environment is not totally compromised, it’s only because you keep it locked in an underwater vault with piranha fish and you keep the servers turned off. It’s an OS based on using NTLM’s challenge response security, which people are still gleefully attacking with new vectors.

You need Kerberos.

Question

We use a lot of firewalls between network segments inside our environment. We have deployed DFSR and it works like a champ, replicating without issues. But when I try to gather a health report for a computer that is behind a firewall, it fails with an RPC error. My event log shows:

Error Event Source: DCOM
Event Category: None
Event ID: 10006
Date: 7/15/2010
Time: 2:51:52 PM
User: N/A
Computer: SRVBEHINDFIREWALL
Description: DCOM got error "The RPC server is unavailable."

Answer

If replication is working with the firewall but health reports are not, it sounds like DCOM/WMI traffic is being filtered out. Make sure the firewalls are not blocking or filtering the DCOM traffic specifically; a later model firewall that supports packet inspection may be deciding to block the DCOM types of traffic based on some rule. A double-sided network capture is how you will figure this out – the computer running MMC will connect remotely to DCOM over port 135, get back a response packet that (internally) states the remote port for subsequent connections, then the MMC will connect to that port for all subsequent conversations. If that port is blocked, no report.

For example here I connect to port 135 (DCOM/EPM), get a response packet that contains the new dynamic listening port to connect for DCOM – that port happens to be 55158 (but will differ every time). I then connect to that remote port in order to get a health diagnostic output using the IServerHealthReport call. If you create a double-sided network capture, you will likely see the first conversation fail, and if it succeeds, the subsequent conversation will be failing. Failing due the firewall dropping the packets and them never appearing on the remote host – that’s why you must use double-sided.


Click me

Question

I know USMT cannot migrate local printers, but can it migrate TCP-port connected printers?

Answer

No, and for the same reason: those printers are not mapped to a print server that can send you a device driver and they are (technically) also a local printer. Dirty secret time: USMT doesn’t really migrate network printers, it just migrates these two registry keys:

HKCU\Printers\Connections
HKCU\Printers\DevModes2

So if your printer is in those keys, USMT is win – and the only kind that live there are mapped network printers. When you first logon and access the printer on your newly restored computer, Windows will just download the driver for you and away you go. Considering that you are in the middle of this big migration, now would be a good time to get rid of these old (wrong?) ways of connecting printers. Windows 7 has plenty of options for printer deployment through group policy, group policy preferences, and you can even make the right printers appear based on the user’s location. For example, here’s what I see when I add a printer here at my desk – all I see are the printers in my little building on the nearest network. Not the ones across the street, not the ones I cannot use, not the ones I have no business seeing. Do this right and most users will only see printers within 50 feet of them. :)

image

To quote from the book of Bourdain: That does not suck.

Question

What are the best documents for planning, deploying, and completing a forest upgrade from Win2000/2003 to Win2008/2008R2? [Asked at least 10 times a week – Ned]

Answer

Here:

Upgrading Active Directory Domains to Windows Server 2008 and Windows Server 2008 R2 AD DS Domains https://technet.microsoft.com/en-us/library/cc731188(WS.10).aspx

If you are planning a domain upgrade, this should be your new homepage until the operation is complete. It is fantastic documentation with checklists, guides, known issues, recommended hotfixes, and best practices. It’s the bee’s knees, the wasp’s elbows, and the caterpillar's feets.

image

 

Moving on to other things not directly sack-related…

There are a couple of interesting takeaways from Black Hat US 2010 this week:

  • We announced our new Coordinated Vulnerability Disclosure process. Adobe is onboard already, hopefully more to come.
  • These folks claim they have a workable attack on Kerberos smart card logons. Except that we’ve had a way to prevent the attack for three years, starting in Vista using Strict KDC Validation – so that kinda takes the wind out of their sails. You can read more about how to make sure you are protected here and here and soon here. Pretty amazing also that this is the first time – that I’ve heard of, at least – in 11 years of MS Kerberos smart cards that anyone was talking attacks past the theoretical stage.
  • Of 102 topics, 10 are directly around Microsoft and Windows attacks. 48 are around web, java, and browser attacks. How much attention are you giving your end-to-end web security?
  • 10 topics were also around attacking iPhones and Google apps. How much attention are you giving those products in your environment? They are now as interesting to penetrate as all of Windows, according to Black Hat.
  • 5 topics on cloud computing attacks. Look for that number to double next year, and then double again the year after. Bet on it, buddy.

Finally, remember my old boss Mike O’Reilly? Yes, that guy that made the Keebler tree and who was the manager in charge of this blog and whom I worked with for 6 years. Out of the blue he sends me this email today – using his caveman Newfie mental gymnastics:

Ned,

I never ever read the Askds blog when I worked there.  I was reading it today and just realized that you are funny.

What a guy. Have a nice weekend folks,

- Ned “I have 3 bosses, Bob” Pyle