Many people are unaware that IIS has a built in limit on URL length. When it tries to process a URL that contains a straight piece that is longer than 260 characters, it's considered invalid, and the server sends out a "Bad Request (Invalid URL)" message, or in some cases "Bad Request (Invalid Hostname). To be more specific, when I say "straight piece", I mean a part of the URL that represents a folder, so a URL can be very long with nesting folders and not cause the error, but this can be a problem with implementations that involve other servers.
Creating an actual website on an IIS machine that will cause this error is almost impossible, as there are file-system limitations in place too. However, when IIS is used as a bridge to other servers, then we run into problems. One specific scenario that can be a problem is when Unix or Linux servers are published using IAG or UAG. IBM's web sphere server, for example, generates very long URLs as part of its design, and while this works perfectly when accessing the site directly, it breaks up completely when it's published using IAG. It's also a scenario that's hard to diagnose correctly, as the error is thrown by the front IIS on IAG without even reaching the IAG filter, so it does not show up in the IAG logs or traces.
IBM are referring to this sort of problems in various KB articles, and there's a well documented solution, but it might also creep up with other products. The solution is to use a Registry key to force IIS to accept URLs with longer pieces. Here's the full procedure:
1. On a client computer, inspect the problematic link or links, to determine the maximum segment length. Look for the longest segments that begins with a slash, and ends with another slash or the end of the URL. For example:
The 13th segment of this URL that starts with "L0DU" is the longest at 348 characters.
2. On the IAG Server, open the registry editor.
3. Navigate to HKLM\System\CurrentControlSet\Services\HTTP\Parameters
4. Create a new DWORD value and name it URLSegmentMaxLength
5. Double click the new value and set it to the length you determined in step 1
6. Close the registry editor.
7. Open the Internet Information Services (IIS) manager
8. Right-click on the local server, select All Tasks and Restart IIS
Setting the correct maximum length is a bit tricky. The server might send slightly longer URLs and cause the problem to re-appear. We recommend setting the value to not be significantly longer than necessary, as it consumes server memory, and also poses a security risk. The limitation was originally implemented to reduce the exposure to malformed URL attacks. IBM's technical documents discussing this issue usually suggest setting the value to 0, which is a no-limit setting. This, of course, is not recommended by Microsoft.
The same behavior might also occur with other IBM products, or other Unix, Linux and MAC based products, but as you can see, it's not hard to solve. One such article is this: http://publib.boulder.ibm.com/infocenter/wpdoc/v6r0/index.jsp?topic=/com.ibm.wp.ent.doc/wpf/inst_ihs.html
The IIS registry change, as well as other modification that affect IIS's behavior are documented here: http://support.microsoft.com/?id=820129
July 19th, 2010:
This information also applies to UAG. Even though the web server running on a UAG server is IIS 7, the registry change still applies, and I have verified that it does indeed solve the problem.