Yahoo! Slurp and googlebot(at)googlebot.com causing Cannot use a leading .. to exit above the top directory

Since I launched my new site www.gamerlobby.net, I've been having this strange error.
As I have added ELMAH logging, I have been picking up a strange error. The error contains the following message:

System.Web.HttpException
Cannot use a leading .. to exit above the top directory.

Reviewing this page, it caused no such error in Internet Explorer nor in FireFox.
Looking at the Server Variables, I saw the following information:

HTTP_FROM: googlebot(at)googlebot.com
HTTP_USER_AGENT: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
or
HTTP_FROM: not present
HTTP_USER_AGENT: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

This has been haunting me for about 2 weeks as it occurs every 2 minutes, and Google is not indexing my site.
It is however mainly produced by Yahoo! Slurp, which seems to poll my site non stop in this case. Google appears to use the "come back later" approach.
So after finally finding some time, I came across the following post Get GoogleBot to crash your .NET 2.0 site

So it turns our certain browser headers are causing this error to occur. By downloading installing fiddler, I could view this error using the following browser header as described in the ToDotNet blog article.

Accept: */*
Accept-Encoding: gzip, x-gzip
User-Agent: Mozilla/5.0

FINALLY, I could reproduce the BUG!! Woohoo!!
So reviewing my page with Fiddler, I could easily see the 500 being thrown that google was experiencing.



So now time for the fix.

 

THE SOLUTION

So the problem boils down to a bug in the HtmlTextWriter class.
Depending on the User-Agent, ASP.NET may use System.Web.UI.Html32TextWriter versus System.Web.UI.HtmlTextWriter. This is aptly referred to as "adaptive rendering"
Now until Microft releases a fix for this, we have a work around using .browser files. With .browser files we have the ability to adjust elements of how asp.net renders based on various parameters.
In this specific case, we want to match for Google or Yahoo Slurp, and use System.Web.UI.HtmlTextWriter to render our page.

So to fix my issue, I created a file named "yahooslurp.browser", and added the following:

<browsers>
   <
browser id="Slurp" parentID="Mozilla">
   <
identification>
      <
userAgent match="Slurp" />
   </
identification>
   <
capabilities>
      <
capability name="browser" value="Yahoo!Slurp" />
      <
capability name="Version" value="4.0" />
      <
capability name="MajorVersion" value="4" />
      <
capability name="MinorVersionString" value="" />
      <
capability name="MinorVersion" value=".0" />
      <
capability name="activexcontrols" value="true" />
      <
capability name="backgroundsounds" value="true" />
      <
capability name="cookies" value="true" />
      <
capability name="css1" value="true" />
      <
capability name="css2" value="true" />
      <
capability name="ecmascriptversion" value="1.2" />
      <
capability name="frames" value="true" />
      <
capability name="javaapplets" value="true" />
      <
capability name="javascript" value="true" />
      <
capability name="jscriptversion" value="5.0" />
      <
capability name="supportsCallback" value="true" />
      <
capability name="supportsFileUpload" value="true" />
      <
capability name="supportsMultilineTextBoxDisplay" value="true" />
      <
capability name="supportsMaintainScrollPositionOnPostback" value="true" />
      <
capability name="supportsVCard" value="true" />
      <
capability name="supportsXmlHttp" value="true" />
      <
capability name="tables" value="true" />
      <
capability name="vbscript" value="true" />
      <
capability name="w3cdomversion" value="1.0" />
      <
capability name="xml" value="true" />
      <
capability name="tagwriter" value="System.Web.UI.HtmlTextWriter" />
     </
capabilities>
   </
browser>
</
browsers>

There are several setting in this snippet I copied from todotnet's blog post, but the main ones we are interested in are:

<browser id="Slurp" parentID="Mozilla">
<userAgent match="Slurp" />
<capability name="browser" value="Yahoo!Slurp" />
<capability name="tagwriter" value="System.Web.UI.HtmlTextWriter" />


Here we noticing that we particularly targeting Yahoo!Slurp, and instructing the rendering engine to use ystem.Web.UI.HtmlTextWriter.
This fixed the issue I was experiencing, and I am no longer receiving unhandled exceptions from this problem. I'm pretty sure Yahoo will start indexing me properly again.

The Result:



This however doesn't solve my Google problem, but i'm pretty sure it'll be solved in the same way.
I will post when I have more information, but for now my errors are back down to a bare minimum.
Please also remember, Yahoo and Google change their browser header over time, so this will have to be adjusted.
I am fairly confident this will be caught in by ELMAH and I can fix the problem in a timely manner.

THANKS Sander from ToDotNet for solving my issue and putting my mind at ease!

posted @ Tuesday, August 26, 2008 5:58 PM

Print
«July»
SunMonTueWedThuFriSat
30123456
78910111213
14151617181920
21222324252627
28293031123
45678910