PDA

View Full Version : Champ car robots.txt



nrc
05-02-05, 05:18 PM
This one is for web geeks only. :)

I just went to archive.org to look at the old Champ car sites and I get a message that it's blocked because the site's robots.txt. For some reason the top of the Champ car site's robot.txt file looks like this:



# The '#' is a comment delimiter.
# To disallow all robots from accessing the site,
# un-comment the following two lines:

User-agent: *
Disallow: /

Why on earth would you want to shut every search engine out of your entire site? :confused:

Spicoli
05-02-05, 05:22 PM
Its those jacktards from **********.

They are beating us at the InnerWeb.

:gomer:

coolhand
05-02-05, 05:24 PM
i dont get it :cry:

jonovision_man
05-02-05, 05:37 PM
This one is for web geeks only. :)

I just went to archive.org to look at the old Champ car sites and I get a message that it's blocked because the site's robots.txt. For some reason the top of the Champ car site's robot.txt file looks like this:


Why on earth would you want to shut every search engine out of your entire site? :confused:

Agreed. If it's a marketing tool, you want to get it indexed by every search tool you can to drive traffic to it.

jono

L1P1
05-02-05, 06:31 PM
It appears to be working too. I just went to google and searched for "Kalkhoven" within champcarworldseries.com. Nothin'. Nada. Zippo.

Searching for "Champ Car World Series" will point you right to the home page thanks to the all the sites that link to it. But Google doesn't have a cached page.

coolhand
05-02-05, 06:43 PM
I liked CART's site for the 2003 season. 04 sucked. but this years is better

nrc
05-02-05, 10:14 PM
i dont get it :cry:
Exactly. Because of the way they have the site configured if you search for something on their site with Google, you don't get it.

coolhand
05-02-05, 11:56 PM
Exactly. Because of the way they have they're site configured if you search for something on their site with Google, you don't get it.

insane :shakehead

Insomniac
05-03-05, 11:47 AM
Is IIS configure out of the box with a defualt robots.txt file? If that's the case, maybe they never modified it and it is unintentional?

racer2c
05-03-05, 12:15 PM
Is IIS configure out of the box with a defualt robots.txt file? If that's the case, maybe they never modified it and it is unintentional?

Not that I can see. Although, they may not even be on IIS. It appears that the robot.txt is simply an internet standard for spidering exclusion. Someone would have had to manually create and exclude old Champ Car dot com pages. Why? Good question.

L1P1
05-03-05, 06:22 PM
Using netcraft.com, I can determine that cart.com has always been an IIS site. It doesn't seem to work for champcarworldseries.com. Perhaps Netcraft respects the robots.txt file.

I can get a single hit searching cart.com for "kalkhoven". So it looks like it wasn't always this way.

Insomniac
05-04-05, 09:09 AM
Not that I can see. Although, they may not even be on IIS. It appears that the robot.txt is simply an internet standard for spidering exclusion. Someone would have had to manually create and exclude old Champ Car dot com pages. Why? Good question.

They all redirect to the same place. The server is running Microsoft-IIS/6.0. If the server is using ASP pages, you can almost guarantee it is IIS. The robots file is a standard. My point was, on the default installation/setup of IIS, that file may be created automatically and they didn't know it should be modified or deleted rather than someone intentionally blocking all (good) spiders. I've never set up an IIS server, so I don't know it the file is there to begin with or not.

racer2c
05-04-05, 09:34 AM
They all redirect to the same place. The server is running Microsoft-IIS/6.0. If the server is using ASP pages, you can almost guarantee it is IIS. The robots file is a standard. My point was, on the default installation/setup of IIS, that file may be created automatically and they didn't know it should be modified or deleted rather than someone intentionally blocking all (good) spiders. I've never set up an IIS server, so I don't know it the file is there to begin with or not.

IIS doesn't create a robot.txt file.

Steve99
05-04-05, 03:36 PM
Could it be a copyright issue with who is/was running Champ Car's website? Or maybe they had a problem in the past with a malacious spider and went overboard?

dando
05-04-05, 07:56 PM
Why on earth would you want to shut every search engine out of your entire site? :confused:
Stealth marketing? :confused: :saywhat:

Either that, or the offshore company that created the site has no clue. :saywhat:

-Kevin