[paulanthony]

Is it a good idea to map old urls to new upon a website update? Or am I best to let search engines find a custom 404 page with links to the rest of the site. The structure of the site im working on has completely changed.

[cam]

You can use mod_rewrite to always direct people back to the site root if the url they requested wasn't found, rather than use 404's (e.g. http://www.teetonic.com/thisisarathersillyexampleurl). That's what we do on TT and it works very well - it also means we don't have to worry as much about mapping old URL's to new ones.

[justal]

Probably best to use mod_rewrite to redirect the visitor to the new page and let the search engines know that the page has been permanently moved with a proper HTTP "permanently moved" redirection (aka 301 error).

BTW cam, your teetonic.com mod_rewrite rule doesn't seem to work if a file extension is added to the url... You then just get a 404 error.

Al.

[cam]

Originally Posted by justal

BTW cam, your teetonic.com mod_rewrite rule doesn't seem to work if a file extension is added to the url... You then just get a 404 error.

Yep, but there wasn't any file extensions beforehand either - we've always used directory-style url's, so that's not an issue

[stickmus]

I read somewhere recently that you should always retain old urls. Not sure why, as I can't remember where I read it, but it was pretty sure of itself.

[cam]

bookmarks

[stickmus]

Fuck bookmarks, I've got my photographic memory to rely on.

[paulanthony]

Originally Posted by cam

it also means we don't have to worry as much about mapping old URL's to new ones.

Great for human visitors - but do the robots get the hump? i.e. do they suspect they may be being spammed? I could redirect back to the site root very easily, im just wondering if throwing a redirect rather than a 404 is the less favourable for the search engines, considering that they will probably remove 404's from the SERPS after a while.

[paulanthony]

http://gsitecrawler.com/articles/error-404-200.asp

Worth a read. Cam I'd make sure you are adding a status code (404) to your custom error page otherwise

Shoot yourself in the foot with a 200-file not found
Assuming you have several URLs which Google now has in the queue to be added to the index and that a part of these are actually file-not-found pages with the result code 200. According to Google's quality guidelines (http://www.google.com/intl/en/webmas...uidelines.html) you should take care that you "Don't create multiple pages, subdomains, or domains with substantially duplicate content." It's listed as a recommendation, but you can believe me, it's more than that: Google will remove your duplicate content - either everything or all but one. Finding duplicate content within your website is really easy with my GSiteCrawler, by the way :-)

So it removes the error pages - great, huh? That's what we wanted in the first place. Well, it actually does a bit more. Every time your site goes against the rules, Google likes it a little bit less. Small things can add up, and 1000's of URLs which your server keeps on passing out (along with 200-all ok) but which Google has to filter out each time it comes visiting WILL make an impact. It probably won't get you thrown out by itself, but it does have an effect on the speeed and frequency of the crawl.

And then we come to the worst-case scenario: Instead of an error page, your server redirects to your start page. Can you see where this is going? So Google doesn't like duplicate content, let's get rid of everything or everything but ... this last URL. In the end you'll either have all those URLs gone (including the start page) or you might end up with an INVALID URL passing the content of your start page. Yes, that does matter, because that invalid URL will not have many external links pointing to it, i.e. it will have a very low pagerank (PR - Google uses this in part to determine the order of the results in the search pages). That was a good idea to redirect to the start page, huh? NOT!