Web Design Forums

ins · 07-06-2007, 16:15

Hi,

I have about 25000 external links on my community website resource database; how easy it is to create a crawler to automatically detect the dead links and report them back to admin for manual removal, and,

How to create and display cache for all these external links, in case the external website is down. Just like Google has.

If you can refer me to some free/cheap software or literature available, it would be great.

Thanks in advance!

Ins

adam c · 08-06-2007, 06:00

Not sure exactly how you'd do this but I'm pretty sure you'd have to use something like Python to do it.
That might be a good place to start looking.

07-06-2007, 16:15	#1 (permalink)
ins Registered User Join Date: Apr 2007 Posts: 5	Cache creator & Crawler to detect dead links on website. Hi, I have about 25000 external links on my community website resource database; how easy it is to create a crawler to automatically detect the dead links and report them back to admin for manual removal, and, How to create and display cache for all these external links, in case the external website is down. Just like Google has. If you can refer me to some free/cheap software or literature available, it would be great. Thanks in advance! Ins

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

08-06-2007, 06:00	#2 (permalink)
adam c berserker than thou Join Date: Sep 2005 Location: Bristol Posts: 450	Not sure exactly how you'd do this but I'm pretty sure you'd have to use something like Python to do it. That might be a good place to start looking.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)