DesignersTalk - View Single Post - Extract Data from site and put into tables?

MikeMackay · 19-04-2006, 18:10

There is some bulky formatting code in there which is a shame, it makes the job of data extraction a lot harder.

After looking through it you probably could extract the data you wanted from the markup MLS generates. You would need a server-side scripting language, such as Perl or PHP, and some regular expressions to tell it what data to remove - this method is known as "Screen Scraping". The downsides to this however, is that every time the company (or whoever is in control of the query output) decides to make a change to the HTML code generated, your script will need to be re-written in places to cater for the new change(s).

I am also guessing the results are generated after a standard form submission so your script would need to be able to mimick that submission (including any user authentication requests) before retrieving and parsing the data.

Some companies offer what's known as API access where you can retrieve data from a database in a specialy formatted way, such as an XML file or tab delimited file. This makes data extraction a lot leaner and quicker without the overheads & worries of Screen Scraping. I'm guessing the MLS system doesn't offer this type of access ?

You would also need to decide the frequency of your extractions. Do you perform the extraction whenever the data is requested or do you run it at a set time every day and update a local file ? The second method is often preferred for data that changes every few hours, in essence you are storing a local 'cache' of the data output. Far less overheads than performing the extraction "On Demand".

Overall, if you are local for a general path to take and a place to start looking, I would suggest PHP as your language of choice. PHP supports everything needed to perform your request.

Hope this helps get you going in the right direction. Perhaps some of DT'ers will chime in and offer some alternative and/or better advice.

- Mike

PS: There could further issues with the system that prevent this procedure/system from working, however, if it is a simple as you have posted then all should be fine.

19-04-2006, 18:10	#5 (permalink)
MikeMackay Yes I look angry. Join Date: Feb 2005 Location: To the left a bit... Posts: 924	There is some bulky formatting code in there which is a shame, it makes the job of data extraction a lot harder. After looking through it you probably could extract the data you wanted from the markup MLS generates. You would need a server-side scripting language, such as Perl or PHP, and some regular expressions to tell it what data to remove - this method is known as "Screen Scraping". The downsides to this however, is that every time the company (or whoever is in control of the query output) decides to make a change to the HTML code generated, your script will need to be re-written in places to cater for the new change(s). I am also guessing the results are generated after a standard form submission so your script would need to be able to mimick that submission (including any user authentication requests) before retrieving and parsing the data. Some companies offer what's known as API access where you can retrieve data from a database in a specialy formatted way, such as an XML file or tab delimited file. This makes data extraction a lot leaner and quicker without the overheads & worries of Screen Scraping. I'm guessing the MLS system doesn't offer this type of access ? You would also need to decide the frequency of your extractions. Do you perform the extraction whenever the data is requested or do you run it at a set time every day and update a local file ? The second method is often preferred for data that changes every few hours, in essence you are storing a local 'cache' of the data output. Far less overheads than performing the extraction "On Demand". Overall, if you are local for a general path to take and a place to start looking, I would suggest PHP as your language of choice. PHP supports everything needed to perform your request. Hope this helps get you going in the right direction. Perhaps some of DT'ers will chime in and offer some alternative and/or better advice. - Mike PS: There could further issues with the system that prevent this procedure/system from working, however, if it is a simple as you have posted then all should be fine.