Data Science · Big data

What are the Best Data Scraping Tools?

Lisa Falcone

October 17th, 2015

I am looking to scrape data from a few websites on a regular basis. Keep in mind that the budget is very modest.

What data scraping tools would you recommend (free or cheap), or do you have any contacts that you'd recommend for this purpose?

Thanks in advance for your feedback.


Peter K Chen

October 17th, 2015

Michael Brill Technology startup exec focused on AI-driven products

October 17th, 2015

As always, it depends on what you need to scrape, your skillset and budget. It's really a big world. I've tried and failed with products like and Kimono and have written maybe 10-20 scrapers... it is highly dependent on your skillset and the nature of the sites you want to scrape. Some take 10 minutes others are basically impossible.

My quick recommendation is that you use Upwork et al to hire a contractor to write your scrapers. They are pretty easy to write if you have the skillset and you can get your basic site scraper for, say, $100.

Armando Vieira Data Scientist, entrepreneur, speaker

October 18th, 2015

the R package rvest is very easy to use and do the work. In python there are plenty of them

Onikepe Adegbola, MD PhD

October 17th, 2015

You might be able to find someone at seo clerks:
Not affiliated with them, but know someone who has used the service and been satisfied

Matthew Watson Principal, InfiniteIQ Consulting, LLC

October 17th, 2015

I have used Outwit Hub with good results.  Affordable and fairly straight forward to use.


October 20th, 2015

Id none of the them meat your requirement than build one,I can help you in that case. 

John Dyrek Medical Economics at Aetna

October 17th, 2015

Kimono Connotate

Stefan Smiljkovic Founder at - Web Studio

October 18th, 2015

There are a lot of tools you can use, but you need to have know technical knowledge.


You can also reach me at to give me more info what you want to scrape, and I will advice you on it.

Mark Watkins Founder, The Hawaii Project

October 17th, 2015 can do this kind of thing. Solid tool if your needs are not super complex

Peter Johnston Businesses are composed of pixels, bytes & atoms. All 3 change constantly. I make that change +ve.

October 18th, 2015

There are two approaches here.

The first is batch - to do a scrape on a one-off or regular basis. This can be a chore, repeating the same task over and over.

The other is track - to dynamically link so that changes in the target site are reflected in the data you have access to.

Increasingly we are moving to this sort of dynamic linking. This again splits into two - those who would be happy for you to track them and those which would not.

For friendly dynamic linking, consider an API. Ask them to share data with you and often give them something in kind as the main payment - a commission, perhaps, or even just recognition of source.

One other thing to consider is doing your own modelling from the data. If you have either dynamic data or regular snapshots to create a timeline, you can start to see what the data is doing over time and predict what it might do in future. Eventually this can get good enough that you are almost in charge, being able to set the figure before they do and simply using their real-time data as confirmation.

As well as scraping tools, you may wish to look into dynamic linking tools and data modelling and prediction.