Scraping

What do you think of scraping as a business model?

Anonymous

January 12th, 2016

I am a technical person working on a side project that works by scraping data from various career websites and builds a database of companies and the relative  popularity of skills over time in those companies.

The idea is that 40% of jobs aren't advertised and that my tool can be used as a prospecting tool for recruiters or jobseekers to find companies that would hire someone with their skillset.

However I'm in 2 minds whether or not building a foundation on scraping other sites is good idea or not. Google, Indeed and SkyScanner have all built successfull businesses, even though they were (probably) violating the terms of sites they scraped at the start.

I would love to hear the communities feedback.

Brad Hall Founder/President, EntertainmentCareers.Net, Inc.

January 14th, 2016

I know this space well.  I would encourage you to shy away from thinking that your business model is"scraping." As you have seen from the comments above, "scaping" has a bit of a negative connotation as it implies to some degree IP theft which is not your goal. It would appear that this project is data mining as you are looking to repurpose data to create a completely different service vs "scraping" and republishing.  The mechanics are inconsequential vs what service you are providing and to whom.    If you provide a valuable service to employers and/or job seekers the companies may even provide you with xml feeds like they  do for indeed, linked in etc so "scraping" is not relevant.  I assume you are in the infant stages and you need data to build proof of concept. Don't get muddled in the how, focus on the what first to see if this is a tool worth building and who is the buyer?  The company itself, the HRIS/ATS side i.e. Taleo, Worday etc. 

Bob Graham Engineering and Software

January 12th, 2016

Do it and see if it gets users and makes money. If it is, then its a good idea.
The only way to know is to try. I like the scraping idea because its fast.

Michael Brill Technology startup exec focused on AI-driven products

January 12th, 2016

Everyone scrapes everything... all the time... everywhere. 

You can do the little ethics dance everyone does or pretends to do.

But the reality is that scraping is the jaywalking of the internet.

Dave Reimherr Founder at Magnificent Marketing LLC

January 12th, 2016

I say go for it and don't look back.

Jenn Steele Director of Product Marketing at Indix

January 13th, 2016

Step 1: Call it "crawling" rather than "scraping"--it's less creepy that way
Step 2: ??? (or just go for it)
Step 3: Profit (or large crash & burn)

Anonymous

January 13th, 2016

Sounds interesting.  Let us know what you uncover.

Gopi Mattel Director, Chennai Area at The Founder Institute

January 13th, 2016

Well now there is an API for scraping websites, perhaps. And it is popular.
http://techcrunch.com/2016/01/12/import-io-raises-13-million-series-a-for-its-data-extraction-platform/
This could make your technology solution easier to solve.

Chris Leonard Scientific Review Manager at SRA International

January 14th, 2016

If you are concerned about violating terms, then adhere to the site's robot exclusion policy (in the robots.txt file).  http://www.robotstxt.org/  If they don't have a robots.txt file, then scrape away.

Rakesh Andey Got a Solution, Need People to Code, Co-Founder at Ruder App,Co-Founder@Instano(Acquired)

January 14th, 2016

There are couple of players already. webscraper.io & import.io works really good. :)