Add a Web Crawler System to walk/scrape multiple pages
i know the structure of the site, and i would like to extract data from it's pages. please give me some way of describing index/category pages, and how to navigate to a target data source page for extracting content
22
votes
AdminJason Swearingen
(PM, PhantomJsCloud.com)
shared this idea
we removed the webcrawler system to better design our v2. a new crawler is planned as an open-source project
-
Anonymous commented
Jason, any updates on this or where projects we can contribute to are.
For page navigation it would be useful for us to limit some how the first requests so they were faster to get to the last page