wscraper.js is a web scraper agent written in node.js and based on cheerio.js a fast, flexible, and lean implementation of core jQuery; It is built on top of request.js and inspired by http-agent.js;
There are two ways to use wscraper: http agent mode and local mode.
In HTTP Agent mode, pass it a host, a list of URLs to visit and a scraping JS script. For each URLs, the agent makes a request, gets the response, runs the scraping script and returns the result of the scraping. Valid usage is:
// scrape a single page from a web sitevaragent=wscraper.createAgent();agent.start('google.com','/finance',script);// scrape multiple pages from a websitewscraper.start('google.com',['/','/finance','/news'],script);The URLs should be passed as an array of strings. In case only one page needs to be scraped, the URL can be passed as a single string. Null or empty URLs are treated as root '/'. Suppose you want to scrape from http://google.com/finance website the stocks price of the following companies: Apple, Cisco and Microsoft.
// load node.js librariesvarutil=require('util');varwscraper=require('wscraper');varfs=require('fs');// load the scraping script from a filevarscript=fs.readFileSync('/scripts/googlefinance.js');varcompanies=['/finance?q=apple','/finance?q=cisco','/finance?q=microsoft'];// create a web scraper agent instancevaragent=wscraper.createAgent();agent.on('start',function(n){util.log('[wscraper.js] agent has started; '+n+' path(s) to visit');});agent.on('done',function(url,price){util.log('[wscraper.js] data from '+url);// display the results util.log('[wscraper.js] current stock price is '+price+' USD');// next item to process if anyagent.next();});agent.on('stop',function(n){util.log('[wscraper.js] agent has ended; '+n+' path(s) remained to visit');});agent.on('abort',function(e){util.log('[wscraper.js] getting a FATAL ERROR ['+e+']');util.log('[wscraper.js] agent has aborted');process.exit();});// run the web scraper agentagent.start('www.google.com',companies,script);The scraping script should be pure client JavaScript, including JQuery selectors. See cheerio.js for details. I should return a valid JavaScript object. The scraping script is passed as a string and usually is read from a file. You can scrape different websites without change any line of the main code: only write different JavaScript scripts. The scraping script is executed in a sandbox using a separate VM context and the script errors are caught without crash of the main code.
At time of writing, google.com/finance website reports financial data of public companies as in the following html snippet:
...
<divid="price-panel" class="id-price-panel goog-inline-block"><div><spanclass="pr"><spanid="ref_22144_l">656.06</span></span></div></div>
...By using JQuery selectors, we design the scraping script "googlefinance.js" to find the current value of a company stocks and return it as a text:
/*googlefinance.js$ -> is the DOM document to be parsedresult -> is the object containing the result of parsing*/result={};price=$('div.id-price-panel').find('span.pr').children().text();result.price=price;// result is '656.06'Sometimes, you need to scrape local html files without make a request to a remote server. Wscraper can be used as inline scraper. It takes an html string and a JS scraping script. The scraper runs the scraping script and returns the result of the scraping. Valid usage is:
varscraper=wscraper.createScraper();scraper.run(html,script);Only as trivial example, suppose you want to replace the class name of
// load node.js librariesvarutil=require('util');varfs=require('fs');varwscraper=require('wscraper');// load your html pagevarhtml=fs.readFileSync('/index.html');// load the scraping script from a filevarscript=fs.readFileSync('/scripts/replace.js');// create the scrapervarscraper=wscraper.createScraper();scraper.on('done',function(result){// do something with the resultutil.log(result)});scraper.on('abort',function(e){util.log('Getting error in parsing: '+e)});// run the scraperscraper.run(html,script);By using JQuery selectors, we design the scraping script "replace.js" to find the
/*replace.js$ -> is the DOM document to be parsedresult -> is the final JSON string containing the result of parsinguse var js-obj = JSON.parse(result) to get a js object from the json stringuse JSON.stringify(js-obj) to get back a json string from the js object*/result={};varimgs=$('img.MyPhotos').toArray();$.each(imgs,function(index,elem){varparentdiv=$(elem).parent();varnewdiv=$('<div class="Hidden"/></div>');$(elem).parent().replaceWith(newdiv)});result.replaced=$.html()||'';Happy scraping!