webmagic-selenium 0.6.1 无法启动

[royal430]

我在使用webmagic-selenium 0.6.1 进行异步下载的时候，报出下面的异常，看了源码，不知为何一定要读下面这个配置。如果您有时间，还请不吝赐教。谢谢

java.io.FileNotFoundException: \Users\Bingo\Documents\workspace\webmagic\webmagic-selenium\config.ini (系统找不到指定的路径。)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at java.io.FileInputStream.(FileInputStream.java:93)
at java.io.FileReader.(FileReader.java:58)
at us.codecraft.webmagic.downloader.selenium.WebDriverPool.configure(WebDriverPool.java:67)
at us.codecraft.webmagic.downloader.selenium.WebDriverPool.get(WebDriverPool.java:192)
at us.codecraft.webmagic.downloader.selenium.SeleniumDownloader.download(SeleniumDownloader.java:78)
at us.codecraft.webmagic.Spider.processRequest(Spider.java:404)
at us.codecraft.webmagic.Spider$1.run(Spider.java:321)
at us.codecraft.webmagic.thread.CountableThreadPool$1.run(CountableThreadPool.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

[code4craft]

使用selenium模块需要另外安装selenium-driver，是这个原因么？

[royal430]

用 webmagic-selenium 0.5.2 不会有这个问题，我也不知道这个配置文件的作用是什么。代码形如

Spider.create(new MfrPageProcessor()).addUrl("url").thread(1)
.setDownloader(new SeleniumDownloader("D:/lib/chromedriver.exe"))
.run();

还请帮再看看，0.6.1有demo吗？

[code4craft]

嗯，有bug，因为引入了 @bingoko 之前提交的代码，增加了GhostDriver来支持phatomjs。
这里有个问题：写死了配置到本地，我改成了可以通过设置参数-Dselenuim_config=/data/webmagic/webmagic-selenium/config.ini的方式来指定。

我本地测试了一下，基本可用了，代码提交到最新master分支。

配置方式见：
https://github.com/detro/ghostdriver/blob/master/test/config.ini

[royal430]

thanks

[krmao]

2018-10-18 09:35:48,054 WARN [us.codecraft.webmagic.Spider] - start
2018-10-18 09:35:48,131 DEBUG [us.codecraft.webmagic.scheduler.QueueScheduler] - push to queue http://gaoxiao.52op.net/fangyan/6195.htm
2018-10-18 09:35:48,133 WARN [us.codecraft.webmagic.Spider] - end
2018-10-18 09:35:48,134 INFO [us.codecraft.webmagic.Spider] - Spider gaoxiao.52op.net started!
Test will use an external GhostDriver
2018-10-18 09:35:48,180 ERROR [us.codecraft.webmagic.Spider] - process request Request{url='http://gaoxiao.52op.net/fangyan/6195.htm', method='null', extras=null, priority=0, headers={}, cookies={}} error
java.lang.IllegalStateException: The driver executable does not exist: /Users/detro/bin/phantomjs-upstream
	at com.google.common.base.Preconditions.checkState(Preconditions.java:177)
	at org.openqa.selenium.remote.service.DriverService.checkExecutable(DriverService.java:117)
	at org.openqa.selenium.phantomjs.PhantomJSDriverService.findPhantomJS(PhantomJSDriverService.java:246)
	at org.openqa.selenium.phantomjs.PhantomJSDriverService.createDefaultService(PhantomJSDriverService.java:182)
	at org.openqa.selenium.phantomjs.PhantomJSDriver.<init>(PhantomJSDriver.java:99)
	at us.codecraft.webmagic.downloader.selenium.WebDriverPool.configure(WebDriverPool.java:140)
	at us.codecraft.webmagic.downloader.selenium.WebDriverPool.get(WebDriverPool.java:196)
	at us.codecraft.webmagic.downloader.selenium.SeleniumDownloader.download(SeleniumDownloader.java:76)
	at us.codecraft.webmagic.Spider.processRequest(Spider.java:404)
	at us.codecraft.webmagic.Spider.access$000(Spider.java:61)
	at us.codecraft.webmagic.Spider$1.run(Spider.java:320)
	at us.codecraft.webmagic.thread.CountableThreadPool$1.run(CountableThreadPool.java:74)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
2018-10-18 09:35:48,185 INFO [us.codecraft.webmagic.Spider] - Spider gaoxiao.52op.net closed! 1 pages downloaded.

迄今 仍然不能用 ......也不知道该怎么用

    compile 'us.codecraft:webmagic-core:0.7.3'
    compile 'us.codecraft:webmagic-extension:0.7.3'
    compile 'us.codecraft:webmagic-selenium:0.7.3'