Skip to content

stopWhenComplete，增加动态修改完成时停止方法。 #1169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sutra merged 1 commit into from
Jun 17, 2024
Merged

stopWhenComplete，增加动态修改完成时停止方法。 #1169

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion webmagic-core/src/main/java/us/codecraft/webmagic/Spider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ public class Spider implements Runnable, Task {

protected AtomicInteger stat = new AtomicInteger(STAT_INIT);

protected boolean exitWhenComplete = true;
protected volatile boolean exitWhenComplete = true;

protected final static int STAT_INIT = 0;

Expand Down Expand Up @@ -598,6 +598,13 @@ public void stop() {
}
}

/**
* Stop when all tasks in the queue are completed and all worker threads are also completed
*/
public void stopWhenComplete(){

@sutra sutra Jun 17, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

增加这个方法是干什么用的？为了让外部调用？系统运行的时候，修改这个变量？

已经有 setExitWhenComplete 了。

@niuxiaozu niuxiaozu Jun 17, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

好吧，只是给exitWhenComplete加了个volatile，然后起了个好看点的方法名，其实不改动也能做到，就是看什么时候轮训线程什么时候刷新缓存了。

@sutra sutra Jun 17, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

为什么需要在运行的时候修改这个值？

@niuxiaozu niuxiaozu Jun 17, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我这边的场景是，多个spider协同运行，一开始就把所有spider启动起来，然后在某个时机，比如主要的spider运行的spider执行结束后，手动停止其他的spider，但是又不想丢弃队列中的任务，所以就需要有个方法使得spider在运行完所有任务后停止。就像线程池的shutdown一样

this.exitWhenComplete = true;
}

/**
* start with more than one threads
*
Expand Down