I'm trying to implement a crawler with System.Threading.Tasks.Dataflow, the core feature is to have a TransformBlock, which accepts an url as input, request the url, parse the response, and find more urls for the same block.
Then I noticed a problem that I cannot complete/stop this pipeline. The following code simulates my problem:
usingSystem;usingSystem.Threading;usingSystem.Threading.Tasks.Dataflow;namespaceConsoleApp1{classProgram{staticvoidMain(){intcount=0;varblock=newTransformBlock<string,string>(uri =>{//await Task.Delay(10);Console.WriteLine($"Downloading {count}");if(Interlocked.Increment(refcount)>10){returnnull;}returnuri;});varlinkOptions=newDataflowLinkOptions{PropagateCompletion=false};block.LinkTo(block,linkOptions,(x)=>!string.IsNullOrEmpty(x));block.Post("");block.Post("");while(true){Console.WriteLine($"Check Count: {count}");if(count>=2){Console.WriteLine("Complete");block.Complete();break;}Thread.Sleep(20);}Console.WriteLine("After Check");block.Completion.Wait();Console.WriteLine("Done");}}}The output is:
Check Count: 0
Downloading 0
Downloading 1
Check Count: 2
Complete
After Check
And program stuck here, block.Completion.Wait() never return.
I've also tried to have two blocks link to each other, yet the result is same.
So what can I do about my code? How can I stop this dataflow?
I'm trying to implement a crawler with
System.Threading.Tasks.Dataflow, the core feature is to have aTransformBlock, which accepts an url as input, request the url, parse the response, and find more urls for the same block.Then I noticed a problem that I cannot complete/stop this pipeline. The following code simulates my problem:
The output is:
And program stuck here,
block.Completion.Wait()never return.I've also tried to have two blocks link to each other, yet the result is same.
So what can I do about my code? How can I stop this dataflow?