No results with specific url

[plompd]

Describe the bug
If I try to scrape the following URL, it is successful:
https://www.werkenbijzicht.nl/vacatures.html

If I try to scrape the same page, but with another URL, it returns no result:
https://www.werkenbijzicht.nl/p/1/vacatures.html

It is the same page, but initiated from the pager at the bottom.

So with the original URL, my scraper only does scrape the first 10 items and if I try to go to the next page, I get this new URL like stated above and that results in nothing.

I cannot see what the difference is or how I should fix this?

• core: 3.0.0

class ZichtSpider extends BasicSpider
{
 public array $startUrls = [
 'https://www.werkenbijzicht.nl/vacatures.html',
 ];

 public array $downloaderMiddleware = [
 RequestDeduplicationMiddleware::class,
 ExecuteJavascriptMiddleware::class,
 [RandomUserAgentMiddleware::class, ['userAgent' => 'Mozilla/5.0 (compatible; RoachPHP/0.1.0)']],
 ];

 public array $spiderMiddleware = [
 //
 ];

 public array $itemProcessors = [
 //
 ];

 public array $extensions = [
 LoggerExtension::class,
 StatsCollectorExtension::class,
 ];

 public int $concurrency = 2;

 public int $requestDelay = 1;

 public function parse(Response $response): Generator
 {
 $items = $response->filter('.actSResContainer .itemContainer')->each(function (Crawler $node) {

 $titleNode = $node->filter('.itemTitle a.cluetips');
 $relAttribute = $titleNode->attr('rel'); // Fetching the rel attribute

 // Regular expression to extract the ID
 $regex = '/id\/(\d+)\//';
 $matches = [];
 $vacancyId = '';

 if (preg_match($regex, $relAttribute, $matches)) {
 $vacancyId = $matches[1]; // The first captured group contains the ID
 }

 return [
 'url' => $titleNode->link()->getUri(),
 'title' => $titleNode->text(),
 'referenceNumber' => $vacancyId,
 ];
 });

 foreach ($items as $item) {
 yield $this->request('GET', $item['url'], 'parseJob', ['item' => $item]);
 }

 try {
 // Attempt to find the next page link. Adjust the selector as needed.
 $nextPageLink = $response->filter('.pageNav a.pnNext');

 if ($nextPageLink->count() > 0) {
 $nextPageUrl = 'https://www.werkenbijzicht.nl' . $nextPageLink->attr('href');
 yield $this->request('GET', $nextPageUrl);
 }
 } catch (\Exception $e) {
 
 }

 }

 public function parseJob(Response $response): Generator
 {

[plompd]

@ksassnowski Do you have any clue?

[ksassnowski]

I had a quick look at their site and I honestly have no clue what they're doing. The same URL sometimes returns results and sometimes doesn't. It might be tied to the current session but I haven't investigated this any further.

I'm turning this issue into a discussion since it's not really a problem with the library.