ExecuteJavascriptMiddleware not waiting long enough - option request for wait until network idle

[chrismcintosh]

I'm finding the execute javascript middleware to be extremely useful but sometimes it doesn't wait long enough and activity is still happening in the DOM.

I ended up having to copy over the middleware and chain on ->waitUntilNetworkIdle() in the constructor

In order for me to be able get the markup from the DOM otherwise lots of critical data doesn't get rendered.

I might have missed this as an option somewhere - there is likely a better way to indicate this as an option rather than an entirely separate middleware file.

<?php

declare(strict_types=1);

/**
 * Copyright (c) 2022 Kai Sassnowski
 *
 * For the full copyright and license information, please view
 * the LICENSE file that was distributed with this source code.
 *
 * @see https://github.com/roach-php/roach
 */

namespace App\Roach\Middleware;

use Psr\Log\LoggerInterface;
use RoachPHP\Http\Response;
use RoachPHP\Downloader\Middleware\ResponseMiddlewareInterface;
use RoachPHP\Support\Configurable;
use Spatie\Browsershot\Browsershot;
use Throwable;

final class ExecuteJavascriptNetworkIdleMiddleware implements ResponseMiddlewareInterface
{
 use Configurable;

 /**
 * @var callable(string): Browsershot
 */
 private $getBrowsershot;

 /**
 * @param null|callable(string): Browsershot $getBrowsershot
 */
 public function __construct(
 private LoggerInterface $logger,
 ?callable $getBrowsershot = null,
 ) {
 $this->getBrowsershot = $getBrowsershot ?: static fn (string $uri): Browsershot => Browsershot::url($uri)->waitUntilNetworkIdle();
 }

 public function handleResponse(Response $response): Response
 {
 $browsershot = $this->configureBrowsershot(
 $response->getRequest()->getUri(),
 );

 try {
 $body = $browsershot->bodyHtml();
 } catch (Throwable $e) {
 $this->logger->info('[ExecuteJavascriptMiddleware] Error while executing javascript', [
 'message' => $e->getMessage(),
 'trace' => $e->getTraceAsString(),
 ]);

 return $response->drop('Error while executing javascript');
 }

 return $response->withBody($body);
 }

 /**
 * @psalm-suppress MixedArgument, MixedAssignment
 */
 private function configureBrowsershot(string $uri): Browsershot
 {
 $browsershot = ($this->getBrowsershot)($uri);

 if (!empty($this->option('chromiumArguments'))) {
 $browsershot->addChromiumArguments($this->option('chromiumArguments'));
 }

 if (null !== ($chromePath = $this->option('chromePath'))) {
 $browsershot->setChromePath($chromePath);
 }

 if (null !== ($binPath = $this->option('binPath'))) {
 $browsershot->setBinPath($binPath);
 }

 if (null !== ($nodeModulePath = $this->option('nodeModulePath'))) {
 $browsershot->setNodeModulePath($nodeModulePath);
 }

 if (null !== ($includePath = $this->option('includePath'))) {
 $browsershot->setIncludePath($includePath);
 }

 if (null !== ($nodeBinary = $this->option('nodeBinary'))) {
 $browsershot->setNodeBinary($nodeBinary);
 }

 if (null !== ($npmBinary = $this->option('npmBinary'))) {
 $browsershot->setNpmBinary($npmBinary);
 }

 return $browsershot;
 }

 private function defaultOptions(): array
 {
 return [
 'chromiumArguments' => [],
 'chromePath' => null,
 'binPath' => null,
 'nodeModulePath' => null,
 'includePath' => null,
 'nodeBinary' => null,
 'npmBinary' => null,
 ];
 }
}

Thank you!

[ksassnowski]

Thanks for pointing this out! I think this would be a sensible change to make in the library itself. Do you want to open a pull request with this change?

[chrismcintosh]

Absolutely, just put one through. Hopefully everything is correct there its my first OSS pull request 😅.

Thanks!

[ksassnowski]

Fixed in the 1.1.1 release. Thanks a lot for the contribution!