Почему Selenium с Geckodriver бесконечно обрабатывает запросы?

Question

maksimwork1980 @maksimwork1980

Почему Selenium с Geckodriver бесконечно обрабатывает запросы?

Сначала у меня выводило
Error: no DISPLAY environment variable specified

spoiler

puttysftp@HostNameMax:/$ java -jar /usr/bin/selenium-server-4.1.3.jar standalone
19:14:45.013 INFO [LogManager$RootLogger.log] - Using the system default encoding
19:14:45.022 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
19:14:46.928 INFO [NodeOptions.getSessionFactories] - Detected 1 available processors
19:14:46.971 INFO [NodeOptions.discoverDrivers] - Discovered 1 driver(s)
19:14:47.027 INFO [NodeOptions.report] - Adding Firefox for {"browserName": "firefox"} 1 times
19:14:47.083 INFO [Node.<init>] - Binding additional locator mechanisms: name, id, relative
19:14:47.166 INFO [GridModel.setAvailability] - Switching node 148a16c3-5601-40b6-8775-45dc5392f39a (uri: http://192.168.0.8:4444) from DOWN to UP
19:14:47.173 INFO [LocalDistributor.add] - Added node 148a16c3-5601-40b6-8775-45dc5392f39a at http://192.168.0.8:4444. Health check every 120s
19:14:47.556 INFO [Standalone.execute] - Started Selenium Standalone 4.1.3 (revision 7b1ebf28ef): http://192.168.0.8:4444
19:14:50.251 INFO [LocalDistributor.newSession] - Session request received by the distributor:
 [Capabilities {browserName: firefox, moz:firefoxOptions: {profile: UEsDBBQAAAAAANiZllTf9RXUMgA...}}, Capabilities {browserName: firefox, firefox_profile: UEsDBBQAAAAAANiZllTf9RXUMgA..., platform: ANY}]
1650629690366   geckodriver     INFO    Listening on 127.0.0.1:48895
1650629691914   mozrunner::runner       INFO    Running command: "/usr/bin/firefox" "--marionette" "-no-remote" "-profile" "/tmp/rust_mozprofileMN4aLn"
Error: no DISPLAY environment variable specified
19:14:52.110 WARN [SeleniumSpanExporter$1.lambda$export$0] - {"traceId": "4268ba544fbca585371f4781fdd63e67","eventTime": 1650629692082538606,"eventName": "exception","attributes": {"driver.url": "http:\u002f\u002flocalhost:48895","exception.message": "Error while creating session with the driver service. Stopping driver service: Could not start a new session. Response code 500. Message: Process unexpectedly closed with status 1\nBuild info: version: '4.1.3', revision: '7b1ebf28ef'\nSystem info: host: 'HostNameMax', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '5.10.0-9-amd64', java.version: '1.8.0_321'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Response code 500. Message: Process unexpectedly closed with status 1\nBuild info: version: '4.1.3', revision: '7b1ebf28ef'\nSystem info: host: 'HostNameMax', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '5.10.0-9-amd64', java.version: '1.8.0_321'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:126)\n\tat org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:84)\n\tat org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:62)\n\tat org.openqa.selenium.grid.node.config.DriverServiceSessionFactory.apply(DriverServiceSessionFactory.java:133)\n\tat org.openqa.selenium.grid.node.config.DriverServiceSessionFactory.apply(DriverServiceSessionFactory.java:66)\n\tat org.openqa.selenium.grid.node.local.SessionSlot.apply(SessionSlot.java:144)\n\tat org.openqa.selenium.grid.node.local.LocalNode.newSession(LocalNode.java:354)\n\tat org.openqa.selenium.grid.distributor.local.LocalDistributor.startSession(LocalDistributor.java:603)\n\tat org.openqa.selenium.grid.distributor.local.LocalDistributor.newSession(LocalDistributor.java:530)\n\tat org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.handleNewSessionRequest(LocalDistributor.java:740)\n\tat org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable.lambda$run$1(LocalDistributor.java:701)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:750)\n","exception.type": "org.openqa.selenium.SessionNotCreatedException","logger": "org.openqa.selenium.grid.node.config.DriverServiceSessionFactory","session.capabilities": "{\"browserName\": \"firefox\",\"moz:firefoxOptions\": {\"profile\": \"UEsDBBQAAAAAANiZllTf9RXUMgAAADIAAAAHAAAAdXNlci5qc3VzZXJfcHJlZigicmVhZGVyLnBhcnNlLW9uLWxvYWQuZW5hYmxlZCIsIGZhbHNlKTsKUEsBAj8DFAAAAAAA2JmWVN\\u002f1FdQyAAAAMgAAAAcAAAAAAAAAAAAAAKSBAAAAAHVzZXIuanNQSwUGAAAAAAEAAQA1AAAAVwAAAAAA\"}}\n"}}

1650629692148   geckodriver     INFO    Listening on 127.0.0.1:48564
1650629692279   mozrunner::runner       INFO    Running command: "/usr/bin/firefox" "--marionette" "-no-remote" "-profile" "/tmp/rust_mozprofilesFyGNO"
Error: no DISPLAY environment variable specified
...

Я установил Xvfb. Запустил:

$ Xvfb :1 -screen 0 640x480x24 -fbdir /var/tmp&
$ export DISPLAY=:1.0

и теперь при парсинге другая ошибка - страница обрабатывается бесконечно долго (и обрывается загрузка после 30 секунд):

spoiler

puttysftp@HostNameMax:/$ java -jar /usr/bin/selenium-server-4.1.3.jar standalone
19:20:08.157 INFO [LogManager$RootLogger.log] - Using the system default encoding
19:20:08.166 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
19:20:09.911 INFO [NodeOptions.getSessionFactories] - Detected 1 available processors
19:20:09.959 INFO [NodeOptions.discoverDrivers] - Discovered 1 driver(s)
19:20:10.018 INFO [NodeOptions.report] - Adding Firefox for {"browserName": "firefox"} 1 times
19:20:10.067 INFO [Node.<init>] - Binding additional locator mechanisms: name, id, relative
19:20:10.164 INFO [GridModel.setAvailability] - Switching node 282dcaea-74b3-422a-b265-af60468241e0 (uri: http://192.168.0.8:4444) from DOWN to UP
19:20:10.166 INFO [LocalDistributor.add] - Added node 282dcaea-74b3-422a-b265-af60468241e0 at http://192.168.0.8:4444. Health check every 120s
19:20:10.517 INFO [Standalone.execute] - Started Selenium Standalone 4.1.3 (revision 7b1ebf28ef): http://192.168.0.8:4444
19:20:13.128 INFO [LocalDistributor.newSession] - Session request received by the distributor:
 [Capabilities {browserName: firefox, firefox_profile: UEsDBBQAAAAAAIaallTf9RXUMgA..., platform: ANY}, Capabilities {browserName: firefox, moz:firefoxOptions: {profile: UEsDBBQAAAAAAIaallTf9RXUMgA...}}]
1650630013244   geckodriver     INFO    Listening on 127.0.0.1:46267
1650630014794   mozrunner::runner       INFO    Running command: "/usr/bin/firefox" "--marionette" "-no-remote" "-profile" "/tmp/rust_mozprofileq31W2U"

Код скрипта:

<?php
namespace Facebook\WebDriver;
use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\Remote\RemoteWebDriver;
require_once('/var/www/sites/product_catalog/php/vendor/autoload.php');
$host = 'http://localhost:4444/';
$capabilities = DesiredCapabilities::firefox();
$driver = RemoteWebDriver::create($host, $capabilities);
$driver->get('http://example.com');
$result = $driver->findElement(WebDriverBy::cssSelector('.sign-in'))->getText();
echo $result;
$driver->quit();
?>

Что теперь можно сделать? Может изучить по быстрому Java и на нём запускать Selenium (Я работаю C++ программистом, а Java и C++ очень похожи. Веб разработка и сервера это всего лишь моё хобби)
P.S. И хотел бы ещё совет, где искать решения самостоятельно. Многие абсолютно разные, некоторые нерабочие.