This package provides a class to crawl links on a website. Under the hood Guzzle promises are used to crawl multiple urls concurrently. Because the crawler can execute JavaScript, it can crawl JavaScript rendered site. Under the hood headless Chrome is used to power this feature.

This package was based on spatie/crawler v2.7.1. Spatie is a webdesign agency in Antwerp, Belgium. You'll find an overview of all our open source projects on our website.

The crawler can be instantiated like this:

Crawler::create()
    ->setCrawlObserver(<implementation of \Dmelearn\Crawler\CrawlObserver>)
    ->startCrawling($url);

Or if a site requires logging in (via a POST request) and using a cookie before being crawled:

Crawler::createLoggedIn(
    $loginUrl,
    [
        'username' => 'myusername',
        'password' => 'mypassword'
    ])
    ->setCrawlObserver(<implementation of \Dmelearn\Crawler\CrawlObserver>)
    ->startCrawling($url);

The argument passed to setCrawlObserver must be an object that implements the \Dmelearn\Crawler\CrawlObserver interface:

/**
 * Called when the crawler will crawl the given url.
 *
 * @param \Dmelearn\Crawler\Url $url
 */
public function willCrawl(Url $url);

/**
 * Called when the crawler has crawled the given url.
 *
 * @param \Dmelearn\Crawler\Url $url
 * @param \Psr\Http\Message\ResponseInterface $response
 * @param \Dmelearn\Crawler\Url $foundOn
 */
public function hasBeenCrawled(Url $url, $response, Url $foundOn = null);

/**
 * Called when the crawl has ended.
 */
public function finishedCrawling();

By default the crawler will not execute JavaScript. This is how you can enable the execution of JavaScript:

Crawler::create()
    ->executeJavaScript()
    ...

Under the hood headless Chrome is used to execute JavaScript. Here are some pointers on how to install it on your system.

The package will make an educated guess as to where Chrome is installed on your system. You can also manually pass the location of the Chrome binary to executeJavaScript()

Crawler::create()
    ->executeJavaScript($pathToChrome)
    ...

You can tell the crawler not to visit certain urls by passing using the setCrawlProfile-function. That function expects an objects that implements the Dmelearn\Crawler\CrawlProfile-interface:

/*
 * Determine if the given url should be crawled.
 */
public function shouldCrawl(Url $url): bool;

This package comes with three CrawlProfiles out of the box:

• CrawlAllUrls: this profile will crawl all urls on all pages including urls to an external site.
• CrawlInternalUrls: this profile will only crawl the internal urls on the pages of a host.
• CrawlSubdomainUrls: this profile will only crawl the internal urls and its subdomains on the pages of a host.

To improve the speed of the crawl the package concurrently crawls 10 urls by default. If you want to change that number you can use the setConcurrency method.

Crawler::create()
    ->setConcurrency(1) //now all urls will be crawled one by one

By default, the crawler continues until it has crawled every page of the supplied URL. If you want to limit the amount of urls the crawler should crawl you can use the setMaximumCrawlCount method.

// stop crawling after 5 urls

Crawler::create()
    ->setMaximumCrawlCount(5) 

By default, the crawler continues until it has crawled every page of the supplied URL. If you want to limit the depth of the crawler you can use the setMaximumDepth method.

Crawler::create()
    ->setMaximumDepth(2) 

When crawling a site the crawler will put urls to be crawled in a queue. By default this queue is stored in memory using the built in CollectionCrawlQueue.

When a site is very large you may want to store that queue elsewhere, maybe a database. In such cases you can write your own crawl queue.

A valid crawl queue is any class that implements the Dmelearn\Crawler\CrawlQueue\CrawlQueue-interface. You can pass your custom crawl queue via the setCrawlQueue method on the crawler.

Crawler::create()
    ->setCrawlQueue(<implementation of \Dmelearn\Crawler\CrawlQueue\CrawlQueue>) 

Please see CHANGELOG for more information what has changed recently.

Please see CONTRIBUTING for details.

To run the tests you'll have to start the included node based server first in a separate terminal window.

cd tests/server
npm install
./start_server.sh

With the server running, you can start testing.

vendor/bin/phpunit

• Freek Van der Herten
• All Contributors

Spatie is a webdesign agency based in Antwerp, Belgium. You'll find an overview of all our open source projects on our website.

Does your business depend on our contributions? Reach out and support us on Patreon. All pledges will be dedicated to allocating workforce on maintenance and new awesome stuff.

The MIT License (MIT). Please see License File for more information.