IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Web crawler (beta) FAQ Web crawler (beta) events logs reference »

› ›

Web crawler (beta) reference

edit

Web crawler (beta) reference

edit

The Elastic Enterprise Search web crawler is a beta feature. Beta features are subject to change and are not covered by the support SLA of general release (GA) features. Elastic plans to promote this feature to GA in a future release.

See Web crawler (beta) for a guided introduction to the Elastic Enterprise Search web crawler.

Refer to the following sections for terms, definitions, tables, and other detailed information.

Crawl

edit

A crawl is a process, associated with an engine, by which the web crawler discovers web content, and extracts and indexes that content into the engine as search documents.

During a crawl, the web crawler stays within user-defined domains, starting from specific entry points, and it discovers additional content according to crawl rules. Operators and administrators manage crawls through the App Search dashboard.

Each crawl also respects instructions within robots.txt files and sitemaps, and instructions embedded within HTML files using canonical link tags, robots meta tags, and nofollow links.

Operators can monitor crawls using the ID and status of each crawl, and using the web crawler events logs.

Each engine may run one active crawl at a time, but multiple engines can run crawls concurrently.

Each crawl is stateless. Crawl state is not persisted, and crawls cannot be resumed.

Each re-crawl is a full crawl. Partial crawls are not supported at this time.

Content discovery

edit

To begin each crawl, the web crawler populates a crawl queue with the entry points for the crawl and the URLs provided by any sitemaps. It then begins fetching and parsing each page in the queue. The crawler handles each page according to its HTTP response status code.

As the web crawler discovers additional URLs, it uses the crawl rules from the crawl configuration, the directives from any robots.txt files, and instructions embedded within the content to determine which of those URLs it is allowed to crawl. The crawler adds the allowed URLs to the crawl queue.

If a page is not linked from other pages, the web crawler will not discover it. Refer to the web of pages in the following image. The crawler will not discover the page outlined in red, unless an admin adds its URL as an entry point for the domain or includes the URL in a sitemap:

The web crawler continues fetching and adding to the crawl queue until the queue is empty, the crawler hits a resource limit, or the crawl fails unexpectedly.

The crawler logs detailed events while it crawls, which allow monitoring and auditing of the content discovery process.

Content extraction and indexing

edit

Content extraction is the process where the web crawler transforms an HTML document into a search document. And indexing is the process where the crawler puts the search document into an App Search engine for searching.

During content extraction, the web crawler checks for duplicate content. The crawler indexes duplicate web documents as a single search document.

The web crawler extracts each web document into a predefined schema before indexing it as a search document. At this time, the web crawler does not provide configuration or control over what content is extracted or how it is extracted.

During a crawl, the web crawler uses HTTP response status codes and robots meta tags to determine which documents it is allowed to index.

Content deletion

edit

The web crawler must also delete documents from an engine to keep its documents in sync with the corresponding web content.

During a crawl, the web crawler uses HTTP response status codes to determine which documents to delete. However, this process cannot delete stale documents in the engine that are no longer linked to on the web.

Therefore, at the conclusion of each crawl, the web crawler begins an additional "purge" crawl, which fetches URLs that exist in the engine, but have not been seen on the web during recent crawls. The crawler deletes from the engine all documents that respond with 4xx and 3xx responses during the phase.

Domain

edit

A domain is a website or other internet realm that is the target of a crawl. Each domain belongs to a crawl, and each crawl has one or more domains.

See Manage domains to manage domains for a crawl.

A crawl always stays within its domains when discovering content. It cannot discover and index content outside of its domains.

Each domain has a domain URL that identifies the domain using a protocol and hostname. The domain URL must not include a path.

Each unique combination of protocol and hostname is a separate domain. Each of the following is its own domain:

http://example.com
https://example.com
http://www.example.com
https://www.example.com
http://shop.example.com
https://shop.example.com

Each domain has:

One or more entry points
One or more crawl rules
Zero or one robots.txt files
Zero or more sitemaps

Entry point

edit

An entry point is a path within a domain that serves as a starting point for a crawl. Each entry point belongs to a domain, and each domain has one or more entry points.

See Manage entry points to manage entry points for a domain.

Use entry points to instruct the crawler to fetch URLs it would otherwise not discover. For example, if you’d like to index a page that isn’t linked from other pages, add the page’s URL as an entry point.

Ensure the entry points for each domain are allowed by the domain’s crawl rules, and the directives within the domain’s robots.txt file. The web crawler will not fetch entry points that are disallowed by crawl rules or robots.txt directives.

You can also inform the crawler of URLs using sitemaps.

Sitemap

edit

A sitemap is an XML file, associated with a domain, that informs web crawlers about pages within that domain. XML elements within the sitemap identify specific URLs that are available for crawling.

These elements are similar to entry points. You can therefore choose to submit URLs to the Enterprise Search web crawler using sitemaps, entry points, or a combination of both.

You may prefer using sitemaps over entry points for any of the following reasons:

You have already been publishing sitemaps for other web crawlers.
You don’t have access to the web crawler UI.
You prefer the sitemap file interface over the web crawler web interface.

Use sitemaps to inform the web crawler of pages you think are important, or pages that are isolated and not linked from other pages. However, be aware the web crawler will visit only those pages from the sitemap that are allowed by the domain’s crawl rules and robots.txt file directives.

Sitemap discovery and management

edit

To add a sitemap to a domain, you must specify it within a robots.txt file. At the start of each crawl, the web crawler fetches and processes each domain’s robots.txt file and each sitemap specified within those robots.txt files.

See Manage sitemaps to manage sitemaps for your domains.

Sitemap format and technical specification

edit

The sitemaps standard defines the format and technical specification for sitemaps. Refer to the standard for the required and optional elements, character escaping, and other technical considerations and examples.

The Enterprise Search web crawler does not process optional meta data defined by the standard. The crawler extracts a list of URLs from each sitemap and ignores all other information.

Ensure each URL within your sitemap matches the exact domain (here defined as scheme + host + port) for your site. Different subdomains (like www.example.com and blog.example.com) and different schemes (like http://example.com and https://example.com) require separate sitemaps.

The Enterprise Search web crawler also supports sitemap index files. Refer to Using sitemap index files within the sitemap standard for sitemap index file details and examples.

Crawl rule

edit

A crawl rule is a crawler instruction to allow or disallow specific paths within a domain. Each crawl rule belongs to a domain, and each domain has one or more crawl rules.

See Manage crawl rules to manage crawl rules for a domain.

During content discovery, the web crawler discovers new URLs and must determine which it is allowed to follow. Each URL has a domain (e.g. https://example.com) and a path (e.g. /category/clothing or /c/Credit_Center).

The web crawler looks up the crawl rules for the domain, and applies the path to the crawl rules to determine if the path is allowed or disallowed. The crawler evaluates the crawl rules in order. The first matching crawl rule determines the policy for the newly discovered URL.

Each crawl rule has a path pattern, a rule, and a policy. To evaluate each rule, the web crawler compares a newly discovered path to the path pattern, using the logic represented by the rule, resulting in a policy.

The policy for each URL is also affected by directives in robots.txt files. The web crawler will crawl only those URLs that are allowed by crawl rules and robots.txt directives.

Crawl rule logic (rules)

edit

The logic for each rule is as follows:

Begins with

The path pattern is a literal string (no metacharacters).

The rule matches when the path pattern matches the beginning of the path (which always begins with /).

If using this rule, begin your path pattern with /.

Ends with

The path pattern is a literal string (no metacharacters).

The rule matches when the path pattern matches the end of the path.

Contains

The path pattern is a literal string (no metacharacters).

The rule matches when the path pattern matches anywhere within the path.

Regex

The path pattern is a regular expression compatible with the Ruby language regular expression engine. In addition to literal characters, the path pattern may include metacharacters, character classes, and repetitions. You can test Ruby regular expressions using Rubular.

The rule matches when the path pattern matches the beginning of the path (which always begins with /).

If using this rule, begin your path pattern with \/ or a metacharacter or character class that matches /.

Crawl rule matching

edit

The following table provides examples of crawl rule matching:

URL path	Rule	Path pattern	Match?
`/foo/bar`	Begins with	`/foo`	YES
`/bar/foo`	Begins with	`/foo`	NO
`/foo/bar`	Begins with	`foo`	NO
`/blog/posts/hello-world`	Ends	`world`	YES
`/blog/world-hello`	Ends	`world`	NO
`/fruits/bananas`	Contains	`banana`	YES
`/fruits/apples`	Contains	`banana`	NO
`/2020`	Regex	`\/[0-9]{3,5}`	YES
`/20`	Regex	`\/[0-9]{3,5}`	NO
`/2020`	Regex	`[0-9]{3,5}`	NO

Crawl rule order

edit

The first crawl rule to match determines the policy for the URL. Therefore, the order of the crawl rules is significant.

The following table demonstrates how crawl rule order affects the resulting policy for a path:

Path Crawl rules Resulting policy

Path	Crawl rules	Resulting policy
`/blog/2021/foo-bar-baz`	1. `Disallow` if `Begins with` `/blog` 2. `Allow` if `Regex` `.*`	DISALLOW
`/blog/2021/foo-bar-baz`	1. `Allow` if `Regex` `.*` 2. `Disallow` if `Begins with` `/blog`	ALLOW

/blog/2021/foo-bar-baz

1. Disallow if Begins with /blog

2. Allow if Regex .*

DISALLOW

/blog/2021/foo-bar-baz

1. Allow if Regex .*

2. Disallow if Begins with /blog

ALLOW

Restricting paths using crawl rules

edit

The domain dashboard adds a default crawl rule to each domain: Allow if Regex .*. You cannot delete or re-order this rule through the dashboard.

This rule is permissive, allowing all paths within the domain. To restrict paths, use either of the following techniques:

Add rules that disallow specific paths (e.g. disallow the blog):

Policy	Rule	Path pattern
`Disallow`	`Begins with`	`/blog`
`Allow`	`Regex`	`.*`

Or, add rules that allow specific paths and disallow all others (e.g. allow only the blog):

Policy	Rule	Path pattern
`Allow`	`Begins with`	`/blog`
`Disallow`	`Regex`	`.*`
`Allow`	`Regex`	`.*`

When you restrict a crawl to specific paths, be sure to add entry points that allow the crawler to discover those paths. For example, if your crawl rules restrict the crawler to /blog, add /blog as an entry point.

If you leave only the default entry point of /, the crawl will end immediately, since / is disallowed.

Robots.txt file

edit

A robots.txt file is a plain text file, associated with a domain, that provides instructions to web crawlers. The instructions within the file, also called directives, communicate which paths within that domain are disallowed (and allowed) for crawling.

The directives within a robots.txt file are similar to crawl rules. You can therefore choose to manage inclusion and exclusion through robots.txt files, crawl rules, or a combination of both. The Enterprise Search web crawler will crawl only those paths that are allowed by the crawl rules for the domain and the directives within the robots.txt file for the domain.

You may prefer using a robots.txt file over crawl rules for any of the following reasons:

You have already been publishing a robots.txt file for other web crawlers.
You don’t have access to the web crawler UI.
You prefer the robots.txt file interface over the web crawler web interface.

You can also use a robots.txt file to specify sitemaps for a domain.

Robots.txt file discovery and management

edit

At the start of each crawl, the web crawler fetches and processes the robots.txt file for each domain. When fetching a robots.txt file, the crawler handles 2xx, 4xx, and 5xx responses differenty. See Robots.txt file response code handling for an explanation of each response.

See Manage robots.txt files to manage robots.txt files for your domains.

Robots.txt file format and technical specification

edit

See Robots exclusion standard on Wikipedia for a summary of robots.txt conventions.

Canonical URL link tag

edit

A canonical URL link tag is an HTML element you can embed within pages that contain duplicate content. The canonical URL link tag indicates the URL of the canonical page with the same content.

Template:

<link rel="canonical" href="{CANONICAL_URL}">

Example:

<link rel="canonical" href="https://example.com/categories/dresses/starlet-red-medium">

Robots meta tags

edit

Robots meta tags are HTML elements you can embed within pages to prevent the crawler from following links or indexing content.

Template:

<meta name="robots" content="{DIRECTIVES}">

Supported directives:

noindex

The web crawler will not index the page.

nofollow

The web crawler will not follow links from the page (i.e. will not add links to the crawl queue). The web crawler logs a url_discover_denied event for each link.

The directive does not prevent the web crawler from indexing the page.

Examples:

<meta name="robots" content="noindex">
<meta name="robots" content="nofollow">
<meta name="robots" content="noindex, nofollow">

Nofollow link

edit

Nofollow links are HTML links that instruct the crawler to not follow the URL.

The web crawler will not follow links that include rel="nofollow" (i.e. will not add links to the crawl queue). The web crawler logs a url_discover_denied event for each link.

The link does not prevent the web crawler from indexing the page in which it appears.

Template:

<a rel="nofollow" href="{LINK_URL}">{LINK_TEXT}</a>

Example:

<a rel="nofollow" href="/admin/categories">Edit this category</a>

Crawl status

edit

Each crawl has a status, which quickly communicates its state.

See View crawl status to view the status for a crawl.

All crawl statuses:

Pending: The crawl is enqueued and will start after resources are available.
Starting: The crawl is starting. You may see this status briefly, while a crawl moves from pending to running.
Running: The crawl is running.
Success: The crawl completed without error. The web crawler may stop a crawl due to hitting resource limits. Such a crawl will report its status as success, as long as it completes without error.
Canceling: The crawl is canceling. You may see this status briefly, after choosing to cancel a crawl.
Canceled: The crawl was intentionally canceled and therefore did not complete.
Failed: The crawl ended unexpectedly. View the web crawler events logs for a message providing an explanation.

HTTP response status codes handling

edit

The following sections describe how the web crawler handles HTTP response status codes for HTML documents and robots.txt files.

HTML response code handling

edit

Code	Description	Web crawler behavior
`2xx`	Success	The web crawler extracts and de-duplicates the page’s content into a search document. Then it indexes the document into the engine, replacing an existing document with the same content if present.
`3xx`	Redirection	The web crawler follows all redirects recursively until it receives a `2xx`, `4xx`, or `5xx` response status code. The crawler then handles that code as indicated in this table.
`4xx`	Permanent error	The web crawler assumes this error is permanent. It therefore does not index the document. Furthermore, if the document is present in the engine, the web crawler deletes the document.
`5xx`	Temporary error	The web crawler optimistically assumes this error will resolve in the future. If the engine already contains a document representing the page, the document is updated to indicate the page was inaccessible. After the page is inaccessible for three separate crawls, the document is deleted from the engine.

Robots.txt file response code handling

edit

Code	Description	Web crawler behavior
`2xx`	Success	The web crawler processes and honors the directives for the domain within the robots.txt file.
`4xx`	Permanent error	The web crawler assumes this error is permanent and no specific rules exist for the domain. It therefore allows all paths for the domain (subject to crawl rules).
`5xx`	Temporary error	The web crawler optimistically assumes this error will resolve in the future. However, in the interim, the crawler does not know which paths are allowed and disallowed. It therefore disallows all paths for the domain.

Web crawler schema

edit

The web crawler indexes search documents using the following schema. All fields are strings or arrays of strings.

additional_urls: The URLs of additional pages with the same content.
body_content: The content of the page’s <body> tag with all HTML tags removed. Truncated to crawler.extraction.body_size.limit.
content_hash: A "fingerprint" to uniquely identify this content. Used for de-duplication and re-crawling.
domains: The domains in which this content appears.
headings: The text of the page’s HTML headings (h1 - h6 elements). Limited by crawler.extraction.headings_count.limit.
id: The unique identifier for the page.
links: Links found on the page. Limited by crawler.extraction.indexed_links_count.limit.
meta_description: The page’s description, taken from the <meta name="description"> tag. Truncated to crawler.extraction.description_size.limit.
meta_keywords: The page’s keywords, taken from the <meta name="keywords"> tag. Truncated to crawler.extraction.keywords_size.limit.
title: The title of the page, taken from the <title> tag. Truncated to crawler.extraction.title_size.limit.
url: The URL of the page.
url_host: The hostname or IP from the page’s URL.
url_path: The full pathname from the page’s URL.
url_path_dir1: The first segment of the pathname from the page’s URL.
url_path_dir2: The second segment of the pathname from the page’s URL.
url_path_dir3: The third segment of the pathname from the page’s URL.
url_port: The port number from the page’s URL (as a string).
url_scheme: The scheme of the page’s URL.

Web crawler events logs

edit

The web crawler logs many events while discovering, extracting, and indexing web content.

To view these events, see View web crawler events logs.

For a complete reference of all events, see Web crawler (beta) events logs reference.

Web crawler configuration settings

edit

Operators can configure several web crawler settings.

See Configuration in the Enterprise Search documentation for a complete list of Enterprise Search configuration settings.

Settings beginning with crawler. are specific to the web crawler.

These settings include:

crawler.crawl.max_crawl_depth.limit
crawler.crawl.max_duration.limit
crawler.crawl.max_unique_url_count.limit
crawler.crawl.max_url_length.limit
crawler.crawl.max_url_params.limit
crawler.crawl.max_url_segments.limit
crawler.crawl.threads.limit
crawler.http.connection_timeout
crawler.http.read_timeout
crawler.http.redirects.limit
crawler.http.request_timeout
crawler.http.response_size.limit
crawler.http.user_agent
crawler.extraction.body_size.limit
crawler.extraction.description_size.limit
crawler.extraction.extracted_links_count.limit
crawler.extraction.headings_count.limit
crawler.extraction.indexed_links_count.limit
crawler.extraction.keywords_size.limit
crawler.extraction.title_size.limit
crawler.workers.pool_size.limit

Refer to the Enterprise Search documentation for an up-to-date list of all configuration settings.

« Web crawler (beta) FAQ Web crawler (beta) events logs reference »