- Elasticsearch Guide: other versions:
- What is Elasticsearch?
- What’s new in 7.7
- Getting started with Elasticsearch
- Set up Elasticsearch
- Installing Elasticsearch
- Configuring Elasticsearch
- Setting JVM options
- Secure settings
- Auditing settings
- Circuit breaker settings
- Cluster-level shard allocation and routing settings
- Cross-cluster replication settings
- Discovery and cluster formation settings
- Field data cache settings
- HTTP
- Index lifecycle management settings
- Index recovery settings
- Indexing buffer settings
- License settings
- Local gateway settings
- Logging configuration
- Machine learning settings
- Monitoring settings
- Node
- Network settings
- Node query cache settings
- Search settings
- Security settings
- Shard request cache settings
- Snapshot lifecycle management settings
- SQL access settings
- Transforms settings
- Transport
- Thread pools
- Watcher settings
- Important Elasticsearch configuration
- Important System Configuration
- Bootstrap Checks
- Heap size check
- File descriptor check
- Memory lock check
- Maximum number of threads check
- Max file size check
- Maximum size virtual memory check
- Maximum map count check
- Client JVM check
- Use serial collector check
- System call filter check
- OnError and OnOutOfMemoryError checks
- Early-access check
- G1GC check
- All permission check
- Discovery configuration check
- Bootstrap Checks for X-Pack
- Starting Elasticsearch
- Stopping Elasticsearch
- Discovery and cluster formation
- Add and remove nodes in your cluster
- Full-cluster restart and rolling restart
- Remote clusters
- Set up X-Pack
- Configuring X-Pack Java Clients
- Plugins
- Upgrade Elasticsearch
- Search your data
- Query DSL
- SQL access
- Overview
- Getting Started with SQL
- Conventions and Terminology
- Security
- SQL REST API
- SQL Translate API
- SQL CLI
- SQL JDBC
- SQL ODBC
- SQL Client Applications
- SQL Language
- Functions and Operators
- Comparison Operators
- Logical Operators
- Math Operators
- Cast Operators
- LIKE and RLIKE Operators
- Aggregate Functions
- Grouping Functions
- Date/Time and Interval Functions and Operators
- Full-Text Search Functions
- Mathematical Functions
- String Functions
- Type Conversion Functions
- Geo Functions
- Conditional Functions And Expressions
- System Functions
- Reserved keywords
- SQL Limitations
- Aggregations
- Metrics Aggregations
- Avg Aggregation
- Weighted Avg Aggregation
- Boxplot Aggregation
- Cardinality Aggregation
- Stats Aggregation
- Extended Stats Aggregation
- Geo Bounds Aggregation
- Geo Centroid Aggregation
- Max Aggregation
- Min Aggregation
- Median Absolute Deviation Aggregation
- Percentiles Aggregation
- Percentile Ranks Aggregation
- Scripted Metric Aggregation
- String Stats Aggregation
- Sum Aggregation
- Top Hits Aggregation
- Top Metrics Aggregation
- Value Count Aggregation
- Bucket Aggregations
- Adjacency Matrix Aggregation
- Auto-interval Date Histogram Aggregation
- Children Aggregation
- Composite aggregation
- Date histogram aggregation
- Date Range Aggregation
- Diversified Sampler Aggregation
- Filter Aggregation
- Filters Aggregation
- Geo Distance Aggregation
- GeoHash grid Aggregation
- GeoTile Grid Aggregation
- Global Aggregation
- Histogram Aggregation
- IP Range Aggregation
- Missing Aggregation
- Nested Aggregation
- Parent Aggregation
- Range Aggregation
- Rare Terms Aggregation
- Reverse nested Aggregation
- Sampler Aggregation
- Significant Terms Aggregation
- Significant Text Aggregation
- Terms Aggregation
- Subtleties of bucketing range fields
- Pipeline Aggregations
- Bucket Script Aggregation
- Bucket Selector Aggregation
- Bucket Sort Aggregation
- Avg Bucket Aggregation
- Max Bucket Aggregation
- Min Bucket Aggregation
- Sum Bucket Aggregation
- Cumulative Cardinality Aggregation
- Cumulative Sum Aggregation
- Derivative Aggregation
- Percentiles Bucket Aggregation
- Moving Average Aggregation
- Moving Function Aggregation
- Serial Differencing Aggregation
- Stats Bucket Aggregation
- Extended Stats Bucket Aggregation
- Matrix Aggregations
- Caching heavy aggregations
- Returning only aggregation results
- Aggregation Metadata
- Returning the type of the aggregation
- Indexing aggregation results with transforms
- Metrics Aggregations
- Scripting
- Mapping
- Text analysis
- Overview
- Concepts
- Configure text analysis
- Built-in analyzer reference
- Tokenizer reference
- Char Group Tokenizer
- Classic Tokenizer
- Edge n-gram tokenizer
- Keyword Tokenizer
- Letter Tokenizer
- Lowercase Tokenizer
- N-gram tokenizer
- Path Hierarchy Tokenizer
- Path Hierarchy Tokenizer Examples
- Pattern Tokenizer
- Simple Pattern Tokenizer
- Simple Pattern Split Tokenizer
- Standard Tokenizer
- Thai Tokenizer
- UAX URL Email Tokenizer
- Whitespace Tokenizer
- Token filter reference
- Apostrophe
- ASCII folding
- CJK bigram
- CJK width
- Classic
- Common grams
- Conditional
- Decimal digit
- Delimited payload
- Dictionary decompounder
- Edge n-gram
- Elision
- Fingerprint
- Flatten graph
- Hunspell
- Hyphenation decompounder
- Keep types
- Keep words
- Keyword marker
- Keyword repeat
- KStem
- Length
- Limit token count
- Lowercase
- MinHash
- Multiplexer
- N-gram
- Normalization
- Pattern capture
- Pattern replace
- Phonetic
- Porter stem
- Predicate script
- Remove duplicates
- Reverse
- Shingle
- Snowball
- Stemmer
- Stemmer override
- Stop
- Synonym
- Synonym graph
- Trim
- Truncate
- Unique
- Uppercase
- Word delimiter
- Word delimiter graph
- Character filters reference
- Normalizers
- Index modules
- Ingest node
- Pipeline Definition
- Accessing Data in Pipelines
- Conditional Execution in Pipelines
- Handling Failures in Pipelines
- Enrich your data
- Processors
- Append Processor
- Bytes Processor
- Circle Processor
- Convert Processor
- CSV Processor
- Date Processor
- Date Index Name Processor
- Dissect Processor
- Dot Expander Processor
- Drop Processor
- Enrich Processor
- Fail Processor
- Foreach Processor
- GeoIP Processor
- Grok Processor
- Gsub Processor
- HTML Strip Processor
- Inference Processor
- Join Processor
- JSON Processor
- KV Processor
- Lowercase Processor
- Pipeline Processor
- Remove Processor
- Rename Processor
- Script Processor
- Set Processor
- Set Security User Processor
- Split Processor
- Sort Processor
- Trim Processor
- Uppercase Processor
- URL Decode Processor
- User Agent processor
- ILM: Manage the index lifecycle
- Monitor a cluster
- Frozen indices
- Roll up or transform your data
- Set up a cluster for high availability
- Snapshot and restore
- Secure a cluster
- Overview
- Configuring security
- User authentication
- Built-in users
- Internal users
- Token-based authentication services
- Realms
- Realm chains
- Active Directory user authentication
- File-based user authentication
- LDAP user authentication
- Native user authentication
- OpenID Connect authentication
- PKI user authentication
- SAML authentication
- Kerberos authentication
- Integrating with other authentication systems
- Enabling anonymous access
- Controlling the user cache
- Configuring SAML single-sign-on on the Elastic Stack
- Configuring single sign-on to the Elastic Stack using OpenID Connect
- User authorization
- Built-in roles
- Defining roles
- Security privileges
- Document level security
- Field level security
- Granting privileges for indices and aliases
- Mapping users and groups to roles
- Setting up field and document level security
- Submitting requests on behalf of other users
- Configuring authorization delegation
- Customizing roles and authorization
- Enabling audit logging
- Encrypting communications
- Restricting connections with IP filtering
- Cross cluster search, clients, and integrations
- Tutorial: Getting started with security
- Tutorial: Encrypting communications
- Troubleshooting
- Some settings are not returned via the nodes settings API
- Authorization exceptions
- Users command fails due to extra arguments
- Users are frequently locked out of Active Directory
- Certificate verification fails for curl on Mac
- SSLHandshakeException causes connections to fail
- Common SSL/TLS exceptions
- Common Kerberos exceptions
- Common SAML issues
- Internal Server Error in Kibana
- Setup-passwords command fails due to connection failure
- Failures due to relocation of the configuration files
- Limitations
- Alerting on cluster and index events
- Command line tools
- How To
- Glossary of terms
- REST APIs
- API conventions
- cat APIs
- cat aliases
- cat allocation
- cat anomaly detectors
- cat count
- cat data frame analytics
- cat datafeeds
- cat fielddata
- cat health
- cat indices
- cat master
- cat nodeattrs
- cat nodes
- cat pending tasks
- cat plugins
- cat recovery
- cat repositories
- cat shards
- cat segments
- cat snapshots
- cat task management
- cat templates
- cat thread pool
- cat trained model
- cat transforms
- Cluster APIs
- Cluster allocation explain
- Cluster get settings
- Cluster health
- Cluster reroute
- Cluster state
- Cluster stats
- Cluster update settings
- Nodes feature usage
- Nodes hot threads
- Nodes info
- Nodes reload secure settings
- Nodes stats
- Pending cluster tasks
- Remote cluster info
- Task management
- Voting configuration exclusions
- Cross-cluster replication APIs
- Document APIs
- Enrich APIs
- Explore API
- Index APIs
- Add index alias
- Analyze
- Clear cache
- Clone index
- Close index
- Create index
- Delete index
- Delete index alias
- Delete index template
- Flush
- Force merge
- Freeze index
- Get field mapping
- Get index
- Get index alias
- Get index settings
- Get index template
- Get mapping
- Index alias exists
- Index exists
- Index recovery
- Index segments
- Index shard stores
- Index stats
- Index template exists
- Open index
- Put index template
- Put mapping
- Refresh
- Rollover index
- Shrink index
- Split index
- Synced flush
- Type exists
- Unfreeze index
- Update index alias
- Update index settings
- Index lifecycle management API
- Ingest APIs
- Info API
- Licensing APIs
- Machine learning anomaly detection APIs
- Add events to calendar
- Add jobs to calendar
- Close jobs
- Create jobs
- Create calendar
- Create datafeeds
- Create filter
- Delete calendar
- Delete datafeeds
- Delete events from calendar
- Delete filter
- Delete forecast
- Delete jobs
- Delete jobs from calendar
- Delete model snapshots
- Delete expired data
- Estimate model memory
- Find file structure
- Flush jobs
- Forecast jobs
- Get buckets
- Get calendars
- Get categories
- Get datafeeds
- Get datafeed statistics
- Get influencers
- Get jobs
- Get job statistics
- Get machine learning info
- Get model snapshots
- Get overall buckets
- Get scheduled events
- Get filters
- Get records
- Open jobs
- Post data to jobs
- Preview datafeeds
- Revert model snapshots
- Set upgrade mode
- Start datafeeds
- Stop datafeeds
- Update datafeeds
- Update filter
- Update jobs
- Update model snapshots
- Machine learning data frame analytics APIs
- Create data frame analytics jobs
- Create inference trained model
- Delete data frame analytics jobs
- Delete inference trained model
- Evaluate data frame analytics
- Explain data frame analytics API
- Get data frame analytics jobs
- Get data frame analytics jobs stats
- Get inference trained model
- Get inference trained model stats
- Start data frame analytics jobs
- Stop data frame analytics jobs
- Migration APIs
- Reload search analyzers
- Rollup APIs
- Search APIs
- Security APIs
- Authenticate
- Change passwords
- Clear cache
- Clear roles cache
- Create API keys
- Create or update application privileges
- Create or update role mappings
- Create or update roles
- Create or update users
- Delegate PKI authentication
- Delete application privileges
- Delete role mappings
- Delete roles
- Delete users
- Disable users
- Enable users
- Get API key information
- Get application privileges
- Get builtin privileges
- Get role mappings
- Get roles
- Get token
- Get users
- Has privileges
- Invalidate API key
- Invalidate token
- OpenID Connect Prepare Authentication API
- OpenID Connect authenticate API
- OpenID Connect logout API
- SAML prepare authentication API
- SAML authenticate API
- SAML logout API
- SAML invalidate API
- SSL certificate
- Snapshot and restore APIs
- Snapshot lifecycle management API
- Transform APIs
- Usage API
- Watcher APIs
- Definitions
- Breaking changes
- Release notes
- Elasticsearch version 7.7.1
- Elasticsearch version 7.7.0
- Elasticsearch version 7.6.2
- Elasticsearch version 7.6.1
- Elasticsearch version 7.6.0
- Elasticsearch version 7.5.2
- Elasticsearch version 7.5.1
- Elasticsearch version 7.5.0
- Elasticsearch version 7.4.2
- Elasticsearch version 7.4.1
- Elasticsearch version 7.4.0
- Elasticsearch version 7.3.2
- Elasticsearch version 7.3.1
- Elasticsearch version 7.3.0
- Elasticsearch version 7.2.1
- Elasticsearch version 7.2.0
- Elasticsearch version 7.1.1
- Elasticsearch version 7.1.0
- Elasticsearch version 7.0.0
- Elasticsearch version 7.0.0-rc2
- Elasticsearch version 7.0.0-rc1
- Elasticsearch version 7.0.0-beta1
- Elasticsearch version 7.0.0-alpha2
- Elasticsearch version 7.0.0-alpha1
Create inference trained model API
editCreate inference trained model API
editCreates an inference trained model.
This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
Request
editPUT _ml/inference/<model_id>
Prerequisites
editIf the Elasticsearch security features are enabled, you must have the following built-in roles and privileges:
-
machine_learning_admin
For more information, see Security privileges and Built-in roles.
Description
editThe create inference trained model API enables you to supply a trained model that is not created by data frame analytics.
Path parameters
edit-
<model_id>
- (Required, string) The unique identifier of the trained inference model.
Request body
edit-
compressed_definition
-
(Required, string)
The compressed (GZipped and Base64 encoded) inference definition of the model.
If
compressed_definition
is specified, thendefinition
cannot be specified.
-
definition
-
(Required, object)
The inference definition for the model. If
definition
is specified, thencompressed_definition
cannot be specified.
Properties of definition
-
preprocessors
- (Optional, object) Collection of preprocessors.
Properties of preprocessors
-
frequency_encoding
- (Required, object) Defines a frequency encoding for a field.
Properties of frequency_encoding
-
feature_name
- (Required, string) The name of the resulting feature.
-
field
- (Required, string) The field name to encode.
-
frequency_map
- (Required, object map of string:double) Object that maps the field value to the frequency encoded value.
-
one_hot_encoding
- (Required, object) Defines a one hot encoding map for a field.
Properties of one_hot_encoding
-
field
- (Required, string) The field name to encode.
-
hot_map
- (Required, object map of strings) String map of "field_value: one_hot_column_name".
-
target_mean_encoding
- (Required, object) Defines a target mean encoding for a field.
Properties of target_mean_encoding
-
default_value
-
(Required, double)
The feature value if the field value is not in the
target_map
. -
feature_name
- (Required, string) The name of the resulting feature.
-
field
- (Required, string) The field name to encode.
-
target_map
- (Required, object map of string:double) Object that maps the field value to the target mean value.
See Preprocessor examples for more details.
-
trained_model
- (Required, object) The definition of the trained model.
Properties of trained_model
-
tree
- (Required, object) The definition for a binary decision tree.
Properties of tree
-
classification_labels
-
(Optional, string) An array of classification labels (used for
classification
). -
feature_names
- (Required, string) Features expected by the tree, in their expected order.
-
target_type
-
(Required, string)
String indicating the model target type;
regression
orclassification
. -
tree_structure
-
(Required, object)
An array of
tree_node
objects. The nodes must be in ordinal order by theirtree_node.node_index
value.
-
tree_node
- (Required, object) The definition of a node in a tree.
There are two major types of nodes: leaf nodes and not-leaf nodes.
-
Leaf nodes only need
node_index
andleaf_value
defined. -
All other nodes need
split_feature
,left_child
,right_child
,threshold
,decision_type
, anddefault_left
defined.
Properties of tree_node
-
decision_type
-
(Optional, string)
Indicates the positive value (in other words, when to choose the left node)
decision type. Supported
lt
,lte
,gt
,gte
. Defaults tolte
. -
default_left
-
(Optional, boolean)
Indicates whether to default to the left when the feature is missing. Defaults
to
true
. -
leaf_value
- (Optional, double) The leaf value of the of the node, if the value is a leaf (in other words, no children).
-
left_child
- (Optional, integer) The index of the left child.
-
node_index
- (Integer) The index of the current node.
-
right_child
- (Optional, integer) The index of the right child.
-
split_feature
- (Optional, integer) The index of the feature value in the feature array.
-
split_gain
- (Optional, double) The information gain from the split.
-
threshold
- (Optional, double) The decision threshold with which to compare the feature value.
-
ensemble
- (Optional, object) The definition for an ensemble model.
Properties of ensemble
-
aggregate_output
-
(Required, object)
An aggregated output object that defines how to aggregate the outputs of the
trained_models
. Supported objects areweighted_mode
,weighted_sum
, andlogistic_regression
.
Properties of aggregate_output
-
logistic_regression
-
(Optional, object)
This
aggregated_output
type works with binary classification (classification for values [0, 1]). It multiplies the outputs (in the case of theensemble
model, the inference model values) by the suppliedweights
. The resulting vector is summed and passed to asigmoid
function. The result of thesigmoid
function is considered the probability of class 1 (P_1
), consequently, the probability of class 0 is1 - P_1
. The class with the highest probability (either 0 or 1) is then returned. For more information about logistic regression, see this wiki article.
Properties of logistic_regression
-
weights
- (Required, double) The weights to multiply by the input values (the inference values of the trained models).
-
weighted_sum
-
(Optional, object)
This
aggregated_output
type works with regression. The weighted sum of the input values.
Properties of weighted_sum
-
weights
- (Required, double) The weights to multiply by the input values (the inference values of the trained models).
-
weighted_mode
-
(Optional, object)
This
aggregated_output
type works with regression or classification. It takes a weighted vote of the input values. The most common input value (taking the weights into account) is returned.
Properties of weighted_mode
-
weights
- (Required, double) The weights to multiply by the input values (the inference values of the trained models).
See Aggregated output example for more details.
-
classification_labels
- (Optional, string) An array of classification labels.
-
feature_names
- (Optional, string) Features expected by the ensemble, in their expected order.
-
target_type
-
(Required, string)
String indicating the model target type;
regression
orclassification.
-
trained_models
-
(Required, object)
An array of
trained_model
objects. Supported trained models aretree
andensemble
.
See Model examples for more details.
-
description
- (Optional, string) A human-readable description of the inference trained model.
-
input
-
(Required, object) The input field names for the model definition.
-
input
.field_names
- (Required, string) An array of input field names for the model.
-
-
metadata
- (Optional, object) An object map that contains metadata about the model.
-
tags
- (Optional, string) An array of tags to organize the model.
Examples
editPreprocessor examples
editThe example below shows a frequency_encoding
preprocessor object:
{ "frequency_encoding":{ "field":"FlightDelayType", "feature_name":"FlightDelayType_frequency", "frequency_map":{ "Carrier Delay":0.6007414737092798, "NAS Delay":0.6007414737092798, "Weather Delay":0.024573576178086153, "Security Delay":0.02476631010889467, "No Delay":0.6007414737092798, "Late Aircraft Delay":0.6007414737092798 } } }
The next example shows a one_hot_encoding
preprocessor object:
{ "one_hot_encoding":{ "field":"FlightDelayType", "hot_map":{ "Carrier Delay":"FlightDelayType_Carrier Delay", "NAS Delay":"FlightDelayType_NAS Delay", "No Delay":"FlightDelayType_No Delay", "Late Aircraft Delay":"FlightDelayType_Late Aircraft Delay" } } }
This example shows a target_mean_encoding
preprocessor object:
{ "target_mean_encoding":{ "field":"FlightDelayType", "feature_name":"FlightDelayType_targetmean", "target_map":{ "Carrier Delay":39.97465788139886, "NAS Delay":39.97465788139886, "Security Delay":203.171206225681, "Weather Delay":187.64705882352948, "No Delay":39.97465788139886, "Late Aircraft Delay":39.97465788139886 }, "default_value":158.17995752420433 } }
Model examples
editThe first example shows a trained_model
object:
{ "tree":{ "feature_names":[ "DistanceKilometers", "FlightTimeMin", "FlightDelayType_NAS Delay", "Origin_targetmean", "DestRegion_targetmean", "DestCityName_targetmean", "OriginAirportID_targetmean", "OriginCityName_frequency", "DistanceMiles", "FlightDelayType_Late Aircraft Delay" ], "tree_structure":[ { "decision_type":"lt", "threshold":9069.33437193022, "split_feature":0, "split_gain":4112.094574306927, "node_index":0, "default_left":true, "left_child":1, "right_child":2 }, ... { "node_index":9, "leaf_value":-27.68987349695448 }, ... ], "target_type":"regression" } }
The following example shows an ensemble
model object:
"ensemble":{ "feature_names":[ ... ], "trained_models":[ { "tree":{ "feature_names":[], "tree_structure":[ { "decision_type":"lte", "node_index":0, "leaf_value":47.64069875778043, "default_left":false } ], "target_type":"regression" } }, ... ], "aggregate_output":{ "weighted_sum":{ "weights":[ ... ] } }, "target_type":"regression" }
Aggregated output example
editExample of a logistic_regression
object:
"aggregate_output" : { "logistic_regression" : { "weights" : [2.0, 1.0, .5, -1.0, 5.0, 1.0, 1.0] } }
Example of a weighted_sum
object:
"aggregate_output" : { "weighted_sum" : { "weights" : [1.0, -1.0, .5, 1.0, 5.0] } }
Example of a weighted_mode
object:
"aggregate_output" : { "weighted_mode" : { "weights" : [1.0, 1.0, 1.0, 1.0, 1.0] } }
Inference JSON schema
editFor the full JSON schema of model inference, click here.
On this page