IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Get transform stats API Update transform API »

› › ›

Create transform API

edit

Create transform API

edit

Creates a new transform.

The API accepts a PutTransformRequest object as a request and returns a AcknowledgedResponse.

Create transform request

edit

A PutTransformRequest requires the following argument:

PutTransformRequest request =
        new PutTransformRequest(transformConfig); 
request.setDeferValidation(false);

	The configuration of the transform to create
	Whether or not to wait to run deferrable validations until `_start` is called. This option should be used with care as the created transform will run with the privileges of the user creating it. Meaning, if they do not have privileges, such an error will not be visible until `_start` is called.

Transform configuration

edit

The TransformConfig object contains all the details about the transform configuration and contains the following arguments:

TransformConfig transformConfig = TransformConfig
    .builder()
    .setId("reviewer-avg-rating") 
    .setSource(sourceConfig) 
    .setDest(destConfig) 
    .setFrequency(TimeValue.timeValueSeconds(15)) 
    .setPivotConfig(pivotConfig) 
    .setDescription("This is my test transform") 
    .setSettings(settings) 
    .setRetentionPolicyConfig(retentionPolicy) 
    .setSyncConfig(syncConfig) 
    .build();

	The transform ID
	The source indices and query from which to gather data
	The destination index and optional pipeline
	Optional, indicates how often to check for updates to the source indices
	The PivotConfig
	Optional free text description of the transform
	Optional transform settings
	Optional retention policy for the data in the destination index
	Details required only when the transform runs continuously

SourceConfig

edit

The indices and the query from which to collect data. If query is not set, a match_all query is used by default.

SourceConfig sourceConfig = SourceConfig.builder()
    .setIndex("source-index")
    .setQueryConfig(queryConfig).build();

DestConfig

edit

The index where to write the data and the optional pipeline through which the docs should be indexed

DestConfig destConfig = DestConfig.builder()
    .setIndex("pivot-destination")
    .setPipeline("my-pipeline").build();

QueryConfig

edit

The query with which to select data from the source.

QueryConfig queryConfig = new QueryConfig(new MatchAllQueryBuilder());

PivotConfig

edit

Defines the pivot function group by fields and the aggregation to reduce the data.

PivotConfig pivotConfig = PivotConfig.builder()
    .setGroups(groupConfig) 
    .setAggregationConfig(aggConfig) 
    .build();

	The `GroupConfig` to use in the pivot
	The aggregations to use

GroupConfig

edit

The grouping terms. Defines the group by and destination fields which are produced by the pivot function. There are 3 types of groups

Terms
Histogram
Date Histogram

GroupConfig groupConfig = GroupConfig.builder()
    .groupBy("reviewer", 
        TermsGroupSource.builder().setField("user_id").build()) 
    .build();

	The destination field
	Group by values of the `user_id` field

AggregationConfig

edit

Defines the aggregations for the group fields.

AggregatorFactories.Builder aggBuilder = new AggregatorFactories.Builder();
aggBuilder.addAggregator(
        AggregationBuilders.avg("avg_rating").field("stars"));  
AggregationConfig aggConfig = new AggregationConfig(aggBuilder);

Aggregate the average star rating

RetentionPolicyConfig

edit

Defines a retention policy for the transform. Data that meets the defined criteria is deleted from the destination index.

RetentionPolicyConfig retentionPolicy = TimeRetentionPolicyConfig.builder()
    .setField("time-field") 
    .setMaxAge(TimeValue.timeValueDays(30)) 
    .build();

	The date field that is used to calculate the age of the document.
	Specifies the maximum age of a document in the destination index.

SettingsConfig

edit

Defines settings.

SettingsConfig settings = SettingsConfig.builder()
    .setMaxPageSearchSize(1000) 
    .build();

The maximum paging size for the transform when pulling data from the source. The size dynamically adjusts as the transform is running to recover from and prevent OOM issues.

SyncConfig

edit

Defines the properties transforms require to run continuously.

SyncConfig syncConfig = TimeSyncConfig.builder()
    .setField("time-field") 
    .setDelay(TimeValue.timeValueSeconds(30)) 
    .build();

	The date field that is used to identify new documents in the source.
	The time delay between the current time and the latest input data time.

Synchronous execution

edit

When executing a PutTransformRequest in the following manner, the client waits for the AcknowledgedResponse to be returned before continuing with code execution:

AcknowledgedResponse response =
        client.transform().putTransform(
                request, RequestOptions.DEFAULT);

Synchronous calls may throw an IOException in case of either failing to parse the REST response in the high-level REST client, the request times out or similar cases where there is no response coming back from the server.

In cases where the server returns a 4xx or 5xx error code, the high-level client tries to parse the response body error details instead and then throws a generic ElasticsearchException and adds the original ResponseException as a suppressed exception to it.

Asynchronous execution

edit

Executing a PutTransformRequest can also be done in an asynchronous fashion so that the client can return directly. Users need to specify how the response or potential failures will be handled by passing the request and a listener to the asynchronous put-transform method:

client.transform().putTransformAsync(
        request, RequestOptions.DEFAULT, listener);

The PutTransformRequest to execute and the ActionListener to use when the execution completes

The asynchronous method does not block and returns immediately. Once it is completed the ActionListener is called back using the onResponse method if the execution successfully completed or using the onFailure method if it failed. Failure scenarios and expected exceptions are the same as in the synchronous execution case.

A typical listener for put-transform looks like:

ActionListener<AcknowledgedResponse> listener =
        new ActionListener<AcknowledgedResponse>() {
    @Override
    public void onResponse(AcknowledgedResponse response) {
        
    }

    @Override
    public void onFailure(Exception e) {
        
    }
};

	Called when the execution is successfully completed.
	Called when the whole `PutTransformRequest` fails.

Response

edit

The returned AcknowledgedResponse acknowledges the successful creation of the new transform or an error if the configuration is invalid.

« Get transform stats API Update transform API »