New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.
« Create inference API Update inference API »
Elastic Docs ›Elasticsearch Guide [8.x] ›REST APIs ›Inference APIs

Stream inference API

edit
IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

Stream inference API

edit

New API reference

For the most up-to-date API details, refer to Inference APIs.

Streams a chat completion response.

The inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the Machine learning trained model APIs.

Request

edit

POST /_inference/<inference_id>/_stream

POST /_inference/<task_type>/<inference_id>/_stream

Prerequisites

edit
  • Requires the monitor_inference cluster privilege (the built-in inference_admin and inference_user roles grant this privilege)
  • You must use a client that supports streaming.

Description

edit

The stream inference API enables real-time responses for completion tasks by delivering answers incrementally, reducing response times during computation. It only works with the completion and chat_completion task types.

The Chat completion inference API and the Stream inference API differ in their response structure and capabilities. The Chat completion inference API provides more comprehensive customization options through more fields and function calling support. If you use the openai service or the elastic service, use the Chat completion inference API.

For more information on how to use the chat_completion task type, please refer to the chat completion documentation.

Path parameters

edit
<inference_id>
(Required, string) The unique identifier of the inference endpoint.
<task_type>
(Optional, string) The type of inference task that the model performs.

Request body

edit
input

(Required, string or array of strings) The text on which you want to perform the inference task. input can be a single string or an array.

Inference endpoints for the completion task type currently only support a single string as input.

Examples

edit

The following example performs a completion on the example question with streaming.

resp = client.inference.stream_completion(
    inference_id="openai-completion",
    input="What is Elastic?",
)
print(resp)
const response = await client.inference.streamInference({
  task_type: "completion",
  inference_id: "openai-completion",
  input: "What is Elastic?",
});
console.log(response);
POST _inference/completion/openai-completion/_stream
{
  "input": "What is Elastic?"
}
Copy as curlTry in Elastic 

The API returns the following response:

event: message
data: {
  "completion":[{
    "delta":"Elastic"
  }]
}

event: message
data: {
  "completion":[{
    "delta":" is"
    },
    {
    "delta":" a"
    }
  ]
}

event: message
data: {
  "completion":[{
    "delta":" software"
  },
  {
    "delta":" company"
  }]
}

(...)
« Create inference API Update inference API »
Was this helpful?
Feedback