Evolution of Workplace Search: Search your private data with Elasticsearch

The standalone Workplace Search product became part of our offering when Swiftype joined forces with Elastic in 2017. Since then, we have added a number of content sources to popular databases, file systems, and other tools to Workplace Search. While users loved that Workplace Search included integrations for various data sources, there was clear feedback that developers wanted the transparency and flexibility to tune the search experience. These tools for tuning were only available when working directly with the Elasticsearch platform.

Internal knowledge search use cases (i.e., being able to search all your private content sources—databases, knowledge bases, file systems, collaboration, or ticketing tools) have increased in  importance to our users with the recent popularity of ChatGPT and generative AI technologies. There is no doubt that search retrieval for private data and LLMs are better together.

Customers that used the Workplace Search product loved the simplicity of ingesting data through content sources, using the indexed data in search experiences, and managing data and searches with built-in tools. These features are not going away but will evolve as part of our Elastic Search offering — some of this work began in early 8.x releases. Content sources have been replaced and elevated by connectors as a type of Elastic integration. These connectors write directly to search optimized Elasticsearch indices, instead of working with abstractions that use hidden indices. By doing so, the ability to utilize platform capabilities like integrating with third party transformer models to transform the indexed data is intuitive and easy to configure. As a result, setting up search experiences is more flexible than ever, and Elasticsearch native features provide multiple means of data and search management.

The most common architecture for internal knowledge search use cases involves bringing private organizational data into Elastic through native connectors and/or self-managed connector clients. Then, a self-managed search application exposes data that your end users are authorized to see through a search experience.

Synchronized content is indexed to regular search optimized Elasticsearch indices that can be used directly in your search experience. By default, connector specific logic and the bundled ent-search-generic-ingestion pipeline extract and transform your data, as configured in your deployment. For more advanced use cases, data can be filtered and transformed using tools like:

• Basic sync rules, which are identical for all connectors
• Advanced sync rules for remote filtering at the data source level
• Self-managed Extraction Service for connector clients
• Customized pipeline filtering, before persistence to Elasticsearch through ingest pipelines (Read about Ingest pipelines in Search for more details.)
• Ingest pipelines are especially powerful as they provide an extremely accessible integration point for ML models such as ELSER, for any private data indexed through connectors.

Once you have your data in Elasticsearch, you can build your own custom search experience with projects like Search UI and SearchKit for frontend components. Our internal knowledge search example app shows how you can use existing tools to get started.

The Javascript client library enables you to easily build or integrate search into your web applications, or use one of the many Elasticsearch client libraries.

We're rapidly adding features to make search more powerful. Building your internal search systems with Elasticsearch gives you access to these capabilities as soon as they're released.

• Elasticsearch supports vector, semantic, hybrid, and generative AI search use cases. Check out some of our semantic and vector search capabilities, including a powerful out-of-the box sparse-vector model. In 8.11 this model is generally available and within a few clicks can be applied to data written using the Elastic connectors or web crawler. 
• With a fully customizable Search Application API based on search templates, this new API abstracts away the learning curve of the Elasticsearch Query DSL, if that is required for your use case. You can dive into the details at any time with the Query DSL.
• New search management tools let you directly manage the results of specific queries, for example boost or bury results and work programmatically with the synonyms management API.

The standalone Workplace Search product will continue to be a supported experience, so while migrating to the new features is recommended, it is not a requirement. Any user looking to elevate their internal knowledge search to include semantic search, vector search or chat prompt based experiences, would benefit from transitioning to use Elastic connectors. There are many compelling advancements for the range of workplace search use cases in the enterprise, supported by Elastic Search. Now would be a great time to think about upgrading your internal knowledge search to make it an amazing experience for your users!