What is sentiment analysis?

Sentiment analysis is a natural language processing (NLP) technique that uses computational linguistics and machine learning to detect the emotional tone behind text data. This allows organizations to identify positive, neutral, or negative sentiment toward their brand, products, services, or ideas.

Core technologies include:

• Natural language processing (NLP): Allows machines to process and interpret human language
• Computational linguistics: Provides linguistic frameworks for text analysis
• Machine learning (ML): Models learn patterns from labeled text to classify sentiment

1. Text ingestion: Raw text data is gathered from a variety of sources, including emails, support tickets, chat logs, social media, and customer reviews. A processing pipeline “ingests” this data.
2. Text preprocessing: The unprocessed text is cleaned and normalized:
   1. Tokenization: The data (text) is split into words or phrases
   2. Lower-casing: The input is standardized
   3. Stop word removal: Common non-informative words are filtered
   4. Stemming/lemmatization: Breaks words down into their most basic forms
   5. Named entity recognition (NER): The process of recognizing proper nouns and entities
3. Feature extraction: Structured numerical representations are created from text:
   1. Bag of Words (BoW) or TF-IDF for sparse vector models
   2. Word embeddings (Word2Vec, GloVe, BERT, etc.) for semantic context
   3. Contextual vectors (e.g., from transformer-based models)
4. Sentiment classification: Text is categorized into sentiment categories using machine learning or deep learning models:
   1. Rule-based models (using sentiment lexicons and linguistic heuristics )
   2. Traditional ML models (Naïve Bayes, SVM, logistic regression)
   3. Neural models (LSTMs, CNNs, transformers)
5. Output and scoring: Every input text has a score along a sentiment continuum (e.g., –1 to +1) or is labeled as positive, negative, or neutral. Next, this sentiment metadata is:
   1. Indexed for filtering or search
   2. Combined for analytics and dashboards
   3. Used to trigger warnings (for instance, when negative sentiment spikes)
6. Feedback + model updates: The model can be fine-tuned or retrained using labeled outputs.

Sentiment analysis is a subcategory of natural language processing, meaning it is one of the many tasks that NLP performs. While sentiment analysis focuses on capturing emotion and opinion within text, NLP is the overarching technology that gives machines the ability to work with human language.

Language-related tasks powered by NLP include:

• NER: Identifying proper nouns such as people, organizations, or places in text
• Part-of-speech tagging: Labeling words with their grammatical roles (noun, verb, adjective, etc.)
• Text classification: Sorting text into categories (like spam vs. not spam)
• Language modeling: Predicting the next word in a sentence or understanding sentence structure
• Text summarization: Generating concise summaries of longer documents
• Machine translation: Converting text from one language to another
• Question answering: Building systems that respond to questions based on text input
• Natural language generation: Creating human-like text from structured data or prompts

Sentiment analysis is a focused use case within the broader discipline of machine learning, typically using supervised machine learning models trained on labeled text data to detect sentiment and opinion within text.

Machine learning, on the other hand, enables systems to learn patterns from data and make predictions or decisions without being explicitly programmed. Some key machine learning tasks include:

• Image classification: Identifying objects or people in images/photos
• Speech recognition: Converting spoken language into text
• Recommendation systems: Suggesting products, media, and more based on user behavior

Fundamentally, sentiment analysis techniques rely on ML techniques such as:

• Classification algorithms: For example, deep neural networks, decision trees, or logistic regression
• Feature extraction: Turning raw text into numeric vectors
• Model evaluation: Performance is assessed using metrics such as recall, precision, and accuracy

AI is a wide-ranging discipline aiming at creating systems able to perform tasks that would normally require human cognitive abilities. Sentiment analysis is a narrow application of AI, specifically within the domain of NLP.

NLP, computer vision, and machine learning are all subfields of AI.

Sentiment analysis builds on core NLP components such as tokenization, parsing, and vector representations of language. It’s often powered by pretrained transformer models (such as BERT or RoBERTa) that have been fine-tuned on datasets labeled for sentiment. In essence, while AI encompasses a range of behaviors, sentiment analysis focuses on AI and NLP methods to analyze emotional tone in textual data.

Data mining is a broad computational process that involves discovering patterns, correlations, and anomalies from large datasets.

Key differences between sentiment analysis and data mining include:

• Methodologies: Sentiment analysis incorporates NLP techniques with supervised or unsupervised machine learning models to interpret language nuances. On the other hand, data mining uses statistical, mathematical, and algorithmic methods that are optimized for pattern discovery across various data formats.
• Output: Sentiment analysis outputs include sentiment classifications or continuous sentiment scores. Data mining outputs include predictive models, clusters, and association rules.
• Data type focus: Data mining deals with diverse data types (e.g., numerical, categorical, and textual data). Sentiment analysis targets unstructured text for emotional insight extraction.

Sentiment analysis can be performed using different approaches: rule-based methods, machine learning models, or a hybrid combination. Each approach can be applied to different types of sentiment analysis tasks, including:

• Fine-grained sentiment analysis
• Aspect-based sentiment analysis (ABSA)
• Emotion detection sentiment analysis
• Intent-based sentiment analysis

Fine-grained sentiment analysis

Also known as graded sentiment analysis, this type refines sentiment into multiple levels rather than just positive, neutral, or negative. Typical categories include very positive, positive, neutral, negative, and very negative. This further granularity can be helpful in specific scenarios and/or industries, such as businesses looking to better understand customer satisfaction levels.

Aspect-based sentiment analysis (ABSA)

This approach focuses on identifying sentiment toward specific aspects or features of a product or service. Take, for example, reviews of wireless headphones. Different aspects could include connectivity, design, and sound quality. ABSA helps businesses pinpoint exactly which parts of their product customers like or dislike.

Emotion detection sentiment analysis

Emotion detection goes beyond polarity to identify specific feelings such as happiness, sadness, anger, or frustration. This type of analysis often uses lexicons to evaluate subjective language.

However, lexicon-based methods can struggle with context or subtle expressions of emotion.

Intent-based sentiment analysis

As the name suggests, intent-based analysis aims at reading the intention behind text. This may allow businesses to identify customer intent and interest levels, such as the intent to purchase, upgrade, cancel, or unsubscribe. Intent detection typically requires training classifiers on labeled data, like customer emails or support queries.

To perform sentiment analysis, you typically follow these steps:

1. Text preprocessing, including tokenizing sentences, lemmatizing to root form, and removing stop words
2. Feature extraction, which can include converting the lemmatized tokens to a numeric representation or generating embeddings
3. Classification, which involves applying a sentiment classifier to your data (This typically uses a specific model or algorithm that works with the extracted features to categorize the sentiment.)

There are also three common approaches to sentiment analysis

• Rule-based sentiment analysis
• Machine learning sentiment analysis
• Hybrid sentiment analysis

Rule-based sentiment analysis

Rule-based sentiment analysis relies on preset linguistic rules and sentiment lexicons to determine the emotional tone of text.

Components include:

• Sentiment lexicons: Dictionaries containing words tagged with sentiment values (positive, negative, neutral)
• Linguistic rules: Sets of handcrafted rules to handle modifiers, such as negations ("not good"), intensifiers ("very happy"), and conjunctions

Process:

1. Tokenization: Break text into tokens (words or phrases).
2. Lexicon lookup: Match tokens against the sentiment lexicon to assign polarity scores.
3. Rule application: Adjust scores using rules that account for context (e.g., negation flips polarity, intensifiers amplify sentiment).
4. Aggregation: Combine individual token scores into an overall sentiment score for the text.

While the benefit of this approach includes easily interpretable results and no need for a large labeled dataset, rule-based sentiment analysis can prove rigid and, at times, it can struggle with subtler nuances such as sarcasm, context, and evolving language use.

Machine learning sentiment analysis

Machine learning sentiment analysis uses algorithms that learn from labeled training data.

Components include:

• Training data: Labeled datasets (e.g., movie reviews, product reviews) used to teach the model which words or phrases correspond to positive, negative, or neutral sentiment
• Features: Numeric representations of text, such as word counts, TF-IDF vectors, or embeddings that capture semantic meaning
• Classification: Models like deep neural networks, naïve Bayes¹, logistic regression, or support vector machines that classify text based on extracted features

Process:

1. Data preprocessing: The text is cleaned and tokenized, stop words are removed, and the text is finally converted into feature vectors.
2. Model training: The features and corresponding sentiment labels are fed back into the ML algorithm to learn patterns.
3. Prediction: The trained model is applied to new text data to predict sentiment labels.
4. Evaluation and tuning: Model performance is assessed by using metrics (accuracy, precision, recall), and hyperparameters are fine-tuned to improve results.

Machine learning approaches have the ability to capture complex patterns and context better than rule-based systems. They also more easily adapt to new language use. However, they tend to require substantial labeled data and computational resources for training.

Hybrid sentiment analysis

Hybrid sentiment analysis leverages the strengths of both approaches by combining rule-based and machine learning methods.

Components therefore include:

1. Rule-based system: Preset linguistic rules and sentiment lexicons that provide interpretable sentiment signals
2. Machine learning model: Algorithms trained on labeled data to capture complex language patterns and context

Process:

1. Preprocessing: The text is cleaned, tokenized, and converted into feature vectors as required by the machine learning component.
2. Rule application: Linguistic rules are applied to identify explicit sentiment indicators and handle modifiers like negations or intensifiers.
3. Machine learning prediction: The ML model analyzes the same or complementary features to detect nuanced sentiment beyond explicit rules.
4. Fusion: Outputs from both rule-based and machine learning components are combined using weighting or voting mechanisms to produce the final sentiment prediction.

Combining the two main approaches can give better results when it comes to domains with subtle sentiment expressions or evolving language use. That said, achieving the right balance between complexity and performance in hybrid systems calls for careful fine-tuning and integration.

Sentiment analysis can provide businesses with actionable insights by identifying:

• The polarity of the language used (positive, neutral, negative)
• The emotional tone of the consumer's response (such as anger, happiness, or sadness)
• Whether the tone conveys urgency
• The consumer's intention or level of interest

As a form of automated opinion mining, sentiment analysis can support a variety of business applications.

Businesses can collect and analyze comments, reviews, and mentions from social platforms, blog posts, and various discussion or review forums to understand how their brand is perceived. Sentiment analysis tools can automate and scale this process. 

Data sources can include:

• Social media (X, Instagram comments)
• Review sites (Yelp, Google Reviews)
• Forums and blogs
• App store reviews

Insights generated from applying sentiment analysis on this data can help companies detect patterns in positive feedback, identify pain points in negative feedback, and gauge urgency and emotional intensity.

Marketing teams often use this approach to refine messaging strategies and monitor brand health and popularity.

Sentiment analysis can be a reliable tool for extracting high-level and aggregated insights about entire markets, industries, or customer segments beyond individual brand sentiment.

Common data sources include:

• News articles and press releases
• Industry reports and analyst commentary
• Financial news and stock market discussions
• Blogs and forums
• Product and service reviews
• Survey and feedback data

Sentiment analysis applications can use these data sources to quantify market sentiment trends, informing risk assessment and product strategy.

Integrating sentiment analysis into an ecommerce platform can improve product search and filtering capabilities. Aside from the aforementioned social media and reviews, further data sources may include:

• Web server logs capturing user navigation paths combined with sentiment-labeled session transcripts
• IoT device logs (e.g., smart home appliances with customer feedback via embedded apps) linked to sentiment tags
• Augmented reality (AR) product interaction feedback (where users’ verbal comments are transcribed and sentiment-analyzed)
• Multilingual sentiment data from international customer support communications

Aggregating and analyzing sentiment signals across diverse textual and semi-structured data sources can be used to benchmark brand and product perception against competitors.

Less conventional data sources may include:

• Patent filings and technical white papers mined for sentiment-laden language 
• Earnings call transcripts analyzed for sentiment shifts and investor confidence signals
• Customer complaint tickets and resolution logs with sentiment annotations
• Influencer content and endorsement sentiment measured via NLP techniques on multimedia transcripts

Sentiment analysis relies on understanding human language, which is by nature complex, ambiguous, and constantly evolving. This makes accurate interpretation a challenging task for automated systems.

Entity disambiguation in business-to-business (B2B) reviews

Distinguishing the sentiment directed at different entities is a common challenge, especially in competitive contexts. In B2B reviews, for instance, similar language may be used to describe your company and your competitors, but the sentiment toward each should be interpreted differently.

The sentiment analysis tool may lack entity disambiguation capabilities, leading to incorrect attribution of positive sentiment to your company when the statement actually refers to a competitor.

Irony, sarcasm, and context

Detecting and understanding irony and sarcasm remains a significant challenge in sentiment analysis.

These forms of expression use positive words to convey negative or opposite meanings, often without explicit textual cues, and this ambiguity may complicate automatic sentiment classification.

Sentiment is highly dependent on context, and identical phrases can carry different sentiment polarities depending on the question or scenario.

Handling sarcasm and irony requires more advanced techniques, such as context-aware models (transformers) and/or multimodal analysis (incorporating tone or visual cues).

Context-dependent sentiment classification often relies on incorporating the prompt or conversation history to correctly interpret responses.

Subjectivity

One of the main challenges of sentiment analysis is the subjectivity of language. Variations in humor, idiomatic expressions, and dialects across cultures can alter meaning.

Due to lexical and syntactic differences, sentiment models trained on one language variant or culture may underperform when applied to others.

Strategies for localization, such as regionally adapted training data and culturally specific lexicons, are essential for the successful application of sentiment analysis.

Sentiment analysis benefits its users with actionable insights. As a tool, its advantages are multiple:

Mine customer emotions at scale

Sentiment analysis tools provide real-time analysis from diverse text sources.

Primary usages include:

• Early detection of negative sentiment spikes and emerging issues
• Crisis management through timely alerts
• Informing PR strategy

The text mining process in this context often involves continuous data ingestion and preprocessing and data visualization tools. Sudden shifts can be flagged by using anomaly detection algorithms for sentiment scoring.

Support predictive analytics models

Sentiment analysis outputs can be integrated as engineered features in predictive modeling pipelines.

A typical workflow includes:

• Sentiment polarity and intensity scores extraction from unstructured text using NLP models or APIs
• Score aggregation over relevant time windows or customer segments to create numeric features
• A combination of sentiment-derived features with structured datasets (e.g., CRM records, transaction logs)
• Supervised machine learning models training (random forests, gradient boosting, deep neural networks) to predict outcomes
• Model validation using metrics such as AUC-ROC, F1-score, or RMSE

Improve product and service development

With sentiment analysis, a data-driven product iteration can be made more efficient through continuous feedback monitoring:

• Implement real-time ingestion of customer feedback from multiple channels (reviews, support tickets, forums) via APIs or streaming platforms.
• Apply NLP preprocessing steps: tokenization, lemmatization, stop-word removal, followed by sentiment classification using rule-based or ML-based models.
• Store sentiment-tagged feedback in a time-series or document database for trend analysis.
• Develop visualization dashboards with metrics like sentiment distribution or volume spikes.

You can build a sentiment analysis system yourself, invest in a third-party provider, or purchase add-ons to integrate in your applications. A variety of software-as-a-service (SaaS) sentiment analysis tools are available, while open source libraries like Python or Java can be used to build your own tool. Often, cloud providers offer their own AI suites.

• Build your own sentiment model
  You can build your own sentiment model using an NLP library, such as spaCy or NLTK. When it comes to a customization, a hands-on approach allows for full control over preprocessing, feature engineering, model architecture, and training data. This said, building your own sentiment model requires expertise in NLP and machine learning, as well as significant investment in data labeling, model training, and tuning. When domain-specific language or fine-grained sentiment nuances require tailored models, a do-it-yourself approach may be the one for you.
• Use turnkey SaaS sentiment analysis solutions
  A prepackaged solution could include Amazon Comprehend, Google AI, or Azure's Cognitive Services. Advantages of a SaaS sentiment analysis tools like these encompass rapid deployment, managed infrastructure, pretrained models, and scalable APIs. However, less control over model internals also means occasionally requiring fine-tuning or domain adaptation through additional training.
• Integrate third-party sentiment analysis models
  You can also choose to upload custom or open source sentiment models into platforms like Elastic's Search AI Platform. By combining Elasticsearch's indexing and search with sentiment scoring to analyze large-scale text datasets, you can develop hybrid architectures, combining pretrained models with custom rule sets or ML enhancements. If you want the flexibility of managing your own models while using a pre-existing and reliable search and analytics infrastructure, this is the way to go.
• Cloud-provider AI suites
  AI and ML suites from cloud providers often include sentiment analysis as part of broader NLP capabilities. These solutions offer easy integration with other services and continuous model updates. However, vendor lock-ins and limited customization may prove challenging.

Launch your sentiment analysis tool with Elastic, so you can perform your own opinion mining and get the actionable insights you need.  

• Learn how to deploy NLP sentiment analysis with Elastic
• Observability and sentiment analysis with the ELK Stack
• How to turn data into actionable insights
• Text classification comprehensive guide
• NLP vs. LLMs
• Machine learning vs. AI

Algorithm: A process or a set of rules that a computer follows

Artificial intelligence: The simulation of human intelligence by machines and computer systems

Computational linguistics: A branch of linguistics that uses computer science theories to analyze and synthesize language and speech

Coreference resolution: The process of identifying all the words that belong to a named entity in a text

Lemmatization: The process of grouping together different inflected forms of the same word

Lexicon: A vocabulary word inventory of a language

Machine learning: A subset of artificial intelligence that, by the use of data and algorithms, allows a computer to learn without prompting

Named entity recognition: The process of recognizing words as proper names or entities

Natural language processing: A branch of computer science that, as a subset of artificial intelligence, is concerned with helping computer systems understand human language

Part-of-speech tagging: The process of marking a word in a text to categorize what part of speech it belongs to (e.g., apple = noun; slowly = adverb; closed = adjective)

Stemming: The process of reducing words to their stem, or root, form

Tokenization: The process of separating a piece of text into smaller units, called tokens

Word sense disambiguation: The process of identifying the sense of word given its use in context

¹ Webb, G.I. "Naïve Bayes." Encyclopedia of Machine Learning and Data Mining, Springer, 2017, https://doi.org/10.1007/978-1-4899-7687-1_581.