Sentiment Analysis with VADER

Source

Sentiment Analysis with VADER

Sentiment analysis estimates whether whether a piece of text is negative, neutral or positive. There are two approaches towards sentiment analysis: binary or polarity based or intensity/valence based. The polarity based only provides information if a certain text is postive or negative. For example, 'good' and 'perfect' would score the same. In contrast, a valence based sentiment analysis also takes the intensity of a word into account, therefore giving a higher value to 'perfect' in the example above.

One of the major applications of sentiment analysis is judge social media streams (Twitter, facebook posts). For example, companies use it to identify negative feedbacks on social media and react/answer to them.

Python provides the excellent Natural Language Toolkit (NLTK) for automatic text processing.Among others, NLTK includes the sentiment analysis package VADER (Valence Aware Dictionary and sEntiment Reasoner).

VADER used a valence based approach for the sentiment analysing, building upon a lexicon of sentiment-related words. When VADER analysis a text, it check for the word in its lexicon, adding their negative or positive sentiments. The final (compound) score is the standardized sum of the ratings.

You can find out more about VADER in the article Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.

To start with VADER, install NLTK (you might need sudo rights to do this).

pip install -U nltk

For further information see http://www.nltk.org/install.html

The first time you use ntlk and VADER, you need to download the VADER lexicon. To do so (dont do that if you run it on the IE Server):

In [1]:
# import nltk
# nltk.download()

Either you now download "all" packages, or just choose the "vader_lexicon" in the category "Models". You only need to download the packages the first time you use them.

Having setup everything your are good to go:

In [2]:
import nltk
from nltk.sentiment.vader import SentimentIntensityAnalyzer

And you can start analysing the first sentence:

In [3]:
text = "This was easy"
SentimentIntensityAnalyzer().polarity_scores(text)
Out[3]:
{'compound': 0.4404, 'neg': 0.0, 'neu': 0.408, 'pos': 0.592}

The most important part of the output dictonary is the 'compound' part, which gives you the overall rating. Positive values indiacte an overall positive rating, negative values the opposite. The 'compound' value can range from -1 to +1.

To automatically process a bunch of text, run VADER in a loop (tricky sentences from the nltk - VADER docs:

In [4]:
tricky_sentences = [
   "Most automated sentiment analysis tools are shit.",
   "VADER sentiment analysis is the shit.",
   "Sentiment analysis has never been good.",
   "Sentiment analysis with VADER has never been this good.",
   "Warren Beatty has never been so entertaining.",
   "I won't say that the movie is astounding and I wouldn't claim that the movie is too banal either.",
   "I like to hate Michael Bay films, but I couldn't fault this one",
   "It's one thing to watch an Uwe Boll film, but another thing entirely to pay for it",
   "The movie was too good",
   "This movie was actually neither that funny, nor super witty.",
   "This movie doesn't care about cleverness, wit or any other kind of intelligent humor.",
   "Those who find ugly meanings in beautiful things are corrupt without being charming.",
   "There are slow and repetitive parts, BUT it has just enough spice to keep it interesting.",
   "The script is not fantastic, but the acting is decent and the cinematography is EXCELLENT!",
   "Roger Dodger is one of the most compelling variations on this theme.",
   "Roger Dodger is one of the least compelling variations on this theme.",
   "Roger Dodger is at least compelling as a variation on the theme.",
   "they fall in love with the product",
   "but then it breaks",
   "usually around the time the 90 day warranty expires",
   "the twin towers collapsed today",
   "However, Mr. Carter solemnly argues, his client carried out the kidnapping \
   under orders and in the ''least offensive way possible.''"
]
In [5]:
for sentence in tricky_sentences:
    sentiment = SentimentIntensityAnalyzer().polarity_scores(sentence)['compound']
    print("{} - score: {}".format(sentence, sentiment))

Most automated sentiment analysis tools are shit. - score: -0.5574
VADER sentiment analysis is the shit. - score: 0.6124
Sentiment analysis has never been good. - score: -0.3412
Sentiment analysis with VADER has never been this good. - score: 0.5228
Warren Beatty has never been so entertaining. - score: 0.5777
I won't say that the movie is astounding and I wouldn't claim that the movie is too banal either. - score: 0.4215
I like to hate Michael Bay films, but I couldn't fault this one - score: 0.3153
It's one thing to watch an Uwe Boll film, but another thing entirely to pay for it - score: -0.2541
The movie was too good - score: 0.4404
This movie was actually neither that funny, nor super witty. - score: -0.6759
This movie doesn't care about cleverness, wit or any other kind of intelligent humor. - score: -0.1338
Those who find ugly meanings in beautiful things are corrupt without being charming. - score: -0.3553
There are slow and repetitive parts, BUT it has just enough spice to keep it interesting. - score: 0.4678
The script is not fantastic, but the acting is decent and the cinematography is EXCELLENT! - score: 0.7565
Roger Dodger is one of the most compelling variations on this theme. - score: 0.2944
Roger Dodger is one of the least compelling variations on this theme. - score: -0.1695
Roger Dodger is at least compelling as a variation on the theme. - score: 0.2263
they fall in love with the product - score: 0.6369
but then it breaks - score: 0.0
usually around the time the 90 day warranty expires - score: 0.0
the twin towers collapsed today - score: -0.2732
However, Mr. Carter solemnly argues, his client carried out the kidnapping    under orders and in the ''least offensive way possible.'' - score: -0.5859
In [6]:
SentimentIntensityAnalyzer().polarity_scores("Have fun")
Out[6]:
{'compound': 0.5106, 'neg': 0.0, 'neu': 0.233, 'pos': 0.767}