How do you feel about those programs which tell you the sentiment of some text? A computer algorithm which examines some text, returning a score on its favourability. On balance I have a negative opinion on them. I explained in this infographic why.
This is not a stagnant space and many minds are working to improve their accuracy. I thought it worthy of reassessment. For my client work I do not use this data to assess media sentiment. Instead I manually check 400 or 25% (whichever is bigger) of the media clips. In this post I explain this process in detail.
Users like the instant results from an auto-sentiment analysis tools. To check 400 clips by hand takes a few hours. Bloomberg and Twitter recently paired up to add Twitter sentiment to their terminals. Thanks for to @sabguthrie for the notification.
A practical experiment:
Being AMEC Measurement Month I collected the Tweets so far this month mentioning #amecmm. From this 1600 Tweets I randomly selected 25% (equal to 400 Tweets). To try and establish a level playing field I tested where I could using just these tweets. I checked the 400 by hand, then used an enterprise paid for tool and an academically based free web tool .
The enterprise tool assigned a negative sentiment to almost two-thirds of the coverage. My own check found only 2% of the sample negative – a large disparity. The academic tool from SentiStrength appeared to be somewhat closer to my scores.
I can’t comment on the enterprise sentiment scoring method as the process is not explained. SentiStrenth’s processes are clearer. It has a list of positive and negative words which are spotted for providing a plus and minus result. To get to an overall favourability I combined these figures assigning neutral to a zero score. This seemed the only way to harmonise the metrics. From my analysis SentiStrength overstated the neutral and understated the positive.
I have not examined the positive and negative word tables in detail. SentiStrengths results are good compared to the enterprise option.
There are a couple of other interesting options. Sentiment140 offers a straight Positive/Negative split. I could not find a way of adjusting date range, it is set to the past week. The classification seems to defer to negative if it includes a negative word like ‘not’ (see below).
The proportional bias on sentiment did not seem unreasonable. There is an API, which for $200 a month will classify up to 1 million tweets a day.
Another tool is Sentiment Viz from North Carolina’s NC State computing department. Put in a keyword and while it won’t condense it to sentiment groups, it will generate cluster diagrams, tag clouds, affinities, etc. You can not select periods, it is set to the past 10 days. Tweet capture seems to be better than Sentiment140.
As you can possible discern from this image below, it uses a similar sentiment word classification table used by others. From the check ran against ‘#amecmm’ the generated results seemed proportional and reasonable. Accurate comparisons are difficult with the set periods and different classification methods.
These are some interesting tools which have a perspective a ‘fast’ PR person could benefit from. My vote still goes to the manual option. Are the automated options getting better? Not dramatically. The top graph compares the two and the deviation is still too great to base judgements on.
As always, I welcome your views and if you would like to not miss future posts please subscribe.
Leave a Reply