Sentiment Analysis with Python

2018-05-18T00:00:00+00:00

I recently had the chance to spend my weekend enhancing my knowledge by joining a local community meetup in Malaysia which is sponsored by Malaysian Global Innovation & Creativity Centre (MaGIC). The trainer was Mr Lee Boon Kong.

Anaconda and Jupyter Notebook

We started by preparing our Jupyter Notebook setup which is running on the Anaconda Python distribution. The installer is 500 MB in size but pretty handy when we started using it.

Anaconda comes with a graphical installer called “Navigator” so the user can install some packages for work. However it did not always work for me on some OSes, so I had to use its command-line based tool “conda”. Conda works like Linux-based package management tools such as apt, dnf, yum, and pacman, so to install a package I would just run conda install <package name>.

Jupyter uses a web browser to allow us to write the code directly in its cell. It is quite helpful for us to debug the code or if we just want to execute it segment by segment independently.

Creating Twitter’s API key

First we need to head to apps.twitter.com.

The following items are needed:

Consumer Key (API key)
Consumer Secret (API secret)
Access Token
Access Token Secret

Using Tweepy, NLTK and TextBlob

from textblob import TextBlob
import tweepy
import nltk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
consumer_token = '<put your token here>'
consumer_secret = '<put your secret here>'
access_token = '<put your access token here>'
access_secret = '<put your access secret here>'
auth = tweepy.OAuthHandler(consumer_token, consumer_secret)
auth.set_access_token(access_token, access_secret)
api = tweepy.API(auth)
public_tweets = api.search("Avengers Infinity War", lang='en')
print("number of tweets extracted: " + str(len(public_tweets)))
for tweet in public_tweets:
    print(tweet.text)
    analysis = TextBlob(tweet.text)
    print(analysis.sentiment)
    print("\n")

If we want to increase the number of tweets to be displayed and analyzed, just change this line to:

public_tweets = api.search("avengers", count=100, result_type='recent', lang='en')

Analyzing Sentiment Score Results

The sentiment score that we got is summarized as follows:

< 0 - Negative sentiment
0 - Neutral
0 - Positive sentiment

By default, the code above uses the English-based library. As a Malaysian, I could not analyze tweets in the Malay language yet. Efforts by the local community are being made to create the Malay-based language corpus for NLTK.

Looking at the TextBlob Component

The Natural Language Processing (NLP) library’s TextBlob did the sentiment processing task.

I did some reading in TextBlob’s documentation. So for example if I declare:

text = '''
I love to read!
'''

I get a sentiment polarity value of 0.5 (positive).

While if I put

text = '''
I hate to read!
'''

I get a sentiment polarity value of -1.0 (negative).

Summary

Overall I am quite satisfied with what I learned during the session. It is good to have one day spent on a technical workshop like this in which we could be super focused on the content without any external distraction.

Kudos to Mr Lee for his effort to teach us about data analysis with Python. Till we meet again, hopefully!

Facebook, Twitter, Google+ sharing with the URL

2014-07-10T00:00:00+00:00

This blog post is intended for the folks who spent way more time displaying social sharing buttons on their websites than originally planned. My buttons were supposed to bring up the Share Dialog for Facebook, Twitter and Google+ platforms. I had the requirement to display a custom logo and a custom description. It seemed easy… until it turned out to be rather difficult.

The appropriate format for Twitter is:

<a href="http://twitter.com/share?text=<%= desc %>&url=<%= url %>">Twitter</a>

Note that Twitter dialog does not include the picture, only the description.

Trouble started with Facebook when I learned that custom parameters in Facebook’s sharer.php url are officially not supported anymore: https://developers.facebook.com/bugs/357750474364812. I tried the format widely suggested on forums, and while custom image was successfully displayed, neither title, not description showed.

<a href="http://www.facebook.com/sharer.php?u=<%= url %>&p[images][0]=<%= img  %>&description=<%= desc %>">Facebook</a>

Facebook documentation hinted that addition of the OpenGraph Protocol standard tags to the of the page may help:

<meta content="http://samples.ogp.me/136756249803614" property="og:url"/> 
<meta content="Chocolate Pecan Pie" property="og:title"/>
<meta content="This pie is delicious!" property="og:description"/> 
<meta content="https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-prn1/851565_496755187057665_544240989_n.jpg" property="og:image"/>

I wasn’t able to get it up and working with sharer.php. After spending considerable amount of time on this I had to give up and acknowledge that the only way to fully customize the dialog would require registering an app and utilizing APP_ID https://developers.facebook.com/docs/sharing/reference/share-dialog.

I anticipated the same kind of trouble with the last button for Google+. And I wasn’t mistaken. The only allowed format for G+ is:

<a href="https://plus.google.com/share?url=<%= url %>">Google+</a>

Despite a few mentions on the web this does not work anymore:

<meta content="<%= desc %>" itemprop="description"/>

Same here. Doesn’t work:

<meta content="<%= title %>" property="og:title"/>
<meta content="article" property="og:type"/>
<meta content="<%= url %>" property="og:url"/>
<meta content="<%= img  %>" property="og:image"/>
<meta content="<%= desc %>" property="og:description"/>

Eventually it turned out that it wasn’t possible to use parameters for G+ link unless you sign up for the API key and use one of the API methods. I wasn’t planning to obtain a key at that time, so I had to simply drop the custom logo and text for G+.

Looks like both Facebook and Google+ took steps to restrict the free usage of their share urls so more people would register their apps with them.

Integrating Facebook SDK and HybridAuth PHP library

2014-06-13T00:00:00+00:00

There are a few different libraries out there for integrating your site with Facebook and other social networking sites. I recently added “Login with Facebook” for a client to their PHP site utilizing the Facebook JavaScript SDK. The documentation on Facebook’s site is pretty good (although it could use a few more examples). Beyond just the login feature, this client also wanted to be able to offer a checkbox for “Post a message to Facebook about your order”. And the way they wanted it done required a PHP library to make calls to the Facebook Graph API directly.

I chose to use the HybridAuth PHP library which is a wrapper for integrating many different social networking sites using a plugin system (Facebook, Twitter, Google, other OpenID services, etc). Likewise, the docs for HybridAuth were sufficient to get the examples up and running for me. The problem was that none of the examples or documentation fit my scenario, where I already have the login set up and working with the JavaScript SDK but want to utilize the PHP library for posting to a user’s feed.

When attempting to connect to Facebook with HybridAuth it kept attempting to log the user in again. The main problem was that the access token received from the JavaScript SDK was not getting passed in correctly to HybridAuth, and so it was attempting to get a new access token. The solution I finally got working seems a little dirty (not going through a standard method call or documented API), but it works. Here is the snippet of PHP code to set the Facebook access token so that HybridAuth will use it instead of fetching a new one:

    // Connect to Facebook and get the user's profile
    try {
        $hybridauth = new Hybrid_Auth( $config );

        // Set some session variables needed for HybridAuth
        Hybrid_Auth::storage()->set( 'hauth_session.facebook.is_logged_in', 1 );
        Hybrid_Auth::storage()->set( 'hauth_session.facebook.token.access_token', $_POST['fb_access_token'] );
        $hybrid_config = require $config;
        $fb_config     = $hybrid_config['providers']['Facebook'];
        $fb_app_id     = $fb_config['key']['id'];
        $_SESSION['fb_'. $fb_app_id .'_access_token'] = $_POST['fb_access_token'];
        $_SESSION['fb_'. $fb_app_id .'_user_id']      = $_POST['fb_uid'];

        // Now we connect to FB using the given access token for this user
        $adapter      = $hybridauth->getAdapter( "facebook" );
        $user_profile = $adapter->getUserProfile();
    }
    catch( Exception $e ) {
        return 'FB Connection Failed: <b>got an error!</b> error message=' . $e->getMessage() . ', error code='. $e->getCode();
    }
    ...

When a user successfully logs in with the JavaScript SDK, you will be given an authResponse object which contains, among other things, the user’s Facebook ID and an access token. I pass these two values to PHP using an HTTPS ajax call (POST as you can see above). These two values are needed by HybridAuth and stored in certain session variables. These must be set in the session before you call $hybridauth->getAdapter() method. The order is important, otherwise HybridAuth won’t use the access token you’ve set and will treat the user as not yet logged in.

Respecting API Call Limit with Radian6

2012-06-29T00:00:00+00:00

A common practice in the online API world is to enforce the call limit. Twitter allows 150 API calls per hour. Shopify has 500 API calls per 5 minutes limit. You may learn how to work with Shopify call limit from this great article.

One of our projects was built around interaction with Radian6 platform. Radian6 is a new and growing data mining service with the default limit of 100 calls per hour. I will describe our take on the call limit implementation.

Introducing the call counter

First, we need to know how many calls have been executed in the current hour. Every external call increments the counter field on a special model until the counter reaches the limit. The counter is reset back to zero at the beginning of every hour.

script/rails g model RadianCallCount

In the migration:

class CreateRadianCallCounts < ActiveRecord::Migration
  def change
    create_table :radian_call_counts do |t|
      t.integer :count
      t.timestamps
    end
  end
end

In db/seeds.rb file

puts "Initializing Radian6 counter"
RadianCallCount.delete_all
RadianCallCount.create(:count => 0)

Let’s roll the counter!

rake db:migrate
rake db:seed

Scheduling the counter reset

It is necessary to reset the counter back to zero in the beginning of each hour otherwise the subsequent calls will not be executed. The excellent ‘whenever’ gem will take care of this.

gem install whenever
cd your_webapp
wheneverize .

In the model:

class RadianCallCount < ActiveRecord::Base
  def self.reset
    RadianCallCount.first.update_attribute(:count, 0)
  end
end

In config/schedule.rb:

every :hour do 
   runner "RadianCallCount.reset"
end

Tracking call count

We will now use the rcapture gem to intercept a call to external API and increment the counter with it.

gem install rcapture

In the module containing all Radian-specific methods:

require 'rcapture'
API_LIMIT = 100
def self.included(base)
  base.extend RCapture::Interceptable
  base.capture_pre :methods => [:authenticate,:tweet_stats] do |cs|
    RadianCallCount.transaction do 
      calls_per_hour = RadianCallCount.first.count 
      allowed = (calls_per_hour < Radian::API_LIMIT)
      cs.predicate = allowed
      cs.return = false 
      RadianCallCount.first.increment!(:count) if allowed
    end
  end
end

The code introduces a simple check before ‘authenticate’ and ‘tweets_stats’ methods. If call count exceeds the allowed limit the method is not executed and the method returns false. Otherwise, the counter increments after the successful method execution. We wrap the code in transaction because the actual count in the database may increase while we are making the API_LIMIT comparison.

Making the limit-aware call

Everything is ready to make the non-blocking API call. I scheduled a twitter statistics update to run every 3 hours:

every 3.hours do
   runner "Article.tweet_stats"
end

The non-blocking calls are suitable for most situations. Sometimes there is a need to just keep trying…

def call_with_timeout(&block)
  timeout = 0.minutes 
  results = false 
  while !results do
    results = block.call
    if !results
      break if timeout >= Radian::API_MAX_TIMEOUT
      Rails.logger.info("Sleeping for 5 minutes")
      sleep(Radian::API_CALL_TIMEOUT)
      timeout += Radian::API_CALL_TIMEOUT
    end
  end
  results 
end
...
call_with_timeout { tweet_stats }

That is all for today. Thanks for your attention. Hope the post was useful.

Developing a Spree Application

2012-06-15T00:00:00+00:00

Mike Farmer presented one of the projects he worked on. He gave us a very detailed overview of integration of a Spree application into Facebook for a large client.

He also described the changes that were made in the Spree project to customize it to the client’s needs.

The most interesting thing in this Spree shop usage is that the company’s clients make extensive use of the admin panel and they can sell their own products for other clients.

Mike showed us a great overview of writing better Rails applications using better object oriented code along with TDD and extensive SASS usage. He also described the great tools he uses including Screen and Vagrant.

Mike also talked about the things he learned during working on this project, especially about Spree and Ruby.

Google+

2011-07-11T00:00:00+00:00

Over the weekend, I dug into Google+ a bit. I wanted to share a few notes about the experience with my coworkers and the world.

Speed

Google has done a great job on performance from what I can tell. They’ve followed their own recommendations on optimization by doing things like implementing CSS sprites, caching static assets, and gzipping content. I can’t do performance tests on my authenticated account at WebPageTest.org, but the perceived performance is great.

User Interface

Parts of the user interface look similar to Facebook. But Google deviated from their norm of utilitarian/pragmatic design according to this article, and I appreciate their focus on aesthetics here. This combined with the speed makes for a delightful user experience.

Circles

Google offers limited sharing functionality with what they call “Circles”, similar to Facebook groups. In my case, I have the following circles:

Friends
Family
End Point
Web People (professional contacts)
Photography

Circles are integrated into every part of Google+, which makes it easy to limit posts, photos, videos or other content to individuals or circles. In addition to limiting sharing permissions, you can also limit viewing “Stream” (akin to Facebook’s News Feed) activity to circles; for example, I might only be interested in looking at my “Friends” stream on the weekend, while I might be more inclined to look at my “End Point” stream during the week when I’m in work mode :)

Conference Calling or “Hangouts”

Yesterday, I tried Google Hangouts with a friend from my Windows laptop. I was required to install software on my laptop to run the chat app. Hangouts are similar to Skype video conferencing but are currently limited to 10 people. The quality of the conference call was comparable to Skype and this feature is an obvious competitor to Skype. Skype has always been a bit finicky on Ubuntu, so hangouts will be a clear replacement for me here if it runs smoothly on my Ubuntu laptop(s). I expect the conference calling feature set to grow over time — it’ll be interesting to see what comes out.

Other Stuff

Jon mentioned that the Android app for Google+ is nice. I haven’t examined Sparks much, but it is similar to sharing in Facebook. I also just read that the iPhone app for Google+ is on its way (from a Google+ contact). Finally, I’d consider shifting my photo backups from Flickr to Picasa if future Picasa updates justify the switch, which would allow me to easily share and tag photos, as well as back-up those 2-3 MB photos!

Conclusion

I’m impressed with the speed and usability of Google+. I’m interested to see the upcoming features and equally as interested to see how much cross-posting is going to happen between Google+ and Facebook, brought up by Dave Jenkins, who has also written about Google+ after he received an early invite.

Ecommerce Facebook Integration Tips

2011-02-23T00:00:00+00:00

Over the past couple of months, I’ve done quite a bit of Facebook integration work for several clients. All of the clients have ecommerce sites and they share the common goal to improve site traffic and conversion with successful Facebook campaigns. I’ve put together a brief summary of my recent experience.

First, here are several examples of Facebook integration methods I used:

Name & Notes	Screenshot Examples
Link to Facebook Page: This is the easiest integration option. It just requires a link to a Facebook fan page be added to the site.	A link to Lots To Live For’s Facebook page was added to the header.
Facebook Like Button: Implementation of the like button is more advanced than a simple link. Technical details on implementation are discussed below. This was integrated on multiple product page templates. The "like" event and page shows up on the user’s wall.	Like button’s were added throughout Paper Source’s site. The screenshot shown above indicates I’ve liked this product page. "Liking" a product page shows up on my facebook wall.
Facebook Like Box: Another example of Facebook integration is adding a “Like Box”. Acting on the Like Box results in liking the actual Facebook fan page. For example, I added a Like Box to Paper Source’s footer. When a user clicks on it, they are Liking Paper Source’s Facebook page and will begin to receive updates from Paper Source’s Facebook page on their wall.	The Facebook Like Box was added to Paper Source’s footer. Clicking on the Like Box is equivalent to liking the Paper Source fan page, shown here.
Sharer: The final example of Facebook integration I implemented for a client was adding “sharer” functionality. This is accomplished by adding a link to “http://www.facebook.com/sharer.php?u=” followed by the URI encoded URL. The user is directed to Facebook and allowed to enter an additional comment and select a thumbnail. This is sent to the user’s facebook wall and will be seen in the user’s friends news feeds.	A Facebook share link is displayed for one of our Spree clients. When you share an item, you are brought to a page much like the page shown in this screenshot to add a comment or select the thumbnail. Two of my Facebook friends shared the same article. I saw this on my News Feed.

In addition to the like button and like box, there are several other types of Facebook plugins including an Activity Feed, Recommendations, Login Button, Facepile, Live Stream, and Comments. Facebook’s documentation on Plugins can be found here.

Plugin Integration: iframe versus xfbml

I’ve used iframe integration for the majority of clients and most recently used xfbml for Paper Source. Implementation with an iframe might be easier and require less templates to be modified. Implementation with xfbml provides more flexibility, claims to have more advanced reporting, and allows you to add event listeners to various actions. Here’s a code comparison of equivalent functionality for the two options:

iframe	xfbml
&layout=button_count" scrolling="no" frameborder="0" style="border:none;overflow:hidden; width:90px;height:21px;" allowTransparency="true">

Gotchas

Throughout development, I’ve learned a few gotchas:

Styling on the Facebook elements is not always easily adjustable. In some cases, the iframe or xfbml element had undesirable styling and the styling of the wrapper div may need to be adjusted to achieve desired results.
Facebook crawls or scrapes the “Liked” pages to retrieve page information such as a title, description and thumbnail. Facebook follows rel=canonical URLs to retrieve this information—so it’s important that these canonical URLs are correct. A couple of Paper Source’s product page templates added an extra forward slash to the URL, and these were followed by Facebook’s crawler, resulting in an infinite redirect loop of URLs with additional forward slashes. The result is that Facebook can not scrape the URL and the like button has a flickering effect when clicked.
Facebook’s crawler caches pages. If there is an error such as the canonical error listed above, it’s recommended to wait 24 hours to test until the page has been re-crawled.

xmlns, firewall issues