Realtime Twitter Streaming using Python APIs

Twitter provides developers with  Streaming APIs  and REST APIs that give complete access real-time twitter. This enables us to query the whole twitter world for specific search criteria and also setup streams . REST APIs are useful for doing individual searches or posting tweets on behalf of users. Stream APIs on the other hand are useful for setting up real-time streams for search criteria as users tweet text containing the keywords.

Twitter gives streaming endpoints to developers, each for different purpose.

Public Streams- Gives access all public data on Twitter. This is meant for following specific topics and Hashtags Private Streams- Gives access to the users account view of the twitter.

Building Twitter Search Streams There are several libraries available in several languages for building twitter apps : https://dev.twitter.com/overview/api/twitter-libraries One thing to note with twitter APIs have rate limiting imposed, this restricts the number of connections that can be made to twitter server for a single app using REST APIs/Streaming. Streaming works better as the connection persists till it is killed. For the REST APIs this restricts the number of queries that be done. Refer : https://dev.twitter.com/rest/public/rate-limits for REST API rate limiting Back-off Strategy need to implemented for Streaming APIs,since the Twitter will disconnect the App if the application make very frequent connection in 15 minutes window. Steps for Streaming & Parsing Twitter data real-time into a MongoDB Collection.

1. Create Twitter APP and obtain the OAuth Keys/Tokens. Twitter client need to be authenticated by the server before accepting any requests.Twitter user OAuth to authenticate its clients, which enables developers to build apps and connect to twitter without sharing user accounts and passwords.. Login to Twitter and Create a new App: https://apps.twitter.com/

  The Keys and Access token will give you the consumer_key, consumer_secret, access_token, access_token_secret  needed to authenticate the your app 2. Python App Code Libraries/Languages for building Twitter Apps :https://dev.twitter.com/overview/api/twitter-libraries. In the example I have used the Tweepy library in Python  for authenticating the client and setting up the Stream. Python has libraries to connect to MongoDB(PyMongo) which is stores data as documents which is very suitable for storing twitter streams as the respose to the APIs is in the JSON format.The JSON Library in python allows parsing the contents of the Json document. See code in github below. Github repo:https://github.com/ashwinaravind/Twitter-MongoDB-Streaming.git