A talk on connecting to the Twitter Streaming API and storing the tweets in MongoDB with Ruby.
I’m looking to use the Twitter Streaming API to scrape all tweets for a few keywords and store them in a database. I would rather not run the workflow locally so I had been looking at ways to use the API with Heroku and MongoLab.
Apr 28, 2013 · Twitter provides a streaming API that you can filter by username, keyword, or geo-location. Using a couple of great Ruby gems, we can store tweets from the streaming API into MongoDB, a NoSQL
High Volume MongoDB with Twitter Streaming API, Ruby on Rails, Heroku setup. Ask Question. Browse other questions tagged mongodb twitter heroku streaming or ask your own question. asked. 6 years, 7 months ago. viewed. 1,019 times How to Implement Twitter Streaming API in …
The write volume that it will handle depends on lots of factors – hardware, indexes, size of each document, etc. Your best bet is to test it in the environment you’re planning to use. If the demands of the write load exceed the capacity of a single mongo server, you can always use just multiple shards.Best answer · 0Completely agree on the need to test this. In general, mongo can handle that many writes, but in practice it depends on the size of your set up, other operations, indexes, etc.
I had to do a similar approach for collecting tons of metrics data.0
Consuming Twitter Streaming API. By Bala Paranj Mar 25, 2017 -Create a rake task that consumes the Twitter Streaming API-Inserts Tweets into a Capped MongoDB tweets collection From the bonjo directory run : ruby twitter_stream.rb, this will continuously consume the Twitter stream for SF.
The json.loads() call converts the string returned from the Twitter API into a json object in Python. Finally, the collection.insert() call inserts the json object into the MongoDB database. From this rather simple change to the Python stream listener all the tweets can be saved into a MongoDB database. Recalling Tweets from MongoDB
Getting Data from Twitter Streaming API
TweetStream gem – reads tweets off of Twitter’s streaming API and provides a Ruby block with the tweets. Twitter gem – for use with the other Twitter API’s – use to get followers for a user, user’s tweet timeline, etc. We can use this to backfill in the history for users we follow on the streaming side.
If you consume a low-volume stream, some libraries (e.g. Java’s GZIPInputStream or many Ruby stream consumers) handle decompression of incoming data poorly, and will need to be overridden in order to decompress the Tweets as they are received, without waiting for a …
I’ve been using Node extensively with the Rawkets multiplayer game, so it seemed logical to use it for scraping the Twitter Streaming API and sending the tweets to MongoDB. What’s even better is that Node has modules for both MongoDB and the Twitter Streaming API (node-twitter), which makes things a lot easier for me.