Eliot’s Ramblings / Streaming Twitter into MongoDB
curl http://stream.twitter.com/1/statuses/sample.json -u: | mongoimport -c twitter_live
One thing that you can do with mongo is have 1 streaming master and 1 read/write master
server A:
./mongod —master —dbpath /tmp/a
server B
./mongod —dbpath /tmp/b —master —slave —source localhost:27017 —port 9999
You can then pipe the stream into server a, and it will only process the live stream.
Server B will replicate all changes. You can also write to it, query on it, etc… This way you can do operations that block writing on server B, but server A will never backlog.
A super good tip with MongoDB !!! :D
