Pollarize architecture at a glance

X-posted from the now-defunct pollarize.me blog

Genesis

Pollarize was bourne out of a 54-hour hacking stint at Startup Weekend in March 2012. The Thursday night before that weekend, I thought about the various tech I knew and wanted to have a good idea how to kickstart our hack before putting pen to paper on the Friday night.

The decision was made: Pollarize was going to be a Spring MVC web app running on an Amazon EC2 instance. I was going to worry about the database later, and maybe get away with using an in-memory map or something for the data.

too much too soon

Come Friday night, I felt a bit different. Why not learn something new, I thought. Besides, AWS might be a bit clunky for a weekend hack. And I’ve heard good things about the Play! Framework and the v2 was released just a few months before.

Today

Fast forward seven months, over 200 deploys, and over 1,000 commits. This is what our API looks like now:

pollarize architecture

Core

The core app is a lightweight Play! 2.0 (Scala edition) web app which exposes a JSON web API, and it makes transactional writes to our operational Postgres database. Most other (unimportant) write operations happen via fire-and-forget calls to other modules via Akka. The IDs for these data are pre-chosen so that subsequent calls will be idempotent.

MongoDB

Writes to the operational database is fired off to a MongoDB actor, which serves as the read-only cache for API reads. The data in MongoDB is already in the format the caller will consume, so it is extremely quick. The cache can be rebuilt asynchronously with the click of a button.

Media

The media module accepts profile pictures and poll images. We use Blitline to get them into our signature circular format (and different sizes), and then we store the results in Amazon S3. It is exposed to the world via Cloudfront. If you have any image processing to be done, definitely give Blitline a shot: they’re very helpful, and the API is a lot of fun to explore.

Notify

Notifications are triggered via an Akka scheduled job in the case of poll expiry and via events in the event of registration, password reset, poll creation from people you follow and follow notifications (yes, we’re a minuscule social network). In general, we use email via Mailgun’s simple API, Pubnub for web users and Urban Airship for our mobile users.

Other

We played with some other tech, but those were discarded for various reasons. We initially used Redis for our social network, but in our case, it was just another piece of your architecture to monitor. Our social network now happily lives in our operational database where it belongs. One day, if Postgres buckles under the graph, we’ll reconsider. Then there’s RabbitMQ. It’s beautifully stable and has a sterling monitoring plugin, but once again: it’s one more thing to manage, and besides, Akka mailboxes will suffice for now.

You bet your PaaS we’re using Heroku

Heroku was perfect for getting started. Have an idea? Code it, commit, push, et voilà. No worrying about builds, deploys or anything like that: one less operational hassle to worry about when you’re building features. However, we do have a chunky $5,000 AWS credit. I’m slowly but surely creating Puppet manifests for all our components and Vagrant has been ideal for testing this locally. Aforementioned is still a nice-to-have for the size of our user base, so I’m just hacking at this in the in-between moments.

Conclusion

So, there you have it. Our architecture at a glance. There will be some more detail later down the line. In fact, I’m already drafting a Play! best practices, jotting down everything I’m learning along the way. Questions and comments on the back of a postcard.