Managing Shell Access In A Cloud Environment – Part I

by Ali Tayarani in

One of the hardest things to design in Operations is a scalable authentication system, especially with a dynamic system inventory like we have at Tapjoy.  We to increase flexibility, a reduce time spent by Operations on user management, increases engineer self-sufficiency, and increase security.

In order to accomplish these goals, we have rolled out OpenLDAP.

Read More

Tracking Mobile Phone Usage During the World Cup Final...

by Weston Jossey in

Yesterday’s World Cup Final may have only been between two countries with a combined population of a little over 120 million people, but it affected billions of people all around the world.

Like most of you, we here at Tapjoy caught plenty of matches over the past month, and we were in no way going to miss the final.

A few of us meandered to the corner bar near our house, and sat down to watch the match.  Like so many around the globe, for most of the match, we were glued to the television.  Football is a sport where it feels like any time you look away, you miss something intense.  It’s not a sport that lends itself to mobile surfing or mobile gaming while the match is on.  As it turns out, we weren’t the only ones who had put down their phones.

Right at half-time, alarms started going off in Librato, which is great tool we use for all things metric driven.  I poked around on my phone and our servers had moved into their reserve capacity (we provision for zone-redundancy, so we always have headroom to handle surges), which isn’t a terribly common occurrence.  I opened NewRelic mobile on my phone (another great service we use for app level drill down information), and I saw that our traffic had shot through the roof.

My wife is much more savvy than I, and she immediately remarked, “Wes, it’s half time.  Everyone just hopped on their phones.”  And, sure enough, it was a trend that would keep repeating itself throughout the rest of the match.

At every break in play, whether it was the end of full time, or the brief intermission between the halves in extra time, we saw surges in traffic.  To illustrate this, I grabbed out a graph of network traffic from one of our public facing facets.  The graph is in eastern time, so you can see the different surges coincide with the stops in play.


In all, it was a non-event for our platform. Our capacity handled the significant surge without issue, and our automated systems handled adding new nodes as necessary throughout the match.  No manual intervention was necessary, other than taking some photos of graphs.

Once I got home, I went back and looked up the previous semi-final matches.  The Brazil vs. Germany match followed similar patterns; however, the Wednesday match between Argentina and the Netherlands had one extra wrinkle in it.  Penalties!

It turns out, penalties trigger remarkable dips in temporary traffic.  If people put away their phones during the matches, then they stick them underwater once penalties begin.

The below graph is a look into how the EU reacted throughout the match on Wednesday.  My personal favorite is the gigantic dip that happens as as penalties take place. This graph is in UTC instead of ET.


Not only is the World Cup incredibly fun to watch, but it also makes for some really fascinating graph


Think this is interesting? We're hiring! See our current openings here.

Getting Exposure to 800 Million Chinese Users

by Jia Feng in ,

Why Tapjoy integrated QQ authenticationAs a large ad-tech company with nearly a half-billion monthly active users, it is critical that we provide the best possible experience to our users. We are always looking out on how we can provide a more seamless and user friendly experience for MyTapjoy, which allows users to earn rewards in our partners apps. Read on to find out why we integrated QQ in China.

Read More

API Orchestration At Tapjoy

by Matt Carbone in , ,

At Tapjoy, we’re currently working on a large scale migration to a Service Oriented Architecture (SOA) for all of our core applications. Over the last few years, our core platform has evolved into what has been dubbed the “monorail” - a large monolithic Rails application. We’re now at a point where we’re seriously feeling some of the negative aspects of the monolith, mostly around excessive complexity and testing/refactoring difficulty which results in reduced speed and agility when building out new features.

Read More