Call Me Maybe - Kyle Kingsbury on Partition Tolerance

Slides

Kyle did an insane amount of work on tooling specifically for this talk to simulate network partitions against a number of databases and then calculate data recovery/healing (or lack thereof). The talk highlighted some tightly held misconceptions and really emphasized how databases often ignore the P in CAP in favor of CA.

Bloom - Neil Conway

Slides

Neil presented the work that he and his colleagues are doing at UC Berkeley on creating a language for writing “disorderly distributed programs”. The culmination of the work is Bloom a language embedded in a Ruby gem DSL that allows easy construction of complex distributed programs. Wether or not Bloom is “production ready” is irrelevant - the beauty is in how simply it can render these normally very large distributed programs.

Smarter Caching - Neha Narula

Slides

Neha showed off Pequod, an intelligent caching system she has been developing with colleagues at MIT and Harvard. The central idea was moving the idea of joins from the DB layer up into the caching layer. With a Twitter-like system as an example, she showed how simple range GETs into Pequod could be used to fetch and collate data at the cache level without touching the database. Though Pequod is not open source yet, the concepts are very interesting and I’m sure well see similar implementations in OSS soon.

Just open a Socket - Sean Cribbs

Slides

Sean gave a detailed rundown of how he improved some common problems in TCP client libraries, specifically for the Riak Ruby client. The primary solution/idea was putting load balancing in the client itself. This spoke directly to me as we’ve been doing a bunch of work on this at PP and Sean’s ideas were novel and very straight-forward.

MORE!

There were so many mind blowing talks at Ricon, I feel bad that I’m not writing them all out. I’m sure Basho will publish the videos soon and we’ll all have a field day. My biggest takeaway from the two days was that, at least in this community, researchers and developers are really starting to work more closely together and the rate at which research is being implemented and integrated into production systems is speeding up. As an implementer, this makes me extremely excited. We have an insane thirst for ideas in this field, and the faster we can test those ideas, the sooner we can solve bigger problems.

I gave a talk a couple of weeks ago where I tried to explain the methodology of Paperless Post’s culture of shipping. It is sung to the tune of Da Dip: when I ship you ship we ship. You can check out the slides and video below:

Automate or Die - Aaron Quint from devopsdays on Vimeo.

Also be sure to check out Bethany’s Talk about Sensu

I wanted to explain a little further about our process and why I think it’s been successful. You don’t have to look far to see or hear people talk about continuous delivery and deployment these days. While I fully believe in the tenants and real benefits of all of these practices and systems, I also feel like everyone is missing the point a bit. When I’ve talked to people at conferences, meetups, etc, a lot (definitely the majority in my very small and unscientific test group) are excited about the ideas and potential results of implementing any kind of automated deployment system but are frankly lost as to where to begin. Worse, the effort that has to go into building or implementing such a huge system seems insurmountable. I want to say that this shouldn’t be the case and I think we’ve proved that at Paperless Post.

Iterative Devops

The first assumption that is incorrect about these kind of projects is that it has to be all or nothing. We, and a lot of other teams, when working on our product are most comfortable and successful when taking an iterative approach to shipping features. We split big projects into small deployable steps, look for the 80% solution, and ship one thing at a time. Why then, on the Ops side are we obsessed with silver bullets? Why are we doing large rewrites and replacements of existing infrastructure? For our case we’ve been able to take the same iterative approach with our developer infrastructure. What I was trying to show in the presentation above was that we didn’t get to where we are today overnight. Rather, it’s taken years, and most importantly, each step was worthwhile and an improvement. We’ve been benefiting since the beginning with even the most simple layer of automation that we added years ago. Don’t wait for the ultimate solution.

Pick the right problems

Of course we all want the ‘push a button’ ideal of an infrastructure. One click deploys, scaling, beer delivery. However, as DevOps Engineers and Leaders that might not be what is actually needed now. While the end result of all that kind of automation can not be argued with, that doesn’t mean it’s the most important problem for your team to solve. At PP, we started with the whining.

It turned out that our developer setup process and our git workflow were actually the most difficult and slow parts of our daily process, and hence what was taking the majority of our dev and ops time. So instead of doing what we wanted first (one click deploys), we focused on the pain points one at a time and made them better. It’s important to remember what I see a the real goal of DevOps - to support the developers and the product they build. In the talk, I presented a ‘DevOps Oath’:

I, [YOU], Do solemnly swear to help developers solve their problems, quickly and without [too many] yak-shaves and make their lives easier, so they can ship code, so that the company can improve, so that we all can be successful. I will do what it takes to avoid repeated tasks, help unblock the blocked, and generally make things work better.

Every team is different

A big question I get whenever I talk about our devtools or other internal tools is ‘is this open source?’ To me that brings up a fundamental point of how I hope people can think about deploying automation into their workflow - every team is different. Even though a lot of people see this as a given, they have no problem grabbing and forcing tools and workflows from other companies and team into their own. This isn’t to say that OSS isn’t valuable and doesn’t have a place in Ops. I really believe, however, that in order to really have a process and tools that work, you need to start with how your team works organically, and then gently apply the tools to that process. Teams are living and breathing organisms, and applying solutions that are too abstract or worse too specific (but in the wrong direction) can be disastrous.

Keep moving

Our process and tools continue to evolve as we run into new blockers or new problems. We’re proud of how far we come, but we know that this is a moving target. Our key to success in this realm has been to just acknowledge that fact, and work with it.

This Valentine’s Day, we saw an unpredicted growth in sender traffic and that, combined with some issues I’ll describe below, led to a cascading series of problems that affected many of our users on a very important day. For most of the day, users were unable to get through the full process of creating and sending a card and, as a company (and a Dev, Ops, and Support Team), we couldn’t be more apologetic and sorry for how this affected our users.

The Problem

By 7:25am EST, when our team began to come online, we immediately noticed that cards were taking a longer-than-average time to render, preventing users from sending. This manifests itself to users as “eternal spinny” (as users often call it), which is on the page after you’ve completed designing your card. We poll the server to see if the images for your card have completed rendering. If they have, we show you a sample of the e-mail that will be sent (usually an envelope with the recipient’s name) with a link to a preview of the card the way a recipient will see it. If the images are not ready, you will see a loading GIF that will refresh once the cards are ready. On a normal day, most users will never see the “spinny” as card rendering takes less than 4 seconds total, on average. On this morning, however, cards seemed to never appear. To the user, this perceived “eternal spinny” was actually a failure on a number of levels, and that continued through to the late afternoon, resulting in most cards not going out until much later in the day.

A little background on “Rendering”

When a user saves a card in our design tool, it goes through a multi-step process before the recipient of the card receives it in their inbox and then sees it as a recipient. The basic workflow is as follows:

• User saves card in design tool
• JSON is sent to the server and is serialized into different PostgreSQL tables and relationships
• A JSON representation is stored in Redis and enqueued in Redis for “rendering”
• A server-side “Renderer” process picks jobs off the queue and turns the JSON into a visual representation of the card (an image)
• The image is put back into a Resque/Redis queue as a Base64-encoded PNG
• Resque picks up the image, writes it to a shared NFS storage and writes a pointer to it (called a MediaFile record) to Postgres – this enqueues another job called “MediaFileConversion”
• MediaFileConversion takes the rendered PNG and, using ImageMagick, turns the image into a series of differently-sized and -shaped thumbnails and JPGs used across the site and on mobile web
• Eventually (about 1 hour later) a “DiskCleaner” process comes around and uploads the on-disk files to Amazon S3, marks them as uploaded in Postgres, and then clears the local files from NFS

We arrived at this multi-step process over the years and, though it has some faults and slow areas, under normal usage it is more than fast enough. By making different parts of the worfklow have different jobs and responsibilities, we’ve been able to scale each as needed.

The Initial Failure

When we came online on 2/14, the Renderer queue was already backed up, with over 1000 jobs in the queue. Worse, it seemed like it was growing much faster than it was processing. Initially, we worked on bringing up more Renderer processes/boxes (we run a number of these within our datacenter and a number in EC2), but even this was not helping. Something was wrong with the Renderer processes themselves – even though they were running, they appeared “red” in our dashboard. On examination, the processes were indeed still running but, at some point, they stopped picking up jobs. A restart turned them all green again, but one by one they quickly turned back to red and the queue continued to grow. This was almost definitely happening throughout the week and the day before, but we never noticed because 1) our daily # of renders is small comparatively and 2) we alert when the queue grows large and when the processes die but not when they “disconnect” in this way.

We began messaging our users through a site-wide alert that creating cards was taking longer than normal. Part of the team divided off and started working feverishly to try to fix the Renderer code that seemed to be causing the disconnections. Unfortunately, before we were able to deploy any changes and because of the way we recently started storing the JSON for the cards in Redis, the queue grew so large that Redis itself filled up and ran out of memory. At 9:48am, it was OOM-killed (by Linux “out of memory” process), bringing the entire site down for a few minutes. Because of an issue with our Redis saving/backup strategy when the process auto-restarted, it came up with an older backup, losing our existing queue and a bunch of other realtime data. We deployed a change to our queueing strategy to take the JSON data out of Redis and have the Renderers instead fetch through our HTTP API, but before the deploy finished, the queue filled up again and Redis was OOM-killed again. This time, before we allowed it to restart, we doubled its memory.

At this point, the Renderers were still not staying connected, but we continued to bring up more Resque workers going from our day-to-day scale of 28 processes to 178. We deployed a number of fixes to the Renderer that put layers of rescuing in place so that no matter what part of the process was failing, it would always continue to request the next job. When the last of these was deployed at 3:26pm, the Renderers finally started to pull off jobs quickly and without disconnecting.

Up until then, we continued to deploy a number of changes to our main app to try to reduce the amount of work the Renderers and other workers were doing. We stopped creating thumbnails only used by certain applications and turned down the amount of polling that clients were doing to check for Renderers. We also noticed that API response times were high and increased our Postgres max connections to account for the higher number of worker connections.

Second Wave

Now that Renderers were staying alive and quickly pushing through the backlog, the next process to get bogged down was the Resque job/queue responsible for saving the images to disk. We brought up more and more workers to try to deal with the backlog, but noticed diminishing returns because they were all operating on shared storage. The process of bringing up and reassigning new workers, though involving very little actual work/effort from the Ops team (thanks to Chef), unexpectedly consumed a bunch of time. We had to pull resources from other sources (our staging clusters). Also, changing the queue distribution required a HUP/restart of the workers, which meant the queues weren’t draining as fast as we hoped.

We continued to make changes to try to get the cards to show up faster (changing what images/steps were required for viewing/sending/previewing) – however, the queue continued to grow. Because, again, we were storing full/large data as part of the queue (this time the rendered images), Redis continued to have problems/get close to capacity. To stem the tide, we stopped the Renderers themselves so that the queue did not continue to grow past Redis’s capacity.

Third Wave

By 7pm, we were back to showing cards within a couple of minutes but the third part of the process, the resizing/thumbnailing queue, had over 30K jobs in it. This meant that the number of files were growing on disk because we had diverted resources from our uploading/cleanup queue to the resizing queue. We started to receive alerts for our shared storage as it approached 100%.

We worked quickly to modify our disk cleaner script to be able to work in parallel over multiple boxes. Unfortunately, at 7:50pm, we ran out of disk space on our shared storage, and the jobs to save/render images began to fail. We had to temporarily stop all workers. Once we got the parallelized disk cleaner running, we were able to reclaim enough space to start the workers again. The key was that the (50) disk cleaners were reclaiming space faster then the media jobs were filling it.

Finally

By 10:00pm, we were back to “normal” and we started to re-enqueue cards/emails/etc. that had not yet been processed or were lost when Redis went down. This took a little over 2 hours and, at 11:30pm, we were able to send out an apology email to our affected users. After a very rough day (to say the least), we were gratified with the general response from our users:

“That is the nicest apology I’ve ever received. Happy Valentine’s Day!”

“Thank you so much for your kind response. I am very impressed with your customer service. You offer the best in email stationary and I will continue to be a big fan and customer!!”

“Thank you. Admirable response. I already knew you provided a good, user friendly service. And now it’s clear you follow good business practices. How refreshing!!”

“Thank you for the coins. I am a loyal customer and I appreciate the indication that you prize excellent customer experience.”

Root Cause

We like to think of ourselves as an ego-less team, and this day really proved it. Wherever you want to point the blame based on the details above, every member of the team clearly felt responsible. We worked together, constantly communicating and trying to keep emotion out of it. Our Support team took the brunt of the user complaints and issues, but at no point did that reflect on how the Dev or Ops teams were treated. Similarly, the Dev/Ops teams were under a huge amount of pressure to “fix it” but this pressure was self-inflicted and we did our best to communicate the current status throughout the day.

It’s hard to point at a real root cause with something like this. We clearly had a number of early failures that took the rest of the day to recover from. The biggest failure, though, was really in planning. We knew this day was coming, but were so focused on shipping all the things that we didn’t step back to adequately test or prepare. Lesson learned.

Post-Mortems

We had two separate post-mortems the day after, one going over the accounts of the day in detail with the Ops and Dev teams, and the other with the management team going over the events at a high level. Both were positive and left us with a large number of next-steps and ways to improve every part of our process.

Next Steps

We have a lot to do to ensure something like this never happens again. We’ve organized the main work under three categories:

• Load and configuration testing
  • We need ways to have automated load tests, because currently we are only really ever testing production load in production.
  • We also had issues with broken chef recipes that could have been caught earlier by Chef integration testing.
• Faster deploys/configuration changes
  • Bootstrapping should only take a couple minutes and nodes should “come up swinging” without an additional deploy.
  • Current deploys are O(N), meaning that the more nodes we add, the slower deploys get.
  • We should be able to reassign/reorganize queues without redeploying Chef or restarting the workers.
• Optimizing the Renderer pipeline
  • Getting off of shared storage.
  • Moving other classes of workers (besides Renderers) to EC2.
  • Giving the renderer more/less responsibility in a live, configurable way.
  • Moving to a more durable queueing system.

On days like this, we’re thankful not only for our team, but for our users. Having scaling problems like this is generally a good problem to have, and it’s up to the entire company to improve and rise to the occasion. We knew about some of the problematic areas in our architecture before Valentine’s, but a crush like this exposed them (as well as new issues) in a clearer way. It’s our job to catch and fix these before our users do and we continue to make steps toward this goal.

If you’re not familiar with Spin, it’s a great example of how UNIX fundamentals (namely, fork(2), sockets, and signals) can be applied in Ruby to cleanly and elegantly solve a modern web development problem: making unit testing fast and fun. It does this by starting a Rails process, then forking off each time you wish to run a test. When your test is finished running in the fork and exits, you get to start again with the original and pristine Rails process, ready for your next test.

The gem I developed is named Rails2Preload, and is now available on GitHub and on RubyGems.org. The gem uses Spin’s hook system to patch the Rails 2 initialization method into two phases: preload and postload. We hope you find it useful! Bug reports, fixes, and contributions are welcome.

The author of the Spin gem is Jesse Storimer, who’s also (by no coincidence, I’m sure!) the author of Working With Unix Processes. If you happen to feel like you missed UNIX 101 like me, I’d recommend it.

Space is limited and RSVP’s are required. Please RSVP here: http://paperless.ly/YpLazi

Continue reading for details on his talk, “Coffee and the Art of Software Maintenance.”

Coffee and the Art of Software Maintenance

Coffee is all about two things: simplicity and passion for your craft. I spent a lot of time in coffee shops, talking and listening to baristas about their passion, their craft, about coffee, getting lots of inspiration for my daily work.

Coffee involves two things: building reproducable results, as every coffee brewed should taste just as great as the previous and the next one. The other is getting the customer excited about that precious cup of caffeine-induced ooze and how it came about.

This talk is an exploration of what makes for a great cup of coffee and how really caring about getting the next cup at least just as great makes you a better developer and even better at making customers happy.

Ops is Dev and Dev is Ops

This was our first full year with a subteam of Operations developers on our Engineering team. Johnny, Bethany, Dan, Alex, and Vipul have tackled some gigantic projects in 2012, including a data center migration aided by the automation framework they have been building for over a year. Other highlights of their hard work include the full automation of our development environment with Vagrant, an amazingly solid and scalable metrics infrastructure with Sensu and Graphite, and, perhaps most importantly, a collaborative workflow for Operations work that looks and feels very similar to how we develop Product features for our web application. Our Ops team members write Ruby and shell code, participate in Code Reviews with Pull Requests, maintain Issues on GitHub for long running projects, and manage features with great aplomb.

As a consequence of the insane gravitational force of our Ops team’s awesomeness, Engineers on other subteams are taking Operational concerns into consideration when developing features. They are making changes to the Chef repo for configuration file changes, coordinating after-hours data migrations, and generally sharing the responsibilities that have traditionally been Ops-team-only burdens. This is both a reflection of the excellence of our Engineers and the great work Ops has pulled off. Major props.

Internal Feedback Rules

Besides an investment in a full Operations team in 2012, one of our other biggest bets was on a suite of internal tools that our CTO Aaron Quint started writing basically as soon as he started working at Paperless Post in 2010. The approach we have taken will be covered in some detail as some members of our Ops team present at DevOps Days 2013 in NYC, but the big lesson is pretty simple, and our statement of it here is hardly novel: Internal Feedback Rules. Do whatever you need to do to shape the data that your application and its environment generate in order to make it more consumable. If your logs are too verbose, fix your logs. If your alerting system is broken, fix it. If you can’t find the right one, fix your process, or write a new one that fits.

During the last weeks of 2012, we now have alerts in our Development Campfire room if our acceptance tests aren’t passing in a given browser, alerts in our Operations room if the latency on key pages exceeds a certain threshold, and daily emails sent containing graphs and KPI numbers to anyone in the company who is interested. Our management and Product leadership rely on our developers to record the right numbers, and our developers trust management and Product leadership to help them interpret the results. Data is our common ground, and as we all work together to record, present, and understand it, the entire process improves.

Community Involvement Works

Working hard on awesome internal tools is more fun if you can share the outcomes. We have been really happy to be able to send our Engineers to great conferences such as Strange Loop, Rocky Mountain Ruby Conf, JSConf, GoRuCo, RubyConf, and more. We encourage our developers to submit proposals and we send people to conferences whether they’re speaking or not. The result has been a lot of information sharing, some great networking, and a chance for us to spread the word about our Engineering team.

Besides sending our team all over the place, we have been using our office space as a meeting ground for groups like NYC Ruby Women, the NYC Chef meetup, and a variety of our own internal tech talks. Having people come to our office for beer and pizza to hear people we admire talk about whatever they want has been an awesome experience. Since we are a group of insanely passionate and dedicated developers our enthusiasm rubs off. It’s been a lot of fun to see young developers learn from our guests, and we’re psyched that people now think of us as a company that gives back to the dev community.

Interns Do Not Fetch Coffee

Our third annual summer internship program helped reinforce what is hopefully by now accepted wisdom: hire interns and get them to actually do things, and both parties will benefit greatly. In addition to travelling to recruit interns, we spend a lot of time putting together the program that they go through once they get here. It’s easy to grind on the recruiting aspect and say, “eh, we’ll figure it out when they get here” but that’s simply a waste of your early investment in finding good interns. Having a program ready that involves as much of your company (yes, not just your engineering team) as possible is the easiest way to reach a productive place as quickly as possible. We find promising, passionate students, teach them as much as we can, give them as many chances to be successful as they can, and then sit back and watch them ship. We blogged about the interns experiences here and we can’t wait to get our new crop in in early Summer 2013.

Embrace New Languages and Frameworks

Our main web application codebase celebrated its 4th birthday this year (oh commit 65ac5ab, you seem so long ago). Since four internet years equals around 400 human years, a lot has changed in the interim. Frameworks change, traffic changes, products change, developers change. One of the most important lessons we re-learned in 2012 was that new is not bad. We’ve tried to embrace technologies that will make it easier for us to maneuver our way towards our goals in 2013. This year we’ve introduced a network service using the Go programming language, moved towards Grunt and other modern JS frameworks for managing our JavaScript code, employed KSS and other modern style frameworks for our CSS, and of course hacked lovingly on our Campfire bot that is written in Node.js. We’ve managed to avoid ‘the big rewrite’ by integrating new tools as they are necessary, and it has paid off. Do yourself a favor and kill that three-year-old code that is causing you and your team a lot of stress. It will pay off in the end.

2012: The Year In Some Numbers

Finally, here are some great figures that sum up our awesome year. Thanks for reading our Engineering blog this year, we hope to hear from you soon!

• Unique animated gifs posted in Campfire: 3,208
• Unique metrics stored in Graphite: ~250,000
• Most individual user-generated cards sent in a single week: ~2 million
• Registered user growth over December 2011: 211% (3.1x)
• Revenue growth over December 2011: 135% (2.3x)
• Number of commits to main app git repo: 44,545
• Number of commits to chef git repo: 4,447
• Peak traffic served: ~300 Mbps
• Number of production deploys across applications: 1,631
• Number of staging deploys accross applications: 10,600

Thanks to David Stamm, Michael Hansen, Tarun Chakravorty, and Alexa Hirschfeld for help with the above numbers

Tuesday, October 30th at 7PM

Paperless Post, 151 W. 25th St, 9th Floor, New York, NY

Space is limited and RSVP’s are required. Please RSVP here: http://paperless.ly/VXuyAn

Alex will be joining us to discuss, in his own words, “Architectural patterns for bridging multiple languages. In a world where we have an increasing number of highly specialized programming languages, how do we let programmers use the best tool for the job while working in/across teams?”

About Alex

Alex is a programmer, writer, angel investor, and an advisor to several startups. He recently helped launch Simple, an online banking service with a focus on better technology, design, and customer service. Previously, he was Platform Lead at Twitter, and has worked on projects ranging from online campaigns to information security. Alex organizes the annual Emerging Languages conference, a showcase of new programming languages. He co-authored O’Reilly’s Programming Scala (2009), and has been writing online for over a decade.

About Paperless Post Tech Talks

Paperless Post Tech Talks are our chance to bring developers and artists whom we admire into our New York office to talk and share knowledge about their work with our team and the larger New York tech community. They are open to the public but RSVP is required as space is limited. There will be time to talk and mingle with the other attendees and the speakers over craft beer and snacks.

Caio Lima

My name is Caio Lima, and I recently finished a four month co-op placement here at Paperless Post. I study Mathematics at the University Waterloo in Canada (birthplace of the frail Blackberry) and I make mobile apps.

Besides learning various ways of making coffee, I have learned an astounding amount of new practices and concepts in software design, all while meeting really cool people and watching the company grow. I have had a few different roles as a member of the PP dev team. For the first month of my co-op term I learned about the architecture and internal workings of their large scale web application and was able to contribute to the project by fixing bugs, streamlining our administrative package creation tool, and ‘JSHint-ing’ part of the code base. For the remaining three months, I learned about design patterns for developing Cocoa applications. The main feature I worked on is the new activity page on the iOS app.

The activity page is where users can view an event’s ‘activity’ and acts as a centralized hub for media and interactivity with your Paperless Post invitation. The page layout differs depending if the viewer is a host or a guest. Hosts and guests can see up to four most recently replied guests, ordered by status, and post comments, but the guest’s view can be limited to the permissions set by the host.

The idea is that this page will eventually get more features or modules and that the layout could change as well, so one of the biggest challenges was setting up the page so that it dynamically generates the UITableView’s data source to make it more malleable and easier for future features to be added. When the UIViewController is loaded, there is a data source array that is constructed to hold the layout of the entire view. It holds arrays of the different data cell types that will go in each slot.

Now maybe this was already familiar to some of you, but to me this was new and I think it is pretty darn cool. This feature allows us to generate the structure of the table view with only an array of integers. It is modular, easy to reorganize, and doesn’t require as much data. When there are many simultaneous tasks, this feature could make a big difference in end user performance. Kudos to Paperless Post’s iOS Developer Sal Randazzo for helping me out. This design pattern has really motivated me to look for other ways of implementing designs in Cocoa and I have learned a bunch of cool features in Objective-C. You can also use the activity page feature to post comments to events, which has been successful so far.

Overall, I’m pretty satisfied with how it turned out, happy with how much I learned and pleased with the amount of ‘real stuff’ I accomplished during this co-op term. Time well spent.

Ryder Moody

Hey! I’m Ryder Moody, and I’m a rising junior at Columbia University in New York City. My work at Paperless Post is on the Accounts/Site Structure team, which means that I get to dig into vastly different parts of the codebase. My background is mostly in Rails and Ruby, but because of my position I’ve been able to play with tech like Sammy.js, Redis, our API, and Twitter/Facebook integration.

Much of what goes into making a successful web based company is knowing what the users want before they do. While user facing features often get all the attention, some of the most interesting projects I’ve undertaken at Paperless Post have been things that users would never see or even think about. How do we give them what they want before they ask? At Paperless Post we do lots of analytics, and the more we know about what every user does on the site, the better we can tailor the experience to them. For instance, we know that we need designers to create more cards for wine tastings, yet we’ve never received an email or phone call expressing this. We know this because I worked on an admin dashboard that gives us the ability to track and see what users are searching for, how many results they get, and when they ran the search. This ensures that our designers don’t waste their time making cards that nobody is looking for, and that existing cards get tagged with all the appropriate phrases that people search for.

I’ve also done a lot of work with Google Analytics. We have a robust system that makes it as easy as possible to add tracking to any element of the site. That’s cool, but figuring out exactly what is tracked and what isn’t would be cumbersome without a nice interface, so I created the “Admin Goggles.”

With one click of a bookmarklet, every tracked element of the site is outlined in red, and by rolling over a link you can see the information that is sent to Google. This removes the guessing from finding Google’s nice graphs for a particular button, and helps us in our goal of gaining more insight into what users do or don’t do around the site.

If you think you’d be a good fit for our Internship program, get in touch! jobs@paperlesspost.com

Most recently, I gave a (more technical) version of the talk at the SF Metrics Meetup which you can watch here:

Aaron Quint (@aq) talks about metrics at Paperless Post and Graphiti. from Librato on Vimeo.

and heres just the slides:

Moving the needle

At Paperless Post our business is very driven by holidays and seasons. For the past 3 years the months between Sept-Dec are our biggest traffic months by far. Last year we set some very large goals for ourselves and our team. The biggest goal, that we talked about loudly throughout the company was expressed as a big number - N Million Cards sent in December 2011. We got close, really close, but we didn’t meet the goal. Though we still had a very successful holiday season, the fact that we promised something and failed was really painful. In January, we looked back and tried to analyze what the real failure was and came to some interesting conclusions about the way we were setting goals.

The clearest failure was that everyone felt very little connection to the goal. Though everyone worked their asses off, it was really hard to see how their work was or wasn’t affecting this big number. No one felt truly empowered to move the needle. The second problem was that the big number itself was actually a conglomeration of a lot of different smaller changes in our numbers and no one change could have produced that goal. Moreover, there were two very different very separate processes at work that would drive total success. We realized if we started to separate them out, they could give so much more clarity to our process.

Quantity vs Quality

The two main categories of metrics as we saw them were separated by how and where they changed our totals. If the end goal was growth of the business in general, we realized that we needed to improve in two distinct areas.

Quality metrics measure the quality of the product. This is often what most startups focus on: Adding and refining features and supporting new and existing users. For us, over the past year, it has also been removing and simplifying. Quality metrics are measured in averages of X per Y. For example average Events created by an Account, or average Coins used per Event. Changes to our product, like simplifying the process of creating events can have a great affect on these numbers. We like to think of this as the velocity of our growth.

However, to grow exponentially (as many companies would like to do) you need to add another factor we call Quantity. Quantity is driven by the sheer volume of people going through the product. Quantity metrics are really growth metrics, they are measured in deltas % of X. For example: % of guests registering for accounts, % of visitors to the homepage creating a card. These aren’t numbers that directly effect our bottom line, instead they drive people into the Quality metrics and the multiplication of the two are what produce the big numbers that let us meet our goals.

Metrics Driven

We’ve tried to shift our process to match these ideas. We want not only the management but everyone to really think about these metrics and push them forward. After breaking down the goals in terms of these smaller consumable deltas in ‘rates’ (as opposed to big numbers), we can attach the metrics we want to change to feature specs and issues. This allows us to have clear, measurable goals attached to features.

Measuring

Once the goals are set up and the development process begins, we measure throughout the process. We have a ton of different tools for measuring and we like to think about it in two separate categories: Accounting vs Trending. The differences between these two are really stark.

Accounting are raw numbers that are exact and not-real time. They come from our production data and they are used for making decisions and measuring our progress in very specific ways. We have internal scripts to pull our data out of PostgreSQL into what we call The KPIs (Key Performance Indicators). We also use a data-warehousing tool called RJ Metrics to get cohort and other deeper analysis.

Trending is relative data, it is completely devoid of real numbers and is all about deltas and changes over time. It is real time and lets us get feedback directly after deploying new code. We have some custom built dashboards for this and then also use Graphite and Graphiti (which we have blogged about here).

The difference is we watch the trends, and we study the KPIs.

Constantly improving

For this upcoming holiday season, we are going to attempt to still drive our team to put out their truly best effort without setting everyone up to fail. We’re going to do this by using these different categories of metrics and tools to set smaller achievable goals that each team can actually change. When combined the goal is all the small changes in our growth rates will make this a very successful year.

We’ve been changing our process continuously over the past couple years and some things have worked and some haven’t. The point is that we are converging on something that truly works and its exciting. We hope to continue to share our learnings with everyone as we improve.

Let’s say you design three pages. You want to make an objective decision about which page is most likely to cause a user that visits it to take an action that you like. You think all the pages are great, but sadly, there is no magic algorithm or application you can send pictures of your great designs to be graded for performance. That’s where multi-armed bandit testing comes in. Wikipedia says: “The multi-armed bandit problem models an agent that simultaneously attempts to acquire new knowledge and to optimize its decisions based on existing knowledge.”

We could run an A/B to determine the success rate of the different registration forms. However, we’d need to leave these views up on our site for a long time to gather data, and probably the results would be inconclusive. Also, we’d be showing a large portion of our users something that doesn’t work well. Instead, we decided to use a machine learning algorithm that exploits the results of the test while the test is still being run.

Multi-armed bandit testing works by choosing the “lever” with the highest expectation of reward. This means that the registration form with the highest probablity of converting a visitor to a registered user is shown the majority of the time. However, we also stir things up a bit and show a random view 10% of the time. This prevents the test from getting trapped at a local maximum (and also guarantees that we don’t only show whichever view first converted a visitor).

This strategy has two benefits. First and foremost, we lose the fewest amount of users to UI that isn’t optimal. After a short initial learning phase, 90% of users are seeing what is currently the best thing you got. Secondly, this serves to verify that the view the system thinks is the best IS actually the best. The test is still running, and gathering data for that particular choice, and it’s gathering the most data around the best choice. You get a high degree of certainty that your registration form works.

Additionally, if after running the test for a while you want to throw another choice into the mix, you can do so without ruining your results because the data you have gathered for each choice are independent of each other.

We hope to use this more widely across the site. In addition to building a library that makes it easy to store trial and reward data in Redis, we also put the data right on our internal dashboard, so that product owners can place bets on which horse is going to win the race.

Here are some of the results for the test running on our splash page. In this test we have three choices, “email”, “facebook”, and “email_form”, which represent three different views we’re displaying to the user. Our implementation yields two statistics for each choice. The first is “successes”, which measures the number of times a choice has been rewarded, meaning in this case that a user registered with that view. The second is “trials”, which is the total number of times that particular choice has been displayed to a user. Percentage is just a measure of successes over trials, giving us the conversion rate for each individual view.

In this case, “email” is the most successful, with a conversion rate of 2.34%. Because it’s currently the most successful view, it’s being shown 90% of the time, which accounts for the large number of trials relative to the number of trials for “facebook”. Looking at these results, we can tell that at some point in recent history “email_form” was the most successful, amassing about 130k trials. This highlights one of the benefits of running multi-armed bandit testing: since “email_form” was the most successful, it was subject to increased scrutiny by the system through more trials and ended up not being the best choice after all.

You may work for a big company, but you couldn’t possibly get your UI in front of more eyeballs without testing like this. 300k users can’t be wrong!

Thursday, August 23rd 2012 at 7PM

Paperless Post,
151 W. 25th St., 9th Floor,
New York, NY

Space is limited and RSVP’s are required. Please RSVP here: http://paperless.ly/S4UhlH

Tiny Code

Alan Kay famously described a half page of code in the Lisp 1.5 Programmer’s manual as “Maxwell’s equations of software”. If you’ve ever built a small Lisp interpreter the amount of semantic power is pretty surprising. But the question remains does “small” equate with “toy”? As engineers we’re consistently fighting complexity - much complexity comes from the sheer amount of code that we’re required to manage. Is it inevitable that code loses clarity as it evolves? I’ve spent nearly two years understanding two pages of code - while I have no definitive conclusions perhaps some of the principles and approaches I’ve discovered can be applied to software development.

About David

David Nolen is a musician and curious programmer living in Brooklyn and currently employed at The New York Times. He has been writing JavaScript applications for nearly seven years, past projects include work for Princeton University, platial.com, The Modern Museum of Art, and shiftspace.org. JavaScript brought him to Lisp and he’s been closely involved in the Clojure community for about three years. When he has a free moment he gleefully fixes bugs and improves the ClojureScript compiler in hopes that he can give back something fun to JavaScript.

About Paperless Post Tech Talks

Paperless Post Tech Talks are our chance to bring developers and artists whom we admire into our New York office to talk and share knowledge about their work with our team and the larger New York tech community. They are open to the public but RSVP is required as space is limited. There will be time to talk and mingle with the other attendees and the speakers over craft beer and snacks.

At first we was a bit confused that people didn’t understand what was so cool about it, until we realized that we never took the time to explain it. In this quick post, we’ll reveal three awesome ways we use Graphiti that wouldn’t be possible without Amazon’s Simple Storage Service (S3) or an equivalent place to serve and store image snapshots.

Several Points in Time

It becomes clear very quickly when you start using Graphite to record data that although it’s straightforward to get information relative to NOW, point in time graphs are harder to come by. Instead of building an extra Javascript or Ruby layer that could interpolate this information for us, we decided to take a somewhat low-tech approach, and added this route to the Sinatra application that powers Graphiti’s backend:

1
2
3
4

post '/snapshot' do
  filename = Graph.snapshot(params[:uuid])
  json :filename => filename
end

The implementation of Graph.snapshot shows what’s going on here: we’re grabbing the Graphite URL, saving the body of the response (the image) as a file, and then uploading it to S3. This allows us to provide each graph with the following dropdown menu:

which lets you recall, view, and share “frozen” point in time graphs that represent what you’d expect them to represent. This way whenever a graph is interesting for some reason, and we want to share it, it is also automatically persisted: a record of it is saved in Redis, and the image is uploaded to an external service. We have a historical record, as above, of interesting graphs. It’s been extremely helpful to see what the differences are between a graph saved a few weeks or months ago and one saved today.

Another obvious benefit to graph snapshotting became clear as soon as we implemented them: we’d be able to view them in places where we didn’t have access to the VPN that our Graphiti instance sits behind. This gave us the opportunity to implement one of our favorite Graphiti related features - daily emailed graph summaries.

Email Reports

Every morning any Paperless Post employee who is subscribed to the Graphiti email list we’ve set up gets an inbox full of graphy goodness:

Each of these emails represents a Graphiti Dashboard, and contains snapshots from that morning of each of the graphs in the dashboard:

This has been very valuable for us across a wide swath of our company. People following product numbers check out product-related graphs, Operations people follow systems-related graphs, and developers who have created dashboards to track data for features can follow graphs they’ve implemented. We all have a common discusion point for the large variety of metrics we collect, and email inbox delivery is appropriately more passive than actively having to visit the site to see the graphs. It’s become a part of many of our morning rituals to check in on our graphs and bring up any issues that might pop up. It greatly aids in correlating seemingly disparate events, and gives us more information, which is always what we’re striving for.

Sensible Chat Logs

The final point here is sort of a humorous one. Immediately after getting data flowing into Graphite, of course we wanted to share our graphs in Campfire. The first time we ever went back into our history to look at a graph we had pasted, we noticed that the graph was strange. It was now relative to the time we were checking Campfire logs, not relative to the time it was first created. So now instead of pasting Graphite links, we post snapshot links:

and when we review Campfire history, the graph history is preserved. This saves us a lot of confusion and keeps Campfire’s narrative history straight.

…but we’re flexible

There is an open issue on Github to remove the S3 dependency from Graphiti. The reason we’ve been hesitant to do this is simply that while having S3 credentials in a config file is a requirement to run the application, they don’t need to be “real.” You can insert some dummy credentials, and then just not use the snapshotting features as outlined above. Our hope is that we’ve been able to explain a little bit better why the decision to include a connection to this service was made, and that you’ll think they’re cool enough to try out. Regardless, we hope you enjoy using Graphiti, and if anyone wants to finish up the patch open on Github to provide alternate storage backends for the images, we’re flexible.

Thursday, May 24th 2012 at 7PM

Paperless Post, 151 W. 25th St., 9th Floor, New York, NY

Space is limited and RSVP’s are required. Please RSVP here: http://paperless.ly/JVA8eG

Sure everyone knows how great and useful github.com is, and all the awesome features and team members that go into making it what it is. But we only see the surface and inside of GitHub an amazing amount of work goes into internal hacks and tools to make the daily lives and work of its team fun and extra-productive. We’re putting Zach to the challenge of showing off some of both the well known and completely unknown hacks, how they work, how GitHub uses them, and how we can learn from their experiments.

About Zach

A Ruby developer with sound fundamentals, firm grasp on the industry, and innovative development approaches are all phrases inapplicable to Zach Holman. He works at GitHub, and hacks on sparkline generators, robot music DJs, and ethically frightening FaceTime + Chatroulette mashups. He blogs, he tweets, he evades his taxes.

About Paperless Post Tech Talks

Paperless Post Tech Talks are our chance to bring developers and artists whom we admire into our New York office to talk and share knowledge about their work with our team and the larger New York tech community. They are open to the public but RSVP is required as space is limited. There will be time to talk and mingle with the other attendees and the speakers over craft beer and snacks.

The History of History

We have not been sleeping for the last three years, of course. Undo history has been in the app, off and on, in various forms, for a long time. Until recently, text input could be undone with Ctrl-Z, although it wasn’t redoable. We switched from per-character undo to batched input undo over the summer, with a little help from the _.debounce feature of underscore.as.

Meanwhile, on the JavaScript side of the application, we had a state-based undo system, which handled every other kind of user action. Two undo systems—odd, but emblematic of one of the key design decisions behind the card creation app.

The Hybrid

Early in the summer of 2011, we switched over from our original pure-Flash card creation app to a new approach we call “the hybrid”: a JavaScript UI wrapping a Flash viewport. Visually, JavaScript handles all the HTML controls outside the viewport, while the Flash viewport renders the card and handles UI interaction that happens within it, such as dragging a photo. More importantly, our JavaScript code can be said to drive the app: it has modules which interact with the server to maintain a canonical card state, and it relays that state to Flash as needed. In turn, it interprets events from Flash and updates the server when necessary. Flash and JavaScript sync constantly, since local communication is cheap; but JavaScript hits the server only when necessary.

Our goal for this separation was to permit advanced Flash rendering techniques such as fast photo filters and high-quality fonts, while doing most UI elements in HTML. JavaScript, HAML, and Sass make UI updates a snap. What’s more, we can easily replace the Flash viewport with a pure HTML5 solution where appropriate… but that’s another blog post.

State Snapshots

When implementing an undo history, having two codebases on two runtimes makes life harder. Crucial state is split across runtimes. Even though JavaScript is the canonical source of model data on the client side, plenty of other state is segregated, including things you might not think of as “state”: object references, event handlers, the starting values of in-progress slider interactions.

When we first tried to implement an undo history, we assumed these issues could be brushed aside. We need to restore object references and event handlers upon undoing a delete? Recreate them! We need to get a delta (a single change amount) from a slider drag (dozens and dozens of little events)? Assign the drag a transaction id! JavaScript should be the sole source of all app data, not just raw model information, and so we should be able to revert to any previous state by loading a snapshot we’ve stored as a JavaScript string.

This may sound cumbersome, but it worked pretty well in practice. Many actions could be undone just by piping model updates from JavaScript to Flash. But although this system showed promise, and its outstanding bugs could have been resolved, we knew we wanted a much more incremental and flexible approach. Instead of serializing the entire application state on every user action, and rewriting the entire application state on undo, why not store only the relevant delta? This would use fewer resources and isolate the effects of each undo step.

Design Patterns to the Rescue

After some internal discussion, we settled on the classic undo implementation: the Command pattern. I first used the Command pattern in Java land, and although I respected its power, I dreaded the tedium of defining customized command classes. In the traditional Command implementation, there might be one for every possible user action, each containing copies of the state necessary to reverse and repeat the action. As I thought about it, however, I realized that JavaScript and its more straitlaced cousin ActionScript 3 were well-suited to the pattern—considerably more so than Java and C++, where the pattern accumulated its original mindshare.

Commands for Controls

There are a few immediate motivations for the pattern in UI programming. One is to attach a single procedure to multiple UI elements—a menu item, a toolbar button, and a keyboard shortcut, for example—with more flexibility than event handlers, such as runtime remapping. In a language with first-class functions, such as JavaScript, this handles itself; so do many other behavioral patterns, such as Strategy… up to a point.

Another is to modify a button’s action bit by bit in response to other factors. Add a Command to a form’s submit button, and alter the command state as the user plays with the form. Sure, the button handler could just read the form, but if you want to divorce command execution from the form code entirely, this is a good architectural choice.

But the overriding reason to use the Command pattern is to implement undo and redo.

Alternatives to Commands: A Chilling Dystopian Hellscape

Without the Command pattern—without a way to visualize actions as storable objects—only two sensible approaches remain. You can save coded representations of the actions to be performed and revoked, such as strings to be interpreted later; or you can save snapshots of the entire state of your app.

Both approaches can be effective. An array of simple action codes such as “element 123abc color = #ff0000”, or more computer-readable data objects such as {targetID: "123abc", action: "move", x: 120, y: 0}, will work for a long time, especially if you write a simple interpreter for such commands. But once you need to undelete a complex object, your undelete command must suddenly contain all the commands necessary to rebuild the object from scratch; or you begin to cache deleted objects for potential resurrection, and must perforce remember to purge them when discarding a history branch.

Keeping a snapshot of the whole state is simple enough, as long as your model is amenable to serialization in the first place. But unless you’re restoring a whole VM state, or rebuilding a purely noninteractive document, it becomes a real task to ensure functionality in all situations. You need a strong canonical source of state data—which we had, in the JavaScript session state. The snapshot approach had legs. But it lacked the one thing we really wanted: isolation. A history event involving text should be replayable even if an unrelated photo has been changed elsewhere; but with a snapshot approach, this was impossible.

The Simplest Command

The classic approach, we decided, was best. A Command, in this pattern, is just that: an object representing a command to be executed—or reversed—by the application. Commands are not UI actions: a click is not a Command, but a click can trigger a command. Commands are not controller or model methods: record.delete() is not a Command, but a Command can invoke that method. At minimum, a Command is an object which has a single execute method. For our purposes, it must also have undo. This is especially simple in JavaScript, where we don’t even need to make a class:

1
2
3
4
5
6
7
8
9

var command = {
  execute: function() {
      releaseTheHounds();
  },

  undo: function() {
    deployDogCatcher();
  }
};

Making Dumb Commands Smarter

“It’s like undo is smarter than do” —overheard on Campfire

Of course, in this example, we assume that releaseTheHounds and deployDogCatcher are global functions that take no parameters. In any real application, we will have commands which alter the program state. Their undo implementation must know what state to revert to. In Java, it was common to write subclasses for each type of command, like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public class ColorChangeCommand extends Command {

  private Potato potato;
  private int oldColor;
  private int newColor;

  public ColorChangeCommand(Potato potato, int newColor) {
    this.potato = potato;
    this.oldColor = potato.getColor();
    this.newColor = newColor;
  }

  public void execute() {
    potato.setColor(newColor);
  }

  public void undo() {
    potato.setColor(oldColor);
  }
}

Then we would use it in an event handler like this:

1
2
3
4
5
6

public void handleColorClick(ClickEvent event) {
  Command command = new ColorChangeCommand(
    PP.getApplication().getCurrentPotato(),
    event.target.getColor());
  command.execute();
}

This subclass is simple, but as you start doing more complicated things, subclasses proliferate, and things get out of hand in a hurry. There are tricks to manage that complexity, but once you start using extra patterns or even full-fledged frameworks, well, “now you have two problems”.

JavaScript and ActionScript 3 sidestep the whole issue. Functions in those languages are closures, which means that they retain references to everything that was in scope when they were defined. All you need is a simple hash to hold a couple of closures, and you’re done. Here’s the above example, as we might write it for Paperless Post.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

function handleColorClick(event) {
  var potato = PP.Current.potato;
  var newColor = $(this).attr("data-color");
  var oldColor = potato.color;

  var command = {
    execute: function() {
      potato.color = newColor;
    },
    undo: function() {
      potato.color = oldColor;
    }
  };

  command.execute();
}

In this case, all the information we need is defined within the scope of handleColorClick; and since that scope is available to the nested functions execute and undo, it remains available to those functions whenever they happen to run. Already you can see a gain in simplicity. As long as you have a good grasp of function scope and closure mechanics, which should be natural for any JavaScript programmer these days, the simplicity stays even when you’re storing complex objects in those closures.

Undo, Redo, Unredo, Reunredo, and Unreunredo

JavaScript and ActionScript make it easy to create a command object along with all its state. The history of the app state—assuming it is entirely user-driven, or that a user-driven component can easily be separated—can be expressed simply as a linked chain of commands. The obvious and correct approach is to embody history as an array, which is used as a stack: push an executed command, pop a command in order to undo it. Push undone commands onto a redo stack; pop those commands one by one to replay history, pushing each redone command onto the undo stack.

To put it another way, as we travel forward through time, we leave behind us a series of Commands: crystals of solidified intent. When we hit Undo, we take a step back, pick up that crystal, give it a solid rap with our magic wand, and time goes backward. Then we set it back down in front of us, on the redo stack. If we go forward again, we tap the crystal and it repeats our undone action for us, making time whole again. But if we do something else, instead, every crystal ahead of us—our redo stack—disappears in the proverbial puff of logic.

Cross-Runtime Issues

This stack-based approach is obvious and correct… until you start dealing with two codebases on two runtimes. That’s when things get tricky. A lot of interactions involve JavaScript simply sending model updates to the Flash viewport, which duly renders them; but when an action originates within Flash, such as a photo drag or text input, JavaScript simply doesn’t have enough information. Text input must be stored as commands holding sophisticated state such as text patches and formatting object fragments. Sending that kind of data to JavaScript would add complexity we don’t need, since we’d need code on the JavaScript side to incorporate fragmentary data into the model. Why implement text patching in JavaScript when the text is only visible in Flash?

My second thought was to give each runtime its own history, with special commands used to hand over control of the undo and redo buttons—literally swapping event handlers, actually, depending on which runtime was “up” to undo the next command. I was thinking of coroutines: the entire command history could be visualized as cooperative multitasking. This quickly led to the approach I actually used.

I didn’t write code for bad approaches, just notes. Never underestimate the debugging power of a notepad!

Main and Subordinate Command Stores

JavaScript was already the canonical source of model information on the client side, the anointed speaker to the database. It made perfect sense to canonize its command store as well, making it the sole keeper of application history. At the same time, we wanted to handle Flash commands within Flash: a photo resize, for instance, involves factors JavaScript should not be aware of, such as the boundary contraints which keep an image in its frame during resizing. User actions performed entirely within Flash should be undone by Flash, even though JavaScript knows the score and calls the tune.

The solution was simple: keep two command stores. On the JavaScript side, the twin undo and redo stacks; on the Flash side, a hash of commands. Commands originating in Flash now do this:

1. User performs action.
2. Flash creates a command which can undo/redo the action.
3. Flash calls historyUpdate on JavaScript, passing the command as an argument.
4. JavaScript creates a command which calls undo(uuid) and redo(uuid) on Flash.
5. JavaScript assigns a new uuid to that command.
6. JavaScript adds the command to its undo stack.
7. JavaScript returns the uuid to Flash; this all took place during execution of historyUpdate.
8. Flash sets the uuid on its own command object.
9. Flash stores that command in its own command hash, keyed by uuid.

In effect, JavaScript maintains the entire command history, but some of the commands delegate directly to commands on the Flash side. Flash stores the commands as an unordered hash, relying upon JavaScript to maintain the sequence.

Breaking with the Past

Sometimes we need to “break” history: destroy some of our stored commands and start over. Any time the user performs a new action, for instance, our redo stack must be cast into the void. Your application might have actions that cannot be undone; when the user opens that door, her history shatters. Depending on your application’s demands, executing a history break can be complex: you need to purge those commands, make them all available for garbage collection, and along with them, any object references and event listeners that might still hold some obscure yet tenacious tie to the outer world. Orphaned event listeners are like samurai ghosts, locked in this world past their time, forever waiting to fulfill an obligation that does not come.

And sometimes, adding history will break your app. If you design with the naive assumption that time travels in only one direction, you will paint yourself into a corner. Adding undo history to our application was not just a matter of wrapping UI handlers in command generators. Every one-way state transition had to become reversible; every act of creation had to contain the blueprint for its own destruction. But by the same token, no object could be deleted without provision for its recovery. In the end, we touched nearly every aspect of the application to implement an undo history, and turned up dozens of new and interesting bugs in the process. If you’re writing a new app that might benefit from undo/redo, I strongly advise building it in at the start.

The Future of History

Undo/redo has gone live, but development is not over. Undo history is an ongoing commitment: every new feature has to tie into the system, and every change to the application can alter the resource usage of the command stacks. But it will be worth it. Undo history is hard to get right, but it’s a huge usability improvement our card senders will notice and appreciate.

I’ve written a blog series to convince you otherwise.

These concepts can be added to everyday code without twisting it into some confusing recursive sculpture, and they can make common tasks easier, faster, and even produce more readable code. The first four posts use examples written in ActionScript, but the concepts are precisely identical in JavaScript; they even use the same library, underscore.js and my AS3 port underscore.as.

Read the whole series here. If you find it helpful (or confusing), let me know.

Though I’ve been using JavaScript professionally for quite some time, and I keep up with the community at large, I have never been to a JSConf before, and many of the people at the conference were new to me. Traveling with Paperless Post CTO and JSConf Alumnus @aq certainly helped, but the openness and accessibility of the conference venue and its participants certainly would have made this conference approachable and fun for anyone.

Instead of attempting to recap the entire experience, which isn’t possible, or even trying to cover all of the amazing talks, which would also be difficult, I thought I’d highlight a few of my favorite moments which I think will properly speak to the quality and success of the two-day conference.

The Return of Play and a Standing Ovation

The community of developers who congregate a few times a year to celebrate JavaScript and all of its incarnations is a thoroughly modern bunch. They are creating cutting edge web application architecture stacks, trying to overthrow the hegemony of native mobile software, and writing 3D graphics simulations in fewer characters than I’ve used in this sentence. While this penchant for the next leads to a lot of innovation and progress, it can also manifest itself in a repetitive way when presented in the conference setting: “Here’s this cool thing I did! It doesn’t really work, but check it out, I did it!” or “Here’s this thing that will certainly be the best thing ever!” are all too common.

Two talks were stark contrasts to this theme, in that they presented ideas that rooted themselves firmly in the trajectory of computing history, and tried to focus as much on ideas as implementation. David Nolen’s talk about Jelly Stains was a trip through the history of his various programming language love affairs, and Daniel Henry Holmes Ingalls, Jr.’s presentation on The Lively Kernel was a hilarious and surprising retelling of the manifestation of a lifetime’s worth of work.

Jelly Stains was a talk about how David Nolen, a JavaScript developer at the New York Times, by chance stumbled upon a series of Computer Science books which manages to share two qualities not often found in the same text: playfulness and denseness. The Little Schemer, The Seasoned Schemer, and The Reasoned Schemer take the reader through a conversational and philosophical journey through the Scheme programming language, a “minimalist LISP” that is also the deus ex machina of another hallmark Computer Science text, The Structure and Interpretation of Computer Programs. Nolen talked about how a relatively unknown “third paradigm” of Programming Languages, Relational or Logic Programming, changed the way he thought about problem solving, and the defining role that the freedom to play had in his transformation. He then took us on a trip back in time, sharing with us the notorious Zebra Puzzle, and illustrated with ease the stark contrasts between Functional, Object Oriented, and Logic Programming. Nolen showed some recent experiments with ClojureScript, which brings Clojure syntax to the JS runtime, and talked about implementing a logic engine in Clojure called core.logic. A trip that started with Javascript and ended with Nolen working alongside his idols was routed heavily through playful terrains that were radically disconnected from the work he was doing in a professional context. This is a lesson we can all use.

The Lively Kernel is “a new approach to web programming” that was founded at Sun Microsystems in 2006 by Dan Ingalls and Antero Taivalsaari. In case Ingall’s name isn’t familiar, read up about his history with the Smalltalk programming language and advances in Virtual Machine technology here. During his talk, Ingalls took the JSConf audience through a tour of the Kernel, which is quite aptly named. Illustrating that every aspect of the system is “alive,” and that it exists in order to make other connections possible, Ingalls used it to make pianos play, elephants dance, and graphs fill with data. His ease at the podium was highlighted by several exhortations that “that’s how it should be,” noting that he has spent a large part of his career trying to make live systems de rigeur, and that our pleasure at seeing things “just work” should be a catalyst for us to actually get up and make them work. I was very impressed that the JSConf community overlooked the lack of visual polish in this presentation and that everyone seemed to gain easy delight from the fact that the simple interactivity obviously rested on a deeply complex and well implemented kernel. Ingalls gave the crowd the gift of vision, and the crowd returned the favor with a well-deserved standing ovation.

Embracing the Low-Level

In addition to the fanciful-but-technical talks, there were a few technical-but-fanciful talks which also went beyond the stereotypical areas of interest for JavaScript programmers. In fact it should be noted that after JSConf it’s no longer worthwhile to harbor stereotypes about “JavaScript programmers” at all – they’re doing all kinds of things you’d never expect. Blowing straight past frameworks, libraries, and even browsers, these talks focused on the Virtual Machines that make JavaScript run so ubiquitously. Stephan Herhut from Intel gave a presentation on Parallel Extensions for JavaScript, while Vyacheslav “Slava” Egorov asked the question Can V8 Do That?

Each of these talks were particularly interesting in how they walked the audience through the process of looking at code in JavaScript and seeing how it manifested itself by the time it got to the assembler code that the Virtual Machines use to “speak” to the host OS. Herhut demonstrated the River Trail parallel data structures that Intel is experimenting with, which could allow for the design of concurrent JavaScript applications that didn’t rely on process concurrency. The idea that a JavaScript developer could look at code that needs to be written and reason about it in terms of data concurrency semantics is a very encouraging one. It means that parallel execution of code paths could be a reality in modern JavaScript implementations, and that developers could write code to execute this way without having to do more than choose the right data structure and model their algorithms accordingly.

Can V8 Do That? gave the audience a chance to peer into the mind of one of the hackers doing work most relevant to modern JavaScript implementations, as presenter Slava Egorov is a member of the V8 team in Aarhus, Denmark. V8 is the JavaScript runtime that both Google Chrome and node.js use to interpret JavaScript code. Egorov took a gentle approach with the audience, encouraging us to not be afraid of the assembler code he displayed on slides, instead telling us that the patterns we saw and explanations he give could lead us to logical conclusions. The basic question is one that is common to developers in any language - to optimize now, or to optimize later? The twist here is that V8 does a lot to optimize our code for us, and Egorov used this fact to debunk some common optimization strategies (such as caching array.length()) and to show us other areas where V8 doesn’t optimize as successfully. The message was a clear and hopeful one - write your code, reason about it, and read up on what V8 can do - it can make you a more confident programmer who writes more efficient code.

Standing in Circles and Nodding

The one thing that I had heard about JSConfs that proved to be very true was that the experiences outside of the conference rooms would be as valuable as the talks themselves. The conference in general has an air of accessibility and in general doesn’t take itself too seriously, and this lends itself to lots of casual conversation. I cannot stress enough how many of the most fun and enlightening moments from the two day conference came from standing in circles with people and chatting. At one point I found myself standing around with beers or coffees in hand with Brian Ford of Rubinius, Slava Egorov, Rick Hudson of Intel, and David Nolen, chatting about the cutting edge Garbage Collection technology used by Rubinius, the JVM and V8. All I could do was smile and nod, knowing that I was in the company of brilliance, and feeling lucky enough to get a glimpse.

What’s even cooler is that when you looked around at JSConf, these kinds of conversations were happening everywhere – speakers made themselves available after their talks or at the sponsored parties, and in general, a congenial air of collaboration amongst peers dominated.

Hope for “the language with a bad upbringing”

After Dan Ingall’s rousing demonstration of the Lively Kernel, I asked him a question about something he hadn’t addressed – what are his opinions and thoughts about working with JavaScript as a language? Since I know he was involved in the implementation of Virtual Machines and was obviously a deep thinker with regard to evaluating Programming Languages, I thought he would have some brilliant insights. His response was more or less that he obviously loved it or he wouldn’t be working with it, and that although JavaScript “had a bad upbringing,” Brendan (referring to Brendan Eich, credited as the creator of the language) had “done a great job with it.”

At first I wanted him to dig some dirt and maybe get a little controversial, but upon reflection, Mr. Ingall’s words depict a hopeful future for the language. I showed up at the conference expecting a lot of whizz-bang, and ended up with a lot of deep ideas. People continue to do astounding things with JavaScript, despite it being a language that is not without its warts and technological and political constraints. Between these speakers and others, the atmosphere, the voracious minds of the community– it’s difficult to leave this conference not convinced that there is a strong foundation and a bright adulthood ahead for JavaScript.

If you’re interested in Paperless Post’s commitment to professional development, we’ve posted about going to conferences before, and keep in mind: we’re looking for more talented people in NYC and SF.

Some Resources

• Daniel Friedman’s Work, including the Reasoned Schemer
• The Zebra Puzzle that David Nolen spoke of
• The Lively Kernel that Dan Ingalls demonstrated
• Intel’s River Trail as provided by Intel
• Google’s V8: an overview

Update 04/18/2012 - The photos from the booth sponsored by Paperless Post are up! Check them out here!

Wednesday, March 28th at 7:00 PM

Paperless Post, 151 W. 25th St., 9th Floor, New York, NY

Space is limited and RSVP’s are required. Please RSVP here

04/18/2012 Update: Greg’s Slides are now online!

About Greg:

Greg Borenstein is an artist and teacher in New York. After years as a web developer and musician he quit to attend grad school at the NYU Interactive Telecommunications Program where he is now a Resident Researcher. His artistic work explores the use of special effects as an artistic medium. He is fascinated by how special effects techniques cross the boundary between images and the physical objects that make them: miniatures, motion capture, 3D animation, animatronics, and digital fabrication. Greg is the author of Making Things See: 3D Vision with Kinect, Processing, Arduino, and Makerbot, published by O’Reilly.

From Greg:

I’ll present a survey of some of my recent work with the Kinect and then talk about two active projects in my current research. The first project explores the use of face tracking and recognition to substitute for QR codes and other fiducial markers in computer vision applications. QR codes are ugly and unusable and replacing them with images of the human face offers a number of interesting design and interaction possibilities. Secondly I’ll talk about a new project I’m undertaking called Makematics. With the rise of 3D scanning, digital fabrication, and advanced computer graphics the field of creative coding requires more and more knowledge of mathematical fields such as linear algebra, differential equations, and topology. There are few resources available for competent programmers (let alone artists and designers) to learn the parts of these fields they increasingly need to do interesting and original work. I’ll talk about some ideas for how to improve this situation and why I think it’s important for the future of art and interactivity.

About Paperless Post Tech Talks

Paperless Post Tech Talks are our chance to bring developers and artists who we admire into our New York office to talk and share knowledge about their work with our team and the larger New York tech community. They are open to the public but RSVP is required as space is limited. There will be time to talk and mingle with the other attendees and the speakers over craft beer and snacks.

Looking forward to seeing everyone in Scottsdale!

Text customization is a big part of our card creation process here at Paperless Post. We want to enable users to easily create beautiful and unique cards. One feature we thought would be cool would be for the user to be able to foil stamp or gold emboss text, which would give text a texturized silver or gold effect.

When assessing how to implement these text effects, we needed a solution that wouldn’t interfere with any existing customization functionality and would also fit easily into our pipeline without slowing everything down. Essentially, our goal was to color the text with the gold or silver texture images, or mask the images with the text.

Currently we are using a simple Flash text field to render all of the text within a card. Flash supports a wide array of options for graphical customization and bitmap filters. One possibility was to add the image to the stage, put the text over it, and either make the text a mask or change the blend mode of the text field. The issue with a solution like this was that it prevented us from having gold, silver and plain text within a single text field. There didn’t seem to be a way to achieve the effect we wanted using a technique like this without overcomplicating everything with duplicate text layers and strange layering hacks.

What we needed was a way to intelligently replace certain text with the texture, while ignoring other parts of the text field. A graphical effect with logic? Sounds like a job for a shader. Flash supports pixel shaders using their proprietary Pixel Bender software. In order to apply a shader written using Pixel Bender, you need to import the shader byte code, create a shader object with the byte code, create a shader filter from that shader object, and finally pass that filter into the filters property of a Flash display object.

1
2
3
4
5
6
7
8

  [Embed("myShader.pbj", mimeType="application/octet-stream")]
  private static var MyShader:Class;

  ...
  var shader:Shader = new Shader(new MyShader() as ByteArray);
  // Set shader parameters here if there are any.
  var shaderFilter:ShaderFilter = new ShaderFilter(shader);
  aDisplayObject.filters = [shaderFilter];

This code loads the byte code for the pixel shader ‘myShader.pbj’ and then passes it into the shader constructor, which creates the executable shader object shader. The shader filter is then created with the executable shader. This shader can then be passed to the filters array of any display object, which will cause the pixel shader to operate on the display object.

The way we implemented our gold and silver text effects was by choosing certain colors that we did not use in our card creation app. When the user selects the gold and silver text effects for a segment of text, that segment of text is assigned a certain color that corresponds to a gold or silver effect. The shader we wrote only targets these colors, and if it is able to find the colors, it replaces that pixel with a sampled pixel from the texture image. This is essentially a smart mask or intelligent blend filter.

Our shader takes in colors and texture images, with each color corresponding respectively to the texture image. So if the pixel shader finds a color1 it will replace it with a sampled pixel from textureImage1. We sample from the image using a 3x3 grid, from an input image called foreground.

1
2
3
4
5
6
7
8
9

  float4 c00 = sampleNearest(foreground, float2(pos.x - 1.0, pos.y - 1.0));
  float4 c01 = sampleNearest(foreground, float2(pos.x, pos.y - 1.0));
  float4 c02 = sampleNearest(foreground, float2(pos.x + 1.0, pos.y - 1.0));
  float4 c10 = sampleNearest(foreground, float2(pos.x - 1.0, pos.y));
  float4 c11 = sampleNearest(foreground, pos);    // center pixel
  float4 c12 = sampleNearest(foreground, float2(pos.x + 1.0, pos.y));
  float4 c20 = sampleNearest(foreground, float2(pos.x - 1.0, pos.y + 1.0));
  float4 c21 = sampleNearest(foreground, float2(pos.x, pos.y + 1.0));
  float4 c22 = sampleNearest(foreground, float2(pos.x + 1.0, pos.y + 1.0));

We sample using a 3x3 grid because we want to replace antialiased pixels that may not have the same color value that we are looking for, but should still be replaced. Because antialiased pixels will be adjacent to the true color of the text, we check whether any color in the 3x3 grid is the target color. We then check to see which sampled pixel is closest to the target color.

1
2
3

  float color1Dist = min(distance(c00.rgb, color1), min(distance(c01.rgb, color1), min(distance(c02.rgb, color1),
    min(distance(c10.rgb, color1), min(distance(c11.rgb, color1), min(distance(c12.rgb, color1),
    min(distance(c20.rgb, color1), min(distance(c21.rgb, color1), distance(c22.rgb, color1)))))))));

If this distance is lower than our target threshold, we set the output pixel to a sampled texture value. Otherwise, we set it to the center of the sampled input matrix.

1
2
3
4
5
6
7
8

  float2 pos = outCoord();
  float4 texture1 = sampleNearest(textureImg1, pos);
  if(color1Dist <= thresholds[0]){
    dst = float4(texture1.r, texture1.g, texture1.b, c11.a);
  }
  else{
    dst = c11;
  }

You can see here that we use the alpha value of the original sampled center pixel for the output pixel. This is to ensure that if the text was originally blending with something in the background, so will the gold or silver text effect.

After trying a variety of other methods of applying these types of effects, this really seems to be the only way to achieve this quality of effect without drastically slowing down the runtime. You can try it out on a card here. Try changing different parts of the text to different metallic colors. All of these metallics are applied with a single shader that is sent a new texture for each metallic color you choose.

If you want to learn more about Pixel Bender, Adobe keeps a list of shaders here; or you can come work/intern with us! Most of this shader work was done by Nick Schaubeck, one of our very talented interns. We’re looking for more talented people in NYC and SF.

As our fearless leader and CTO Aaron Quint mentioned in our previous blog post, we are completely addicted to metrics. One of the cornerstones of this new mindset is statsd, a node.js server for aggregating and sending information to a Graphite server. If you don’t know what statsd or Graphite are, read the above blog post. If you don’t know what node.js or V8 are, you should check them both out as they are very interesting projects!

As a consequence of a big push to automate our software infrastructure, we have tried to tie as many of our dependencies as possible to pre-compiled packages for our current deployment platform of choice, CentOS. The only available packages for our version of CentOS were for the 0.6.x stable branch of node. Even though we were running the 0.4.x branch (compiled by hand) up until that point, we decided to give it a shot.

After our upgrade, we noticed that the node statsd process on our isolated statsd box would balloon in memory usage, measured by resident set size, from a manageable number in the tens of megabytes to upwards of a gigabyte. Eventually, the process would consume so much memory that it would get shut down by the OS, and for the time that the process was down, we would miss out on our stats getting transmitted properly to Graphite. Besides making us aware that we should try to find a way to improve the availability of our statsd, we were very suspicious.

At first we assumed that we had accidentally created a memory leak in the statsd process, because we are actively maintaining our own fork at this point and we couldn’t find many other mentions of similar issues. After some serious head scratching and investigation, Aaron tracked it down to node itself, by stripping away the extraneous bits and observing how node handles sending and receiving UDP packets.

Here’s the client test code, commented for your convenience:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

// Require the necessary node modules
var util = require('util'),
    dgram  = require('dgram');

// Set up the UDP client, create a dummy message, and zero out a counter
var client = dgram.createSocket("udp4");
var message = new Buffer("stats.counters.test|1");
var sent = 0;

// Define a send function that will be executed in a timed loop.
function send() {
  process.nextTick(function() {
    client.send(message, 0, message.length, 8125, "localhost", function() {
      sent++;
    });
    send();
  });
};
send();

// Set up an interval timer that will periodically send the number of sent
// packets to the console
setInterval(function() {
  console.log('Sent ', sent);
}, 1000);

// Output the client start to the console
console.log('Started sending to', port);

And the server, equally simple but also commented:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

// Require the necessary node modules
var util = require('util'),
    dgram  = require('dgram');

// Set up the server and a counter for the number of received packets, and
// define a callback function to be executed when a packet is received.
var packets = 0;
var receive = function(msg, rinfo) {
  packets++;
};
var server = dgram.createSocket('udp4', receive);
server.bind(8125);
console.log("Listening for UDP packets on", port);

// Print out the process memory usage, the number of packets received,
// and force a GC run every 2 seconds
setInterval(function() {
  console.log("\n*** RSS: ", process.memoryUsage().rss / 1024 / 1024, "mb")
  console.log("packets:", packets);
  // A hook to force gc through V8's --expose_gc option
  if (gc) gc(true);
}, 2000);

This simple client/server set up lets us observe what happens when we throw a bunch of UDP packets at a node server, and do nothing else with them. When we ran the above client and server code against the 0.6.x stable version of node, we got the following output (note that this is a sample from the “middle” of the log, you can note by the number of packets sent):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

*** RSS:  64.64453125 mb
packets: 95354

*** RSS:  93.68359375 mb
packets: 113610

*** RSS:  56.98046875 mb
packets: 114268

*** RSS:  55.50390625 mb
packets: 114278

*** RSS:  74.39453125 mb
packets: 131329

*** RSS:  60.00390625 mb
packets: 131329

*** RSS:  58.69140625 mb
packets: 132717

*** RSS:  100.3359375 mb
packets: 147266

This simple UDP client/server implementation leaked memory profusely. Even more damning is the fact that after the client was killed and packets stopped being transmitted to the server, the RSS did not drop significantly, even after many cycles. After we had proof of some memory weirdness in the node process and verified that this issue didn’t exist in the 0.4.x branch of node, we did the next natural thing: opened an issue on GitHub.

Thanks to the transparency and activeness of node’s development process, we were able to get the direct attention of the maintainers to explain our issues. It turns out that the 0.6.x branches introduced major changes in node’s entire networking layer via the adoption of a new library called libuv. Ryan Dahl described libuv on the node.js blog:

libuv’s purpose is to abstract platform-dependent code in Node into one place where it can be tested for correctness and performance before bindings to V8 are added. Since Node is totally non-blocking, libuv turns out to be a rather useful library itself: a BSD-licensed, minimal, high-performance, cross-platform networking library.

The 0.5.x branch of node.js had given the node community the opportunity to test these changes with an unstable release, but since UDP isn’t used as commonly as TCP by node.js users, its datagram functionality was sort of lost in the shuffle. More “important” and oft-used protocols were covered first, and first class memory management was one of the features that TCP got that datagrams didn’t.

It turned out that the node core team knew that datagrams needed some love, and they were happy to have some help. The TCP strategy for memory management made it much easier to avoid memory fragmentation because it used slab allocation. For an excellent, illustrative explanlation of memory fragmentation, see this Stackoverflow answer. Slab allocation solves fragmentation issues by allocating larger chunks of memory at once, to be used only by objects of the same type and size, alleviating memory gaps of different sizes being created by deallocation.

Because we had committed to using node in production, and we try as hard as possible to be good open source citizens, I decided to challenge myself to make this change. Mind you I haven’t touched C++ in years, I have zero familiarity with the V8 C++ interface, and I’m not an avid user of node.js, but I still really wanted to do it. If you’re wondering why, I was motivated by my desire to give back to the open source community, and by my dedication to keep Paperless Post’s operational overhead to a minimum. I was presented with a clear path, and was ready for the adventure. Plus, we needed our stats!

I started to dig in, read some materials online, and got a good sense at how the interactions worked between V8, libuv, and node.js. Because the Internet is awesome, I was able to continue the conversation from GitHub on Twitter and then on IRC with a few of the node and libuv maintainers. They patiently explained some of the crucial pieces I needed to know, and after some major head-banging, a lot of rubber duck programming, and some heavy testing, my patch was ready.

After all of that, it turns out that:

1

1 files changed, 59 insertions(+), 9 deletions(-)

could result in a change from the above log to:

1
2
3
4
5
6
7
8

*** RSS:  14.59375 mb
packets: 4074145

*** RSS:  17.60546875 mb
packets: 4081558

*** RSS:  17.64453125 mb
packets: 4081568

If you’re interested in the specifics of the patch, you can see them here along with the pull request. The patch isn’t that exciting - I simply ported the slab allocation code from libuv’s stream_wrap.cc file which handles streaming networking connections to the udp_wrap.cc file which specifically pertains to datagrams. I’m happy to report that the next stable version of node, v0.6.8 (changelog here) shipped with my patch, and that the memory usage of the node process under heavy datagram load shows a stable memory profile.

There have been many times where I have not bothered to get involved in a project, and instead settled for suboptimal performance or degraded functionality in a software library. I wanted to share this example of when I decided to step up, and I hope that reading this story reminds people like me who have made a career out of using open source software that it’s possible to get involved in projects of any size – you just need to be persistent, patient, and highly tolerant to pain.

Want to be patching, shipping, and learning every day? We’re hiring in NYC and SF.