Next month, a group of people from the Paperless Dev team (including myself, @aq) will be attending GothamGo in NYC. I’m very excited to meet other gophers and get to talk shop. In thinking about attending I realized that though we’ve been running a large scale Go application in production for over a year, we’ve been relatively quiet about our work.
How I got Go’ing
It’s been over three years since Blake Mizerany started talking my ear off about Go. Blake is one of the people that I tend to listen closely to when talking about programming, so when he told me I needed to spend some time learning Go, I started looking for an opportunity to dabble. I immediately liked its simplicity, the robustness of its standard library, and its speed. It took me a while, however, to find the real gateway project that would take me over the initial hump.
A lot of my work at PP over the past 5 years has revolved around our rendering and image pipeline. When we deliver cards to users they come as rendered images of the creative designs they’ve customized in our design tool. The rendering, processing, and storage of these images is by far the most intensive part of our infrastructure and always one the parts that needs the most upkeep and rethinking as we grow. For a very long time we were able to rely on shared storage (NFS) and ImageMagick to do the storage and processing of these images (KISS), but that started to show some big faults under pressure. An idea started to come together when looking at the problem and playing with Go. What if we could process these images in memory, and then store them somewhere distributed - that would probably be a lot faster and more reliable than reading and writing them to NFS, right? That’s when I discovered the magic of cgo.
cgo and magick
cgo is not one of the most often talked about or touted features of Go, but as someone who’s tried to and failed at working with Ruby’s C-ext support, cgo really seemed like some alien gift from the future. Even though Go’s standard library was great, the number of packages to solve common problems was small (especially 2 years ago - this has changed). With cgo, however, any problem that’s been solved in C is easily ported to Go. In fact, because of how easy, fast, and no-effort the compilation and deployment of Go wrappers around C are, it makes you want to wrap everything. So I started diving through the C source to ImageMagick (specifically MagickCore) to try to wrap our common use cases in simple Go methods. This became magick. It took some work in the background, but I proved part of my idea - that you could work on these images in memory (with a simple API) and it would be (much) faster than our current methods.
From there, it took a little while to convince the team to start work on the bigger idea: using magick to build an image storage and manipulation service in Go. Myself and my former colleague @mrb worked on the project, which we call Agency, on and off for 6 months before we took it across the finish line in October of last year. Since rolling it out Agency has stored and processed over 85 million images, and does so using about 1/4 of the resources of our previous pipeline. I’m happy to say it’s up there with one of the the most stable and predictable parts of our infrastructure.
This year I’ve worked a bunch on refactoring and improving Agency as its usage has grown. We’ve extracted some more pieces of it in an attempt to make the pattern of simple TCP request/response services grow within our team and infrastructure. This project (we call it tcpez) is forming the foundation of another new project that we’re working on currently that should vastly improve our rendering pipeline. We’re busy testing it, but we’re planning on open sourcing tcpez in the very near future.
After 3 years of dabbling and one year where its been my primary focus, I’ve come to love working with Go. It has a simplicity that allows for a really quick route from idea to implementation. It has some rough edges for sure (type inference, error handling), but hitting them is pretty rare if you program “within the lines”. As a programming environment, it’s unbelievably good: great tools (testing, pprof), fast and no brain complication, and great standard library. And with every other language that I’ve taken on very seriously, I enjoy programming with it. Go is not meant for solving every problem, but the problems it solves (concurrent networked services) it solves really well.
At Paperless Post, we’re investing in Go as we think it hits a sweet spot for key parts of our application. We still use and love Ruby, too. If you’re interested in working on either of these kind of problems come find and talk to us at GothamGo in NYC or a couple days later at RubyConf in San Diego.