Saving Sprockets

What do you do when a maintainer leaves a project with over 51 million downloads? That is what we had to consider this year when Sprockets lost the developer responsible for more than 70% of the commits. We'll explore this and more through my RailsConf 2016 talk and the transcript below.

I've spoken in 12 countries on 5 continents and this is my favorite talk to date. This talk was difficult. The ideas were in my head, but I couldn't get the words right for the longest time. This was an emotional talk for me . Writing this talk turned into a soul searching journey through open source. What does it mean to be a maintainer? What does it mean to leave a project? What does it mean to respect and help a maintainer? The road was long and perilous but I'm very happy with the talk. Hopefully you'll join me on this archeological expedition.

This version has intro music and closed captions. Transcript and slides are below.

Here's what some people had to say about my talk.

  • "That was an epic talk on many levels" - @_tankard

  • "Every single Ruby developer should watch @schneems's [Saving Sprockets] talk" - @samphippen

  • "good talk man, every conf should have sth like that until people learn how this oss thing works :)" - @solnic

  • "Thoroughly enjoyed, would attend again, ⭐️⭐️⭐️⭐️⭐️ / ⭐️⭐️⭐️⭐️⭐️" - @chrisarcand

  • "I loved it [...] but I am biased in favor of smart ideas full disclosure" - @searls

  • "Go home people, @schneems just had the best slide analogy of #RailsConf" - @cecycorrea

  • "Don't quote me boy" - Eazy-E



This is the text of my talk with some minor adaptation to make sense when reading without slides. It's about 5,000 words or the length of two of my normal blog posts.

We want to start out asking the question, Why does Sprockets need saving? For those of you who haven't been around since May of 2011, it is the premiere feature of Rails 3.1, and Sprockets is the Asset Pipeline. Sprockets actually came first, before it was ever wrapped up into Rails. One thing I wanted to mention before we get too carried away, is that you don't need to look like Indiana Jones in order to maintain open source, it just so happens that he is my Sprockets spirit animal.

So, from 2011 to 2016 Sprockets has had 51 million downloads, and I'd like to put that into perspective. Rails has had 65 million downloads, so Sprockets is pretty close, and, of that entire library, one developer is responsible for 2027 commits, which happens to be about 68% of Sprockets. That's one person. Compared, in contrast, to a Ruby hero, Rafael Franca, who has over 5000 commits on Rails. This accounts for only 0.9% of Rails. 51 million downloads, one developer, and one day, Josh left. "I'm cutting it, I'm out, I'm gone."

So, when something like this happens, what should we do? Should we, as a community, abandon Sprockets? There's a lot of people who said "I don't like Sprockets", and "it's got problems". To them, I ask: what are the problems? Do you know what they are? Because we can't fix what we can't define, and if we want to attempt a re-write, then a re-write would assume that we know better. We still have the same need to do things with assets, so we don't really know better.

I think we should stick with Sprockets and make it better. Assets are really the easy part of Sprockets. There's a whole bunch of edge cases. Also, Sprockets has a really well-defined and established API.

Losing maintainers is inevitable, and it's not always expected. Jim Weirich was the creator of a amazing library that we've all used called Rake, and in 2014, Jim passed away very suddenly. It wasn't like anybody saw this coming. He wasn't working with someone to pass on the software to for a long period of time. And so, whether a maintainer suddenly walks away or they pass away, it hurts. We, as library consumers, have to cope with it, and there's a lot of different ways that we do that. We might go through a period of denial and say something like "they're going to come back". We are going to get this person back into our lives. You might be angry and say, "leaving is selfish", or "that was such a jerk thing to do". You might start bargaining and say "Maybe if we hire them, they'll work on it full time and we can get them to come back", and eventually, acceptance. "They're not going to come back, who's going to take this over?"

The number one rule, is that a maintainer does not owe you anything.

A maintainer does not owe you Anything

Not even an explanation. If you're going to leave a project, or someone is leaving a project, it's a very personal decision. I actually reached out to Josh and said, "Hey, man, let's talk about this. I'm giving a talk at RailsConf. I need some content, gotta help me out," and Josh didn't want to talk about it, and I want to respect that wish, and I also want to respect what he's done, which brings me on to the number two rule.

The number two rule, is that you owe a maintainer respect.

Some people will say things like, "Oh, but I really hate this project." It is possible to critique software without demonizing the creator, and, as a matter of fact, I'm going to critique the crap out of Sprockets. Notice word choice was intentional here. I'm critiquing and I'm not criticizing. I aim to be productive with the words that I'm using. I want to find what is bad and then make it better. Originally, when Sprockets fell into my lap, and somebody said, "hey, do you want to be on Sprockets core?" I was like, "Sprockets, why did it have to be Sprockets?" You are not your software. Josh gave years of his life to the project. No matter what you think of the project, or what you think of how it was maintained, I want you to give a thank you to Josh.

Rule number three, is that words without actions are empty.

I want you to be actionable with your critiques and think about this. For example, we have a Hacker News comment which says, "unless they add this feature to Node, I see this as ugly and barely usable"

When I read that, that's not going to make me want to go out and help them. Instead, they could've easily said, "Hey, this is great, this is amazing, I love it. It looks like they don't have this thing I need, and, as a matter of fact, I can't use it and here is my use case" and that's actionable feedback. You can critique without criticizing. So, I want you to ask yourself, "is this comment adding anything?"

Hyperbole in comments and blog posts is good for laughs and fake internet points but it doesn't help. I want you to be honest with your critiques. I want you to be productive. Here is the babeljs creator tweeting a screenshot, again, coincidentally, from Hacker News, and it reads, "Babel sucks. I never thought I could hate something so strongly."

Wow, that's really going to encourage that guy to go out and fix all of your problems. You might disagree and you might have very strong opinions, and those opinions might be very negative. Even if that's the case, this software is in your life for a reason, and if you can figure out why those things hurt, why you are having those negative feelings, and you can enumerate that in a productive way then it helps everyone. Complaining by itself accomplishes nothing. When I started the talk, I wanted to touch on "how do we keep a maintainer longer?" Or if you're maintaining software, how can you stick around longer? To do that we need to look at what maintainers want. We also need to do our homework and ask ourselves "is there any value in a maintainer sticking around?" All maintainers will one day leave and we can either have a maintainer that just mic drops and you never see them again, or we can have somebody who's passing the torch and doing a graceful hand-off.

While I'm working on Sprockets, there's so many times that I say "this is absolutely batshit insane. This makes no sense. I'm going to rip this all out. I'm going to completely redo all of this." And then, six hours later, I say "wow, that was genius," and I didn't have the right context for looking at the code. Maintainers are really historians, and these maintainers, they help bring context. We try to focus on good commit messages and good pull requests. Changelog entries. Please keep a changelog, btw. But none of that compares to having someone who's actually there. A story is worth 1000 commit messages. For example, you can't exactly ask a commit message a question, like, "hey, did you consider trying to uh..." and the commit message is like, "uh, I'm a commit message." It doesn't store the context about the conversations around that.

So, maintainers are historians, and we can keep those maintainers longer by giving them what they want. Maintainers want respect. They want to be appreciated. They also want help, and I know all of you are thinking, "Ugh, this is the part where he's going to be like, asking me to help, and I really don't want to do that." Or, maybe you already are helping. Maybe you're saying, "I don't have enough time", or "ughh just fix all of the things for me it will be faster if you do it". I'm here to say that if you have five minutes to snap-to-face-to-fours-tagram, then you have five minutes to help open source. You can contribute to docs. You can read the guides. You can fix typos. Maybe you found a really surprising behavior. Was that behavior documented? If not, then go ahead and add it to the guide. If you have five minutes to help, then you can submit a bug report. Seriously, the maintainers have no clue that things are broken. You might be thinking, "oh, there's thousands of people using Rails and all of them have reported this thing." No.

The question of "why is Sprockets bad?" I don't know. Nobody actually gives me actionable bug reports. So, if you have five minutes to help, then please let us know what your problems are in a productive way.

Critique over criticism.

Another thing you can do to help is sign up for a service that I wrote and maintain called CodeTriage. You can go there and sign up for a project you care about and it will send you an issue in your inbox once a day. It's a very actionable way to get started. When you get the issue, you can ask common questions like "what version were you running on?" or "Was this working previously?" Let's step back, would you rather the maintainer of that project spent the time fixing bugs or would you rather they spent the time asking for insanely small minutiae on the issues? Anyone can ask those questions. And it might only seem like you're giving a minute or two out of your day, how could that be impactful? If you give a minute, you are actually saving a minute of a maintainer's time. A little bit of help can go a long way. And if you don't help, then who will? It also has the benefit of exposing you to different parts of projects, which helps you grow as a developer.

If you have 10 minutes to help, include an example app to reproduce the problem. Example apps are amazing. I get all these bug reports that are like, "well, first I run rails new," and then I go and try the instructions and come back an hour later "sorry, couldn't reproduce," and then they respond, "oh yeah, I forgot to add this other thing," and then I try it, and couldn't reproduce, and I waste hours of my life that I could be spending fixing bugs or writing new features. As the reporter, you waste hours of your life. Nobody's happy. Instead, you can ask and say "here's an application that is going to reproduce my problem." Make a new project with the bare minimum to get the bug to show. Put it on<username>/ExampleApp if you don't have that yet. You can even choose ExampleApp1 or ExampleApp2 as a repo name. I'm not picky. If you give a minute of your time, then you're going to save a minute for a maintainer.

I personally challenge you, if you haven't already, please try and make it your goal to produce one example app this year. It is so helpful.

If you have 30 minutes to help, you can try fixing a bug. Anybody's bug or your bug. It's not as hard as it sounds, just timebox it. Even if you don't fix it, then you're guaranteed to learn something. You're guaranteed to read other people's code. You're going to be navigating and debugging other people's code, which happen to be highly marketable skills. With all of this, I know you're like, "Okay, uh, I don't want to do that like, every time," and that brings me back to club soda.

I drink club soda at home, and I don't like putting the whole thing in my refrigerator. So instead, what I do is I put like three or four in to get cold. Then when I pull one out, I put one back in. However, sometimes I run out of club soda. How did this happen? Is somebody stealing my club soda? Is my dog drinking my club soda? The rule is one in, one out. It's pretty simple. But clearly it's not sustainable. Instead, what I found, is that if I put in two cans instead of just one can that I somehow end up with 3-4 cans in the refrigerator. Now I always have club soda. You got it? It all makes sense. No?

So what I'm saying is, you don't always have to contribute to open source. You don't always have to make an example app, but just every once in a while please go the extra mile.

These are all different ways that we can help a maintainer. Ways we can make their job a little bit easier. So how do we transition from one maintainer to another maintainer? Well, what is a maintainer? We've talked about this. A maintainer is somebody who knows the stories. A maintainer is someone who's going to take 5, 10, 30 minutes out of their day to help. If a maintainer is somebody who helps, and the act of helping preserves history, then maybe the act of helping is the answer to keeping a maintainer. Also, the act of helping is the key to creating maintainers.

If you have people familiar with your code, whenever you actually go through that hand-off process, people aren't just starting from zero.

The next question we have is, how can we foster a culture for helping? How can we get more people to help? If you are a maintainer, you want people in your project. If you're using that project, you want more people helping and contributing because that makes it better. So, how do we foster that culture for helping? We talked about what maintainers want, but we never really talked about what the helpers want.

Well, helpers want documentation. They want sane code. They want what regular users want: good user experience. Non-magical code, backwards compatibility, good deprecations, reliable tests. These are all things that are interesting to them. So let's look at one and compare it on the Sprockets chart.

Documentation. Sprockets has 73% documented methods. Seventy-three percent of all methods are documented. That's a lot. That's really up there. On a side-note, I think that method documents are kind of like unit tests. They are very focused on one part, and don't necessarily tell the whole story, so it is possible for those comments to get a little bit out of sync with reality. I also highly recommend keeping a README. A README I see as something more like an integration test. It's going to tell a little bit more of the whole story, and if we look at Sprockets, well, Sprockets has about 2,600+ words. That's a pretty long blog post. That's a pretty substantial README. That's a pretty long story. If I'm here, telling you that helpers love docs, and I'm telling you that Sprockets has docs, why doesn't Sprockets have anybody helping?

I put on my design research hat. I went to design research school and I learned about user stories. So, we are going to actually consider the people using our product. I want to introduce you to Pedro. Pedro enjoys long walks on the beach. Favorite food is bagel bites, and he's building the next Uber for goldfish. Pedro is a Rails user, and Pedro cares about the Rails interface. This is going to be, how do I get one file to require another one? I want to know what I actually have to type into my project to get it to work. Now, I don't care about all that other stuff. I just want the things I need now.

Next up is Pat. Pat is addicted to ES6. I know. Pat loves to fly fish and Pat is a plugin developer for Sprockets. By the way, did you know that Sprockets has plugins? Okay, well, they're not called plugins, they're called processors and transformers and compressors and like 20 other things, but it does have a plugin system. And Pat cares about the processor interface. They maintain one. They want to know that, whenever we pass the hash of things to them, what is going to be in it? And what can I do with it? What should I do with it? Pat wants this documented when Pat is working on their plugin.

Finally, Diana has a dog named Exception and hates mustard. Both of these are highly relevant to Diana's job as a Rails developer, i.e. somebody actually developing Rails, somebody actually building an asset pipeline. Diana cares about the low-level interface. What does that mean? Diana cares about what are the classes she can use? What are the methods on those classes? All of them. If she wants to be able to disable gzip. Here you go.

These are all different people with very different needs who need different documentation. Don't make them hunt down the documentation that they need. When I started working on Sprockets, somebody would ask, "is this expected?" and I would say honestly, "I don't know, you tell me. Was it happening before?" And through doing that research, I put together some guides, and eventually we could definitively say what was expected behavior. The only way that I could make those guides make sense is if I split them out, and so, we have a guide for "building an asset processing framework", if you're building the next Rails asset pipeline, or "end user asset generation", if you are a Rails user, or "extending Sprockets" if you want to make one of those plugins. It's all right there, it's kind of right at your fingertips, and you only need to look at the documentation that fits your use case, when you need it.

We made it easier for developers to find what they need. Also, it was a super useful exercise for me as well. One thing I love about these guides is that they live in the source and not in a wiki, because documentation is really only valid for one point in time. Otherwise, you end up in the wiki like, "if you're using this version, do this, if you're using this version, do this," and it's 20 versions and it's no good. Helpers love contributing to docs, so you know what? We can make more docs. We can make our docs better. Those docs are going to be the gateway drug to code contributions.

The next thing I want to talk about is sane code and realtalk. Sprockets was designed to solve problems, and sometimes, when it's putting out a fire, it kind of feels like it's starting other fires. Maybe making additional problems that you didn't see before, and you don't know why it fails. The reason you don't know, is because Sprockets isn't talking to you. How does code talk? Code can speak to you through errors, and I'm not talking about "something broke", or "no method error, on nil". I'm talking about: I want my error to say, "this broke". I want my error to say, "ID key is missing", look here, "this is the thing that you're missing". Good errors are instructive, and Sprockets will have better errors. It doesn't yet. I do care about this. I am the owner of a gem called, wait for it, Sprockets Better Errors. That gem was merged into Sprockets Rails, but yes, I had some good ideas for better errors in Sprockets itself.

The other way that code can speak to us is through deprecation. Now, deprecating something in a code comment is not enough. Right now, Sprockets is doing this. They have a little code comment, and they're like, "by the way, this method is now deprecated." We will just delete it, and it won't be available, and you never knew that because A) No one is casually reading the method documentation, and B) Who has the time. It's not as though every single time you upgrade every version of Sprockets, you're going to read every method comment in the documentation. Are you going to do that? No. You cannot just sit back and break your API, especially when you have 51 million downloads. That is unacceptable. Since your code knows when somebody's using a deprecated interface, we can yell at them.

We have these things called deprecations. I wrote a detailed guide on using Deprecations in your library.

Sprockets 3.X will have deprecations before we go to Sprockets 4.X. We have a branch. We've started working on this. If you've not implemented deprecations in your own project, it's super simple. You say "hey, the thing you're using is deprecated. Use this other thing that's not deprecated, and here's where you were using it." It's kind of like a three-step process. So deprecations are going to nudge people into the right behavior. They're going to help get people to upgrade, and they also help with API design, because, if you can't write a good deprecation, then guess what? The interface probably wasn't the best.

In the talk I mention an hash key based API, turns out I was wrong. More information can be found at this PR. This just further underscores the importance of having a maintainer who knows what is up around to sanity check things for you.

This is my favorite section coming up. I hope you're paying attention. Sprockets suffers from something that I like to call the god object problem. It has this one main class that has all of these concerns mixed in with it. It's one object with 105 methods. It's using a lot of them, and you ask yourself, "where did that method come from?" and you look at this source code, maybe it came from


This is my personal favorite,


Is mixed into


Which is then mixed into


Which is then included in


Which is inherited by


Which is then wrapped in cache by


It's impossible to just glance at something and know how things are interacting. You change this one method that you thought was only being used in this one part of the project, and something else breaks.

For more information about how Sprockets work, I highly recommend you go to Rafael's talk

What is the solution to god objects? We can move logic over to helper classes. For example I introduced the URITar class while I was adding new functionality. It takes an absolute path and trims it down to a relative path, or it can take a relative path, and make it an absolute path. We need this for storing things in the cache. The beautiful thing about this is it has a couple extra methods that make it a little bit cleaner that are not actually exposed to that god object API. We can expose only the things we need. So it's going to minimize that god object API and it also, hopefully, produces small, easy-to-read files.

You can look at that file and say to yourself "I vaguely understand a tar utility can expand or compress a file; maybe it's related to that, but for URIs". Ideally, this produces readable code and readable code also attracts helpers who read code, believe it or not. As a side-note, I will say that Ruby is object oriented, if you're not super comfortable with objects and classes, please spend a little bit of time there. It's totally worth checking out. Sandi Metz has a book that I have totally not read, but she's given a ton of conference talks, which I have seen, and you should watch the talks, and I'm sure the book is amazing as well. As well as Katrina Owen, who has done a ton of Refactoring talks. If you want to see how can we make this better, how can we make this more readable, as well as is an actual place where you can go and try out your skills. You actually refactor things there. It's pretty cool. They're also working on a book together which I'll actually read when it comes out.

Helping takes commitment, and we do need to respect that. How are different ways that we can respect that commitment? When somebody gives a pull request, even if it's not the best pull request, we, as maintainers, can say "thanks for submitting this". That person cared enough, and you can help them to help you. You can explain the reasons why you're not merging it, or help them to get to a place where you can merge it. You can also help guide them and say "I don't really care about that thing. How about you look at this other thing which I really care about?" And it's a way to get them on board. If you just close an issue, dismiss it, and then lock it: that is not how you attract people to help you.

What else do people want? People want recognition. Rails has this great leader board. That's actually the reason why I had my first commit under Rails, I wanted to just be on the board, period. Maybe you don't have a leader board, but you can still give recognition.

Maybe your helpers want pizza.

There's a fun story where, when I introduced a feature, I actually broke Windows on a minor release of Sprockets, and I had a developer come to me tell me that I broke the build on Windows. I had no idea what to do. How do I fix this? He helped explain the problem, we worked through it, we pushed a release. Later I went to him and I said, "okay, obviously you care about Sprockets, you care about Windows, you have a Windows machine. Can you help me get the Sprockets tests running on Windows?" He was hesitant to commit that much time to a pretty thankless task. I wanted to show how much I would appreciate it so I offered to buy him a pizza for his efforts. I'm not joking. That's not like, hyperbole. "I will actually order delivery for you for a pizza to your home". Well, a couple weeks later, he did it, and he did not happen to, in those couple of weeks, reach out to me and mention "hey, by the way, I live in Germany". But he did live in Germany. I now know lots about ordering pizzas for delivery in Germany and explaining to credit card companies what open source is. I'm happy to do things for people who help me. The cost of the pizza was trivial compared to the time spent on the feature, but the gesture was worth more. We mentioned acknowledgement. Well, thank you, thank you, Daniel, for doing this, and Sprockets is now tested on a Windows CI server via appveyor.

All of these things we've talked about: good docs, clean code, those are ideals we can strive for. What happens when you actually have the scenario where you inherited a project? Where the precious maintainer might not have done all of these things? They just mic-dropped? What are you going to do?

You can start by finding something that needs fixing. I call this bug-driven development. I keep on talking about example apps because the only way to get started is with an example app that allows you to reproduce the problem. If I didn't mention, example apps are amazing, and you should probably make it a goal to make one this year.

Once you have an example app, reproduce the problem and then repeat. Every single bug that you fix, you're going to learn a little bit more about the codebase, and eventually you're going to start seeing non-bug problems. Eventually you're going to be a lot more comfortable. An example of this for me was Source Maps. Source Maps are a thing in Sprockets 4, and it was half-finished when Josh stepped away, and when I got this project, I had no idea what they were. Somebody would report a bug. I would try and fix that bug and it made the tests break, and I had to step back and ask if the tests are even reliable? I've got no clue. So where do we start? I put on my archaeologist hat, which was totally the inspiration for this talk, and I started research. I looked at the Mozilla RFC. I got out evernote and I started taking notes, and I learned a whole lot about source maps. I eventually was able to take those notes and actually turn them into guides. I took all of that information and said "I don't know this, so other people probably don't know this. Let's not make them work as hard". I made those notes into a guide and I put it in the Sprockets source tree. If you're interested in reading it, it's totally rough, but it works. I've even used it for my own reference more than a few times. So if you want to know what a source map is, then I can tell you, go read my Source Maps guide. In this process I ended up having to borrow from some other projects, I actually used other projects from the technology which shall not be named (NPM).

I used uglifyjs to verify my encoding and make sure that my encoding tests were valid. I used source-map to verify that my decoding tests were correct, and then I got the tests to a place where they could pass. So, is Source Maps finished? No. I need more bug reports that are actionable.

We've got to wrap up soon. With all of that being said, where do we go? Because maintainers won't be around forever. I won't be around forever. So, I need help. I need help maintaining the history of Sprockets. I don't need you to know everything. I don't need you to go out there and fix all of the problems. You know what? Sometimes I might just need you to help not say bad things about me on the internet. If somebody's like, hey, trash-talking me ask them to clarify their positions into a critique.

We can preserve these stories by getting involved, and if you don't get involved, then who will?

It's open source.

You can say, well, maybe somebody else will. Guess what? That somebody else is you, and we all need to step up.

We can take five minutes. We can just read those guides. We can write some docs. We can open some issues. We can create example apps, which I've totally not mentioned before.

It's only



I invite you to join me, and together we can become maintainers, we can become helpers, and together, we can Save Sprockets.

The Straight Dope on Deprecations

The road to stability is paved with good deprecations. A deprecation is a warning message that tells a user they’re using some piece of code or interface that will go away soon. In this post, we’ll peel back the seemingly simple veneer of deprecations, and we will learn when and how to use deprecations effectively.

Keep Reading on Codeship

Ruby Hero 2016

This year I was lucky enough to be one of 9 rubyists chosen to receive the Ruby Hero award. I'm also only one of 2 Heroes to be pictured wearing a hat this year.

I've been describing the award as "The Oscars" for Ruby programming for those not in the community. This description went over really well, however my Mother-In-Law got a bit confused and asked if I won because I "wore a hat to my talk this year" (i.e. dressed up as Indiana Jones).

Thankfully, the award has nothing to do with my ability to act.

I'm incredibly grateful to be a part of the Ruby community. I started programming Ruby as my first language back in 2006. I thought I would be super rich by making a website and selling it, instead I found something even better. I found an amazing and supportive community. It's impossible to quantify how friendly, and helpful a large group of self-identified people are. If I could, then Ruby would be at my personal number one spot.

There were a few bad apples. In my first programming job interview, a CTO told me "I couldn't program" to my face. I got over it. By pushing past the bad I was able to get to the warm collaborative center of the community. I eventually quit my mechanical engineering job and got a software dev position. I became active with I started teaching programming and even taught at the University of Texas once. I've written heavily on tech topics at this blog, and I enjoy staying active in open source.

For the most part I've gotten nothing but encouragement and help. It is so easy to be mean, snarky, or cynical. There is so much negative in the world. It takes a great deal more effort to be productive and helpful.

I was inspired by the idea of nominating a "Ruby Hero" this year to start a personal experiment. I like the idea that people get up and share that they have someone who made an impact in their lives. It's sad that there were 500 or so nominations but only a few got to be recognized. I wanted to show those around me that they're actively making the world a better place. Nothing says "thanks" quite like a hand written card, so I printed off a bunch of postcards:

I intended to hand them out blank for others to write on, but I ended up writing so many myself that I went through my whole stash. I had to go buy some regular cards at a grocery store just to have enough. It felt really good to give out "thanks", and once I started I couldn't stop. Here's an action shot:

I want to keep this up. I'm exploring ways to encourage more people to speak up about the good others have done for them. I've printed out a whole bunch of postcards and I'll be at OSCON May 16-19 in Austin (my home town!), find me and grab one. Then consider someone who made your day, week, or year; and let them know.

Before I get too carried away, I would like to thank Heroku for taking a chance on me over 4 years ago. They also employ another of the 2016 Ruby Hero's, Koichi Sasada, who works on Ruby core. Heroku has a history of supporting the Ruby and the Ruby community. They've supported my interests in speaking and open source. I'm proud to work with such an amazing team.

This past year has been a whirlwind. I had a son, I got into Sprockets Core, and now this award. I sincerely appreciate this honor and I plan to keep contributing. I also see the award as more than recognition. I see it as a challenge. It says to me "strive to be more". It says "go out there knowing that people are watching you now". It says "set an example and live with purpose". While this award is a career and community high for me, I don't see it as the end of my journey. It's a beginning. Thank you all for reading and for making Ruby such an amazing community.

Container-Ready Rails 5

Rails 5 will be the easiest release ever to get running on Heroku. You can get it going in just five lines:

$ rails new myapp -d postgresql
$ cd myapp
$ git init . ; git add . ; git commit -m first
$ heroku create
$ git push heroku master

These five lines (and a view or two) are all you need to get a Rails 5 app working on Heroku — there are no special gems you need to install, or flags you must toggle. Let's take a peek under the hood, and explore the interfaces baked right into Rails 5 that make it easy to deploy your app on any modern container-based platform.

This article originally published on the Heroku Blog.

Production Web Server as the Default

Before Rails 5, the default web server that you get when you run $ rails server is WEBrick, which is the only server that ships with the Ruby standard library. For years now Heroku has recommended against using WEBrick as a production webserver mostly due to performance concerns, since by default WEBrick cannot handle more than one request at a time. With the addition of ActionCable to Rails 5, the Rails team needed a web server that could handle concurrent requests, so they decided to make Puma webserver the new default. Now, when you deploy a Rails 5 app without a Procfile in your project and Heroku boots your application using $ rails server, you'll get a performant, production-ready web server by default.

Note: if you're upgrading an existing Rails app, you'll want to manually add Puma to your app.

In addition to shipping with Puma, Rails also generates config/puma.rb and efforts were made to allow Puma to read this config file when it's booted by the $ rails server command. This feature is baked into Puma 3.x+, which allows Rails to configure Puma around the number of threads being generated.

Active Record will generate a pool of five connections by default. These connections are checked out from the pool for the entire duration of the request, so it's critical that for each concurrent request your webserver can handle, you need that many connections in your connection pool. By default, the Puma server starts with up to 16 threads. This means that it can be processing up to 16 different requests at the same time, but since Active Record is limited to five connections, only five of those requests will have access to the database at a time. This means eventually you'll hit this error:

ActiveRecord::ConnectionTimeoutError - could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it

The solution was to tell Puma that we only want five threads by default. We also wanted a way to re-configure that count without having to commit a change to git, and redeploy for it to take effect. So by default Rails specifies the same number of threads in Puma as Active Record has in its connection pool:

# config/puma.rb

# Puma can serve each request in a thread from an internal thread pool.
# The `threads` method takes a minimum and maximum.
# Any libraries that use thread pools should be configured to match
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum, this matches the default thread size of Active Record.

threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }.to_i
threads threads_count, threads_count

Note: For a production service there is little benefit to setting a minimum thread value.

Now when you deploy, your Puma thread count will match your Active Record thread count so you won't get timeout errors. Later the default for Active Record was adjusted to take advantage of the RAILS_MAX_THREADS environment variable. When you scale your Puma thread count via that environment variable, the Active Record connection pool automatically does the right thing.


On Heroku, we recommend you specify how to run your app via the Procfile — if you don't specify a Procfile we will set a default process type for you. Since Heroku apps run inside containers, they need to know which port to connect to, so we set the $PORT environment variable. The buildpack will specify a web process command if you don't provide one. For example, if you're deploying a Rails 2 app without a Procfile, by default your app would run:

$ bundle exec ruby script/server -p $PORT

In Rails 5 you can now use the $PORT environment variable to specify what port you want your app to connect to. This change doesn't really affect how your app runs on Heroku, but if you're trying to run inside of a logic-less build system it can help make it easier to get your application to connect to the right place.

Serving Files by Default

Prior to Rails 4.2, a Rails app would not serve its own assets. It was assumed that you would always deploy behind some other kind of server such as NGINX that would serve your static files for you. This is still the default behavior, however, new apps can have the static file service turned on via an environment variable.

# config/environments/production.rb

config.public_file_server.enabled = ENV['RAILS_SERVE_STATIC_FILES'].present?

Heroku will set this value when you deploy a Ruby app via the Heroku Ruby Buildpack for Rails 4.2+ apps. Previously you would have to either set this value manually or use the rails12factor gem.

STDOUT Logging

The default logging location in Rails has always been to a file with the name of your environment so production logs go to logs/production.log. This works well for a traditional deployment but when deploying to a container-based architecture, it makes retrieving and aggregating logs very difficult. Instead, Heroku has advocated for logging to STDOUT instead and treating your logs as streams. These streams can then be directly consumed, fed into a logging add-on for archival, or even used for structured data aggregation.

The default hasn't changed, but starting in Rails 5, new apps can log to STDOUT via an environment variable

if ENV["RAILS_LOG_TO_STDOUT"].present?
  logger           =
  logger.formatter = config.log_formatter
  config.logger =

This value can be set by the container or the platform on which your Rails app runs. In our case, the Ruby buildpack detects your Rails version, and if it's Rails 5 or greater will set the RAILS_LOG_TO_STDOUT environment variable.


Support for connection to the database specified in $DATABASE_URL has been around since Rails 3.2, however, there were a large number of bugs and edge cases that weren't completely handled until Rails 4.1. Prior to Rails 4.1, because the DATABASE_URL integration was not 100% of the way there, Heroku used to write over your config/database.yml with a file that parsed the environment variable and returned it back as in YAML format. You can see the contents of the "magic" database.yml file here. The biggest problem is that this magic file replacement wasn't expected. People would add config keys for things like pool which specifies your Active Record connection pool, and it would be silently ignored. So they had to resort to hacks like this code to modify the database configuration

# Hack, do not use with Rails 4.1+

Rails.application.config.after_initialize do

  ActiveSupport.on_load(:active_record) do
    config = ActiveRecord::Base.configurations[Rails.env] ||
    config['pool']              = ENV['DB_POOL']      || ENV['MAX_THREADS'] || 5

Even then, you need to make sure that code gets run correctly in all different ways your app can be booted. For example, if you're preloading your app to take advantage of Copy on Write, you'll need to make sure this code runs in an "after fork" block. While it works around the issue, it normally meant that configuration was spread around an application in many places, and often resulted in different behaviors for different types of dynos.

After the 4.1 patch, Rails merged configuration from the config/database.yml and the $DATABASE_URL environment variable. Heroku no longer needed to over-write your checked-in file, so you can now set pool size directly in your database.yml file. You can see the database connection behavior in Rails 4.1 and beyond explained here.

This allows anyone who does not need to configure a database via an environment variable to run exactly as before, but now anyone connecting using the environment variable can keep additional Active Record config in one canonical location.


At around the time that Rails 4.1 introduced $DATABASE_URL support, Rails was introducing the secret token store as a new feature. Prior to this feature, there was one secure string that was used to prevent Cross-site request forgery (CSRF). Lots of developers forgot that it was in their source, and they would check that into their git repository. It's never a good idea to store secrets in source control, and quite a few applications that were public on GitHub were vulnerable as a result. Now with the introduction of the secret key store, we can set this secret token value with an environment variable.

# Do not keep production secrets in the repository,
# instead read values from the environment.
  secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>

Now we do not need to check secure things directly into our application code. With new Rails 4.1+ apps you are required to provide a secret via the SECRET_KEY_BASE environment variable, or to set the value some other way.

When deploying a Rails 4.1+ app, Heroku will specify a SECRET_KEY_BASE on your app by default. It is a good idea to rotate this value periodically. You can see the current value by running

$ heroku run bash
Running bash on issuetriage... up, run.8903

To set a new key you can use

$ heroku config:set SECRET_KEY_BASE=<yournewconfigkeyhere>

Note: That this may mean that people who are submitting a form in the time between the key change will have an invalid request as the CSRF token will have changed.

Safer Database Actions

One of the scariest things you can say to a co-worker is "I dropped the production database". While it doesn't happen often, it's a serious enough case to warrant an extra layer of protection. In Rails 5, the database is now aware of the environment that it is run in and by default destructive actions will be prevented on production database. This means if you are connected to your "production" database and try to run

$ rake db:drop

Or other destructive actions that might delete data from your database you'll get an error.

You are attempting to run a destructive action against your 'production' database
if you are sure you want to continue, run the same command with the environment variable

While not required to run on Heroku, it's new in Rails 5, and might save you from a minor catastrophe one day. If you're running on a high enough Postgres plan tier, you'll also have the ability to rollback a database to a specific point in time if anything goes wrong. This is currently available for different durations for all plans Standard and above.

Request IDs

Running a Rails app with high traffic can be demanding, especially when you can't even tell which of your log lines go together with a single Request. For example three requests could look something like this in your logs:

Started GET "/" for at 2016-01-06 20:30:21 +0000
  Rendered welcome/index.html.erb within layouts/application (0.1ms)
Started GET "/" for at 2016-01-06 20:30:22 +0000
Started GET "/" for at 2016-01-06 20:30:23 +0000
  Rendered welcome/index.html.erb within layouts/application (0.1ms)
Processing by WelcomeController#index as HTML
Completed 200 OK in 5ms (Views: 3.8ms | ActiveRecord: 0.0ms)
Processing by WelcomeController#index as HTML
  Rendered welcome/index.html.erb within layouts/application (0.1ms)
Completed 200 OK in 5ms (Views: 3.8ms | ActiveRecord: 0.0ms)
  Processing by WelcomeController#index as HTML
Completed 200 OK in 5ms (Views: 3.8ms | ActiveRecord: 0.0ms)

With Rails 5, the request ID will be logged by default, ensuring each request is tagged with a unique identifier. While they are still interleaved it is possible to figure out which lines belong to which requests. Like:

[c6034478-4026-4ded-9e3c-088c76d056f1] Started GET "/" for at 2016-01-06 20:30:21 +0000
[c6034478-4026-4ded-9e3c-088c76d056f1]  Rendered welcome/index.html.erb within layouts/application (0.1ms)
[abuqw781-5026-6ded-7e2v-788c7md0L6fQ] Started GET "/" for at 2016-01-06 20:30:22 +0000
[acfab2a7-f1b7-4e15-8bf6-cdaa008d102c] Started GET "/" for at 2016-01-06 20:30:23 +0000
[abuqw781-5026-6ded-7e2v-788c7md0L6fQ]  Rendered welcome/index.html.erb within layouts/application (0.1ms)
[c6034478-4026-4ded-9e3c-088c76d056f1] Processing by WelcomeController#index as HTML
[c6034478-4026-4ded-9e3c-088c76d056f1] Completed 200 OK in 5ms (Views: 3.8ms | ActiveRecord: 0.0ms)
[abuqw781-5026-6ded-7e2v-788c7md0L6fQ] Processing by WelcomeController#index as HTML
[abuqw781-5026-6ded-7e2v-788c7md0L6fQ]  Rendered welcome/index.html.erb within layouts/application (0.1ms)
[abuqw781-5026-6ded-7e2v-788c7md0L6fQ] Completed 200 OK in 5ms (Views: 3.8ms | ActiveRecord: 0.0ms)
[acfab2a7-f1b7-4e15-8bf6-cdaa008d102c]  Processing by WelcomeController#index as HTML
[acfab2a7-f1b7-4e15-8bf6-cdaa008d102c] Completed 200 OK in 5ms (Views: 3.8ms | ActiveRecord: 0.0ms)

Now, if you have the logs and you find this unique ID, you can filter to only look at information from that request. So a filtered log output would be very clear:

[c6034478-4026-4ded-9e3c-088c76d056f1] Started GET "/" for at 2016-01-06 20:30:21 +0000
[c6034478-4026-4ded-9e3c-088c76d056f1]  Rendered welcome/index.html.erb within layouts/application (0.1ms)
[c6034478-4026-4ded-9e3c-088c76d056f1] Processing by WelcomeController#index as HTML
[c6034478-4026-4ded-9e3c-088c76d056f1] Completed 200 OK in 5ms (Views: 3.8ms | ActiveRecord: 0.0ms)

In addition to this benefit, the request can be set via the X-Request-ID header so that the same request could be traced between multiple components. For example, a request comes in from the Heroku router which assigns a request id. As the request is processed we can log that id, then when the request is passed on to Rails, the same id is used. That way if a problem is determined to be not caused in Rails, it could be traced back to other components with the same ID. This default was added in PR #22949.

This is another feature that isn't explicitly required to run on Heroku, however, it will make running an application at scale much easier.


Rails 5 is the easiest to use Rails version on Heroku ever. We also hope that it's the easiest version to run anywhere else. We're happy that the power of "convention over configuration" can be leveraged by container-based deployment platforms to provide a seamless production experience. Many of these features listed such as request IDs and destructive database safeguards are progressive enhancements that will help all app developers regardless of where they deploy or how they run in production. Heroku has been committed to providing the best possible Ruby and Rails experience from its inception, whether that means building out platform features developers need, automating tasks via the buildpack, or working with upstream maintainers. While we want to provide an easy experience, we don't want one that is too "magical". By working together in open source we can make software easier to deploy and manage for all developers, not just Heroku customers.

If you haven't already, try upgrading to Rails 5 beta.

Check out this Dev Center article for more information on getting started with Rails 5.x on Heroku.

Optimist's Guide to Pessimistic Library Versioning

Upgrading software is much harder than it could be. Modern versioning schemes and package managers have the ability to help us upgrade much more than they do today.

Take for example my post on Upgrading to Rails 5 Beta — The Hard Way. Most of the time was spent trying to find all the different libraries my app was using that weren’t compatible yet with Rails 5 and upgrade them.

What if somehow bundle update rails could have known they didn’t work and instead find a version that did work? It turns out that it can, but it requires some pain on the maintainer’s part. In this post, we’ll look at different strategies for declaring dependencies in libraries, why one is dominant right now, and how we might be able to make major version bumps easier into the future.

Keep Reading on Codeship