Conduct unbecoming of a hacker

Warning: this post wanders around a lot. Sorry.

There was an article a bit ago about opening presentations in old versions of Keynote:

“This presentation can’t be opened because it’s too old. To open it, save it with Keynote ’09 first.” – I was greeted with this message today when I was about to publish few more presentations on Slideshare about Knowledge Management.

Apparently most of HN thought Apple’s oversight was a glaring omission:

We’re not talking “backwards compatibility philosophy” here, we’re talking user experience 101 (something Apple used to hold in the highest regard). – tomalsky

It’s ridiculous that you cannot open the previous version of a Pages file in the current version of Pages! What an absolute joke. – chris_wot

If I were Apple, right now I’d be more concerned with losing customers to Android devices. – Sillhouette

So some background information for you: The business model of most of my software, and the model of a great many people who write software for Apple platforms, is “filling in the gaps”. Reminders not quite cutting it for you? Here are a few hundred competitors. Quicktime Export for iOS not quite doing the trick? Here are a plethora of video encoder products. Apple Maps a little annoying for whatever reason? Enter Google Maps. Safari’s Reading List not working as well as you like? Let me introduce you to Instapaper ¹.

Fixing niche annoyances in Apple software is probably the soundest, surest, and safest business model for the Apple ecosystem that exists. I’ll give you one of my secrets: this is one of the things I look for when deciding what to invest in. If it addresses something wrong in Apple’s ecosystem it’s a safer bet.

It is in that spirit that not just a solution to this problem, but a microbusiness, emerges. If it is really true that people are really frustrated about the inability to open old Keynote documents, it is very straightforward to do something about it:

• Implementation: an online conversion tool, that lets users upload old Keynote documents. These documents are then farmed out to a Mac Mini running several versions of Keynote. The user can be charged a few dollars for the conversion using whatever billing product HN is recommending this week. If we are feeling especially intrepid, we could write very simple iOS/Mac apps that use in-app purchase and upload the documents to the service. Take it from somebody who builds this kind of business for a living: we are looking at one or maybe two weekends.
• Marketing: Google the error message. Find people who are complaining about it. Reply to them that your service solves their problem. Repeat for the first page or two of Google. You have now reached everybody who uses Google to find their problems, e.g., everybody that can possibly be reached.

So the argument goes as follows: if this is a product omission, Apple’s loss is your gain. You are one or two weekends away from solving the problem, helping a lot of people, and making a tidy profit. All the smug satisfaction of winning an internet argument, except this one comes with a cash prize. (Just in case smug satisfaction isn’t your thing.)

If, however, this sounds like a waste of a weekend, a lot of work for something that not many people would pay for, that is because it seemed that way to Apple too. Apple does not run a charity; they do not implement features because that would be the nice thing to do. Apple implements features on the theory that people want to pay for them. If you are arguing that Apple “should have” implemented this feature, you are also arguing that there are people who want to buy it, and that is a point that is fairly easy to prove.

I want to give HN some credit because it may not be immediately clear how a dialog box in a proprietary program is a problem that third-party software developers can solve. Indeed, several point the fingers at closed-source software as the problem which I think is an incorrect, but not entirely unreasonable view. It would be easiest to fix this with source code, but it is sufficiently easy to do it with a conversion service. And certainly the model of filling in gaps in Apple software is an extremely sound one.

And yet an entire thread ostensibly full of hackers continues to whine to nobody in particular that somebody else won’t solve the problem for them. Boo.

We’re all talk

Back when I joined the hacking community, it was about making things. There were mailing lists, and before them there were dead-tree magazines, and they would be full of things that people had made. And there was discussion about those things that people made.

Now, it has always been a critical culture. The phenomenon of the top HN comment that complains about OP is not a new phenomenon, it is older than HN and it is older than the Internet. Various people occasionally argue that we need to be “less critical” and “less abrasive” and I am sorry to be so critical and abrasive, but I think that viewpoint is dumb. Making critical remarks is how we make code better.

However that only counts if the remarks actually do make the code better. At one time, this was more true than it is today. Minimally, criticisms were sent to a person who could do something about it, not about them, to the Internet in general.

And when that person elected to do nothing in particular, the canonical reply was to write a patch yourself. And that patch was posted for discussion, and people would poke and prod at it, and see how good of a patch it was. Did it solve the problem? Did it introduce any new issues?

And if nobody was willing to write the patch, then that was the end of the discussion. A problem is not a real problem if nobody is willing to solve it.

And if you did continue to whine that nobody else would solve your problem for you, you got booted from the list. People did whine, in a sense, but it was very different. Stallman is probably the canonical example: he whines more than just about anyone, but his whining is neatly interspersed by patches, and he is probably one of the most prolific hackers of his generation. In spite of the fact that his complaints are extremely abstract–copyrights, patent law, etc.–they are at once concrete, because he has actually built the copyleft utopia he advocates for. Not blogged about how other people should build it, or gotten into flamewars about how good it might be–he built it. The GNU project is an actual project with actual software and actual users, and now we can post that to the list for discussion. Stallman wrote a patch, and today you can poke and prod at it.

This idea that people just whine and whine until they run out of breath is relatively new. One of the first times I really saw it in full force, accepted by most of the community, was with the App Store Wars in 2008. This was the time that Apple instituted app review for all iOS apps and for one reason or another this was going to End Computing As We Knew It. Blog articles were written and heated debates were had. Even pg got in on the action. But as far as I can tell, everybody complaining just eventually ran out of oxygen. There was no mass customer exodus, and there was not even a mass hacker exodus. Android did eventually rise to become a viable competitor to iOS, but it seems to me this had more to do with pricing and carrier partners than an ideological struggle; certainly the toppling of Apple’s kingdom over this issue never happened. At best, a few egregious app review problems got fixed.

There was another tempest in a teacup when Apple got serious about their 30% cut. Article after article about how now, this time, Apple must correct this injustice or face the terrible consequences of software developers rising up to demand money that is rightfully theirs. Or something. What actually happened is that Kindle removed their online store so now you had to use (the horror!) Safari to buy books, and everybody else just raised their prices 30%. The end. No uprising, no injustices corrected, nothing.

Is it my point that Apple was right and everybody was wrong? No, not really. My point is that in the 80s and 90s when the hacker community was in crisis, the response was to write a patch. Not to whine endlessly in the blogosphere. When we were threatened by Ma Bell and Microsoft (which, kids these days forget, were way way scarier than Tim Cook can even dream of being), we wrote GNU and Linux and the BSDs. And you can laugh about things like “Linux on the desktop” all you like, but these projects are all seriously impressive achievements in their own right, that fundamentally shifted the needle on the software industry. Hackers didn’t topple Microsoft, but they seriously threatened it, and they won several battles, like the battle for servers and the battle for the Internet. Hackers of today can’t threaten anyone, or win anything. We’re all talk.

Inevitably someone will say I am painting the landscape with a revisionist’s brush. But I ask you: where is our generation’s equivalent of a Linux project–one that threatens and wins battles against the monopolist? Where is our generation’s GNU project—one that presents an alternative to the proprietary business models that have concerned hackers lately (e.g. IAP and 30% cuts)? I submit to you that yesteryear’s hackers would have done more, and in fact they did more, than we are doing.

I will give us credit for one thing: we came together and did something about SOPA. But if the shining example of our community is maintaining the status quo, we are in trouble.

The pestilence spreads

This “bikeshedding culture” wouldn’t be so bad if you saw it only in hacker discussion spaces, like HN, because at worst you can just add them to your hosts file. However I am sorry to report that the malaise has now infected places of actually writing code, so the problem is now unavoidable.

I have been working on a project lately that requires me to rope together various FOSS projects and extend them in logical ways. And so the last few months have been a lot of this:

Hello folks, I need [obviously useful feature]. I realize that this is a lot of work, and you’re not going to get to it any time soon, and I need it, so I’m going to do it this week myself. I plan to do the X followed by the Y followed by the Z, and then contribute that back under [your license]. Does that basically sound mergeable to you, or are there other things we should discuss?

(Un)fortunately I have decided not to name the guilty, which makes the next part of this narrative unfalsifiable. For those of you joining us from very well-run projects like git, the Linux Kernel, FireFox, WebKit, etc., it may even be hard to believe. However if you ever have the misfortune to venture into the GitHub Graveyard of projects that aren’t quite popular (and even a few that are, in a way) you will see at once the force I am wrestling with.

My email is inevitably met not with acceptance, nor with constructive discussion, but with some attempt to derail the entire enterprise. Here are some real examples, paraphrased by yours truly:

1. I think it should be done some other way, even though the other way obviously doesn’t work for you and so far nobody has ever been found who is willing to implement it that way
2. I don’t want to solve this problem without also solving [unrelated problem X], your proposal doesn’t address [unrelated problem X], therefore I am inclined to reject it
3. I don’t know you and there might be a bug in your patch. This patch is too important to leave to somebody new. At the same time it is not important enough for any of the core committers to get to it.
4. Defend this proposal. You’re telling me you “need” encryption in an internet communications library, or you “need” unicode support in an object storage library. I don’t believe you. We’ve gotten along just fine for N months without it, and we’ll get along for another 2N months just fine thanks.
5. Look, we’ve already implemented [sort-of related feature] even though it’s buggy and doesn’t cover your usecase. That decision was complicated and people were arguing about it for years and I really don’t want to go through that jungle again. If you wanted to do it this way you should have spoken up two years ago.

In some cases I have made this proposal to my first-choice project, gotten one of the numbered responses, then went to the second and third-choice projects only to get a different rejection at each one. I track most of these in a spreadsheet, and many of them get in long flamewars that run on for months after I’ve unsubscribed, forked, and shipped the solution I originally proposed. Most of those flamewars come to nothing, but a few of them even end up implementing an equivalent solution to the one I proposed, all that time later. Exactly one project in my dataset ever reached a decision that actually improved on my proposed solution, and so far that decision was made in theory only–a separate flamewar erupted between the spec writers and the implementers, and that flamewar continues to this day. Meanwhile my code shipped in 2012.

Is my point that I am the best hacker, mwahahaha? No. These are largely pretty easy patches that most software developers could write in a few days. I do not claim any special talent in writing them. I simply claim that while everybody else was arguing, I was the one who did write them. And without exception, every single argument over one of these patches caused a delay and produced no benefit, at best. I am sorry to report the facts today are that flamewars, not patches, are king.

Sadly, this experience goes beyond just me. Rachel Nabors has even drawn a comic about it:

Even more sadly, Rachel’s experience is often treated as a feature, not a bug. Ilya Grigorik argues with a straight face that “surprise patches are mostly a burden” and he manages to discover more reasons to reject patches than can apparently be listed in an article dedicated to that topic. Perhaps Grigorik works on projects where competent maintainers and reviewers are consistently available to provide legitimate design guidance ahead-of-time. But my spreadsheet is evidence that at best reviewers are unavailable for ahead-of-time guidance and at worst these discussions are an exercise in derailing patches, not improving them.

What I’m trying to say is that our propensity to argue about everything–from politics to tech companies to some feature in presentation software–has now infected that one thing we are actually here to do: hack. You cannot “just hack” anymore, you have to hack and then rhetorically prosecute your hacking against the null hypothesis at frequent intervals.

Who cares? Let people argue if they think it’s fun

Well, let me be clear: I am in favor of recreational arguing. Just read this blog; that is an exercise in the discipline. ².

However, there are some limits. It’s all fun and games until somebody loses an eye. One problem is that when all the hacker spaces are infected with patchless argumentation, we discourage all the up-and-coming hackers who do, actually, write patches. The Rachels of the world are confused: is this what programming is about? It’s about winning the mailing list thread? We also attract people who are good at arguing, instead of people who are good at patches. Those are mistakes, and probably enormous ones, but the effect is hard to prove.

However there is a more visible problem: arguing infects the patches. When your experience leads you to believe that all patches terminate in a discussion that invokes Godwin’s law, you start to take that into account in the design phase. “If this patch isn’t going to be accepted,” you think to yourself, “I should plan on maintaining a long-term fork.”

And if you plan on maintaining a long-term fork, dumb things start to make sense. I maintain significantly more than 10 forks of OSS projects that I meticulously keep up with new upstream versions. Trying to do all that as a single developer leads to really dumb design decisions, like using monkey patching to inject code into a different file to avoid merge conflicts when upstream changes that file. In several cases my changes take the form of a Turing-complete program that dynamically modifies the upstream source code in several attempts until it passes tests. I’m not kidding.

That is the real cost of our flamewar culture. Avoiding unpleasant comment threads is something that disaffected people can do sort-of successfully. However when it is easier to write Turing-complete update scripts than it is to resolve our differences on the mailing list like gentlemen then you have a real problem. And the problem of discouraging the future generation of hackers is probably a much bigger one, although I have no way to know.

Proposed patch

I have some concrete ideas on what we can do. I am not sure they are any good, but I post them to the list so we can poke at them.

One thing we could do pretty easily is create a social rule against whining as such. There should be a canonical reply to whining, like “patch or GTFO”, to encourage redirecting the energy from whining into doing something constructive. Certainly I think there is room for vigorous exchange of views for what the solution looks like. However I think we should have a strong social norm against the endless “Company X really needs to do ABC” type discussions that plague our community and produce neither change in Company X nor any comprehensive response from the hacker community. The place to have that argument is in a bar or something, not in a hacker space. Hacker spaces should be about making things, not whining that other people should make them. Perhaps it is okay every now and again, when the entire community is temporarily shocked by e.g. the Snowden revelations, but the discussion should rapidly converge on doing something, and the whiners should get redirected to /dev/null at a reasonably appropriate time.

However “patch or GTFO” only works if patches are indeed being accepted. Secondly, and this is a little trickier, I think we need to develop a social norm about the “weight of evidence” and the “burden of evidence” of a patch. What I mean is that in a court case, sometimes witness testimony and DNA evidence conflict, and we have developed the rather sensible legal tradition of considering the DNA evidence more reliable, which seems obvious now but it was up in the air for awhile.

When a patch is proposed to the 90%+ of projects that aren’t super popular, there should be a presumption that the patch should be merged. Now, this constitutes a rather large shift in thinking for some people, so hear me out. When somebody submits a patch, that is a tangible claim from a real software developer that doing a specific thing improves the software. Furthermore, that software developer was sufficiently motivated by whatever the problem was to check out the source code, figure out how to build it, write the patch, at least kinda test it, and submit it back. What we have in a patch is a fairly robust and high-effort claim that such-and-such should be done.

What ordinarily happens in such a situation is that the patch is subjected to criticism. Now, this criticism is “valid” in the sense that it accurately reflects the opinions of those who give it. But, I think in the ordinary case with the ordinary project that criticism comes from a place of substantially less effort and less directly relevant experience than was required to produce the patch it condemns. Consider the following outcomes, which happen with some regularity:

• No discussion or action on the patch after a reasonable period. Perhaps the project is unmaintained, or maintained poorly. In the absence of any opposing viewpoint whatsoever, it seems obvious to me that the contributor’s vote to merge should carry.
• Objections from NIMBYs who may not even be software developers. These should be heard, of course, but efforts should be undertaken to quantify how representative of the userbase they may be, and whether the impact is really as substantial as it sounds on the mailing list.
• Objections from the style guide police. It is better to have fixes and features than it is to have code with the desired amount of whitespace. If the style guide is so important, either they can correct patches themselves in a reasonable amount of time or else they can release tools like autopep8 for this purpose.
• Objections that the problem should be solved another way, but that are accompanied without any volunteers to do it that way. A patch in the hand is better than two in the bush. If somebody does end up doing it the “right” way someday, git revert is only 10 keystrokes. The fact that someday a better patch might appear is not an argument against merging an adequate patch right now.
• Bare assertions that there is no need for the feature, when the fact that somebody wrote a patch should be primae facie evidence that the feature was needed

Now I don’t want to give the impression that it is impossible to rebut the merge presumption.

• The patch has an obvious security vulnerability
• The project has a pre-existing, documented, design decision or policy against the feature
• The patch violates a pre-existing, documented backwards compatibility policy. In this case there should be a focus on merging the patch in a way compliant with the policy, like in the next major version, etc.
• There is actual evidence, not mere speculation, that the patch has significant downsides. For example, benchmarks showing degraded performance on an important platform.
• The patch causes test failures

Notably absent from my list of objections is any concern for project stability. It may surprise you that plenty of “stable” projects are liberal in accepting patches. Consider Mozilla Firefox, certainly an important project:

In FireFox, a whopping 82% of patches are accepted on first review. In fact, FireFox reviewers seem quite a bit less concerned about patch quality than the patch writers themselves: after already passing review, many contributors saw the need to make further adjustments even though this was not required. The story is similar for another stable FOSS darling child: “WebKit developers frequently ‘obsolete’ their own patches and submit updates before they receive any reviews at all”. It seems that patch authors are actually quite conservative.

Meanwhile, less than four percent of Firefox or WebKit patches are completely abandoned. These projects are apparently pretty good at “growing” poor quality patches into better ones.

If FireFox and WebKit can do it, so can you. They have more reason to be worried about NSA backdoors, buffer overflows, feature regressions, and performance regressions than 99.9% of projects. If your project’s accept rate is under 50% than something is seriously wrong.

When you see these “we set a high bar for patches” projects in the wild, inevitably it means one of two things. Either the project is a traditional, commercial, corporate-backed, cathedral-style software project, which just happens to be on GitHub because that is what the cool kids are doing these days. Or else it is a lone wolf maintainer with enough hubris to believe that starting a project makes him some kind of code review expert. (I am qualified to say mean things here because, too often, the behavior I am condemning is my own.) Linus Torvalds probably is the world’s ranking code review expert, and he delegates everything possible to other people. The Linux contributing guide specifically says that Linus should only review trivial patches and that complicated stuff should be sent to a deputy. That is your clue that concentrating reviews in one person is not what real experts do. ³

However, human behavior is slow to change “what works”, even when, as in this case, the argument for change is strong. As a compromise position, a project could create a “sandbox” branch that has the “lax” merge policy that I am advocating. The sandbox branch is frozen at intervals and merged into stable, with the effect that patches which survive long enough in the sandbox branch are considered stable. Basically, the Debian release model, applied to individual projects.

So what would we gain from adopting a “patches should merge” presumption? The first thing we gain is more patches. People submit patches on the theory that they will be merged, and if we increase the probability of merging than we increase the number of contributors and increase the project mindshare. I can tell you from my personal experience, I don’t submit a lot of things because after the hard work of fixing X, the very last thing I want is to get dragged into an argument about how well I did it. The result is I only submit the small fraction of my patches that are worth the hassle of defending.

But secondly, we gain by converting flamewars into patch wars. Today, code reviews can be derailed simply by hitting reply on your iPhone in the grocery store checkout line. But under the system I propose, the best way to prevent a patch from merging is to submit a better patch ⁴. Either you do, in which case now we have a fair fight between two concrete and specific ideas. Or you don’t, in which case I question whether you had a serious objection after all.

Really, what I’m asking is this: Which is more convincing? Concrete computer code authored by someone with first-hand knowledge of the defect? Or the bare assertion that something is wrong with it? I mean, either one might be correct. But the first is better supported.

I think that patches, in ordinary circumstances, are strictly better than comments, and we should treat them that way. Our behavior at present is diametrically opposed to this view. A person who wants to contribute a patch first feels out the mailing list about the whole enterprise, then they discuss the design details, then they write the patch, then everybody picks it apart. That is a 3:1 discussion:patch ratio, and there are many opportunities for patches to come under fire from the keyboard of armchair observers. This is far from a meritocratic system where the best code wins. In fact it is a system where the code that can be most easily defended wins, and where the people best at arguing win. That sounds a lot like the legal system, and it is about as effective.

Some projects do better under a BDFL!

Probably. The direction of this rant is not so much “everything should be this way” but rather that your project probably isn’t Python. We have all kinds of software in our community: proprietary software, OSS software, “free as in freedom” software, popular software, niche software, and in my experience very few of them operate with “patches >> whining” as a guiding social norm. They probably exist, and I am ignorant of them. But if I am ignorant of them, so are many people.

Really what I’m asking you to consider is if these principles make sense a lot of the time. If so, we should follow them a lot of the time. We should refine them and give them a cool-sounding Three-Letter Acronym like everything else in this industry and mark applicable projects that we run, and applicable discussion spaces, that subscribe to these principles.

Hacking should be about making things. And yet a great many of our institutions are set up to discourage, distract, destroy, and derail the making of anything. It’s time we called it what it is: conduct unbecoming of a hacker.

---

Incidentally, if you made it to the end of this post, whether you thought it was good or you thought there are some things I missed, I need your help. My usual circle of editors were too busy this week to give me adequate feedback on this article. If you or someone you know is interested in reading Sealed Abstract posts like this and making comments so I can improve them prior to publication, please send me an e-mail.

---

Want me to build your app / consult for your company / speak at your event? Good news! I'm an iOS developer for hire.

Like this post? Contribute to the coffee fund so I can write more like it.