One of the remarkable things about computers is how quickly things change. Of course, all good computer scientists get drilled into their heads at a young age that Moore’s Law will not fix a poor algorithm and may very well run out. Even so, the idea that something can double for any amount of time, in any observable period, is insane. And Moore’s Law seems to be picking up in some areas, like mobile, even as it may be slowing down in others. We do not double anything else in eighteen months, unless it is Randall Monroe’s religion.
People talk a lot about how the software side of things is in trouble, that writing software has not really improved, and does not really improve. Famous people like Fred Brooks write seminal books with titles like “No Silver Bullet” that drill into you that this software thing is pretty difficult. And this is true, but it is mostly true relative to the astonishing progress in the hardware space. If you set the improvements of software development relative to, say, the improvements to the field of structural engineering, software is simply fantastic. Absolute staples like GCC and CVS are in their mid-twenties. I don’t know what kind of major changes the doctors and lawyers and electricians have been dealing with in the last 25 years, but I can promise you it doesn’t hold a candle to the change in the programming profession marked by the compiler, the debugger, and source control.
And we’re just getting started. Things like high level languages and garbage collection algorithms and elliptic curve cryptography and distributed source control and static code analysis and unit testing are big deals. Even if you think that half of those are crap, the fact that you know what they are is huge. Most professions don’t have that many contenders for improvement in many decades. Even humble problems that have probably occurred to you remain low-hanging fruit that are the gateways to huge improvements in the programmer’s condition.
This idea–that we can improve the way in which we write the tools that we use to improve the way in which we write the tools–is the closest thing to a self-sustaining singularity that humans have ever devised.
It also leads to very strange results. If you are in charge of a large number of doctors, for example, and want to divide them sensibly between treating patients now and doing research that may lead to cures in the future, you will assign many more doctors to the “treat now” than the “research” column.
But if you are in charge of programmers, it is almost foolish to allocate anyone to build “end user” software. The productivity you get from one lone Linus Torvalds banging out git translates into tens of thousands of man-years saved on every subsequent project, or, as the case may be, on every subsequent research project which then snowballs into subsequent research projects and thus, the singularity. And even if Linus is a bit of an outlier and most people’s research is a bit more mediocre–say being a few times better some of the time–the gain still exceeds the investment, often by a wide margin.
If you are in charge of programmers, and you are not doing the bare minimum “end user” work that you can get away with–you are foolish. If an investor finds a goldmine that produces 50% returns year-over-year, for decades, they do not put 5% of their portfolio there. They’d go out of business. That’s what we do in software.
Another way to say the same thing is that we have written most of the software that we need for the society in which we live. We have about enough social networks now, and enough point of sale systems, and enough social photo whatever apps, and enough accounting systems. What we do not have enough of is self-driving cars, computational knowledge engines, package-delivery drones, and task-scheduling AIs. The “bugs” in life–in the broad strokes I think–are mostly closed. There are not many problems that 21st century western middle-class laymen have that mere application of Python can solve.
What remains are the “feature requests.” The “refactors”. The task that remains for software is not to increase the magnitude of the vector of progress, but instead to change its direction; not to create better tools for society to solve its problems, but to change the society so that the tools are unnecessary. We should not be coming up with more podcasts and voice-activated whatsits to recover one’s commute time; we should eliminate the driving. We should not be creating more apps to help navigate the grocery store; we should eliminate the need to go there in the first place. We should not be designing e-mail clients that display massive amounts of e-mail marginally more effectively; we should be eliminating the amount of e-mail people need to receive. We should not be creating more “power” to-do list methods; we should be thinking of ways to constrain our tasks so that a list is not required to manage them. This is progress, this is better–the other kind of change is just more, just piling on. Like the software that was sold on the basis of the longer bullet-point feature list. It’s not good, it’s just longer.
Obviously we can no more set out to build a self-driving car today than we can set out to build a warp drive. But this fact an obscures an important truth; that we must learn to walk before we can run. Software engineering is still at the stage where someone with a laptop in a broom cupboard somewhere can raise the tide for all ships. (By comparison, to move physics forward, it takes several acts of Congress. And even then, it’s not enough.) That time you spend to marginally improve the garbage collection or to tweak the unit testing framework leads to better tools that the computer vision guys can use to build the cars. And the huge payoff of just a small amount of research and the interest that it rapidly accrues means that very small improvements in writing software are actually very big. We are only a few doublings away from cars or package drones.
This idea–that process improvement is of tremendous value–is an idea that I think many programmers internalize, as many of us automatically and without any prompting spend as much time as we can find building and improving the tools that we use. But the consequence of it being so easy to invest in tools quietly is that we actually do it quietly, and management is never clued in, either to the success of the system, or the extent to which getting anything done depends on it, out of the (sometimes rightly-placed) fear that they might kill it.
This is all fine if our goal is to write Yet Another Point of Sales System, as a few days here and there is all the time you really need to build wonderfully awesome metatools for that problem. But if you have your sights set a little higher, your research project might be measured in man-months or man-years. And that is not, really, a scale that can be effectively hid from management (unless you are working on Graphing Calculator). Needing management to approve a research project is an unexpected condition, and so an exception is thrown.
Naturally, the easiest way out is just to do it on your own time with people who already agree that it is a good idea, rather than going to bat for it with a hostile audience. Maybe, someday, after it is already proven to work out, management will buy in. And this is how, essentially, all the important things like Linux and Git and WebKit get done.
In some sense, this is a victory, as all the ships rise, and we are one step closer to the Sci-Fi Future. All fine and good.
But in another sense, I think it is kind of a defeat. The ships rising becomes a natural force that we expect to happen rather than a culture that we must cultivate. We just expect “them” to continue to release new versions of, say, LLVM, or Python, or SQLite, or valgrind. “Them”. “The LLVM maintainers.” Not the brilliant coders Jeff and Frank who we take out for beer every third Wednesday and recommend for consulting gigs and are honored to know personally. Not the heroes that we aspire to become, or the trailblazers to whom we must later pay it forward. If anything, we know them as those stupid idiots who introduced a bug into the new release (a bug we did not even bother to file), not the luminaries who work tirelessly to make the lego bricks we depend on every day to make buildings out of.
And then we wonder why managers are hostile to our own research projects that are unproven and speculative.
This broken, backwards system has produced some really incredible results, like git and LLVM. But I do not think it scales all the way up to self-driving cars. To solve the new problems, we need to do real research. We need to be throw tens of thousands of man-years down research black holes. We need an order-of-magnitude shift that moves the majority of developers out of the “end user” column and into the “tools and research” column.
I think we need to turn the entire pyramid on its head. We have to change the way we think about “The LLVM maintainers.” We should be thinking about the people who maintain the packages that we use, and we should ourselves respect them in a way that is commensurate with how their tools have improved our lives. Because if we cannot respect their contributions to our lives, we cannot expect our efforts to contribute to the wellbeing of others to be respected.
Second, if we believe the gospel, we should preach it. The only reason any software gets written around here at all is because we owe a tremendous debt to Jeff and Frank. Just try and imagine for a minute, making a list of all the software packages you use in one day directly (ignoring the huge problem of things you indirectly use like Mach or WebKit). What fraction of that software has you or your company sponsored, whether through licensing fees or otherwise? 1%? less?
Would your manager be surprised by this figure? To discover that the only reason you ever get anything done at all is because somebody did some unfunded research? They shouldn’t be. They should be hearing about it from you. Daily.
Third, we should clue the managers in more when we work on the little research projects ourselves, because it communicates to them that research is important and good, instead of what we actually tend to communicate, that it is something to be hidden away. If you are tinkering with the template engine, it should be in a memo! Research is good! You do not hide a light under a bushel.
I think we wonder sometimes why so much of software is Yet Another Lolcats or social network or photo sharing whatsit. Hackers wonder why “the best minds of [our] generation are thinking about how to make people click ads” instead of, let’s say, SpaceX, or curing cancer. But can you really blame the VCs and the managers and the owners for putting money into the social whatsits? What else have we left them to fund? They cannot fund a compiler, because you cannot compete with LLVM. They cannot fund a better clang-analyze, because they have never even heard of it or how much money it saves, because nobody has bothered to tell them. They cannot fund a unit testing tool, because nobody has told them about unit testing, because everyone was afraid that they would mandate it. They cannot fund experimental for-other-developer projects, because not even the experiments that succeed are well-respected by the programmers who use them, let alone does any programmer ever pay for them. They fund lolcats because it is all that we have left over for them to fund.
This is not to say that we have not made the right choices in the past, to develop free compilers that eliminate their competition, to keep clang-analyze a secret because management might rightly kill it. Rather, I am saying that these choices do not scale to the future. They do not get us to driverless cars. They may have taxied us onto the runway, but they will not get us into the air. We need to start moving people en-masse out of the “end user software” column and into the “research” column. Where we want to go, you can’t just throw a few extra weekends at the problem anymore.
This is a tall order, to change an industry, to admit that the way we have organized ourselves is fundamentally wrong. And a lot of people have come to the conclusion, and I think it’s a fairly reasonable conclusion, that we’re probably not going to make it. That we will not change. That the future of software is in lolcat apps and building Yet Another Whatever that we put in the pile with the rest of the whatevers, and that we will keep getting the slow trickle of LLVM releases, but no package drones. And I fear that they are right.
But if we are just picking sides, I’ll be on the side with the cool sci-fi gear.
Want me to build your app / consult for your company / speak at your event? Good news! I'm an iOS developer for hire.
Like this post? Contribute to the coffee fund so I can write more like it.
Tags 
- Subscribe via e-mail
Comments
Comments are closed.