Of Wizards and Magical Machines

People want things to be cheaper and easier and faster than they are.

That doesn’t seem very extraordinary, does it? I mean, it’s not really news. In fact, the opposite would be news–if people wanted things harder, slower, or more expensive. You may not even believe it is true. People in fact pay more for the same good if it is packaged slightly nicer, for instance. How do you explain that?

However, today we are not going to deal with small 20% trifles. We are dealing with orders-of-magnitude, of $8 against $80,000. It is not incorrect to say that people prefer to spend 8.

An interesting fact about software is that the price of it is essentially a step function: if there exists software that does what you want, you can get it from the App Store for $5. If there is not, the cost to produce that software starts at $10,000, and is probably closer to $100,000 for many applications. What people desperately, desperately want is some way to pay about $500 for something marginally better than a prepackaged solution and marginally worse than a custom solution. They want to take some existing program that they have, slap a feature or two on it, and call it a day. Unfortunately (at least in closed source world), it doesn’t work like that, you have to replicate existing software, it’s expensive and hard. So you have one group that pays the $10,000 under protest and another group that goes home empty-handed. So far this should all be review.

The interesting bit is that all software is like this. I don’t mean to suggest, of course, that there isn’t a Photoshop or Word clone available at every conceivable price point. But once you hit anything that even smells like a niche, there are only two prices: affordable and oh wow that’s expensive.

Of course this dichotomy, in the strict sense, is tautological. “You can either afford a new car or you cannot.” But that is not the kind of statement I am making–perhaps you walk into the car dealership expecting to pay $15,000 and end up paying $20,000. But you do not walk in expecting to spend $1.50. People routinely make that mistake in custom software–they have a budget that is four, five, six, seven zeroes wrong.

Another way to see this is to consider Siri. Suppose you want a virtual assistant that has pretty reasonable English spoken word comprehension. Your choices are, quite literally: put a bunch of PhDs on salary for a decade, or: buy an iPhone. It’s either very expensive, or free with a new phone. There is no $10,000 model.

The economics of this lead to a very interesting situation: essentially all software is subsidized. You are using an operating system that is subsidized by Berkley, typing on a keyboard subsidized by IBM, doing a heck of a lot of things subsidized by Xerox, a network subsidized by DARPA, a web browser subsidized by Google, Apple, and Netscape, etc. To a nontrivial extent, Python is subsidized by Google, Linux is subsidized by IBM, C++ is subsidized by AT&T, and [arbitrary software package you use] is subsidized by [arbitrary large entity]. Even if their only contribution is to feed and water and do laundry for the open-source developers working on the project, it’s subsidies all the way down.

When the cost of something is subsidized, people tend to forget the true cost and they start to think only about the perceived cost. It is easy to forget, when you install Google Chrome, the millions of man-hours of engineering effort, the billions of dollars, the decades of development time, the companies that have risen, grown, and withered away, that have produced those 35 megabytes. Not to mention the legal battles that have threaten to bankrupt hundred-billion-dollar-corporations, the IP lawyers, people to keep the air conditioning cold and to sweep the floors, and so on.

As software developers, we’ve forgotten all this. Even to a developer, the dizzying yet invisible depth of computing borders on actual magic. We’ve forgotten what web browsers cost. We don’t value web browsers. Or text editors. Or HTTP servers. Or operating systems. Or shells. Or programming languages. Or compilers. Because they’re old hat. Just pull it down from GitHub.

And if we’ve forgotten about it, imagine the users! To a user, the economies of scale in software are insufficiently distinguishable from magic. If we can accept that ordinary people download billion-dollar web browsers for free on a daily basis, is it really a such a stretch that they should expect their custom software to be merely five zeroes cheaper? Far stranger things have happened, and are happening, and continue to happen to ordinary people month after month in this industry. When you set their custom software package next to things that they already use–that were built by Xerox, Apple, Google, DARPA, and so on, the problem is absolutely miniscule. The frame of reference is that software developers are wizards who use magical machines to produce arbitrarily fast, cheap, and awesome software that improve everyone’s life by magic and the whole cost part of the equation is entirely a black box that somebody else somewhere else has funded.

And so now you start to see the first crack in the statement. People want things to be cheaper. But in lots of other fields, people have some kind of intuitive sense as to what something costs–they would not try to buy a car for $1.50. But as we have discussed, in software, the intuitive sense of the cost–to whatever extent it exists at all–is wrong. But not just a little bit wrong–the kind of wrong that would lead you to believe a billion-dollar software package is free. And when you remove this intuitive sense of cost, the whole train derails. People want–nay, demand, quite angrily in fact–arbitrarily awesome software.

At this point you may wash your hands of the matter and say “Well, they’re not going to get it. It costs what it costs. They will either decide it is worth it or not. Let me get back to coding my web app now.” But I think the problem is more pervasive than that, and has far-reaching implications that are quite concerning.

For starters, there are some requests that are not really feasible to decline. If you’ve watched Doctorow’s fantastic talk about the coming war on general-purpose computing, or read Skala’s incredible “What Colour Are Your Bits”, you know that there is a pretty big disconnect between lawmakers and technologists, and this disconnect is merely the 0.9 beta, that the real war on technology is coming. But if you ignore, for a minute, your ethical feelings about IP issues, and what laws like SOPA mean for us as an industry and personally as software developers, and look at it purely from a feasibility point of view:

So today we have marketing departments who say things like “we don’t need computers, we need… appliances. Make me a computer that doesn’t run every program, just a program that does this specialized task, like streaming audio, or routing packets, or playing Xbox games, and make sure it doesn’t run programs that I haven’t authorized that might undermine our profits”. And on the surface, this seems like a reasonable idea — just a program that does one specialized task — after all, we can put an electric motor in a blender, and we can install a motor in a dishwasher, and we don’t worry if it’s still possible to run a dishwashing program in a blender. … “can’t you just make us a general purpose computer that runs all the programs, except the ones that scare and anger us? Can’t you just make us an Internet that transmits any message over any protocol between any two points, unless it upsets us?”

SOPA has been discussed to death on ethical and legal grounds. But when you look at SOPA or encryption laws, or another arbitrary electronic freedom issue, through the lens of “feasibility”, it simply reduces to “solve this hard technical problem”: build me a box that does X, or that doesn’t do X. But as we’ve discussed, nontechnical people have no feeling for the cost involved in this. “We can index the entire corpus of human knowledge, but we can’t build a computer that doesn’t run certain programs? We’re the US government!” And so it’s one thing for a few crazy people to have a poor idea of what it costs to build an iPhone app, but it’s another thing entirely when those crazy people get elected and are now in the full-time job of evaluating technical costs, and are mandating technical projects into existence by legislation.

And if that is the extreme example, consider the common case. I have been involved in far too many projects where the development timeline and costs for a project were decided, not investigated. You may be working in such an organization now. I regularly participate in conference calls with people who explain to me how long it will take to do my job. In fact, that is the state of the industry. Perhaps eventually, the project will fail, and you can whip out some e-mail from six months ago and say “I told you so.” But more likely, you will work 60-hour weeks for a while, and then get blamed when the project goes awry. We have billion-dollar web browsers for free, why should it take 3 months to write an inventory system? That’s pretty much how those things go.

Now consider hiring. As a hiring manager, perhaps your frame of reference is no longer web browsers–perhaps it’s, say, Microsoft Office licenses. You’ve licensed Excel at $300/seat for your office of 300 people for total damage of $30,000. Now you need some other arbitrary internal custom app. Custom software development should be more expensive than off-the-shelf licensing, right? Hmm… perhaps… 3x more expensive. So $90,000. Some HR intern puts $60k-90k plus benefits on a job posting on Monster.com, and that’s the story of every in-house software development position ever.

In reality, the size of the problem is a step function: it’s actually $10,000, $100,000, or $1M. If it is a $10,000 problem, some developer spends ten months a year reading Reddit. If it is $100,000, some developer is a bad negotiator. If it is a $1M problem, you start to see articles about how there is a developer “shortage”. These articles are greatly amusing to me, partly because I have this romantic notion of characterizing my fruitless search for a $1.50 car as a “car shortage” and being called for comment by NYT reporters.

As technologists, we’ve run a pretty awful PR campaign. Things started out OK–we billed computers as the awesome machines that everybody should have, and that part has been a success. Computers are great! People love them! But the marketing campaign worked too well. Now people believe that computers are infinitely awesome–and we have no hope of meeting anyone’s expectations any more. It’s a runaway train that’s hit escape velocity, zooming into outer space. Anything less than miraculously perfect is a failure.

Louis CK has a routine called “Everything’s amazing; nobody’s happy”.

We live in an amazing world and it’s wasted on the crappiest generation of spoiled idiots that don’t care. Because this is what people are like now, they’ve got their phone they’re like ‘Urgh it won’t [work].’ Give it a second, its going to space! Can you give it a second to get back from space? Is the speed of light too slow for you?
I was on an airplane and there was high speed internet on the plane for you, that’s the newest thing I know that exists. They’re like ‘open up your laptops and go on the internet’, so I’m on YouTube and then it breaks down. The guy next to me was like ‘This is bullshit!’ Like how quickly the world owes him something he knew existed only ten seconds ago.
Flying is the worst one, because people come back from flights and act like it was a horror story. They act like their flight was a cattle car in the 40s in Germany. That’s how bad they make it sound. They’re like ‘This was the worst day of my life, first of all we didn’t board for twenty minutes. and then we got on the plane and they made us sit there, on the runway for forty minutes.’
Oh really? What happened next? Did you fly through the air incredibly like a bird? Did you partake in the miracle of flight you non-contributing zero? You’re flying, it’s amazing. Everybody on every flight should just constantly be like oh my god, this is amazing. You are sitting in a chair, in the sky.
…
People say there’s delays on flights. Delays, really? New York to California in five hours. That used to take 30 years to do that, and a bunch of you would die, and have a baby, you’d be with a whole different group of people when you got there. Now you watch a movie and take a dump and you’re home.

We have a crisis of expectations management. The bar is moving at the rate of Moore’s law in the 90s, but unfortunately that type of sustained innovation appears not to be sustainable. Developers are in the unfortunate position now of being compared and evaluated by the incredible hockey curve we saw in the dot-com boom. If we alt-tab back to Vim and say “Well, they’ll find out their expectations are unrealistic sooner or later,” we are participating in the problem.

I’m reminded of this anonymous rant on HN:

Software, as Marc Andreesen said in his famous editorial, is becoming a larger and larger part of our daily lives. Given how important software is, then, you’d expect computer programmer – the task of developing and maintaining software – to be one of the highest-paid, most-wanted jobs. It isn’t. Why not?…

Is programming a top job from a social perspective? Again, no. Congress includes not a single programmer, and to my knowledge, it never has. Almost all big companies are run by MBAs. Even Microsoft, arguably the canonical software company, is run by a non-programmer from Stanford Business School.

Are programmers top government advisors? Are they national heroes? Do doctors and lawyers and policemen tell their children that, if they work hard and practice, one day they can grow up to be a programmer? No. Obviously not.

The reasons are many, but perhaps one is that as a discipline we essentially have no spine. We do not pipe up in meetings and say the timeline is unrealistic. We do not put our foot down and demand a nice work computer and four monitors. Doctors and lawyers do not put up with this kind of thing; developers do. And when the project fails we move on to the next failing project. We have the power to insist on a successful project but not the will.

And I think open-source software deserves some of the blame as well. Some of the gains we’ve made in free and open software have been made at the cost of accidentally promoting the idea that the work of software engineers is easy and valueless. I am not, of course, advocating against open source–but perhaps we could throw up a little nag screen on install? “Hey, hope you are enjoying FireFox, thousands of people have been working hard on it since 2003. If you enjoy using this software, consider making a donation to the Mozilla Foundation and/or ChildsPlay/EFF/etc.” Not like the BSD fourth clause, of course, not a licensing requirement. Just something that would give an average middle-manager who installs it from firefox.com an idea of the scope of the undertaking to produce these incredible tools. On par with Wikipedia.

To outsiders, we are wizards with magical machines–but we have somehow failed to capture the aura of respect and admiration normally bestowed upon that office. We stereotypically prefer to avoid conflict and let others set the direction, whether successfully or unsuccessfully, without much input. But I believe this strategy is beginning to work against our interests as masters of a craft, and further is ultimately detrimental to the quality of software we produce and the happiness of our users. The fiction is that software pervades more and more industries, and so our power grows. But the reality of the invasion is the opposite: that good software engineering is trespassed upon by a gaggle of unrealistic deadlines, incompatible business processes, and impossible expectations, and we sit on the sidelines and allow it to happen.

---

Want me to build your app / consult for your company / speak at your event? Good news! I'm an iOS developer for hire.

Like this post? Contribute to the coffee fund so I can write more like it.