The Optimism Queue

What happens when the tools finally catch up with your imagination?

During the dot-com years, there was a particular kind of failure I remember almost as clearly as the excitement of what we were building. We were creating applications and attracting audiences faster than the hardware and bandwidth could always keep up. You would click on something and wait, usually without a useful explanation of what was happening. The site hadn’t necessarily crashed. It had accepted more work than it could handle, and your request was somewhere in a growing line of requests that might not get answered before you gave up. The technical problem was server overload. A 2001 Berkeley paper described services slowing so dramatically under that load that they appeared to have crashed, even while the system was still trying to work through the queue.[1]

I’ve been thinking about that lately because I’m beginning to recognize a little of myself in it. I’m a self-professed idea guy, which sounds harmless enough until you combine it with an eternal optimist and a general belief that there is very little we can’t figure out how to do. An idea rarely stays small in my head for very long. I start thinking about how it could work, what else we could use it for, and who might benefit from it. Before I’ve really considered whether it belongs in my life, I’m already figuring out how to make room for it.

I sometimes joke that I’m missing whatever gene makes people look for the downside before getting excited. I suspect it’s the same gene that allows them to have an interesting thought without immediately sharing all of it with whoever happens to be nearby. It’s not that I’m incapable of seeing risk or applying logic. Those things just tend to arrive a little late to the conversation, usually after I’ve explained why the idea could work and what we could do with version three. People sometimes tell me, “Your passion is obvious,” and I generally choose to take that as a compliment, although I realize it may occasionally be a polite way of asking, “Are you going to shut up at some point?”

The thing is, the excitement is real every time. There’s this rising feeling when the pieces start coming together, followed by that familiar thought: This one is it. Maybe it’s a business, maybe it’s a better way to handle something at work, or maybe it’s an answer to a question that has been bothering me. Whatever it is, for a moment I can see how it might all fit. That can happen dozens of times in a week, sometimes what feels like hundreds, and somehow the last round of excitement does very little to temper the next one. You’d think I would recognize the feeling and make it wait outside for a few minutes. Usually, I invite it right in.

That’s where the merry-go-round gets dangerous. I have an idea, convince myself it has merit, and get to enjoy the feeling of what might happen if it works. Following it a little further reveals another possibility, which is interesting enough to deserve a look of its own. Somewhere along the way, I’ve already received a fair amount of satisfaction without having finished much of anything. I’ve had the pleasure of discovering an answer, at least in my imagination, and now there’s another one to explore. It doesn’t feel like going around in circles when you’re doing it. Each turn seems to show you something you hadn’t seen before.

For most of my life, the world supplied some of the restraint I didn’t naturally provide for myself. I might need a programmer, a budget, a skill I didn’t have, or someone who knew enough about the subject to tell me where I was going wrong. I spent a lot of time trying to get around those limitations, finding existing tools and connecting pieces that would get me closer to what I had in mind. But there was still a considerable distance between an idea and something I could actually try. Some ideas survived the trip, and others faded while I was busy with the rest of my life. I didn’t appreciate the friction at the time, but it was doing some filtering for me.

Now I can follow a question almost as soon as it occurs to me. With AI, I can investigate a suspicion, test an instinct, sketch out an application, or start taking apart a process that seems more complicated than it should be. I can get something in front of me while I still remember what excited me about it in the first place. It may need a lot of work, and it may turn out to be wrong, but I have something to examine instead of another note about something I might someday look into. For someone who has spent years trying to shorten that distance, this is pretty wonderful.

It also makes my particular problem harder to argue with, because some of the optimism is justified. I really can get further than I could before. I can learn something, build something, and occasionally finish something that would once have required considerably more time or help. So when the next idea shows up, I have recent evidence that pursuing it might be worthwhile. This isn’t simply a matter of a machine telling me all my ideas are brilliant. Sometimes I have a working result sitting in front of me, and it’s hard not to wonder what else I could do.

Of course, getting something to work is only one of the questions. Whether it solves a problem worth solving, whether anyone else needs it, and whether I want to be responsible for what comes next are separate matters. A promising first version doesn’t answer those questions, although it can make me feel as though I’m well on my way. By then, I’ve started to care about it. I’ve seen enough to imagine the finished thing, which makes it much harder to put down than it was when it was just a passing thought.

This is where I start recognizing the old server problem. Every encouraging result creates another request for my attention. Something needs to be checked, a decision needs to be made, or a person needs to be involved before it becomes useful outside my own screen. A simple experiment can turn into a project before I’ve consciously decided to take on a project. Meanwhile, everything I was already responsible for is still there. I may be able to do more work, but I’ve also become much better at generating work for myself.

I think of this as optimism overload. My queue isn’t full of things I dread doing. A lot of it is full of things I would genuinely love to get back to. Each one had its moment when I could see what it might become, and I still remember enough of that feeling to resist letting it go. That’s what makes the queue so easy to grow. I can explain why nearly everything in it is worth keeping, without ever quite explaining when I’m going to do it.

“Pending” is a very comfortable word for an optimist. I haven’t abandoned the idea, and I haven’t decided it was a bad one. I just haven’t gotten back to it yet. That sounds perfectly reasonable when we’re talking about one thing. Across enough things, though, I’ve made an unreasonable number of promises on behalf of a future version of myself who apparently has fewer responsibilities and much more free time. At some point, that guy should probably get a say.

It doesn’t help that a new idea rarely takes its place at the back of the line. It has all the appeal of the older ideas without any of their accumulated difficulties. I haven’t yet found the awkward part, the tedious part, or the part that requires a conversation I’d rather not have. It’s still mostly possibility, and possibility is my favorite part. I can come up with a very convincing explanation for why it deserves a little attention right now, even though I’ve used much the same explanation for several other things that are still waiting.

The cost can be difficult to see because there’s often useful work happening throughout this process. I’m learning, solving problems, and making things that genuinely do something. But I can also get to the end of a busy day and realize that the most important commitment hasn’t moved very far. I’ve been working; I just haven’t necessarily been working on what most needed me. That’s an uncomfortable distinction when you’ve spent the day feeling productive and have plenty of evidence to show for it.

Engineers have ways of dealing with overload that don’t involve allowing the queue to grow forever. They limit what gets admitted, give some requests priority, and sometimes deliberately drop work so the rest of the system can keep functioning. One of those approaches is called “load shedding.”[2] I understand the logic immediately when we’re talking about a server. Applied to my own ideas, I find it considerably less appealing. My first instinct is to look for more capacity, a better tool, or a clever way to get everything through. I would much rather solve that problem than decide which things aren’t going to happen.

I don’t want to become someone who demands a business case for every interesting thought. I enjoy exploring, and some of the best things I’ve done began with a question that didn’t have an obvious destination. What I need to get better at is letting an exploration be enough. I can answer a question without turning the answer into a new undertaking. I can build a prototype, learn something from it, and stop there. Enjoying the possibility doesn’t have to mean volunteering to bring the whole thing into existence.

For the ideas that do become commitments, I need to give the less exciting questions a little more room. What would have to be true for this to work? What am I assuming because I want it to be true? What will get less attention if I take this on? I don’t have to lose my enthusiasm to ask those things, but I do have to pause long enough to hear the answers. Otherwise, I’m asking for a risk assessment while already reaching for the tools.

Naturally, as I think about all of this, I can feel another idea forming: a better system for managing the ideas, probably using AI. I can already imagine how it would work, which is both funny and slightly concerning. So I’m not writing this as someone who has figured it out. I’m trying to catch myself somewhere between the excitement and the commitment, while there’s still an opportunity to choose what happens next.

I still like being an idea guy. I like believing we can figure things out, and I don’t want to lose the joy of seeing something take shape that didn’t exist a few hours earlier. I would just like more of that joy to come from finishing the things I care about, and a little less of my time to be spent reassuring myself that I’ll get back to them. I spent years wishing the tools could keep up with my imagination. Now that they’re getting closer, I’m starting to think my part of the work is deciding what deserves to go into the queue—and allowing a good idea to remain a good idea without making it another promise.

Sources

[1] Matt Welsh, David Culler, and Eric Brewer. SEDA: An Architecture for Well-Conditioned, Scalable Internet Services. Proceedings of the 18th ACM Symposium on Operating Systems Principles, 2001.

[2] Google. Site Reliability Engineering, “Addressing Cascading Failures.”