BACK TO BLOG

Support · The Job · Incident Response

“It’s Broken” Is Not A Ticket. “Works For Me” Isn’t An Answer.

Will Farrar · · 5 min read

The ticket says "Portal is broken, please fix ASAP." Priority: Critical. Category: Other. That is the entire ticket.

So you do what everyone does. You ask for steps to reproduce. You wait a day. They answer "it's still broken." You open the portal, it loads fine, you resolve it Cannot Reproduce, and you both walk away privately convinced the other one is the problem.

I have been on both ends of that exchange for twenty-five years, and most of what I believed about it is wrong. There is real research here going back to 2008, and almost nobody in operations has read it.

The comfortable theory

Ask any IT department why the Cannot Reproduce queue is full and you get the same answer: users don't know how to report problems. Train them. Add required fields. Make them pick a severity.

Tidy theory. It also puts the fault entirely outside the department, which is usually a sign to go check it.

What developers want, and what reporters can actually give

In 2008 researchers did something nobody had bothered to do: they asked both sides. They contacted 2,226 people across the Apache, Eclipse and Mozilla projects and got 466 responses — 130 usable from developers, 215 from reporters. Developers were asked which items in a report helped most; reporters, which were hardest to give. The work is What Makes a Good Bug Report?, published at FSE 2008; the figures below are from the authors' full technical report.

On the developer side, steps to reproduce won and nothing came close: 83%. Stack traces, 57%. Test cases, 51%. Observed behavior, 33%. Screenshots, 26%. Expected behavior, 22%.

Those percentages are conditional: of the developers who said they use a given item when fixing bugs, the share who also ranked it among their three most helpful. Which makes the bottom of the list startling.

Severity: 0%. Hardware: 0%. Product 5%, operating system 4%, component 3%.

Zero. Of the developers who said they look at the severity field, not one ranked it among the three most useful things in a report. We have built an industry of intake forms around a required dropdown the people receiving it do not use.

Now the reporter side. Hardest to provide: test cases, 75%. Steps to reproduce, 51%. Code examples, 43%. Stack traces, 24%. At the easy end: observed behavior, 2%; expected behavior, 3%.

Put those two lists side by side and the gap is the whole story.

Asking a user for reproduction steps is asking for the hardest thing on the list. Asking what they saw and what they expected is asking for the easiest — and it is where triage actually starts.

That changed how I answer a vague ticket. "Steps to reproduce, please" is a request for homework. "What did you see, and what did you expect instead?" is a question a human being can answer in one line while annoyed.

The part that broke my assumption

Here is where it stops being about ticket hygiene.

In 2014 researchers at the University of British Columbia pulled 188,319 bug reports across six projects — Firefox, Firefox Android, Eclipse, MediaWiki, Moodle, and one proprietary mobile codebase — and found 32,124 resolved as non-reproducible at some point. On average, 17% of all reports; Firefox ran 28%.

Then they hand-classified 1,643 of them to find out why. If the comfortable theory were right, one bar would dwarf the rest.

It doesn't. Insufficient information accounted for 14%.

The largest cause, at 45%, was inter-bug dependencies — the report was a duplicate, or the behavior depended on another defect already sitting in the tracker. Environmental differences came second at 24%. Conflicting expectations, 12%.

A second team repeated the exercise in 2020 on 576 non-reproducible reports from Firefox Core and Eclipse JDT, with a different taxonomy and 13 professional developers validating the categories. Same shape. Duplication came out on top again — 26.8% in Firefox, 31.3% in Eclipse. Missing information was 13.8% in Eclipse and 1.5% in Firefox.

Two independent studies, different methods, same conclusion. When you close a ticket Cannot Reproduce, the likeliest reason is not that the reporter failed you. It is that the thing they hit is already known under another number, or lives in a corner of the environment you weren't standing in.

Be careful how far you take that. Environmental differences — cached data, an extension, a stale setting, a different network path — are information somebody could in principle have captured. Fold that generously into the reporter's column and you are still only at 38%, and the biggest single cause stays something the reporter could not possibly have known. Nobody outside the department has a copy of the tracker.

And it is expensive. In the 2014 data, non-reproducible reports stayed active a median of 154 days against 40 for everything else. They don't get closed fast. They get closed slowly, after everybody has touched them.

Why it ended up this way

Intake forms were designed by the people who buy ticket systems, not by the people who work the queue. Severity exists because an SLA clock needs a number to start from; the field survives because it is billable, not because it is useful.

And triage — the step that determines the shape of the fix — got handed to whoever picks up the queue first, usually the least experienced person in the room, holding a form that asks about operating system and never asks what the person saw.

Google's SRE book is blunt about the order of operations in its troubleshooting chapter: triage first, diagnose second, and "stopping the bleeding should be your first priority."

Simon Tatham wrote the clearest sentence on any of this in 1999: "the aim of a bug report is to enable the programmer to see the program failing in front of them." Everything else on the form is overhead.

What I'd actually do

Ask the two easy questions first. What did you see, and what did you expect instead. Reporters rate both trivial to answer. Stop opening with the hardest item on the list.

Search the tracker before you ask for anything. Duplication leads both studies. If the top cause of your Cannot Reproduce pile is a lookup you didn't do, asking the user for more detail is the wrong move at the wrong moment.

Collect the environment, don't interview for it. Mozilla's bug writing guidelines ask reporters to retest in a clean profile and attach their troubleshooting data — the point being it's a captured artifact, not a question. Browser, build, version, extensions, network path: your tooling already knows. Ask for none of it.

Make severity advisory, not required. Zero percent. Let the reporter say how much it hurts in a sentence instead.

One issue per ticket, and write the summary as the problem. Mozilla again: a summary "should explain the problem, not your suggested solution." Cancelling a file copy crashes the file manager is a ticket. Software crashes is a mood.

Add one field: can reproduce at will, sometimes, or never. Three options that split your queue into three completely different investigations, and almost no form asks for it.

Cannot Reproduce almost never means what it says. Most of the time it means nobody has looked hard enough yet, and the likeliest place the answer is hiding is your own tracker.

0 Share
Will Farrar

Will Farrar

Hands-on CTO with 25+ years in regulated healthcare IT. Author of the Prompt Mode workbook and creator of the companion app. Writes about the parts of the job — good and hard — that don't fit on a resume. More about Will →

BACK TO ALL ARTICLES