What You Refuse to Check Decides the Quality of a Linter
I built a checker for a configuration directory. The time went not into adding rules, but into deciding what not to add. Things you could detect are easy to think of. That was never the constraint. One false positive is enough to get the tool thrown out A checker is asymmetric. A miss goes unnoticed. The cost is only that you did not learn something you could have. A false positive stops the reader and demands a decision: is this actually wrong? And once someone has been burned, they read every finding with suspicion. Twice, and the tool comes out of CI. So a checker that calls a valid configuration broken is worse than no checker. Better ten rules with no false positives than thirty with one. That is obvious in the abstract and hard in practice, because while you are writing the code, every “oh, I could check that too” pulls in the other direction. No citation, no rule So I fixed one condition for adding a rule: Only check what the official documentation states outright — as an error, as skipped, or as ignored. If the documentation does not say it, the rule does not go in, however wrong the pattern looks. What this buys is that the judgement stops living in my memory. “I’m fairly sure that form was invalid” is not a citation, and my memory goes stale the moment the tool it describes releases a new version. In the implementation, every finding carries its reason: export interface Finding { severity: “error” | “warn”; file: string; line?: number; /** what is wrong, in one sentence / message: string; /* why that can be claimed — includes the source URL */ because: string; } Making because required is the point. A rule you cannot justify cannot be written, because the type will not let you leave the field out. If no source comes to mind, the rule never gets implemented. The tests enforce it too: for (const f of findings) { if (!f.because.includes(“https://”)) fail(no source: {f.message}); } One finding without a source URL fails the suite. The four things I decided not to check Each of these is easy to detect. That was not the question. 1. Unknown keys The settings file has a published JSON schema. Diffing against it would let me flag unfamiliar keys as possible typos. I did not, because the documentation says this: The schema can lag behind the newest CLI releases, so a validation warning on a recently documented key doesn’t mean your configuration is invalid. Which means the people who adopt new features first are the ones who get the false positives. You would lose your most engaged users in order of engagement. 2. Model names It is tempting to validate the value of model. But no exhaustive list of valid values exists in the documentation — only a handful of examples. Examples are not an enumeration. Not appearing in an example is no basis for calling a value invalid. Dropped. 3. File path patterns In permission rules, the leading characters change the anchor four different ways: Pattern Anchored at //path the filesystem root ~/path the home directory /path the settings file’s own location — not an absolute path path the current directory The third is the trap: /Users/alice/file is not an absolute path here. On top of that sits gitignore-style globbing. I was not confident I could implement it correctly, so it is not checked. False positives come out of exactly the places you were not sure about. 4. Booleans restricted to true / false Frontmatter booleans. It is tempting to accept true and false and reject the rest. The documentation lists yes / no / on / off / 1 / 0 as valid too. Obvious in hindsight for YAML — and had I implemented it without checking, it would have been a false positive. Do not emit “probably wrong” Anything I cannot decide passes in silence, with no warning at all. “Just mentioning it for your information” looks considerate and reads, from the other side, as offloading the judgement onto the user. The output gets longer, and the errors that matter get buried in it. There are only two severities: error — a documented violation. It does not work until you fix it. warn — it works, but it is deprecated or will break later. There is deliberately no info, because an info level becomes the place where “I couldn’t decide” goes to live. Give it a home and everything undecided piles up there. Weight the valid input more heavily than the broken one Two fixtures: Input Expected Result deliberately broken config findings 22 valid config nothing at all 0 The second row is the one that matters. And the “valid” fixture deliberately includes things designed to trip the checker: a key that looks like it belongs to a release that has not happened yet schema, which exists only for editor completion a setting nested inside another, where writing it at the top level would be an error A single finding from any of those means the design is wrong. The failure messages follow the same asymmetry: FALSE POSITIVES on a valid tree: 1 Misses are reported plainly as “not detected”. False positives get the shouting. The output itself decides which one you fix first. Summary A checker’s value is not how much it detects. It is that it does not cry wolf. One false positive costs you trust; two costs you your place in CI. Fix the condition for adding a rule at “can I cite this?” Put because in the type and unjustifiable rules become unimplementable. If you are not confident you can implement it correctly, do not check it. No info level. It becomes the dumping ground for undecided cases. Make “zero findings on a valid tree” the most important test, and seed that tree with things designed to trip you. Deciding what to pass over in silence turned out to be harder than deciding what to catch.