Debugging on Nick Liu - Software Engineer

Five wrong answers in one day. One nearly deleted 10 GB of coursework.

nickboy@users.noreply.github.com (Nick Liu) — Mon, 17 Aug 2026 00:00:00 +0000

`mdls kMDItemLastUsedDate` returned `(null)` for Microsoft Word. I read the null as "never opened" and put Office on a removal list: 10.1 GB, four apps. One last check saved me. My home directory held 150 Office documents, a conference presentation edited two weeks earlier, and a PowerPoint lock file, which only exists while the file is open. The proof that the null was misleading had been sitting in my own diagnostic report for an hour.

That was one of five. In a single day of hardening this machine, five different tools told me things that were not true. None of the answers looked like an error. Each one arrived as a clean, confident finding, and under each one a check had quietly failed or asked the wrong question. All five had the same shape underneath. Once I could name the shape, I stopped falling for it.

Case 1: null means “I don’t know”, not “no”
#

The plan was simple: free disk space by removing apps that had never been used. The evidence came from Spotlight metadata:

$ mdls -name kMDItemLastUsedDate -raw "/Applications/Google Chrome.app"
2026-08-15 20:18:18 +0000
$ mdls -name kMDItemLastUsedDate -raw "/Applications/Microsoft Word.app"
(null)

Chrome had a date. Word did not. The conclusion looked solid, because the same command told the two apps apart: Chrome is in use, Word is not.

It was not solid. kMDItemLastUsedDate is null when an app has never been opened, and when Spotlight’s metadata store has been reset. My store had been reset. I even knew that, because four hours earlier I had written this line into my own diagnostic report:

12:14:16.629 lsd _CSStoreCreateWithURL failed with error ... "The file
 com.apple.LaunchServices-33554481-v2.csstore couldn't be opened
 because there is no such file."

The LaunchServices database was rebuilding at boot. In one section of my report, this was a strange log line. In another section, its consequences became hard evidence for deletion. I never connected the two.

What answers “is this app still needed” is user data:

$ find ~/Documents ~/Desktop ~/Downloads -iname '*.docx' -o -iname '*.pptx' | wc -l
 150
$ ls -t ~/Documents/**/*.pptx | head -3
cs6460-final-GT.pptx
~$cs6460-milestone2-GT.pptx # lock file: exists only while the file is open
cs6460-milestone2-GT.pptx

Documents, lock files, container contents. Things a person made on purpose. An index field is different: the OS is free to throw it away and rebuild it, and while that happens the field tells you nothing.

Case 2: a query that never ran is not a query that found nothing
#

My environment audit reported “no git remotes configured” for a repository that had pushed four signed commits and opened a pull request earlier the same day.

The check behind that sentence was this:

$ cd ~ && git remote -v
fatal: not a git repository (or any of the parent directories): .git

These dotfiles are managed with yadm, and yadm keeps its bare repository in ~/.local/share/yadm/repo.git. $HOME is the work tree, not a git directory. Plain git in $HOME cannot answer the question at all. Then “cannot answer” got recorded as “the answer is none”.

$ yadm remote -v
origin git@github.com:nickboy/dotfiles.git (fetch)
origin git@github.com:nickboy/dotfiles.git (push)

The tool did not lie. fatal: not a git repository and empty output are two different results, but the code reading them treated both as “nothing there”.

Case 3: a 404 from the wrong endpoint
#

Same audit, same day: “branch protection could not be determined”. The check had asked the classic API:

$ gh api repos/nickboy/dotfiles/branches/master/protection
{"message":"Branch not protected","status":"404"}

That is a real 404, and the message reads like a finding. But GitHub has two protection mechanisms, and this repo uses the newer one:

$ gh api repos/nickboy/dotfiles/rulesets --jq '.[] | "\(.name) \(.enforcement)"'
Main Branch Protection active

A branch protected by a ruleset returns 404 from /branches/*/protection. Any tool that only checks the old endpoint will report every ruleset-protected branch as unprotected, in a sentence that sounds like it actually checked.

Asking the right endpoint paid off twice, because the true answer held a detail the false one would have hidden. The ruleset requires a pull request and four status checks, with no bypass actors. But required_approving_review_count is 0, and there is no signature requirement. “Protected” here means “must go through a PR and CI”. It does not mean “must be reviewed”. My signed commits are a habit; the server never checks them. Recording how strong a control is beats recording that one exists.

Case 4: the number was real, the units were not
#

My health check started with a headline number: WindowServer was eating 62.4% CPU, the biggest single CPU user on the machine. The number came from ps:

$ ps -Ao pid,%cpu,comm | grep WindowServer
 436 62.4 .../SkyLight.framework/Resources/WindowServer

On macOS, ps %cpu is a decaying average. It leans hard toward the last minute, not toward the life of the process. The cumulative figure is CPU time divided by elapsed time:

$ ps -o time=,etime= -p 436
 26:03.57 01:02:14 # 1563s CPU / 3734s elapsed = 41.9%

So 42%, not 62%. The next morning brought a bigger correction, after sixteen hours of normal use:

WindowServer 6.6% (16.4 hours)
ghostty 3.3% (16.4 hours)

Six point six. My original 62% came from an hour in which I was the load. I was running diagnostics nonstop, with Activity Monitor open and a backup copying files in the background. I had measured the observer and blamed the system. The whole “WindowServer is the problem” section of that report was an accident of timing.

Case 5: the benchmark that measured nothing happening
#

ll felt slow. The alias behind it is eza with a long list of flags, including --git. I benchmarked it and got a number that ended the discussion:

eza --all --long --git --icons ... 8.4 ms

Eight milliseconds. Nothing to fix. Except the same command, in the same directory, also reported this:

$ eza --all --long --git ... ~/Downloads | wc -lc
 0 0

Zero lines, zero bytes, exit 0, and 128 ms of system time. The benchmark had run inside the sandboxed shell my coding agent uses, and that sandbox blocks the metadata and git reads that --long --git needs. eza was making syscalls, and the sandbox refused every one of them. My benchmark timed the result with great precision: a program doing nothing, very fast.

I re-ran everything in a real terminal:

directory	size	with `--git`	without
`nickboy.github.io`	47 entries, 263 tracked	68.8 ms	13.9 ms
`claudedeck`	125 tracked	49.0 ms	20.8 ms
`Downloads`	622 entries, not a repo	49.6 ms	39.8 ms

Inside a repo, --git costs 2x to 5x, and the cost follows the repository, not the directory being listed: eza resolves git status for every tracked file to fill one column. Listing a 47-entry folder inside a 263-file repo pays for all 263 files. My everyday alias paid that price in every project directory, which is exactly where I type it most.

The flag moved to an opt-in alias, llg. It also came out of ls, l, lt and tree completely, because the git column only renders in long view. For those four, the output is byte-identical with and without the flag. I verified that instead of assuming it:

$ [[ $(eza --git ... | md5) == $(eza ... | md5) ]] && echo "no-op"
no-op

Dead config, carried for months, costing a git query on every call.

The common shape
#

Five cases, one sentence:

A query that failed and a query that found nothing are different results.

Every case was a tool reporting an absence. In none of them had anyone checked whether the question ever reached the thing it was asking about.

“Absence of evidence is not evidence of absence” is an old phrase. The working version is more direct: evidence gathered by a broken method is not evidence at all, and a broken method usually exits 0.

The test that could not fail
#

The same day turned up a bug that had survived for a year because the test guarding it was written backwards.

Ghostty on macOS reads both ~/.config/ghostty/config and ~/Library/Application Support/com.mitchellh.ghostty/config. My bootstrap script had symlinked the second to the first, so the same file was parsed twice. For single-value keys the last value wins, so nothing looked wrong. But repeatable keys append:

$ ghostty +show-config | grep -c '^custom-shader = '
4 # the file declares 2

Two extra full-screen shader passes on every frame, for a year. And the test suite stayed green the whole time, because it contained this:

GHOSTTY_LINK="$HOME/Library/Application Support/com.mitchellh.ghostty/config"
if [ -e "$GHOSTTY_LINK" ] || [ -L "$GHOSTTY_LINK" ]; then
 run_test "Ghostty config is symlink" "[ -L \"$GHOSTTY_LINK\" ]"
fi

Two problems in four lines. The assertion checks the opposite of the correct rule, because the symlink should not exist at all. And the assertion sits inside a guard, so when I deleted the symlink, the test did not fail. It silently stopped running. The first post in this series was a catalog of safety nets that had quietly died. This test is the sharpest example I have found since: it disappeared at the exact moment it finally had something to say.

The replacement tests the property instead of the mechanism:

run_test "Ghostty config is not parsed twice (custom-shader not duplicated)" \
 "[ '$LOADED_SHADERS' -eq '$CFG_SHADERS' ]"

Whatever causes a double parse next time (this symlink, a different symlink, an include directive, some future Ghostty change), the count stops matching and the test goes red. I put the symlink back once, on purpose, and watched the test fail before I trusted it.

Lessons
#

Null, empty, and 404 are answers about the query, not always about the world. Before treating one as a finding, confirm the query reached the right place.
Decide “is this app still needed” from data a person created. An index the OS can reset is not a record of anything.
Know which average your tool reports. ps %cpu on macOS leans toward the last minute; cumulative load is CPU time divided by elapsed time.
Measure where the work really happens. A sandbox that blocks the syscalls under test produces a fast, precise, meaningless number.
Never measure a system while you are the heaviest thing running on it.
Test the property, not the mechanism. The symlink was one way to get a double parse; the count catches all of them.

References
#

eza (--git cost measured on v0.23.5)
yadm (bare repo location: ~/.local/share/yadm/repo.git)
GitHub rulesets API (rulesets are invisible to the older /branches/*/protection endpoint)
Ghostty configuration (macOS reads both the XDG path and the Application Support path)
ps(1) on macOS (%cpu is documented as a decaying average)
The repo and its test suite: nickboy/dotfiles

My commands vanished with exit 0. The culprit was a file named env.

nickboy@users.noreply.github.com (Nick Liu) — Thu, 13 Aug 2026 00:00:00 +0000

`env -u VAR command` did nothing. Exit code 0, no output, no error. A different command, same thing. A different variable, same thing. Any invocation that started with `env` just quietly evaporated. What finally made the problem visible was a `git init` that reported success while creating no `.git` directory at all.

That was a year ago. The case closed last month, and the culprit was not uv, not some third-party installer, not anything exotic. It was this repo’s own bootstrap script. The thing that caught it was the regression test I had written for the original incident.

This post is three silent failures from the same repo, plus that ending. They share a shape worth naming.

Case 1: the executable that should have been sourced
#

uv’s installer drops a file at ~/.local/bin/env. It is a PATH snippet, a few export lines meant to be sourced by your shell profile. It is not a program.

At some point, something gave it an executable bit. I could not figure out what at the time, so that part of the case stayed open. But the consequences were mechanical from there:

Every env -u X cmd, every shebang-adjacent env trick, every script that spelled out env for portability: all of them ran a five-line PATH snippet and stopped. With exit 0, because the snippet ran fine. It was just never asked to run your command, since sourcing semantics and executing semantics are different contracts.

The fix was one line:

chmod -x ~/.local/bin/env

Finding the line took the better part of a debugging session. So the incident became a permanent test, with the story written into the comment:

# uv's installer drops a PATH snippet at ~/.local/bin/env meant to be
# SOURCED; if it ever becomes executable it shadows /usr/bin/env (PATH
# puts ~/.local/bin second) and silently swallows every 'env ... cmd'
# invocation with exit 0. That exact trap cost a debugging session.
run_test "env resolves to /usr/bin/env (no executable shim)" \
 "[ \"\$(command -v env)\" = /usr/bin/env ]"

There is also a generalized version: a scan that flags any executable in ~/.local/bin shadowing a system command. The specific trap gets a named test; the class of trap gets a sweep.

Case 2: the fixture that reformatted my real repo
#

I was writing a fixture test for the secret scanner: create a throwaway git repo in a temp directory, plant a fake credential, confirm the scanner catches it. Standard stuff.

The test runs inside the yadm pre-commit hook. yadm exports GIT_DIR and GIT_WORK_TREE for its own purposes, and here is the part I did not respect enough: those environment variables outrank git -C <path>. The -C flag changes the working directory; the environment still decides which repository you are talking to.

So the fixture’s git init reinitialized the real yadm repository. It flipped core.bare and broke the repo’s configuration. Recovery turned out to be one setting, with zero history lost:

yadm gitconfig core.bare false

But the lesson went into the suite in capital letters:

# CRITICAL: every call strips GIT_DIR/GIT_WORK_TREE — the yadm
# pre_commit hook exports them, and an ambient GIT_DIR silently
# redirects fixture git commands at the REAL yadm repo (this once
# flipped its core.bare).
GL_GIT="env -u GIT_DIR -u GIT_WORK_TREE git"
$GL_GIT -C "$GL_TMP" init -q

The suite now also runs a hostile-environment simulation: it deliberately plants a decoy GIT_DIR and confirms the fixtures ignore it. And the fake credential is assembled at runtime, so no secret-shaped literal ever exists in the repo for a scanner to trip on:

printf 'aws_key = "%s%s"\n' "AKIA" "ZZZQK9X2M4P7L3TQ" \
 > "$GL_TMP/leak.txt"

Case 3: the file manager that shrugged
#

yazi 26 renamed a fetcher config field from id to group. My config still said id. When yazi hits a config it cannot parse, it does not error; it falls back to its presets and carries on looking almost normal. The git and mactag columns just stopped being enriched, and I only noticed weeks later, by accident.

The canary here is two words:

yazi --version # exits 1 when the config fails to parse

That exit-code behavior is something I tested, not something I read somewhere. It now runs in the test suite and again after upgrades, as a schema check. When yazi next renames a field, the failure will be loud and same-day instead of silent and discovered by luck.

The ending: a year later, the case closed itself
#

Here is where case 1 gets its resolution, and it did not come from detective work.

An unrelated idempotency audit of the bootstrap script added one final step to it: after bootstrapping, run the full test suite as self-verification. The very first self-verification run came back red.

The bootstrap script contained a blanket chmod +x ~/.local/bin/*. The “scripts must be executable” convention, applied to a whole directory. It had just re-armed the uv trap, a year after I defused it, and it had presumably been the original armer all along. One line took down three tests at once: the env resolution test, the PATH shadow sweep, and the gitleaks fixtures, whose env -u calls were being swallowed again.

# before: the whole directory gets the executable bit.
# uv's source-me env snippet is in there, so the trap gets
# re-armed on every bootstrap run.
chmod +x ~/.local/bin/*

# after: only scripts yadm actually tracks get chmod'd.
# untracked files (including uv's env) never gain the bit.

The full lifecycle reads like this: symptom, defusal, regression test, then a year of quiet, then the test catches the original root cause in the act of reoffending, and the root cause gets fixed. The gap between step three and step four was twelve months. The gap between “bootstrap gained self-verification” and “cold case closed” was a few minutes.

The common shape
#

All three cases had no error message. All three returned success. All three were a system helpfully falling back to something instead of failing: a snippet standing in for a binary, an environment variable standing in for a flag, presets standing in for a config.

Silent fallback is the expensive kind of bug. A loud failure costs you five minutes. A quiet substitution costs you weeks, because nothing marks the moment where reality diverged from your model of it.

Institutionalizing the paranoia
#

Every incident in this post exists in the test suite as a named check. The suite grew from 74 checks to 105 across this overhaul, and almost all of that growth is “we got bitten here” items. Case 1 is the proof that the habit pays twice: the test written to protect the future is the same one that closed the past.

The other byproduct is a small document, docs/upgrade-watch.md, pairing every config file that tracks an upstream schema with a canary that fails loudly:

| Config file | Canary |
| --------------------------- | ----------------------- |
| ~/.config/yazi/yazi.toml | yazi --version |
| ~/.config/atuin/config.toml | atuin doctor |
| ~/.tmux.conf | throwaway-server parse |
| ~/.local/bin/env (uv) | suite `env` check |

Lessons
#

Environment variables outrank the flags on your command line. git -C does not protect you from an ambient GIT_DIR.
A file that is meant to be sourced becomes a command-swallowing trap the moment it gains an executable bit and a good PATH position.
Tools that silently fall back to defaults on a bad config need a canary with a real exit code. Test the canary’s behavior; do not assume it.
Write the regression test even when the root cause is still unknown. Mine waited a year and then caught the culprit red-handed.
Bootstrap scripts should only touch files they own. A wildcard chmod is a loaded gun pointed at the future.

References
#

uv (the installer that ships the ~/.local/bin/env snippet)
yazi (the config-schema rename landed in yazi 26; exit-code canary behavior verified on this machine)
yadm hooks documentation (the pre-commit hook environment where GIT_DIR/GIT_WORK_TREE are exported)
The repo and its test suite: nickboy/dotfiles

My diagrams rendered on refresh and vanished on click. The head never loaded.

nickboy@users.noreply.github.com (Nick Liu) — Tue, 11 Aug 2026 00:00:00 +0000

A reader clicking from my homepage to a post with a diagram got a block of raw mermaid source. The same reader pasting that post's URL directly got a rendered diagram. Same page, same build, same browser. The difference was the click, and the bug had been live on this site for months across every diagram, every math formula, and every chart, because I had only ever tested pages by loading them directly.

Tested with Blowfish 2.10x · htmx 2.0.10

Two features, both reasonable, one collision
#

This site has htmx’s hx-boost on the body: internal navigation swaps page content in place instead of doing full page loads, which keeps transitions smooth. Separately, the Blowfish theme is smart about heavy libraries: mermaid, KaTeX, and Chart.js bundles are only included on pages that use them, injected into the <head> of exactly those pages.

Each feature is sensible alone. Together they have a hole in the middle:

hx-boost replaces the body and updates the title; it does not merge the incoming head. So when you navigate from a page without diagrams to a page with them, the one script tag that would have loaded the mermaid bundle is in the part of the document that gets thrown away. My re-render code even had a guard, if (typeof mermaid !== 'undefined'), which is exactly the kind of guard that turns a loud failure into a silent one. It skipped, politely, every time.

Direct loads worked because the full document parses normally. Which is why months of my own testing missed it: I test pages by URL. Readers arrive by link.

The fix: steal the bundle URL from the response you already have
#

The swapped-in page’s HTML passes through htmx’s hands, head included, before the body is extracted. The fix reads the bundle URL out of that response and injects the script once:

function ensureVendorThenRender(evt, srcPattern, isReady, render) {
 if (isReady()) { render(); return; }
 var xhr = evt && evt.detail && evt.detail.xhr;
 var html = xhr && xhr.responseText;
 var match = html && html.match(srcPattern);
 if (!match || injectedVendors[match[0]]) return;
 injectedVendors[match[0]] = true;
 var s = document.createElement("script");
 s.src = match[0];
 s.onload = render;
 document.head.appendChild(s);
}

Called from the htmx:afterSettle handler with a pattern like /\/js\/mermaid\.bundle\.[^"']+\.js/, once per library. Fingerprinted filenames come along for free since the pattern matches whatever hash the current build produced.

Charts needed a second layer. The theme’s chart shortcode wraps its new Chart(...) call in a DOMContentLoaded listener, and after an htmx swap that event is ancient history; the listener registers and waits forever. So the chart path re-inserts each inline chart script with addEventListener temporarily patched to invoke DOMContentLoaded callbacks immediately, plus a Chart.getChart() check so nothing draws twice. It is not elegant. It is honest about the constraint: the shortcode’s markup is the theme’s, and the fix has to meet it where it is.

If you want the pattern’s shape as a takeaway: conditionally loaded assets and partial navigation are individually fine and jointly broken, and neither feature’s documentation will warn you about the other. Any site combining per-page bundles with hx-boost, Turbo, or SPA-style routing has this class of bug available to it.

The tests were their own debugging story
#

I added three Playwright regression tests, each walking the failing path on purpose: land on a listing page without the library, click through to a post with it, assert the rendered artifact exists (pre.mermaid svg, .katex, a live Chart.getChart() instance). Two traps surfaced on the way, and they cost more time than the fix itself.

First, the clicks kept landing on the wrong page. Blowfish’s article cards stretch the post link across the whole card and stack tag links on top of it, and Playwright’s coordinate-based click aimed at the card’s center, which happened to be a tag chip. The katex test spent several runs waiting for math on /tags/georgia-tech/. The cure is locator.dispatchEvent('click'), which delivers the event to the exact anchor regardless of what floats above it, plus a waitForURL assertion so a wrong destination fails fast with a readable error instead of a timeout.

Second, my shell habit was eating the exit code. npx playwright test | tail reports tail’s status, which is always zero, and a failing suite sailed through my terminal looking green twice before I noticed. Redirect to a file and check the real exit code. CI was never fooled, because CI does not pipe; only my eyeballs were.

The affected pages, for the record, included every post in my ML math series and the diagram in the hooks post. All fixed by the same three functions, all guarded by the same three tests.

Lessons
#

Per-page asset loading plus body-swap navigation is a bug class, not a bug. If your head is conditional and your navigation is partial, budget for the collision.
A typeof x !== 'undefined' guard converts a crash you would notice into a skip you will not. Guards need a fallback path, not just an exit.
Test the click path, not only the URL path. Direct loads exercise the document pipeline; in-site navigation exercises a completely different one.
In tests, dispatch events to the element you mean. Coordinate clicks trust the visual stack, and the visual stack lies on layered layouts.
Never pipe a test runner’s output when you care about its exit code.

References
#

htmx: hx-boost attribute
Chart.js: Chart.getChart API
Playwright: dispatchEvent
The fix and tests: layouts/partials/extend-footer.html and tests/site-enhancements.spec.ts in this site’s repo

Debugging on Nick Liu - Software Engineer

Five wrong answers in one day. One nearly deleted 10 GB of coursework.

Case 1: null means “I don’t know”, not “no” #

Case 2: a query that never ran is not a query that found nothing #

Case 3: a 404 from the wrong endpoint #

Case 4: the number was real, the units were not #

Case 5: the benchmark that measured nothing happening #

The common shape #

The test that could not fail #

Lessons #

References #

My commands vanished with exit 0. The culprit was a file named env.

Case 1: the executable that should have been sourced #

Case 2: the fixture that reformatted my real repo #

Case 3: the file manager that shrugged #

The ending: a year later, the case closed itself #

The common shape #

Institutionalizing the paranoia #

Lessons #

References #

My diagrams rendered on refresh and vanished on click. The head never loaded.

Two features, both reasonable, one collision #

The fix: steal the bundle URL from the response you already have #

The tests were their own debugging story #

Lessons #

References #

Case 1: null means “I don’t know”, not “no”
#

Case 2: a query that never ran is not a query that found nothing
#

Case 3: a 404 from the wrong endpoint
#

Case 4: the number was real, the units were not
#

Case 5: the benchmark that measured nothing happening
#

The common shape
#

The test that could not fail
#

Lessons
#

References
#

Case 1: the executable that should have been sourced
#

Case 2: the fixture that reformatted my real repo
#

Case 3: the file manager that shrugged
#

The ending: a year later, the case closed itself
#

The common shape
#

Institutionalizing the paranoia
#

Lessons
#

References
#

Two features, both reasonable, one collision
#

The fix: steal the bundle URL from the response you already have
#

The tests were their own debugging story
#

Lessons
#

References
#