The tunnel was up. The socket existed. Nothing was on the other end.

nickboy@users.noreply.github.com (Nick Liu) — Thu, 03 Sep 2026 00:00:00 +0000

The daemon reported the tunnel as established. The local socket file was right there on disk with the right permissions. The target sat at `offline` forever. Meanwhile `connect to /home/nick/.config/herdr/sessions/main/herdr.sock port 0 failed` was being written to a stderr stream that nothing in my process ever read, several times a second, for as long as I left it running.

Tested with herdr 0.8.0 · macOS

HerdDeck drives agents on a second machine. My laptop has the Stream Deck and the daemon; the desktop runs herdr and the agents actually doing work. They are joined by ssh -N -L <local.sock>:<remote.sock> host, a Unix-domain socket forward, and the daemon speaks the same protocol to a forwarded socket as it does to a local one.

That symmetry is the entire reason remote support was cheap. It is also what made this bug possible, because “I have a socket” is a much weaker statement than it looks.

`ssh -L` binds before it connects
#

A local forward binds the listening end immediately. It does not, and cannot, verify that anything is listening on the far side, because nothing tries to reach the far side until you open a connection through it.

So the moment ssh -N -L starts, you have a socket file. It is a real socket. You can connect() to it. And when you do, ssh opens a channel, asks the remote sshd to reach the target path, the remote sshd fails, and ssh writes a message about it to its stderr and closes your channel.

My daemon’s definition of “established” was the existence of that file.

Every failure mode on the far side therefore looked identical and looked fine. Remote herdr not running: tunnel up, target offline. Wrong absolute path in the config: tunnel up, target offline. Session socket not created because that named session had never been started on the remote: tunnel up, target offline.

Three different problems, one symptom, and the symptom actively pointed away from the cause, because the tunnel was the part reporting healthy.

The error was there the whole time
#

The failures are asynchronous. ssh does not fail at spawn; it fails per channel, later, on stderr. I was reading stderr during spawn to catch startup errors, then dropping the stream on the floor once the process was up.

Everything the debugger needed was in the part I stopped reading.

Two changes. First, stderr is drained for the tunnel’s entire lifetime into a small per-target ring buffer, capped, so a chatty failure cannot grow without bound and the last few dozen lines are always available for diagnostics. Second, and more important, establishing a tunnel now ends by proving it:

bind local socket -> ping-probe THROUGH the socket -> declare established

The probe is a real protocol ping through the forwarded socket with a 4-second timeout. If it fails, the tunnel is not established, regardless of what the filesystem says, and the failure joins the retry path with the stderr tail attached to the error.

The observable difference is the whole point. Before: tunnel green, target offline indefinitely, zero clues. After: it fails within seconds with channel 1: open failed: connect failed and the full -L path mapping in the log, which names the wrong path directly.

ControlMaster pollution, in both directions
#

This one I would never have found by reading code, because the bug lived in the interaction between correct code and my personal SSH config.

Most developers have ControlMaster auto set. It multiplexes SSH sessions to the same host over one connection, which is why your second ssh to a box you are already on connects instantly. It is a good default and I have had it on for years.

A long-lived tunnel joining that multiplexing arrangement fails in two directions, and the second is nastier.

Direction one: the tunnel dies when an unrelated master connection closes, or when its ControlPersist timer expires. Your daemon’s connectivity is now coupled to whether you happen to have a terminal open.

Direction two: if the tunnel gets there first, it becomes the master. Every subsequent interactive ssh to that host attaches to a connection owned by a background daemon. Restart the daemon and you drop the shells of anyone attached to it.

The fix is to refuse to participate:

// Own the connection lifecycle: with the user's ControlMaster
// auto config the tunnel would otherwise mux onto (or become) a
// shared master and its forward would die with that master's
// ControlPersist timer (found live in Tier-1 testing).
"-o", "ControlMaster=no",
"-o", "ControlPath=none",
"-o", "ExitOnForwardFailure=yes",

A tunnel must own its connection. Borrowing one couples its lifetime to something the daemon does not control and cannot observe.

Not every failure deserves a retry
#

launchd starts the daemon at login, which on a laptop means before Wi-Fi has associated and well before a VPN is up. The first tunnel attempt fails, and in the original code that failure was permanent until I restarted the daemon by hand.

Retrying everything is the wrong correction. Retrying a wrong passphrase forever is how you get an account locked out, and retrying a host-key mismatch forever is how you sit in a loop ignoring a security warning. So failures get classified by what ssh said:

ssh stderr contains	Class	Behaviour
`Permission denied`	permanent	no retry, surfaced as `auth`
`Host key verification failed`	permanent	no retry, surfaced as `auth`
`Bad configuration`	permanent	no retry
`Bad owner or permissions`	permanent	no retry
everything else	transient	capped jittered retry, 15s to 10min

Connection refused, timeout, network unreachable and DNS resolution failures all default to transient, and DNS is the interesting one. Could not resolve hostname looks permanent and is completely ambiguous under a VPN that has not come up yet. The default is to retry, because the cost of retrying a genuinely dead hostname is a log line every ten minutes, and the cost of not retrying is a daemon that is dead until you notice.

The loud error fires once per target rather than once per retry, which is the difference between a diagnostic and a log flood.

localhost is a valid far side
#

The technique that made most of this testable, and the part I would steal if I were reading someone else’s post:

“Remote” here means the far side of an ssh streamlocal forward. It does not mean “a different machine”. Point the forward at localhost and you get a genuine SSH forward, with a genuine remote sshd, a genuine second herdr socket path, and genuine channel-open failures, on one laptop with no second machine and no network.

Wrong path, remote daemon down, socket missing, sshd refusing connections, first-connect-before-network: all of those reproduce against sshd on localhost. Only latency and actual link loss need real hardware.

One caveat about `--remote`, because the obvious guess is wrong
#

herdr has a --remote flag and it does not do what you would assume from the outside.

It makes the human terminal experience a thin client. It does not expose a local API socket that proxies the remote server. The local herdr-client.sock is a TUI attach endpoint, so a daemon cannot piggyback on it and needs its own forward.

There is a consequence to that which bit me later. herdr auto-syncs versions between client and server, and that sync fires on a human --remote attach. It never fires for the daemon’s forward. So a machine you rarely attach to by hand can quietly drift to an older protocol while the daemon happily keeps talking to it. Every target is therefore pinged and version-checked on connect and degraded to a warning state on mismatch, rather than assumed compatible.

When not to guess
#

I want to end on the one place in this system where the correct behaviour is to do nothing, because multi-target is what creates it.

Each herdr server tracks its own focused pane. With two targets connected, two panes can each legitimately report that the user is looking at them, and the daemon has no way to know which screen you are actually in front of.

The physical Enter key sends a keystroke to the focused agent. Guessing wrong there does not display something incorrect. It submits someone else’s prompt, to a different agent, on a different machine, with whatever half-written text was sitting in that pane.

getFocusedAgent(): AgentSnapshot | undefined {
 if (this.focused) {
 return this.agentsList.find((a) => sameAgent(a, this.focused));
 }
 const herdrFocused = this.agentsList.filter((a) => a.focused);
 // More than one target can each report a focused pane; only an
 // unambiguous single candidate is safe to act on.
 return herdrFocused.length === 1 ? herdrFocused[0] : undefined;
}

An explicit slot press always wins, because that is you telling it. Absent that, one candidate is a fallback and two candidates is nothing. The refusal has its own test, with the reasoning written into the test body, because it is the kind of behaviour a future refactor would happily “fix” into a [0].

Lessons
#

A bound socket is not a connected socket. ssh -L binds before anything reaches the far end, so the file existing proves only that ssh started.
End every “establish” with a probe that exercises the thing you are claiming works. Reachability is the claim, so make a round trip and let that be the definition.
If a process fails asynchronously on stderr, drain stderr for its whole life. Reading it only at startup means the exact bytes naming your bug are being written and discarded while you debug.
A long-lived tunnel must own its SSH connection. Muxing onto a user’s ControlMaster couples the daemon to a terminal, and winning the race to become the master is worse, because now your restart drops their shells.
Classify failures before retrying them. Auth and host-key failures should never retry; timeouts, refusals and DNS should, because DNS under a sleeping VPN is indistinguishable from DNS against a dead host.
Log the loud error once per target, not once per retry. A diagnostic that repeats every fifteen seconds is a log flood that people filter out.
localhost is a valid far side. Most of a remote-path test matrix runs against sshd on the same machine, and only latency and link loss need real hardware.
When ambiguity could take a destructive action, do nothing and test that you do nothing. Sending Enter to the wrong agent submits someone else’s prompt.

References
#

ssh_config(5) for ControlMaster, ControlPath, ControlPersist and ExitOnForwardFailure; ssh(1) for -L and streamlocal forwarding
OpenSSH release notes on Unix-domain socket forwarding: https://www.openssh.com/txt/release-6.7
herdr: https://herdr.dev
The rebuild that made remote almost free: I deleted six subsystems by swapping one dependency.
HerdDeck tunnel manager, ssh args, stderr ring and the probe: packages/daemon/src/tunnel.ts
HerdDeck registry, retry classification and backoff: packages/daemon/src/registry.ts
Remote-path static review findings: docs/plans/2026-08-08-remote-path-review.md

Controlmaster on Nick Liu - Software Engineer

The tunnel was up. The socket existed. Nothing was on the other end.

ssh -L binds before it connects #

The error was there the whole time #

ControlMaster pollution, in both directions #

Not every failure deserves a retry #

localhost is a valid far side #

One caveat about --remote, because the obvious guess is wrong #

When not to guess #

Lessons #

References #