A Dead Connection That Looks Alive: Shellby's Liveness Token

Shellby is an SSH client that also runs on mobile. Mobile has a problem few SSH clients face honestly: connections die silently, and you don’t notice.

When iOS backgrounds the app, the system suspends the process and drops the TCP connection within tens of seconds to a few minutes. Network switches, server outages, sleep/wake cycles — all make an SSH connection die without a sound. And note: without a sound, no event notifies you. This post is about detecting a dead connection, and a counterintuitive trap: “the command returned” doesn’t mean the connection is alive.

The old “frozen” behavior

In the earliest implementation, a dropped connection looked like this: the underlying channel’s output stream (an AsyncStream<Data>) silently ends, the read loop exits, and… nothing after that. The session sits on its last screen, your keystrokes have no echo, the UI looks perfectly normal — but nothing can actually be sent. The user’s experience is “frozen, and no way to recover” — the only escape is to close the tab and reconnect, losing all context.

Two problems: you can’t tell it dropped, and even if you could, there’s no in-place recovery.

Two ways to die need two kinds of detection

Disconnects actually come in two flavors, handled separately.

Flavor one: the stream ends naturally. The remote sent FIN/RST, the process exited cleanly — in this case the output stream ends. Detecting it is easy: after the read loop for await data in channel.output finishes, check whether it was a deliberate cancel (use Task.isCancelled to distinguish “I disposed/rebound it” from “a real drop”), and if it’s a real drop, mark the session disconnected. This is the fast path.

Flavor two: TCP hangs silently. The network dropped, the server vanished, the device slept and woke — in this case TCP gives you no signal, and the output stream doesn’t end. It just hangs. Relying on “the stream ended” will never detect this class. You have to actively probe.

The trap: execute “fakes success”

The naive way to probe is: send a command every few seconds, see if it returns. Returns → alive, times out → dead.

This has a fatal trap, learned the hard way.

The problem is that the underlying execute (run a command, get the result) swallows channel errors and returns. It returns on a non-zero exit; it even “returns quickly” in the instant right after the connection dies, before NIO has marked the channel inactive, while isConnected is still true. In other words, a connection that’s already dead will still return you a (empty, wrong) result from execute.

If you judge liveness by “did execute return,” you’ll classify a pile of dead connections as alive. The connection is clearly gone, yet the heartbeat is all green lights.

The token: only trust that the server actually replied

The fix: don’t look at whether the command returned; look at whether the server actually spoke.

Send a command with a unique token:

echo __shellby_alive__

Then only when __shellby_alive__ actually appears in stdout is the connection considered alive. A timeout, or a return without the token, is judged dead.

The distinction is subtle but crucial: execute returning only proves “this local call ran to completion”; the token appearing in stdout proves “the command actually reached the server, the server actually ran it, and the result actually came back” — a full round trip. The former can fake success on a dead connection; the latter can’t. A liveness probe isn’t asking “did my call finish,” it’s asking “is the other end still there,” and those two must be distinguished by a signal that travels the entire path.

A few anti-jitter details:

• Two consecutive failures before declaring death, to avoid false positives from a single network hiccup;
• Each round trip has an 8-second timeout, with a TaskGroup racing the command against a timer, first-to-finish wins;
• The heartbeat interval is configurable (default 30s, options 15/60 or off), and the heartbeat loop re-reads the setting every round, so a change or toggle takes effect immediately.

There’s an honest tradeoff too: any probe sends real traffic, so running the heartbeat ≈ running SSH keepalive — it keeps idle connections alive (incidentally defeating NAT or server idle timeouts) but it can’t revive a connection that’s already dead, only discover it sooner. Turn the heartbeat off and you’re back to stream-end detection only, which can’t sense silent hangs. That tradeoff is put directly in the settings for the user to choose.

In-place reconnect: swap the channel, keep the session

Detecting a dead connection is only half; the other half is recovery — and it has to preserve context.

The key design: on reconnect, reuse the same session object and the same terminal render surface, and only swap the underlying channel. The scrollback and screen buffer all live in the render surface; swapping the channel doesn’t clear the screen, so after reconnect your earlier output and your half-typed input are all still there.

The flow: mark “reconnecting” → tear down the old SSH connection → re-authenticate and open a new PTY channel → have the session “rebind” to the new channel → mark “connected” → restart the heartbeat on the new connection. Rebinding just restarts the read loop; it doesn’t touch the screen buffer.

One decoupling technique worth noting: the object that manages sessions doesn’t hold the data layer (doesn’t know how to establish a connection). Reconnect capability is injected via a closure, provided by a higher layer. First connect and reconnect share the same connection-building path. The session manager only decides “when to reconnect”; “how to reconnect” is someone else’s job — a clean boundary of responsibility.

Make “dropped” and “recovering” both visible

Finally, surface the state to the user, because the old version’s biggest problem was “frozen and won’t tell me.”

• On drop/reconnect, overlay a translucent layer: icon + “Connection dropped / Reconnecting…” + reason + a “Reconnect” button;
• The tab’s status dot is three colors: green connected, orange reconnecting, red dropped;
• Three manual reconnect entry points (tab context menu, split-pane title, iPhone menu) — if auto-detection ever fails, the user always has a manual escape hatch.

Takeaways

Building reliability for mobile SSH, the lessons that carry over:

• Silent hangs are the norm on mobile — stream-end detection can’t catch them, so you must actively probe;
• The biggest probe trap is “the command returned ≠ the connection is alive” — the low-level call fakes success on a dead connection. Use a signal that travels a full round trip and only holds if the other end actually replies (a unique-token echo), not “did the call return”;
• Recovery must preserve context: reuse the session and render surface, only swap the underlying channel, lose no history on reconnect;
• State must be visible: drops, reconnects, and failures all shown to the user, always with a manual reconnect escape hatch.

In one line: to know whether a remote is still there, don’t ask “did my request finish,” ask “did you reply.”