There's Only One Kind of Tunnel: Shellby's Port Forwarding

Shellby supports four kinds of SSH port forwarding: local -L, remote -R, dynamic -D (SOCKS proxy), and multi-hop -J (ProxyJump). This post is about folding these four forms into one unified tunnel model, and a NIO event-loop trap I hit doing bidirectional forwarding.

One decision: tunnels don’t depend on the terminal

First, the most important architecture decision: all tunnels are independent, persistent, auto-reconnecting, dedicated SSH connections — they don’t depend on any terminal session.

Many SSH clients make port forwarding a “terminal connection accessory” — you open a shell, add a forward on that connection, and closing the shell kills the forward. Shellby does the reverse: a tunnel has its own dedicated SSH connection, and if you close every terminal, the tunnel keeps running.

Why? Because the typical use of port forwarding is “run in the background persistently” — forward a database port to localhost, keep a SOCKS proxy up. You want these to stay alive, not bound to whichever terminal window happens to be open. Decoupling tunnels from terminal sessions gives tunnels their own lifecycle: independent start, independent reconnect, independent traffic counting.

The cost is that a tunnel has to manage its own connection. There’s an app-level tunnel manager (shared across windows) that manages each independent tunnel’s state machine (connecting / active / failed), the SSH handle, and a “rebuild closure.” Building a tunnel only does connect + startForward — no shell, no terminal session — it’s a connection purely for forwarding.

Four forwards, one protocol

The four forwards unify through one Forward protocol, dispatched by the session by forward type. Their implementations differ quite a bit:

• Local -L: start a ServerBootstrap listening on a local port; each inbound connection opens a directTCPIP channel on SSH, with a pair of GlueHandlers doing bidirectional forwarding.
• Dynamic -D (SOCKS5): a hand-written subset of RFC 1928 — no-auth only, CONNECT only, IPv4/domain/IPv6 supported, BIND and UDP ASSOCIATE return “unsupported.” After the handshake completes, the SOCKS handler is removed from the pipeline and glue forwarding takes over.
• Remote -R: reuses Citadel’s high-level remote-forward capability in a cancellable Task; stopping cancels the Task, letting Citadel internally send cancel-tcpip-forward.
• Multi-hop -J: see the next section.

A hand-written SOCKS detail trap: the instant the handshake completes, the client may have already sent application data alongside the handshake packet (pipelined), and when swapping handlers you have to carry that leftover data over as-is and write it to the target — you can’t drop it. This kind of “residual data at the moment of protocol switching” is the easiest thing to miss when hand-writing network protocols.

There’s also an App Store compliance note: dynamic SOCKS is explicitly positioned for dev/ops use, and must not be advertised for circumventing network restrictions — so only a subset sufficient for dev/ops was built, no UDP, no BIND.

The NIO trap: cross-EventLoop writes must hop back to the peer’s loop

The core of bidirectional forwarding is that pair of GlueHandlers: data read on the local end is written to the remote, data read on the remote is written to the local. Sounds simple, but there’s a NIO trap.

The problem: the two ends of the forward may run on different EventLoops. When one end reads data and needs to write it to the other end, if you directly call the peer channel’s write on the current loop — NIO silently drops it, or the behavior is undefined. Because NIO’s rule is: a channel’s operations must execute on the EventLoop it belongs to.

The fix is that when forwarding data, you must use loop.execute to hop the write operation onto the peer channel’s own EventLoop before executing. One execute line, but without it the symptom is “forwarding works sometimes, drops data” — the hardest kind to debug, because it doesn’t error, it just quietly drops.

With an async networking framework, “which thread/loop you execute on” matters as much as “what you execute.” The framework’s concurrency model isn’t background knowledge, it’s a constraint you carry with every line of IO code you write.

There’s also a handler-timing trap: local forwarding must install the SSH-side glue in the channel-initialization callback, not after the channel activates — otherwise the first few bytes arriving right when the channel activates get dropped before the handler is in place.

Multi-hop: authenticate per hop, and detect loops

The implementation of ProxyJump multi-hop (local → jump A → jump B → target): connect the first hop directly, then “jump” hop by hop — open a directTCPIP channel to the next hop on the current connection and complete the SSH handshake, hop after hop, finally jumping to the target. Each hop authenticates and verifies the host fingerprint independently. When a hop fails, the error marks clearly which hop and which host failed — for multi-hop troubleshooting, not telling you which link broke is as good as nothing.

The jump chain is built by walking up from the “jump host” field in the host config, and here you have to detect loops: walk up the chain with a seen-set, and if you hit a duplicate host ID, throw “the jump chain has a loop,” rather than recursing infinitely. Users can configure a loop (A’s jump is B, B’s jump is A again), and the code can’t trust the config to be loop-free.

Reconnect: first-connect failure and a drop need different handling

Independent-tunnel auto-reconnect has a counterintuitive but important distinction: a first-connect failure doesn’t auto-reconnect; only a drop after having connected once does.

The logic: after the first connect succeeds, start a persistent Task that probes whether the connection is still up every 10 seconds, and on a drop reconnects with exponential backoff (starting at 2s, doubling, capped at 30s). But a tunnel that fails to connect the first time — usually a port conflict or a config error — is marked failed directly, no auto-reconnect, for the user to fix.

Why? Because “connected once then dropped” is most likely a temporary problem (network jitter, a server restart) where reconnecting makes sense; while “never connected at all” is most likely a config problem, where blind reconnecting just bangs against the same wall over and over, flooding the failure log. Distinguish “transient failure” from “config error” — retry the former, stop and report the latter for a human to fix — this distinction saves a mountain of pointless reconnect storms.

An honest shortcoming: -R can’t count traffic

The tunnel panel shows each tunnel’s live up/down rate. But one kind of tunnel shows “—”: remote forward -R.

Because traffic counting relies on the GlueHandler that all bidirectional forwarding passes through — data goes through here, so it’s tallied. Local -L and dynamic -D both pass through glue and can be counted; but remote -R goes through Citadel’s high-level capability, its data doesn’t pass through our own glue, and it can’t be counted.

So it honestly shows “—” instead of inventing a fake number. A feature you can’t deliver, marked clearly as undeliverable, is more trustworthy than faking a number that looks usable — a user seeing “—” knows there’s no data here, but a user seeing a fake number would make decisions on it.

Takeaways

Building port forwarding, a few takeaways:

• Tunnels independent of the terminal: to let forwarding run persistently in the background with its own lifecycle, don’t bind it to a terminal session;
• In an async networking framework, “where you execute” is a first-class constraint: NIO cross-EventLoop writes must loop.execute back to the peer’s loop, or you silently drop data;
• Don’t trust user config to be loop-free: multi-hop jumps must actively detect loops and mark the failure position per hop;
• Distinguish transient failure from config error: reconnect on a drop, stop and report on a first-connect failure, don’t let config problems trigger reconnect storms;
• If you can’t do it, mark it honestly: -R can’t count traffic, so show “—,” don’t fake a number.