Chinese input over remote control starts with choosing who composes

Chinese input in Lynsi’s phone remote control turned out to involve considerably more than forwarding the characters typed on a Mac. This is a development account of the work in October: getting key positions into the phone’s input method, discovering why candidates appeared on the phone but disappeared from the mirrored picture, and keeping a second path that composes text locally on the Mac.

The implementation and its recorded validation are the subject here. The current working tree is not evidence of what is in a particular store binary.

Pinyin is state, not a sequence of finished characters

When someone types nihao, they have not yet committed the text they want. The input method owns the pinyin buffer, candidate list, and selection. Space, digits, and Backspace operate on that state.

Interpreting the keys on the Mac and forwarding NSEvent.characters blurs that ownership. The Mac’s input source has already assigned meaning to the event, while the phone is still expected to run its own composition process.

The first path therefore sends physical key positions and lets the phone compose. RemoteKeyboard.swift maps Mac virtual key codes to phone key codes without reading the event’s characters. For N I H A O, those codes are:

2030 → 2025 → 2024 → 2017 → 2031

Space arrives as 2050, leaving candidate selection to the phone. Shift used in a chord is a modifier; Shift pressed and released on its own is a separate stateful gesture for switching Chinese and English. The implementation remembers intervening keys or clicks so that releasing Shift after typing a capital cannot accidentally switch the input method.

Editing shortcuts need translation too. Mac ⌘C and ⌘V become the phone’s Ctrl chords. Unmapped commands such as ⌘Q and ⌘W remain local. Key position, editing conventions, and composition state each have a defined place to be interpreted.

Bluetooth supplies keyboard identity; another channel carries the keys

That path has a prerequisite: the phone must believe a hardware keyboard is attached.

The October experiment on a Mate X5 established that, after registering a physical keyboard, the Xiaoyi input method entered hardware-keyboard mode and could compose pinyin from key events injected through UiTest. The implementation consequently publishes a Bluetooth HID keyboard service from the Mac, while sending actual keystrokes through the remote-control UiTest channel. It does not deliver those keystrokes as BLE HID input reports.

Separating identity from data also separates their lifetimes. An authenticated mirroring connection does not prove that the keyboard still exists in the phone’s input-device list.

One brief outage made the distinction visible. The Mac’s Bluetooth log recorded Disconnecting GATT for unused LE device; the phone then removed the keyboard input device. The LAN session stayed authenticated and video frames continued to arrive. Allowing typing solely because remote control was online could send the rest of a pinyin word after hardware-keyboard mode had disappeared.

Direct typing therefore requires both a ready control session and an actual report that the phone has the keyboard. Losing the keyboard blocks typing while leaving mouse control available.

Keystrokes cannot inherit the mouse’s discard policy

Remote control already needed limits on old actions. A delayed click can hit an entirely different button after the user changes pages. Coordinate-based actions consequently have queue and freshness constraints.

Applying the same policy to pinyin would silently change the word. Drop one key from nihao and every subsequent key may execute correctly while the composition is already wrong. The input method has no concept of a letter being discarded as an optimization.

RemoteControlSession gives typing different admission rules:

Action	Semantics the queue must preserve
Clicks and other coordinate actions	Check freshness and display changes to avoid acting on old coordinates
Ordinary typing	Preserve order without merging or aging out keys; reject new keys visibly when 48 are waiting
Auto-repeat from a held key	Avoid adding repeats while keys are waiting, keeping actions such as Backspace near the phone’s processing pace

When the keyboard disappears, pending keystrokes are removed. Reconnecting does not replay an old pinyin fragment. Execution checks keyboard presence again because it may have changed after admission.

A keyboard generation counter protects another edge. A daemon may reject a chord and trigger a CLI fallback. If the keyboard disconnects and reconnects during that transition, the old request must not revive simply because presence is true again. A restored Boolean is not the original input context.

The first explanation for missing candidates was wrong

Once composition worked, the next report was that Xiaoyi’s candidates were visible on the phone but absent from the Mac’s mirrored picture.

The initial investigation treated this as a capture-interface limitation. There was no SDK switch allowing the app to force inclusion of another application’s input-method window. That established a limit on what the app could control; it did not establish that the system could never include that window.

The device record from October 10 corrected the conclusion. On that phone, the screen-sharing notification exposed a screen-recording privacy-protection setting. Turning it off made the hardware-keyboard candidate strip appear in the mirror. Turning it back on hid the strip. The choice applied to that sharing session and needed attention again for a new one.

The scope matters: this was the observed Mate X5 system, input method, and sharing session. It was not a test of every device, nor of whether a software keyboard could be captured while hardware-keyboard mode normally kept it hidden. A bounded observation should remain bounded.

Compose on the Mac, then send finished text

Candidate visibility depended on a phone system setting, so a second path remained useful. A native NSTextField at the bottom of the picture lets macOS handle composition and candidate selection. Once the user finishes the text, the Mac submits the whole string through the remote-control text channel.

This path needs no Bluetooth keyboard identity: the phone receives finished text rather than pinyin to interpret. It has a different cost. The UiTest text path used here borrows the phone’s clipboard, so the UI warns that it will be overwritten. Clipboard synchronization temporarily suppresses the injected content’s echo by hash, preventing it from returning as a new copy operation.

Return also has an explicit meaning. In this field it submits the composed string; it does not additionally inject Enter on the phone. Otherwise, placing text in a chat field might also send the message.

Clearing the local field requires a similarly narrow contract. sendRemoteText first checks that control is ready and idle, and that the text is nonempty, contains no NUL, and fits within 4096 UTF-8 bytes. The field clears only after local acceptance. If admission fails, the draft remains. Acceptance is not an acknowledgment that the phone has displayed the text.

The October 10 Mate X5 acceptance record covers Mac-side candidate selection, whole-string insertion on Return, draft preservation across mode changes, and retaining text when the channel is busy. Existing regression code separately constrains the boundaries: no silent raw-letter fallback without a keyboard; phone composition must ignore the Mac event’s characters; Mac composition must not append a phone-side Enter. Those are historical records and source evidence, not newly repeated device tests during this writing.

The useful decision from this work is to choose which device owns composition before choosing what crosses the connection. Sending keys requires ordering and keyboard-lifetime rules. Sending finished text requires submission, clipboard, and draft-preservation rules. The difficult parts of remote input sit in that ownership of state.