Admin Remote Servers, WebSSH, and WebSFTP

Title

feat(admin): add remote servers and browser SSH/SFTP

Target Branch

main

Summary

This change adds admin-managed remote SSH servers, SSH public-key credentials, browser-local SSH terminals, and browser-local SFTP file operations. The Server creates a one-time WebSocket ticket and brokers encrypted bytes through the existing Agent relay. It does not terminate SSH, store SSH passwords or private keys, inspect terminal output, or expose a Server-side file API.

The browser libssh2 transport now registers its custom send/receive callbacks before handshake, maps an empty WebSocket receive queue to Emscripten/WASI EAGAIN (-6) instead of the public libssh2 error code (-37) or Linux EAGAIN (-11), and decodes WASM heap pointers into readable SSH/SFTP error text. SFTP directory parsing also compares POSIX file-type bits exactly so regular files are no longer misclassified as symlinks.

The WebSocket byte stream now waits for onopen before delivering the client SSH banner. This removes a startup race where libssh2 sent its banner while the browser socket was still CONNECTING, causing the transport to fail and libssh2 to report Failed getting banner.

@verdigris/libssh2.js@0.1.9 declares ssh2_session_hostkey in its TypeScript definitions but does not export the matching WASM function. The browser client now extracts the server public-key blob from the pre-encryption KEX reply and falls back to that key when the binding is unavailable, preserving interactive host-key verification without weakening SSH security.

The package also declares string arguments for functions whose actual exports use raw C pointers and lengths. The client now allocates NUL-terminated UTF-8 strings for password authentication, public-key authentication, PTY, disconnect, and SFTP paths; public-key authentication uses the actual seven-argument ABI. The host-key confirmation dialog renders the fingerprint as a wrapped standalone code block instead of embedding a long value in a normal sentence.

The ninth round adds lrzsz (ZMODEM) transfers inside the browser terminal and stored credential secrets that let WebSSH/SFTP authenticate without a dialog. Credentials now come in two types, ssh_public_key (optionally storing the private key and passphrase) and password; secrets are sealed with the existing AES-256-GCM auth.SecretStore and delivered to the owner’s browser exactly once at session creation. Schema expands from v12 to v13 with four nullable credentials columns.

User Impact

  • Administrators and authorized users can maintain remote SSH server records and SSH public-key credentials.
  • Remote-server lists show joined Agent and credential display names plus Agent online state.
  • SSH passwords are entered per session and retained only in browser memory.
  • WebSSH verifies the SSH host-key fingerprint interactively before continuing.
  • WebSFTP supports directory browsing, upload, download, rename, delete, and progress display over the same authenticated SSH connection.
  • Single-file browser transfers are capped at 1 GiB before transfer begins.
  • The 远程服务器 page lists the caller’s own active SSH sessions and can disconnect one, which releases the active-session quota immediately instead of waiting for session_ttl.
  • A terminal whose remote shell exited prints the remote exit code or terminating signal and offers 重新连接 back to the matching server’s authentication dialog.
  • Empty lists render the normal Element Plus empty illustration on every route instead of an unstyled oversized graphic.
  • 密钥管理 offers two credential types; an ssh_public_key credential can additionally store its private key, and a password credential stores only a password. Both are encrypted at rest and reported as 已保存 / 未保存.
  • A remote server bound to a credential that holds a secret is tagged 自动认证, and clicking SSH 连接 or SFTP 文件 enters directly without the password dialog. Without a usable secret the manual dialog still appears, so nothing regresses for existing rows.
  • The terminal supports lrzsz: sz <file> downloads to the browser and rz opens a multi-file picker and uploads to the remote working directory. A transfer panel shows direction, name, progress and 取消传输, and keystrokes are not forwarded to the remote while a transfer runs.

API / Schema / Configuration Impact

  • API adds credential CRUD and one-time private-key extraction, remote-server CRUD, and WebSSH ticket/session operations under /api/v1.
  • API adds GET /api/v1/ssh-sessions, a cursor-paged list of the caller’s own active WebSSH sessions; owner scope is enforced in the service and administrators also see only their own sessions.
  • Schema expands from v11 to v12 with credentials, remote_servers, and webssh_sessions; no existing table is modified.
  • Schema expands from v12 to v13 with four nullable credentials columns (secret_ciphertext, secret_nonce, secret_key_id, secret_version); no existing column or index is modified and no row is rewritten.
  • API extends credential create/replace/patch with a write-only secret object, adds hasSecret to credential reads, adds nullable credentialType/credentialHasSecret to remote-server reads, and adds a one-time auth object to the WebSSH ticket response.
  • Configuration gains no new key: credential secrets reuse TUNNELMESH_TOKEN_ENCRYPTION_KEY / TUNNELMESH_TOKEN_ENCRYPTION_KEY_ID. Without them, storing a secret fails with 503 and auto-authentication degrades to the manual dialog. ZMODEM needs no server-side switch.
  • Configuration adds server.webssh controls for enablement, ticket/session TTL, active-session limits, open/idle timeouts, and maximum WebSocket message size.
  • Nginx documentation adds a long-lived, unbuffered /ws/webssh/ location.

Security and Authorization Impact

  • Remote-server ownership and Agent policy are enforced by the service layer; handlers do not trust client-supplied ownership.
  • WebSSH tickets are random, hash-stored, single-use, and time-limited.
  • Browser origin validation occurs before WebSocket upgrade and ticket consumption.
  • Passwords and private keys used for a browser SSH connection remain in browser memory and are never sent to the management API.
  • Private-key extraction is an explicit one-time HTTPS operation: the Server parses the key in memory and returns only the public key and fingerprint; private-key material is not persisted or logged.
  • Audit records contain resource IDs and result classes, not credentials, terminal bytes, or SFTP file content.
  • Runtime shutdown now terminates every process-local WebSSH bridge and its Agent stream.
  • Credential secrets are decrypted only for credential.OwnerUserID; administrators can list a credential but cannot read another user’s secret.
  • The decrypted secret is attached to the session-creation response after the idempotency record is written, so a replay returns the ticket without any auth material. The response carries Cache-Control: no-store, and the credential.revealed audit event records identifiers only.
  • Secret ciphertext, nonce, key id and plaintext are absent from list/detail responses, audit details, logs, Prometheus, traceroute and error messages.
  • ZMODEM bytes are brokered like every other WebSSH byte: the Server never parses the protocol, buffers a file, or records a file name.

Test Evidence

The following checks passed during implementation:

go test ./internal/server -run 'TestWebSSH' -count=1
go test ./... -count=1
go test ./internal/storage -run 'TestMySQLV11ToV12|TestSQLiteV11ToV12' -count=1
cd web && npm test -- --run
cd web && npm run build
./scripts/verify-web-embed.sh

The frontend suite reported 25 files and 138 tests passing. The Go suite passed all packages. The E2E test verified one-time ticket consumption, browser-to-TCP bytes, TCP-to-browser bytes, and close of both streams. A separate red-green test verified runtime shutdown terminates active WebSSH bridges.

The v11-to-v12 schema regression test verifies that indexed WebSSH timestamp columns use VARCHAR(32) in both the full DDL and the MySQL migration, preventing MySQL Error 1170 when creating the new indexes. The MySQL migration also repairs tables partially created with TEXT columns by a failed earlier attempt, then tolerates already-created duplicate indexes on retry.

Final mandatory verification also passed:

go test ./... -count=1
go test -race ./...
go vet ./...
go build ./cmd/...
git diff --check
cd web && npm test -- --run && npm run build
rsync -a --delete web/dist/ internal/server/web_dist/
./scripts/verify-web-embed.sh

go test -race ./... passed all packages; the slowest package was internal/server at 249.906 seconds. The final frontend run again reported 133 tests passing and a successful Vite build. git diff --check, go vet, go build, and the embed comparison produced no errors. Docker image construction was not executed in this development environment because no docker, podman, buildah, or nerdctl executable is installed.

A later red-green regression fix for the browser handshake added seven WebSSH tests covering callback registration, WASI EAGAIN, heap-pointer error decoding, KEX host-key fallback, SFTP pointer errors, directory-name decoding, and POSIX type bits. A separate byte-stream regression test verifies that the SSH banner waits for the WebSocket to open and fails clearly if the socket closes first. Real Chrome WASM smoke tests against a local OpenSSH server completed the SSH handshake (WEBSSH_WASM_HANDSHAKE_OK) and extracted the host key successfully (HOSTKEY_OK:sha256:...). Fresh verification passed:

cd web && npm test -- --run
cd web && npm run build
rsync -a --delete web/dist/ internal/server/web_dist/
./scripts/verify-web-embed.sh
go test ./internal/server -run 'TestWebSSH' -count=1
go vet ./...
git diff --check

The frontend suite reported 25 files and 141 tests passing, and the WebSSH-focused Go package test passed.

The 2026-09-12 password-authentication deployment regression was fixed with red-green tests for the real WASM ABI and host-key dialog. A real Chrome test against a local SSH server returned PASSWORD_AUTH_OK after interactive-equivalent host-key verification. Final verification passed:

cd web && npm test -- --run
cd web && npm run build
rsync -a --delete web/dist/ internal/server/web_dist/
./scripts/verify-web-embed.sh
go test ./... -count=1
go test -race ./...
go vet ./...
go build ./cmd/...
go build -o ./tunnelmesh-server ./cmd/tunnelmesh-server
git diff --check

The frontend suite reported 25 files and 150 tests passing. The race suite passed all packages, with the slowest package internal/server completing in 305.886 seconds. The final tunnelmesh-server SHA-256 is 18b696f72485ac7e57a4b755a38f9555e768a2682b9726b8028cf4ced2b6863d.

A second deployment regression round fixed three production issues reported from the browser console and UI:

  • The embedded static server now pins Content-Type: application/wasm for .wasm assets. Host mime databases without a .wasm entry previously made http.FileServer sniff application/octet-stream, so browsers rejected streaming compilation and logged Incorrect response MIME type before falling back to ArrayBuffer instantiation.
  • main.ts now imports the Element Plus message and message-box styles explicitly. On-demand style imports only cover components referenced from templates, so programmatic ElMessageBox/ElMessage rendered without panel or overlay styles in production builds.
  • The WebSSH terminal is now created only after the session reaches connected, so xterm measures real cell sizes and receives keyboard focus via an explicit terminal.focus(). The shell channel also exposes onExit, and the view reports remote shell termination with a localized notice instead of staying connected with a dead channel.

Red-green tests cover the WASM content type, the style imports, shell exit notification, and the view ordering/focus markers. A real Chrome smoke test against a local SSH server verified password authentication and the input path (PASSWORD_AUTH_OK, INPUT_ECHO_OK).

A third round fixed three admin-console defects reported from production use:

  • SSH 会话创建失败 had no actionable reason. The connect dialog replaced every rejection with the generic remoteServers.sshFailed string, so administrators could not tell a disabled server from an offline agent or an exhausted session quota. writeAPIError puts the stable identifier in data.error while the envelope msg is only the HTTP status text, so the new web/src/views/remote-server-errors.ts maps APIError.domain (remote server is unavailable, remote server agent is unavailable, remote server credential is unavailable, remote server agent is offline, webssh active session limit reached, webssh username is required, resource is forbidden) to localized remoteServers.errors.* messages, keeps an unmapped identifier readable for troubleshooting, and falls back to the generic string for non-API failures. RemoteServers.vue calls sshErrorMessage(error, t).
  • Cross-owner WebSSH requests answered 500 instead of 403. writeWebSSHError sent ErrResourceForbidden through writeStorageError, so POST /api/v1/remote-servers/{id}/ssh-sessions, GET and DELETE /api/v1/ssh-sessions/{id} returned 500 with the internal text resource is forbidden, unlike the remote-server and credential APIs which already answer 403. writeWebSSHError now maps ErrResourceForbidden and auth.ErrForbidden to 403 with a non-leaking forbidden body.
  • The active-session limit answered 500 instead of the documented 409. docs/api/openapi.yaml already promises 409: Idempotency conflict or active-session limit reached, but the inline error fell through writeStorageError. The limit is now the sentinel ErrWebSSHActiveSessionLimit, which writeWebSSHError maps to 409 while keeping the stable identifier in data.error so the console can localize it. No OpenAPI change was required because the implementation now matches the published contract.
  • Quota exhaustion after a Server restart. webssh_sessions rows left in active by a previous process could never be served again, but the sweeper only reclaimed them after the 8-hour session_ttl, so max_active_sessions_per_user=5 filled with dead rows and every new session failed. WebSSHSessionRepository.CloseActiveByNode and WebSSHSessionService.CloseStaleLocalSessions now close those rows once during runtime startup, before any ticket is issued, and log webssh_stale_sessions_closed with the affected count. The startup hook is guarded by server.webssh.enabled and aborts startup if the cleanup fails.
  • 远程服务器 / 服务令牌 / Server 节点 tables forced a page-level horizontal scrollbar. .tm-page used display: grid with an implicit auto column, so the track grew to the table’s max-content width instead of the viewport. It is now grid-template-columns: minmax(0, 1fr), which keeps wide tables scrolling inside their own card while the fixed="right" action column stays reachable.

The administrator menu entry 下载 was also renamed to 发行管理 (Releases in English) and moved to the last position, after 审计日志. Documentation was updated in docs/user-guide/server-admin.md (new 发行管理 section, SSH failure reasons, node-restart session cleanup), docs/README.md, and docs/deployment/binary-release.md.

Round-three verification:

cd web && npx vitest run src/tests/shell.spec.ts src/tests/design-system.spec.ts src/tests/remote-servers-view.spec.ts
cd web && npm test -- --run
cd web && npm run build
rsync -a --delete web/dist/ internal/server/web_dist/
./scripts/verify-web-embed.sh
go test ./internal/storage -run TestWebSSHSessionCloseActiveByNode -count=1
go test ./internal/server -run 'TestWebSSHSessionServiceClosesStale|TestWebSSH' -count=1
go test ./... -count=1
go test -race ./...
go vet ./...
go build ./cmd/...
go build -o ./tunnelmesh-server ./cmd/tunnelmesh-server
git diff --check

Red tests were confirmed failing before implementation: TestWebSSHSessionCloseActiveByNode, TestWebSSHSessionServiceClosesStaleLocalSessionsAfterRestart, TestWebSSHAPIForbiddenReturns403 (create status = 500, want 403: {"code":500,"msg":"Internal Server Error","data":{"error":"resource is forbidden"}}), TestWebSSHAPIActiveSessionLimitReturnsConflict (second create status = 500, want 409), the remote-server-errors.spec.ts domain-mapping suite, the shell.spec.ts menu-order and rename assertions, and the design-system.spec.ts .tm-page assertion.

After implementation the frontend suite reported 26 files and 164 tests passing, go test ./... -count=1 passed every package, and go test -race ./... passed with internal/server completing in 340.095 seconds. go vet ./... and git diff --check are clean, and npx vue-tsc --noEmit stays at the pre-existing 172-error baseline with no new errors in the touched files. scripts/verify-web-embed.sh reported web/dist and internal/server/web_dist match, and the rebuilt binary embeds the new bundle. The final tunnelmesh-server SHA-256 is 70bc83223f589b6b38df4ef8355a1bbbeed34b263322cc7d44b58e71f758649c.

Pre-existing flake, now fixed (test-only): two go test ./... -count=1 runs reported TestServeClientSessionResetsStreamAfterShortRelayWrite: RESET count = 0, want 1. internal/server/ws_client.go is untouched by this branch. The RESET is enqueued from the per-stream relay pump goroutine (ws_client.go:530) while ServeClientSession returns on EOF and drains the fair writer in a defer, so the assertion depended on a goroutine ordering the implementation does not promise. scriptedClientTransport.sent is now guarded by a mutex with a sentFrames() snapshot accessor (the unsynchronized slice was also a latent data race between the writer goroutine and the test), and the assertion uses a new bounded waitForResetCount poll. Verified with 200 repetitions plus two full-package runs; production behavior is unchanged.

A fourth round closed the remaining horizontal-overflow reports and one embedding defect found while verifying in a real browser:

  • 服务令牌 still overflowed because it never used the shared page grid. Tokens.vue was the only console view whose root element lacked tm-page; its scoped .tokens-page { display: grid; gap: 16px; } duplicated the old unbounded rule. The view now roots in tm-page tokens-page, the duplicate rule is deleted, and the same latent pattern was hardened in Downloads.vue (.release), WebSFTP.vue (.file-card, .transfer-panel), WebSSHTerminal.vue (.terminal-card) and Tokens.vue (.scope-list) by giving every single-column grid an explicit grid-template-columns: minmax(0, 1fr).
  • A new guard test makes the whole class unregressable. web/src/tests/layout-overflow.spec.ts asserts that every console view roots in tm-page, that no view/component/layout style block declares display: grid without explicit columns, that tokens.css keeps the bounded page grid, and that any el-table whose column widths exceed ~1270px pins its action column with fixed="right". It failed first on Tokens.vue and Downloads.vue .release, then passed.
  • //go:embed web_dist silently dropped _-prefixed assets. Go embed excludes files starting with _ or ., and Vite emits lodash helper chunks such as _baseClone-*.js. The missing chunk made the browser fall back to index.html (200 + text/html), the ES module load failed, and every route importing it rendered a blank shell. internal/server/web.go now uses //go:embed all:web_dist, and TestEmbeddedWebDistContainsEverySourceFile diffs the embedded FS against the source directory so any future omission fails the build. Red output before the fix: embedded web_dist is missing 1 source file(s): [assets/_baseClone-CZsh75pW.js].
  • Layout verified in a real browser, not just by CSS assertions. Headless Chrome at the reported 1521x731 viewport, authenticated against a local SQLite server with 12 seeded tokens, measured all 12 console routes: documentElement.scrollWidth == clientWidth everywhere (no page-level horizontal scrollbar), /tokens rendered 12 rows with the table scrolling inside its card (scrollWidth=1410 vs clientWidth=1231) and the action column pinned right. A screenshot confirms the rendered result.

A fifth round fixed the quota lockout reported as “活跃 SSH 会话数已达上限,请先断开闲置会话后重试”:

  • Completed or dead WebSSH sessions never released their quota. WebSSHBroker.Handle only wrote closed in the agent_unreachable branch; when the bridge ended normally — browser tab closed, remote sshd exited (including host-side access control dropping the session seconds after the MOTD) — the DB row stayed active until the 8h session_ttl sweeper. With max_active_sessions_per_user=5, five such connections locked the user out, and no API or UI lists or closes own sessions. The owning node now durably closes the session whenever the bridge returns, with a close_reason derived from which side ended first: remote_closed, client_disconnected, server_closed, or bridge_closed. Red before the fix: session status = active, want closed after the remote stream ended and the browser-disconnect twin.
  • Two new broker tests cover both endings and pin the close reasons; the existing e2e and idle-deadline tests still pass. Documentation in docs/user-guide/server-admin.md now explains quota release, the close reasons, and that a restart reaps legacy rows via webssh_stale_sessions_closed.

A sixth round implemented the approved plan docs/superpowers/plans/2026-09-12-webssh-session-management.md (T1-T7), giving users a self-service remedy for an exhausted quota:

  • Rows left active by an older build still had no user-visible escape. Round five stopped new leaks, but legacy rows — or a browser that dies right after consuming its ticket — occupied the quota until session_ttl, and nothing listed a caller’s own sessions. WebSSHSessionRepository.ListActivePage now pages one owner’s active sessions by ID using the shared cursor contract (pageArgs, decodeCursor, ORDER BY id LIMIT limit+1), mirroring credentialRepo.List. Rows whose expires_at has passed are excluded because the sweeper reclaims them without a bridge, so offering them would disconnect nothing. Red first: db.WebSSHSessions().ListActivePage undefined (type WebSSHSessionRepository has no field or method ListActivePage).
  • GET /api/v1/ssh-sessions is owner-scoped in the service, not in the handler. WebSSHSessionService.List forces ownerUserID = actor.UserID and never reads an owner from the request; administrators also list only their own sessions, because cross-user session management needs its own authorization and audit model. The route answers 405 for non-GET on the collection. Red first: post status = 404, want 405. docs/api/openapi.yaml gains the path plus WebSSHSessionPage/WebSSHSessionPageEnvelope.
  • The console can now release the quota itself. 远程服务器 gains an 活跃会话 card (target server, created, expiry, confirmed 断开) that calls the existing DELETE /api/v1/ssh-sessions/{id}; no new close path was added. Session rows carry IDs only, so names resolve from the loaded server list first and per-ID second, and an unreachable or deleted server falls back to the raw ID with an 未知服务器 tag. The server list is awaited before sessions resolve, which removes a redundant per-session fetch on first paint.
  • A closed terminal now explains itself. TerminalChannel.onExit delivers { status, signal } read from ssh2_channel_get_exit_status / ssh2_channel_get_exit_signal before the channel handle is freed, reusing the pointer-or-string decoding from the session error path and degrading to null when a build lacks the bindings rather than inventing an exit code. The terminal prints the reason and offers 重新连接, which routes to /remote-servers?ssh=<serverId>; the view opens that server’s auth dialog and clears the query. Only the non-sensitive server ID survives a close, so credentials are still dropped.
  • Verified in a real browser, not only by source-marker specs. Headless Chrome at 1521x731 against a local SQLite Server with two seeded active sessions (one pointing at a nonexistent server): the card rendered both rows, resolved 验证服务器 from the loaded list, tagged the unknown target, issued no redundant fetch for the known server, and kept documentElement.scrollWidth == clientWidth (sessions table 1263/1263; the servers table still scrolls inside its own card at 1390/1263). Clicking 断开 and confirming produced 200 DELETE /api/v1/ssh-sessions/webssh-live-1, a 会话已断开 toast, a refreshed single-row list, and status=closed, close_reason=user_closed in the database. /remote-servers?ssh=<id> opened 连接 SSH with the default username prefilled and an empty query string afterwards. No console exception was recorded in any step. Restarting the Server logged webssh_stale_sessions_closed count=2 node_id=local, confirming the round-five startup reaper on a live process.
  • Suite state after the round: go test ./... -count=1 and go vet ./... clean, go test -race ./... clean, git diff --check clean, frontend 27 files / 177 tests passing, npm run build plus scripts/verify-web-embed.sh reporting web/dist and internal/server/web_dist match. npx vue-tsc --noEmit reports 179 errors project-wide; it is not part of the build or test gate and the codebase already carries this class of error. This round adds exactly four: three DefaultRow table-slot errors in RemoteServers.vue (lines 23, 24, 37), the same Element Plus slot-typing pattern already present in that file and in WebSFTP.vue, and one untyped-mock error in ssh-client.spec.ts:566 matching the pre-existing _ssh2_sftp_* assertions in the same spec.
  • Suite state after the round: go test ./... -count=1 and go vet ./... clean, go test -race ./... clean, git diff --check clean, frontend 27 files / 177 tests passing, npm run build plus scripts/verify-web-embed.sh reporting web/dist and internal/server/web_dist match. npx vue-tsc --noEmit reports 179 errors project-wide; it is not part of the build or test gate and the codebase already carries this class of error. This round adds exactly four: three DefaultRow table-slot errors in RemoteServers.vue (lines 23, 24, 37), the same Element Plus slot-typing pattern already present in that file and in WebSFTP.vue, and one untyped-mock error in ssh-client.spec.ts:566 matching the pre-existing _ssh2_sftp_* assertions in the same spec.

A seventh round fixed the empty-state rendering defect reported as a page-sized black shape on every empty list:

  • Root cause: style entries are injected only for template-resolved components. unplugin-vue-components with ElementPlusResolver imports a component’s CSS when it resolves the tag from a template. DataState.vue imports ElEmpty and ElSkeleton explicitly in <script setup>, and StatusTag.vue imports ElTag, so the resolver never sees them. el-empty.css entered the bundle only through lazy route chunks that use <el-empty> in a template (WebSSHTerminal, AgentDetail, Dashboard); on every other route the empty state rendered without .el-empty__image sizing and without the --el-empty-fill-color-* variables. Measured in a real browser before the fix: image box 1263x1375 and first SVG path fill rgb(0, 0, 0) — the giant black shape in the report. The same gap left loading skeletons and tags unstyled on routes whose chunk never loads those styles.
  • The fix follows the convention already used by AppShell.vue (menu), PageHeader.vue (breadcrumb) and main.ts (message/message-box): import the style entry next to the explicit component import. DataState.vue now imports the alert, button, empty and skeleton style entries; StatusTag.vue imports the tag style entry, making both components self-sufficient on every route.
  • Guard test. web/src/tests/build.spec.ts gains loads styles for components imported explicitly in script setup, asserting those five style imports. It failed first on the missing alert import, then passed.
  • Verified in a real browser against an empty database. /credentials, /tokens, /audit-logs, /remote-servers and /agents now render the bounded illustration: image box 160x174 (was 1263x1375), first path fill rgb(220, 223, 230) (was black), empty block 294px tall (was 1418px), and no page-level horizontal scroll on any of them.
  • Suite state after the round: frontend 27 files / 178 tests passing, npm run build plus scripts/verify-web-embed.sh matching, and go test ./... -count=1, go vet ./..., git diff --check clean. No Go behavior changed, so the previous -race result still applies. An eighth round fixed the report “ssh可用,切到sftp不能用 / 断开连接后应回到主机列表 / 活跃会话列表放到页面下方”. Unlike earlier rounds this one could only be diagnosed against a live SSH target, so a throwaway end-to-end harness (outside the repository, under /tmp/tm-e2e) was built: a local SSH host implemented with golang.org/x/crypto/ssh plus github.com/pkg/sftp (read-only, loopback, password tmuser/tmpass), the real Server and Agent binaries, and headless Chrome driven over CDP. It reproduced the failure exactly and then proved the fix: six checks pass, including a 23-entry directory listing over the tunnel, the terminal showing the remote prompt after returning from SFTP on the same connection, no second host-key dialog, and disconnect landing back on /remote-servers.

  • SFTP died on the first non-blocking stall. libssh2 reports “would block” for handle-returning calls (_ssh2_sftp_init, _ssh2_sftp_open, _ssh2_sftp_opendir) with a NULL handle plus LIBSSH2_ERROR_EAGAIN in the session errno; the return value stays 0. The existing retryEAGAIN/retrySFTP loops only recognise the int code -37, so SFTP startup aborted immediately with SFTP initialization failed: Would block starting up channel even though the tunnel was healthy. A new retryHandleEAGAIN retries while the session errno is EAGAIN and the handle is NULL. Red first: promise rejected "Error: SFTP initialization failed: libssh…" instead of resolving.
  • A successful realpath was reported as an error. libssh2_sftp_realpath returns the resolved string length on success, but the code asserted “non-zero means failure”, so /tmp/tm-e2e (length 11) became SFTP real path failed and the file manager stayed empty. Only negative results are failures now. Red first: the new reads the resolved path when realpath returns its length test rejecting.
  • Failures were invisible. WebSFTP.vue set status='error' but not loadError, so a failed connect rendered the disabled toolbar plus “目录为空” — indistinguishable from an empty directory. DataState now owns the error surface (:error="loadError", retry re-runs the handshake when the view never connected), and openSFTP includes libssh2’s errno/message instead of the bare “SFTP initialization failed”. Red first: expected [Function] to throw error including 'SFTP initialization failed: subsystem…' but got 'SFTP initialization failed'.
  • The direct SFTP entry never authenticated. Opening SFTP straight from the server list only creates a pending ticket; WebSFTP.vue reused store.activeSession and threw missingSession. It now bootstraps the SSH connection itself (same flow as the terminal) and publishes it to the store, so the terminal opened afterwards reuses it. The host-key confirmation moved to a shared web/src/webssh/host-key.ts (useHostKeyVerifier) and the WebSocket URL builder to websshWebSocketURL in byte-stream.ts, so both entry points present identical prompts and URLs.
  • Returning from SFTP to the terminal demanded a new one-time ticket. WebSSHTerminal.vue now opens another shell channel on store.activeSession when the session IDs match; the store keeps the non-sensitive remoteServerId on the active session so the reconnect button still works after a remount.
  • 断开连接 now leaves the page. Both views expose disconnectAndLeave, which closes the session and routes to /remote-servers; staying on a dead session only invites a reload that can never succeed because the ticket is consumed.
  • 活跃会话 moved below the server table on /remote-servers: the list is the primary task, the session card is a recovery surface. A view test asserts the card order so it cannot drift back.
  • Suite state after the round: frontend 27 files / 191 tests passing, npm run build plus scripts/verify-web-embed.sh matching, go test ./... -count=1, go test -race ./..., go vet ./... and git diff --check clean. npx vue-tsc --noEmit still reports the pre-existing 177 errors and adds none.

A ninth round implemented the approved plan docs/superpowers/plans/2026-09-12-webssh-zmodem-credential-auto-auth.md (T1-T8): lrzsz/ZMODEM transfers in the browser terminal, and stored credential secrets that let WebSSH/SFTP authenticate without a dialog.

  • Schema v12 to v13 is expand-only. credentials gains four nullable columns (secret_ciphertext, secret_nonce, secret_key_id, secret_version) in migrations/ddl.sql, migrations/incremental/v0012_to_v0013/{mysql,sqlite}.sql, SchemaVersion=13 and docs/operations/schema-upgrades.md. Two latent migration-test bugs surfaced while writing the v13 red test and were fixed: the SQLite migration tests must INSERT INTO schema_meta(id,version) because an UPDATE matched zero rows and silently skipped the migration, and scanCredential reads secret_version through sql.NullInt64 so a NULL no longer errors.
  • One sealed blob, not nine columns. A credential secret is stored as a single AES-256-GCM JSON document ({"password":…} or {"privateKey":…,"passphrase":…}) through the existing auth.SecretStore, so adding a future secret type needs no DDL. Limits are enforced at the boundary (password 4 KiB, private key 64 KiB, passphrase 1 KiB) and a type change clears the stored secret. Without TUNNELMESH_TOKEN_ENCRYPTION_KEY the service returns ErrCredentialSecretUnavailable and the API answers 503; there is no silent plaintext fallback. Reads only ever expose hasSecret.
  • Decryption is owner-scoped, not admin-scoped. CredentialService.GetWithSecret refuses every caller except credential.OwnerUserID, so an administrator cannot read another user’s SSH password or private key even though they can list the credential.
  • The plaintext never reaches the idempotency store. createWebSSHSession builds the ticket inside a.mutate and attaches WebSSHAuthPayload afterwards, so the persisted response body holds no secret and a replay with the same Idempotency-Key returns the ticket without auth. The one-time response is sent with Cache-Control: no-store and audited as credential.revealed with identifiers only. Missing resolver, missing secret, or a non-owner all fail open to the existing manual password dialog instead of failing the connection.
  • ZMODEM runs entirely in the browser. zmodem.js@0.1.10 is wrapped by a single module, web/src/webssh/zmodem.ts (createZmodemBridge), whose Sentry splits remote bytes into terminal output and protocol state. Detection is auto-confirmed (a ZMODEM magic sequence can only appear because the user ran sz/rz), the header is suppressed from the terminal, lrzsz’s trailing OO is stripped, receive resolves to a Blob download, and send drives a hidden multi-file input through Zmodem.Browser.send_files. Keystrokes are dropped while a transfer is active, abort() sends ZABORT, and the shared web/src/webssh/download.ts helper removes the duplicated Blob/anchor code from WebSFTP. Switching the terminal byte path to Uint8Array plus a streaming TextDecoder also fixed multi-byte characters split across chunk boundaries.
  • Auto-authentication is visible before it is used. RemoteServerDisplay adds nullable credentialType/credentialHasSecret, rendered as a “自动认证” tag on /remote-servers; clicking SSH or SFTP on such a row skips the dialog and falls back to it when the response carries no auth or the connection fails.
  • End-to-end proof against real lrzsz. The throwaway harness in /tmp/tm-e2e was upgraded so the SSH target is a real pty shell (creack/pty + /bin/zsh) with homebrew sz/rz on PATH. Twelve checks pass under headless Chrome over CDP: the auto-auth tag, no auth dialog, host-key confirmation, terminal connect with the stored password, keystrokes, a byte-for-byte 256 KiB sz download, panel teardown with the prompt returning, a byte-for-byte 2 KiB rz upload, SFTP reuse listing the uploaded file, disconnect returning to the list, and the manual dialog for a server without a credential.
  • A pre-existing flake in internal/agent was fixed because it broke the gate. TestConnectionControllerRequiresServerAckAndRestartsFailedConnection slept 20 ms and then asserted runner.count() >= 2, while supervise restarts after its own 20 ms backoff and Snapshot().Active is already 1 before the first Runner call; the test failed roughly two runs in three. The new waitConnectionStarts polls the recorded starts under a 2 s deadline. Test-only change, 20 consecutive -count=20 runs green.
  • Suite state after the round: frontend 30 files / 213 tests passing, npm run build plus scripts/verify-web-embed.sh matching, go test ./... -count=1, go test -race ./..., go vet ./... and git diff --check clean. npx vue-tsc --noEmit still reports the pre-existing 177 errors and adds none. A tenth round implemented the approved plan docs/superpowers/plans/2026-09-12-webssh-bulk-transfer-flow-control.md (T1-T8): bulk sz/rz transfers died mid-flight, and a browser refresh dead-ended on “credentials missing”.

  • Root cause: the local Agent relay path had no flow control in either direction. The protocol and the Agent already had the primitives, but internal/server/agent_relay_transport.go never used them. Download: the WebSSH broker left relay.StreamRequest.InitialWindow at zero, which the Agent reads as “unlimited”, so it never created a sendState and flooded the Server; the Server’s bounded inbound channel overflowed, enqueue returned relay.ErrBackpressure, dispatch treated that as fatal and reset the stream, and the SSH channel hit EOF - which the terminal rendered as a genuine 远端退出码:0. Upload: the Agent does enforce a 256 KiB receive window and resets the stream on overrun, but the Server’s Write never consumed send credit and dropped every inbound WINDOW_UPDATE, so any upload larger than the window was killed. Both directions are now symmetric window accounting.
  • One shared credit contract. New internal/protocol/window.go holds MaxStreamFrame (32 KiB), DefaultServerReceiveWindow (512 KiB), DefaultAgentReceiveWindow (256 KiB) and DefaultWindowUpdateThreshold (128 KiB); internal/agent/session.go now aliases them instead of keeping private copies, so the two ends cannot drift. No new frame type and no capability negotiation were needed: OPEN_STREAM.Window and WINDOW_UPDATE already existed, and DefaultServerReceiveWindow is only advertised when the caller asked for none.
  • A second-order bug found while fixing the first: the buffer was counted in frames while the window is counted in bytes. After window accounting was correct, transfers still reset between 0.3 and 1.1 MiB. The Agent charges its window per byte but emits one frame per pty read, and a pty read is often far smaller than 32 KiB, so a peer that stayed strictly inside its 512 KiB window could still send thousands of small frames into a 16-slot channel. The inbound path is now a byte-budgeted FIFO whose budget equals the advertised window, with ErrBackpressure retained only as a guard against a window-ignoring peer. Regression test TestAgentRelayStreamAcceptsManySmallFramesWithinWindow sends 511 KiB in 1 KiB frames to a reader draining one byte at a time; before the fix it failed at exactly 16384 bytes with relay: backpressure.
  • Deviation from the plan (recorded, not silent). The plan had the WebSSH broker set the default window. It is applied in AgentRelayTransport.OpenStream when InitialWindow == 0 instead, so every current and future caller of the local relay path is covered by one code path rather than each remembering to set it.
  • Cross-node relay audited, no change needed (T4). The gRPC relay path already blocks in Write and is governed by HTTP/2 flow control, so backpressure propagates to the caller naturally. Adding a second window layer there would have been redundant and would risk deadlock against the transport’s own credit.
  • The browser converted transient bursts into session kills. web/src/webssh/byte-stream.ts threw as soon as bufferedAmount passed 4 MiB, and web/src/webssh/ssh-client.ts closed the stream when the SSH receive queue passed 4 MiB - both trivially reachable by a legitimate sz. Writes now wait for the socket to drain above a 1 MiB high-watermark and only fail past a 64 MiB hard guard, and the receive bound was raised to 64 MiB. Because a waiting writer could otherwise be overtaken by a later one, writes are chained FIFO; reordered bytes would corrupt the SSH stream, and there is a test for the ordering specifically.
  • A local transport failure no longer masquerades as a remote exit. ShellExitInfo gains transportError, populated from the client’s transport error when the shell drain loop breaks. The terminal renders it as a red alert with the real reason instead of printing a fabricated exit code, and exitReason refuses to report a status or signal when the transport died.
  • Refresh recovery. The one-time ticket and the password live only in page memory, so a reload always lost them and the view threw webssh.missingSession with no way forward. Three helpers in web/src/stores/webssh.ts now remember only the non-sensitive remote-server ID in sessionStorage; on reload both the terminal and the SFTP view route to /remote-servers?ssh=<id>, which already re-opens the connect flow and auto-authenticates when a credential secret is stored. An explicit “disconnect” clears the record so a deliberate close is not undone by a later refresh. No credential material is persisted, and the helpers tolerate a browser that blocks sessionStorage.
  • The E2E harness moved into the repository (T7). It previously lived in /tmp/tm-e2e and was rebuilt by hand each round. It is now test/e2e/webssh/: a dependency-free CDP driver (lib/cdp.mjs), stack lifecycle and config resolution (lib/stack.mjs), deterministic generated fixtures instead of committed multi-megabyte blobs (lib/fixtures.mjs), a readable scenario script (run.mjs), and a documented README. Every path, port and payload size is environment-overridable; it builds the three product binaries from the checkout and skips the ZMODEM checks with a clear SKIP when lrzsz is absent. The throwaway SSH host is a nested Go module (test/e2e/webssh/sshhost/go.mod) so creack/pty, pkg/sftp and x/crypto/ssh stay out of the root go.mod - the product must never ship an SSH server. It was 14 checks at the end of that round, adding the refresh-recovery assertion and a “browser console stayed clean” gate that fails the run on any console error or uncaught exception; round 11 later added the SFTP upload check, so the current total is 15.
  • All temporary [diag] instrumentation was removed from webssh_broker.go, ssh-client.ts, byte-stream.ts and WebSSHTerminal.vue, and the web bundle plus internal/server/web_dist were rebuilt afterwards.
  • Verification. Go: go test ./... -count=1, go test -race ./..., go vet ./..., git diff --check. Frontend: npm test -- --run (30 files / 219 tests), npm run build, scripts/verify-web-embed.sh matching. Browser E2E from the repository checkout: 14/14 PASS, including a byte-for-byte 4 MiB sz download and a byte-for-byte 1.5 MiB rz upload against real lrzsz, with no console errors. npx vue-tsc --noEmit still reports the pre-existing 177 errors and adds none. All counts in this bullet are as of that round; round 11 raised them to 15 E2E checks and 222 frontend tests.

An eleventh round fixed a production report: WebSFTP uploads failed with the generic “文件传输失败” banner. A new E2E check reproduced it in-repo, and the diagnosis found three defects of one family plus the relay kill switch that actually killed the user’s sessions. This round was unplanned triage, so it is recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-sftp-upload-partial-write.md, with the red-light evidence labelled as observed during implementation rather than re-run.

  • The real production root cause: the relay’s WINDOW_UPDATE mirror queue killed bulk uploads. handleAgentFrameGeneration applies inbound WINDOW_UPDATE credit to sendState and then mirrors the frame into controlCh (capacity 16) for ReadControl consumers. The WebSSH/SFTP broker only ever calls Read, never ReadControl, so on any upload large enough to earn more than 16 updates the mirror overflowed and the dispatch failed the stream with protocol: flow-control window exhausted; the bridge closed, sshd saw the connection drop, and the UI reported “SFTP 通道已关闭”. Evidence from a temporary bridge diagnostic: first=protocol: flow-control window exhausted, with the on-disk upload truncated at 768 KiB - sshd’s flush lag behind the ~2 MiB of consumed credit (16 updates x 128 KiB). Downloads never hit this because Server-to-Agent credit travels through WriteControl, not controlCh, which is exactly why only uploads died. The mirror is now best-effort: when the queue is full the frame is dropped, because the authoritative credit already lives in sendState and consumeSendWindow reads it directly. Regression test TestAgentRelayStreamSurvivesUndrainedWindowUpdates sends 40 undrained updates and then requires Write to succeed; before the fix it failed with the exact production error.
  • libssh2_sftp_write partial returns truncated every upload (browser, pre-existing). The session runs non-blocking, so libssh2_sftp_write returns the bytes consumed so far - a positive value smaller than the request, not EAGAIN - whenever the channel window or the tunnel stalls. retrySFTP only retries EAGAIN, and uploadFile treated any short write as fatal, which is precisely the generic “文件传输失败” the user saw: the bigger the file or the slower the path, the more certain the failure. The handle write now loops, feeding the unconsumed remainder at pointer + written, with a 5 s zero-progress stall guard so a dead tunnel surfaces as an error instead of spinning the retry loop forever.
  • libssh2_channel_write partial returns silently dropped terminal bytes (browser, pre-existing, latent). The same non-blocking semantics apply to the shell write path, where the remainder after a partial write was discarded - a long paste could lose keystrokes with no error anywhere. It now uses the same consume-the-remainder loop and stall guard.
  • The E2E harness gained an SFTP upload check (now 15 checks). It uploads a 3 MiB deterministic payload through the real file picker, waits for the row to appear in the listing, and compares the on-disk sha256 inside the throwaway host’s SFTP root. sshhost SFTP is writable only when TM_SSHHOST_SFTP_READONLY=0; the harness sets it because the SFTP root is a temp directory it owns, while manual runs of the throwaway host stay read-only. The check failed at 768 KiB before the relay fix and passes byte-for-byte after it.
  • Verification. Go: go test ./... -count=1 with all 23 packages ok, go vet ./..., gofmt -l internal/ cmd/ test/ and git diff --check clean; go test -race ./internal/server ./internal/agent -count=1 -timeout 25m ok at 427 s and 4.3 s (the full-suite -race was validated earlier the same day and needs -timeout 30m as documented). Frontend: npm test -- --run 30 files / 222 tests, npm run build, scripts/verify-web-embed.sh matching. Browser E2E re-run in a fresh working directory against binaries rebuilt from the current source: 15/15 PASS including the 3 MiB SFTP upload landing byte-for-byte (on-disk sha256 identical to the fixture), the 4 MiB sz download and the 1.5 MiB rz upload, with a clean browser console. All temporary [diag] instrumentation removed again after the diagnosis.

A twelfth round fixed a production report that only reproduces on real networks: ZMODEM transfers corrupted in flight, with rz answering Unhandled header: ZRPOS / Peer aborted session and a finished sz dying with PROTOCOL: Only thing after ZFIN should be "OO" when the shell prompt arrived. The fix is browser-only and is recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-zmodem-write-serialization.md.

  • Root cause: concurrent outbound writers plus the round-11 partial-write continuation. The terminal view wires ZMODEM’s synchronous sender callback as fire-and-forget (void channel.write(bytes)), and zmodem.js calls it several times per consumed chunk (ZACK, ZFIN, data subpackets), so one SSH channel routinely carries several in-flight write() promises. Round 11 turned each write into a loop that resends the unconsumed remainder after an await; under channel-window pressure - routine on a WAN link, absent on localhost - two writers’ continuations interleave and the peer receives torn frames. The red unit test captured the exact wire signature: [1,1,2,2,1,2,1,2] instead of [1,1,1,1,2,2,2,2]. Torn frames explain every reported symptom: rz fails CRC and sends ZRPOS (which zmodem.js does not implement), and the browser’s mangled ZFIN reply makes lrzsz sz exit without ever sending OO, so the receive session meets the prompt’s 27,93 (ESC ]) where OO belongs.
  • Fix: one writer at a time per channel. A new createWriteGate() promise chain serializes every libssh2 write-path call per channel: shell write/resize/close (extracted to writeShell/resizeShell/closeShell), and the whole SFTP surface - open, handle read/write/close, readDirectory (held across opendir, every readdir round and the close), unlink, rmdir, rename, realpath and close. A rejected task does not poison the queue; close queues behind in-flight operations instead of tearing the channel down mid-continuation; writeShell drops the remainder when shellClosed flips while it waits for window credit. The gate is microtask-level, adds no perceptible latency, and has no configuration.
  • The frozen-terminal symptom was a second defect. When zmodem.js threw, consumeFromRemote discarded the session together with the parser’s unconsumed buffer - in the post-ZFIN case that buffer is exactly the shell prompt, so the terminal sat on a blank line until the next keystroke. The catch now replays _input_buffer (only for a receive session past _got_ZFIN; a send session’s leftover is peer protocol bytes) plus any chunks behind the one that threw.
  • Why E2E never caught it, and what guards it now. On localhost the channel window never tightens, libssh2 always consumes whole buffers, and the continuation loop runs once - the interleaving window does not exist. The invariant is therefore pinned by unit tests: never interleaves concurrent shell writes when libssh2 consumes partially, never interleaves an SFTP write with a concurrent SFTP request, and returns the parser leftover to the terminal when a receive session dies after ZFIN. The troubleshooting guide states explicitly that a green E2E is not evidence for this fix.
  • Verification. Frontend: the two webssh specs 46/46, full suite 30 files / 225 tests, npm run build plus scripts/verify-web-embed.sh matching. Browser E2E in a fresh workdir against rebuilt binaries: 15/15 PASS including the byte-for-byte 4 MiB sz, 1.5 MiB rz and 3 MiB SFTP transfers with a clean console. No Go code changed this round, so the same-day go test ./... -count=1 (23 packages ok) and go test -race ./internal/server ./internal/agent -count=1 -timeout 25m (427 s / 4.3 s) results stand. npx vue-tsc --noEmit remains a non-gate; the project-wide count of pre-existing errors is 194 and none of them sits on a line added this round.

A thirteenth round moved the ZMODEM transfer panel above the terminal and diagnosed a production report of a download whose progress froze at 291 KiB of 17.2 MiB with no error.

  • Panel placement. The progress bar, file name and cancel button rendered below the terminal, so a scrolling terminal pushed them out of view mid-transfer; the user guide has promised “above the terminal” since the feature landed, and the implementation had drifted. The fix is a pure template reorder in WebSSHTerminal.vue (the panel block now precedes the terminal element inside the same grid), guarded by a new source-order assertion keeps the transfer panel above the terminal in web/src/tests/zmodem-terminal.spec.ts. No logic, style or i18n changes; the E2E harness locates the panel by selector and is position-independent.
  • Frozen-progress diagnosis, no code change. A 17 MiB sz passes end-to-end on localhost (15/15 E2E with TM_E2E_DOWNLOAD_BYTES=17 MiB), and the relay’s inbound FIFO was re-audited for lost wakeups: the stale wake token is drained under the same mutex that guards the append, and the append always signals, so a blocked Read cannot miss bytes. The reported signature instead matches the round-12 torn-outbound defect on a bundle that predates it - a corrupted ZACK makes lrzsz pause mid-send with no error until its own timeout - and the screenshot’s session login (00:39) predates the round-12 bundle (built 00:56). The troubleshooting guide now lists frozen progress as a third manifestation of that defect and tells operators to confirm the deployed bundle contains the single-writer gate before treating a freeze as a link problem; if it recurs after redeploy, Server logs and the browser console are the next evidence.
  • Verification. Frontend 30 files / 226 tests, npm run build plus scripts/verify-web-embed.sh matching, browser E2E 15/15 PASS in a fresh workdir (/tmp/tm-e2e-r13) with a clean console. No Go code changed this round.

A fourteenth round fixed the production sequence “sz freezes, then the whole page shows SSH 通道已关闭”: the freeze and the channel close are one causal chain, and both halves are browser-side. Recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-zmodem-stall-channel-policy.md.

  • The freeze amplifier: per-subpacket progress events. The offer input handler emitted one progress event per ZDATA subpacket - tens of thousands of Vue reactive updates and el-progress repaints for a 17 MiB file. While the main thread churns, the ZACK writes that keep lrzsz sending are delayed, sz pauses mid-window, and the panel freezes at an arbitrary percentage. Progress is now reported on a 64 KiB byte step or a 500 ms quiet interval, with the first subpacket always reporting so tiny files leave 0 B immediately (an existing test caught the first version of this, which starved a 34-byte transfer of any event).
  • The channel close: the write-failure handler overreached. toPeer closed the terminal on any write rejection, including the round-11 stall guard’s “SSH shell write stalled after N of M bytes” - which is exactly what a paused sz produces once the outbound window tightens. The channel was never dead: the read loop would have reported the real end state later. The catch now checks whether a ZMODEM session is active; if so it ends the transfer locally and shows ZMODEM 传输失败:…, keeping the shell. Channel liveness stays owned by the read loop.
  • New bridge primitive abortLocal(). Ending a transfer because the transport failed must not push a ZABORT into the pipe that just proved undeliverable - that write would reject and recurse into the same handler. abortLocal() clears the session, suppresses the pending accept() rejection, emits aborted so the panel closes, and touches nothing outbound.
  • Verification. Targeted specs 72/72 including the new throttles progress events for bulk transfers (200 KiB produced ~200 events before the fix, under 10 after) and abortLocal ends the transfer without sending to the peer; full frontend suite 30 files / 229 tests; npm run build plus scripts/verify-web-embed.sh matching; browser E2E 15/15 PASS in /tmp/tm-e2e-r14 with a clean console. No Go code changed this round.

A fifteenth round fixed two further production reports against the round-14 bundle: a 153 MiB sz that ended before completing (“没传输完,又被关闭了”), and a terminal that stopped echoing after a transfer while the identical PROTOCOL: Only thing after ZFIN should be “OO” banner stacked up once per keystroke. The fix is browser-only and is recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-zmodem-sentry-lifecycle-stall-policy.md.

  • The vendored Sentry keeps the session that threw, so the terminal dies permanently. zsentry.js routes every input to this._zsession while it is non-null and clears it only from _after_session_end(), which is bound to the session’s own session_end event. zsession.js’s _consume_first() throws (a string, not an Error) when a post-ZFIN byte is not OO, so session_end never fires and the library exposes no reset. Round 12 correctly nulled the bridge’s own session, but the Sentry still owned the dead one: every later byte was re-fed and re-threw, producing one identical banner per keystroke and never reaching the terminal - exactly the stacked alerts and the frozen pane in the report. createZmodemBridge now builds its Sentry through a makeSentry() factory held in a let and recreates it on all three abandonment paths: the consumeFromRemote catch, abort() (whose current.abort() can throw before firing session_end) and abortLocal() (which deliberately sends nothing, so session_end can never fire). The vendored library is not patched; keeping the workaround inside the bridge is what makes a future zmodem.js upgrade reproducible.
  • A missing OO after ZFIN is teardown noise, not a failed transfer. ZFIN is only exchanged after ZEOF and OO only follows ZFIN, so by the time that throw is reachable every file byte is already spooled, completed has been reported and the browser has saved the file. Reporting it as error told operators that a finished 153 MiB download had broken and invited a retry of a file already on disk. The catch now classifies the failure using the very state the leftover replay already read (role === 'receive' && session._got_ZFIN), arms stripNextOverAndOut so a late OO is still eaten, replays the buffered prompt, and emits ended - after the replay, so the progress panel only closes once the prompt is on screen. Mid-transfer throws still report error unchanged. The round-12 test that asserted this error reached the UI was rewritten to assert the new contract.
  • The stall deadline killed large transfers, so protocol writes opt out of it. A bulk sz accumulates megabytes of ZACK backlog in the outbound direction while lrzsz pauses to retransmit; the round-11 writeStallLimit = 5000 then threw SSH shell write stalled after N of M bytes and the round-14 toPeer policy ended the transfer, which is the user’s “没传输完,又被关闭了”. TerminalChannel.write now takes options?: { stallLimitMs?: number } with 0 meaning no deadline, and only toPeer passes it. Keystrokes keep the 5 s bound on purpose: a keystroke that cannot be written in 5 s really does mean a dead channel, and the view relies on that rejection to close the terminal. A genuine tunnel failure still surfaces, because libssh2 returns a negative code, assertResult throws, and the round-14 “abort the transfer, not the channel” policy applies.
  • Removing the deadline forced close() out of the write gate. close() queued behind shellGate, so a single unbounded stalled writer would hang await channel?.close() in closeTerminal() forever and leak the libssh2 handles - a new failure mode that removing the deadline would otherwise have introduced. close() now calls closeShell() directly, and closeShell() flips shellClosed synchronously, which is what makes the in-flight writer unwind at its next check. To keep that safe, retryEAGAIN gained an optional isCancelled predicate and returns 0 (the “no progress” value the write loops already understand) instead of touching a handle the caller may already have freed; without it a retry attempt could land on a channel that _ssh2_channel_free had just released.
  • Verification. Red first: four new assertions failed before the implementation - timed out waiting for ended; got ["detected","receive-offer","progress","completed","error"] (twice, for the ended contract and the passthrough contract), expected Error: SSH shell write stalled after 0 of 3 bytes to be null, and the missing stallLimitMs: 0 source marker. Two new tests passed pre-fix and are kept as regression guards: parses shell output normally after abortLocal (the abandoned receive session happens to route invalid bytes back as garbage, so the old code did not error there either) and still reports a stalled shell write once the default deadline passes (proves the deadline removal is opt-in). Targeted specs 60/60; full frontend suite 30 files / 234 tests, up from 229; npm run build plus scripts/verify-web-embed.sh matching, with stallLimitMs present in the embedded WebSSHTerminal asset; browser E2E 15/15 PASS in /tmp/tm-e2e-r15, including transfer panel closes and the prompt returns with a clean console. No Go code changed this round, but the gates were re-run for completeness: go build ./..., go vet ./..., gofmt -l internal/ cmd/ test/ and git diff --check clean, go test ./... -count=1 with 0 failures across 18 tested packages (23 total). npx vue-tsc --noEmit remains a non-gate; the project-wide count of pre-existing errors is still 194 and none sits on a line added this round.

A sixteenth round fixed a production report of the terminal page collapsing into the “已断开 / SSH 通道已关闭” empty state in the middle of an sz download, together with the operator directive that a channel must not be declared dead before thirty seconds of evidence. The fix is browser-only and is recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-webssh-dead-channel-window.md.

  • The only sub-30s “channel is dead” decision in the whole stack was the browser’s 5 s write-stall guard. An audit of every time-based liveness decision found exactly one under thirty seconds: writeStallLimit = 5000, shared by the shell write loop and the SFTP write loop. On the Server side the only timer is server.webssh.idle_timeout, defaulted to 5 minutes; the 64 MiB receive-queue and WebSocket backpressure guards are capacity bounds, not timers. The tail of a bulk sz keeps the outbound window congested for longer than five seconds, so a keystroke written right after the transfer - queued behind a protocol write that is still waiting for window credit - hit the guard, threw SSH shell write stalled after N of M bytes, and the keystroke path’s catch closed the whole session. The guard is now 30_000 ms, with a comment stating that it is the dead-channel decision rather than a latency budget; a genuinely dead tunnel still surfaces earlier through the WebSocket close and the read loop, so thirty seconds is only the fallback bound.
  • A leftover protocol write could close a living session after the transfer ended. Round 15 lets toPeer writes wait for window credit without a deadline, so one can still be inside the write gate when the transfer reports ended or aborted. Its late rejection - a negative libssh2 result once the peer winds down - reached the toPeer catch’s inactive branch, which called closeTerminal(): a rejection with nothing to do with shell health killed the session and produced exactly the reported empty state (no transport alert, no remote-exit block). That branch now does nothing; channel liveness stays owned by the read loop and by the keystroke path, consistent with the round-14 principle that write failures must not adjudicate liveness.
  • Verification. Red first: three assertions failed before the implementation - expected Error: SSH shell write stalled after 0 of 3 bytes to be null at the 20 s mark, expected SFTPError: SFTP write stalled after 0 of 64 bytes to be null at the 20 s mark, and the source assertion that the toPeer slice must not contain closeTerminal(). The two stall tests were rewritten from the round-15 4 s/2 s shape to 20 s silent then 35-36 s failing, and the SFTP stall test’s fake-timer loop was extended past the new window. Targeted specs 60/60; full frontend suite 30 files / 234 tests (unchanged count: three tests rewritten, none added); npm run build plus scripts/verify-web-embed.sh matching; browser E2E 15/15 PASS in /tmp/tm-e2e-r16 with byte-for-byte sz, rz and SFTP transfers and a clean console. No Go code changed this round; gates re-run for completeness: go build, go vet, gofmt -l internal/ cmd/ test/, git diff --check clean and go test ./... -count=1 with 0 failures across 18 tested packages (23 total).

A seventeenth round made the embed sync impossible to forget and settled the recurring Nginx question. Every round since the admin UI shipped ended with the same release reminder - npm run build, then rsync into internal/server/web_dist/, then rebuild the Server binary - because a build that skipped the middle step silently shipped the previous bundle: the binary rebuilds, the embedded assets do not change, and every frontend fix looks unreleased.

  • npm run build now performs the embed sync itself. The script is vite build && node scripts/sync-web-dist.mjs; the new web/scripts/sync-web-dist.mjs mirrors web/dist/ into internal/server/web_dist/ with mirror semantics (clear then copy), because hashed chunk names change on every build and merged leftovers would be embedded forever. It is plain Node rmSync/cpSync rather than rsync so the step also works on Windows release hosts, takes optional source/target arguments for tests, and exits non-zero with a sync-web-dist: source build output missing message when vite build has not run - a half-written or empty tree must never reach the embed directory. Manual rsync -a --delete remains documented as a verification and recovery path only.
  • Nginx needs no frontend-route configuration in the default deployment. The SPA history fallback is served by the Server’s embedded static handler, so location / only needs proxy_pass; the /api/ and /ws/* locations exist for proxy precedence and WebSocket upgrade, not for frontend routing. try_files $uri $uri/ /index.html; and the application/wasm type addition are required only in the alternative “Nginx serves web/dist statically” mode. Both facts are now stated explicitly in docs/deployment/nginx.md and docs/deployment/frontend.md, which previously left operators guessing.
  • Verification. Red first: syncs the build output into the Go embed directory as part of build failed on the old "build": "vite build" script, mirrors the source tree into the embed directory and deletes stale files failed with the script missing (stale app-old.js survived a merge), and fails when the build output is missing failed until the guard message existed. Green: web/src/tests/build.spec.ts 8/8; full frontend suite 30 files / 237 tests; npm run build now prints sync-web-dist: mirrored … -> …/internal/server/web_dist and scripts/verify-web-embed.sh reports the trees match. No Go code changed; go build ./..., go vet ./... and the embedded-asset test were re-run clean.

Release Steps

  1. Back up SQLite or MySQL and verify restore.
  2. Deploy the new Server with storage.auto_init=true so schema v11 advances to v12 and v12 advances to v13.
  3. Keep server.webssh.enabled=false during rollout if a staged release is required.
  4. Verify /health/ready, schema_meta.version=13, the three WebSSH tables, and the four credentials.secret_* columns.
  5. Configure exact security.allowed_origins and the Nginx /ws/webssh/ location.
  6. Inject TUNNELMESH_TOKEN_ENCRYPTION_KEY (base64 or hex, 16/24/32 bytes) and TUNNELMESH_TOKEN_ENCRYPTION_KEY_ID identically on every Server node before credential auto-authentication is offered to users.
  7. Enable server.webssh, sign in to the admin console, and complete one SSH, one SFTP, one sz download and one rz upload smoke test.

Rollback Steps

  1. Disable server.webssh.enabled.
  2. Close active browser sessions and roll back the application to the previous release.
  3. Retain the v12 tables and the v13 credentials.secret_* columns; do not issue reverse DDL or manually change schema_meta.version. A v12 binary ignores the new columns, stored secrets simply become invisible, and browser SSH/SFTP falls back to the manual password dialog.
  4. If exact schema restoration is mandatory, restore the pre-upgrade database backup.
  5. Keep TUNNELMESH_TOKEN_ENCRYPTION_KEY unchanged across the rollback window; removing it makes stored secrets undecryptable even after upgrading again.

Reviewer Focus

  • Ticket authentication order, one-time consumption, and durable close behavior.
  • WebSocket origin checks and /ws/webssh/ precedence over SPA fallback.
  • Browser memory handling for passwords, private keys, and host-key confirmation.
  • SFTP chunking, 1 GiB preflight limit, progress reset, and shared-connection close behavior.
  • v11 to v12 migration parity between MySQL and SQLite.
  • Nginx timeout and buffering settings for long-lived browser sessions.
  • Owner scoping and cursor semantics of GET /api/v1/ssh-sessions, including that administrators list only their own sessions.
  • Exit status/signal are read while the libssh2 channel handle is still valid, and the optional-binding fallback returns null instead of a fabricated code.
  • Owner-only decryption in CredentialService.GetWithSecret, and the fail-open path that returns no auth instead of an error.
  • Ordering of a.mutate and the auth attachment in createWebSSHSession; the stored idempotency body must never contain plaintext.
  • Shape validation in validateCredentialShape: a password credential rejects a private key, an ssh_public_key credential rejects a password, and a passphrase requires a private key.
  • Type change clearing the stored secret, and the 503 fast failure when no encryption key is configured.
  • v12 to v13 migration parity between MySQL and SQLite, including the corrected SQLite schema_meta seeding in the migration tests.
  • ZMODEM lifecycle in createZmodemBridge: single finish, abortedByUser suppressing spurious errors, keystroke gating, and dispose() on close and unmount.
  • Relay window symmetry in agent_relay_transport.go: send credit consumed before every DATA frame, receive credit released only after Read hands bytes to the consumer, and finish waking a blocked Write.
  • The byte-budgeted inbound FIFO: budget equals the advertised window, Read drains the wake token while still holding the stream lock so no wakeup can be lost, and readBuf is compacted rather than resliced.
  • FIFO serialization of browser WebSocket writes; a waiting writer must never be overtaken by a later one.
  • ShellExitInfo.transportError propagation, and that exitReason reports nothing when the transport died.
  • sessionStorage target memory holds only the remote-server ID; verify no credential material can reach it on any path.
  • Nested-module isolation of test/e2e/webssh/sshhost: confirm go build ./... and go vet ./... at the root still ignore it and that the root go.mod gained no SSH server dependency.
  • WINDOW_UPDATE handling in handleAgentFrameGeneration: credit must reach sendState before the mirror attempt, and a full mirror queue must neither fail the stream nor lose credit.
  • The SFTP and shell write loops: remainder offsets, the stall guard bound, and that a negative result still goes through assertSFTPResult/assertResult unchanged.
  • That uploadFile’s short-write assertion remains as a defensive check now that the handle write guarantees full consumption.
  • createWriteGate semantics: a rejected task must leave the queue usable, the gate must span multi-call operations (the SFTP handle write loop and the whole readDirectory sequence), and no gated task may call another gated function (deadlock check).
  • Fire-and-forget callers (toPeer in the terminal view, keystroke onData) get strict ordering without awaiting; verify no caller depends on write resolving before the bytes are queued.
  • writeShell dropping the remainder on shellClosed, and shell/SFTP close queueing behind in-flight operations instead of preempting them.
  • The zmodem leftover replay: receive role and _got_ZFIN only, and the tail-chunk arithmetic (offset + ZMODEM_CHUNK_BYTES) for chunks behind the one that threw.
  • DOM order in WebSSHTerminal.vue: the transfer panel must stay before ref="terminalElement" so progress and cancel remain visible while the terminal scrolls; the grid must not let it overlap the transport alert.
  • The toPeer failure policy: transfer-active means abortLocal() plus an error message and a living shell; transfer-inactive keeps the old closeTerminal(); verify no path closes the channel from a write rejection while a session is active.
  • Progress throttling thresholds (64 KiB / 500 ms) and the unconditional first report; abortLocal must never call toPeer, and must suppress the pending accept() rejection exactly once.
  • Sentry lifecycle in createZmodemBridge: all three abandonment paths (the consumeFromRemote catch, abort(), abortLocal()) must recreate the Sentry, and suppressed/suppressNextTerminalWrite must be reset together with the session so a fresh Sentry cannot silently swallow the next terminal write.
  • The finishedBeforeThrow classification reads _got_ZFIN off a receive session only. Verify that a send-session protocol failure still reports error, that stripNextOverAndOut is armed before the replay, and that ended is emitted after the leftover reaches the terminal.
  • stallLimitMs is opt-in and defaults to writeStallLimit. Confirm no caller other than toPeer passes 0, and that the keystroke path still depends on the rejection to close the terminal.
  • close() bypassing the write gate: closeShell() must remain synchronous up to shellClosed = true, and retryEAGAIN’s isCancelled must be evaluated before every operation() call so no write can touch a freed channel handle.

  • The dead-channel window: writeStallLimit is the policy constant for both the shell and SFTP write loops; confirm nothing else re-introduces a shorter timer and that the keystroke path still closes the terminal once the window elapses.
  • The toPeer catch must have exactly two outcomes: transfer-active aborts the transfer with a banner, transfer-inactive does nothing. Verify no path closes the session from a protocol-write rejection, and that a truly dead tunnel is still closed by the read loop / WebSocket close.

  • The embed sync in npm run build: confirm CI and the Dockerfile multi-stage build still produce a matching internal/server/web_dist (the Dockerfile performs its own sync and must not double-delete a mounted volume), and that scripts/verify-web-embed.sh stays in the release checklist as the independent check.

Integration Status

The feature is wired into the Server runtime, management API, embedded Vue bundle, and Agent relay path. Generated web/dist and internal/server/web_dist are ignored and are not part of the source diff.