Admin Remote Servers, WebSSH, and WebSFTP
Admin Remote Servers, WebSSH, and WebSFTP
Title
feat(admin): add remote servers and browser SSH/SFTP
Target Branch
main
Summary
This change adds admin-managed remote SSH servers, SSH public-key credentials, browser-local SSH terminals, and browser-local SFTP file operations. The Server creates a one-time WebSocket ticket and brokers encrypted bytes through the existing Agent relay. It does not terminate SSH, store SSH passwords or private keys, inspect terminal output, or expose a Server-side file API.
The browser libssh2 transport now registers its custom send/receive callbacks before handshake, maps an empty WebSocket receive queue to Emscripten/WASI EAGAIN (-6) instead of the public libssh2 error code (-37) or Linux EAGAIN (-11), and decodes WASM heap pointers into readable SSH/SFTP error text. SFTP directory parsing also compares POSIX file-type bits exactly so regular files are no longer misclassified as symlinks.
The WebSocket byte stream now waits for onopen before delivering the client SSH banner. This removes a startup race where libssh2 sent its banner while the browser socket was still CONNECTING, causing the transport to fail and libssh2 to report Failed getting banner.
@verdigris/libssh2.js@0.1.9 declares ssh2_session_hostkey in its TypeScript definitions but does not export the matching WASM function. The browser client now extracts the server public-key blob from the pre-encryption KEX reply and falls back to that key when the binding is unavailable, preserving interactive host-key verification without weakening SSH security.
The package also declares string arguments for functions whose actual exports use raw C pointers and lengths. The client now allocates NUL-terminated UTF-8 strings for password authentication, public-key authentication, PTY, disconnect, and SFTP paths; public-key authentication uses the actual seven-argument ABI. The host-key confirmation dialog renders the fingerprint as a wrapped standalone code block instead of embedding a long value in a normal sentence.
The ninth round adds lrzsz (ZMODEM) transfers inside the browser terminal and stored credential secrets that let WebSSH/SFTP authenticate without a dialog. Credentials now come in two types, ssh_public_key (optionally storing the private key and passphrase) and password; secrets are sealed with the existing AES-256-GCM auth.SecretStore and delivered to the owner’s browser exactly once at session creation. Schema expands from v12 to v13 with four nullable credentials columns.
User Impact
- Administrators and authorized users can maintain remote SSH server records and SSH public-key credentials.
- Remote-server lists show joined Agent and credential display names plus Agent online state.
- SSH passwords are entered per session and retained only in browser memory.
- WebSSH verifies the SSH host-key fingerprint interactively before continuing.
- WebSFTP supports directory browsing, upload, download, rename, delete, and progress display over the same authenticated SSH connection.
- Single-file browser transfers are capped at 1 GiB before transfer begins.
- The 远程服务器 page lists the caller’s own active SSH sessions and can disconnect one, which releases the active-session quota immediately instead of waiting for
session_ttl. - A terminal whose remote shell exited prints the remote exit code or terminating signal and offers 重新连接 back to the matching server’s authentication dialog.
- Empty lists render the normal Element Plus empty illustration on every route instead of an unstyled oversized graphic.
- 密钥管理 offers two credential types; an
ssh_public_keycredential can additionally store its private key, and apasswordcredential stores only a password. Both are encrypted at rest and reported as 已保存 / 未保存. - A remote server bound to a credential that holds a secret is tagged 自动认证, and clicking SSH 连接 or SFTP 文件 enters directly without the password dialog. Without a usable secret the manual dialog still appears, so nothing regresses for existing rows.
- The terminal supports lrzsz:
sz <file>downloads to the browser andrzopens a multi-file picker and uploads to the remote working directory. A transfer panel shows direction, name, progress and 取消传输, and keystrokes are not forwarded to the remote while a transfer runs.
API / Schema / Configuration Impact
- API adds credential CRUD and one-time private-key extraction, remote-server CRUD, and WebSSH ticket/session operations under
/api/v1. - API adds
GET /api/v1/ssh-sessions, a cursor-paged list of the caller’s own active WebSSH sessions; owner scope is enforced in the service and administrators also see only their own sessions. - Schema expands from v11 to v12 with
credentials,remote_servers, andwebssh_sessions; no existing table is modified. - Schema expands from v12 to v13 with four nullable
credentialscolumns (secret_ciphertext,secret_nonce,secret_key_id,secret_version); no existing column or index is modified and no row is rewritten. - API extends credential create/replace/patch with a write-only
secretobject, addshasSecretto credential reads, adds nullablecredentialType/credentialHasSecretto remote-server reads, and adds a one-timeauthobject to the WebSSH ticket response. - Configuration gains no new key: credential secrets reuse
TUNNELMESH_TOKEN_ENCRYPTION_KEY/TUNNELMESH_TOKEN_ENCRYPTION_KEY_ID. Without them, storing a secret fails with503and auto-authentication degrades to the manual dialog. ZMODEM needs no server-side switch. - Configuration adds
server.websshcontrols for enablement, ticket/session TTL, active-session limits, open/idle timeouts, and maximum WebSocket message size. - Nginx documentation adds a long-lived, unbuffered
/ws/webssh/location.
Security and Authorization Impact
- Remote-server ownership and Agent policy are enforced by the service layer; handlers do not trust client-supplied ownership.
- WebSSH tickets are random, hash-stored, single-use, and time-limited.
- Browser origin validation occurs before WebSocket upgrade and ticket consumption.
- Passwords and private keys used for a browser SSH connection remain in browser memory and are never sent to the management API.
- Private-key extraction is an explicit one-time HTTPS operation: the Server parses the key in memory and returns only the public key and fingerprint; private-key material is not persisted or logged.
- Audit records contain resource IDs and result classes, not credentials, terminal bytes, or SFTP file content.
- Runtime shutdown now terminates every process-local WebSSH bridge and its Agent stream.
- Credential secrets are decrypted only for
credential.OwnerUserID; administrators can list a credential but cannot read another user’s secret. - The decrypted secret is attached to the session-creation response after the idempotency record is written, so a replay returns the ticket without any auth material. The response carries
Cache-Control: no-store, and thecredential.revealedaudit event records identifiers only. - Secret ciphertext, nonce, key id and plaintext are absent from list/detail responses, audit details, logs, Prometheus, traceroute and error messages.
- ZMODEM bytes are brokered like every other WebSSH byte: the Server never parses the protocol, buffers a file, or records a file name.
Test Evidence
The following checks passed during implementation:
go test ./internal/server -run 'TestWebSSH' -count=1
go test ./... -count=1
go test ./internal/storage -run 'TestMySQLV11ToV12|TestSQLiteV11ToV12' -count=1
cd web && npm test -- --run
cd web && npm run build
./scripts/verify-web-embed.sh
The frontend suite reported 25 files and 138 tests passing. The Go suite passed all packages. The E2E test verified one-time ticket consumption, browser-to-TCP bytes, TCP-to-browser bytes, and close of both streams. A separate red-green test verified runtime shutdown terminates active WebSSH bridges.
The v11-to-v12 schema regression test verifies that indexed WebSSH timestamp columns use VARCHAR(32) in both the full DDL and the MySQL migration, preventing MySQL Error 1170 when creating the new indexes. The MySQL migration also repairs tables partially created with TEXT columns by a failed earlier attempt, then tolerates already-created duplicate indexes on retry.
Final mandatory verification also passed:
go test ./... -count=1
go test -race ./...
go vet ./...
go build ./cmd/...
git diff --check
cd web && npm test -- --run && npm run build
rsync -a --delete web/dist/ internal/server/web_dist/
./scripts/verify-web-embed.sh
go test -race ./... passed all packages; the slowest package was internal/server at 249.906 seconds. The final frontend run again reported 133 tests passing and a successful Vite build. git diff --check, go vet, go build, and the embed comparison produced no errors. Docker image construction was not executed in this development environment because no docker, podman, buildah, or nerdctl executable is installed.
A later red-green regression fix for the browser handshake added seven WebSSH tests covering callback registration, WASI EAGAIN, heap-pointer error decoding, KEX host-key fallback, SFTP pointer errors, directory-name decoding, and POSIX type bits. A separate byte-stream regression test verifies that the SSH banner waits for the WebSocket to open and fails clearly if the socket closes first. Real Chrome WASM smoke tests against a local OpenSSH server completed the SSH handshake (WEBSSH_WASM_HANDSHAKE_OK) and extracted the host key successfully (HOSTKEY_OK:sha256:...). Fresh verification passed:
cd web && npm test -- --run
cd web && npm run build
rsync -a --delete web/dist/ internal/server/web_dist/
./scripts/verify-web-embed.sh
go test ./internal/server -run 'TestWebSSH' -count=1
go vet ./...
git diff --check
The frontend suite reported 25 files and 141 tests passing, and the WebSSH-focused Go package test passed.
The 2026-09-12 password-authentication deployment regression was fixed with red-green tests for the real WASM ABI and host-key dialog. A real Chrome test against a local SSH server returned PASSWORD_AUTH_OK after interactive-equivalent host-key verification. Final verification passed:
cd web && npm test -- --run
cd web && npm run build
rsync -a --delete web/dist/ internal/server/web_dist/
./scripts/verify-web-embed.sh
go test ./... -count=1
go test -race ./...
go vet ./...
go build ./cmd/...
go build -o ./tunnelmesh-server ./cmd/tunnelmesh-server
git diff --check
The frontend suite reported 25 files and 150 tests passing. The race suite passed all packages, with the slowest package internal/server completing in 305.886 seconds. The final tunnelmesh-server SHA-256 is 18b696f72485ac7e57a4b755a38f9555e768a2682b9726b8028cf4ced2b6863d.
A second deployment regression round fixed three production issues reported from the browser console and UI:
- The embedded static server now pins
Content-Type: application/wasmfor.wasmassets. Host mime databases without a.wasmentry previously madehttp.FileServersniffapplication/octet-stream, so browsers rejected streaming compilation and loggedIncorrect response MIME typebefore falling back to ArrayBuffer instantiation. main.tsnow imports the Element Plusmessageandmessage-boxstyles explicitly. On-demand style imports only cover components referenced from templates, so programmaticElMessageBox/ElMessagerendered without panel or overlay styles in production builds.- The WebSSH terminal is now created only after the session reaches
connected, so xterm measures real cell sizes and receives keyboard focus via an explicitterminal.focus(). The shell channel also exposesonExit, and the view reports remote shell termination with a localized notice instead of stayingconnectedwith a dead channel.
Red-green tests cover the WASM content type, the style imports, shell exit notification, and the view ordering/focus markers. A real Chrome smoke test against a local SSH server verified password authentication and the input path (PASSWORD_AUTH_OK, INPUT_ECHO_OK).
A third round fixed three admin-console defects reported from production use:
- SSH 会话创建失败 had no actionable reason. The connect dialog replaced every rejection with the generic
remoteServers.sshFailedstring, so administrators could not tell a disabled server from an offline agent or an exhausted session quota.writeAPIErrorputs the stable identifier indata.errorwhile the envelopemsgis only the HTTP status text, so the newweb/src/views/remote-server-errors.tsmapsAPIError.domain(remote server is unavailable,remote server agent is unavailable,remote server credential is unavailable,remote server agent is offline,webssh active session limit reached,webssh username is required,resource is forbidden) to localizedremoteServers.errors.*messages, keeps an unmapped identifier readable for troubleshooting, and falls back to the generic string for non-API failures.RemoteServers.vuecallssshErrorMessage(error, t). - Cross-owner WebSSH requests answered 500 instead of 403.
writeWebSSHErrorsentErrResourceForbiddenthroughwriteStorageError, soPOST /api/v1/remote-servers/{id}/ssh-sessions,GETandDELETE /api/v1/ssh-sessions/{id}returned500with the internal textresource is forbidden, unlike the remote-server and credential APIs which already answer403.writeWebSSHErrornow mapsErrResourceForbiddenandauth.ErrForbiddento403with a non-leakingforbiddenbody. - The active-session limit answered 500 instead of the documented 409.
docs/api/openapi.yamlalready promises409: Idempotency conflict or active-session limit reached, but the inline error fell throughwriteStorageError. The limit is now the sentinelErrWebSSHActiveSessionLimit, whichwriteWebSSHErrormaps to409while keeping the stable identifier indata.errorso the console can localize it. No OpenAPI change was required because the implementation now matches the published contract. - Quota exhaustion after a Server restart.
webssh_sessionsrows left inactiveby a previous process could never be served again, but the sweeper only reclaimed them after the 8-hoursession_ttl, somax_active_sessions_per_user=5filled with dead rows and every new session failed.WebSSHSessionRepository.CloseActiveByNodeandWebSSHSessionService.CloseStaleLocalSessionsnow close those rows once during runtime startup, before any ticket is issued, and logwebssh_stale_sessions_closedwith the affected count. The startup hook is guarded byserver.webssh.enabledand aborts startup if the cleanup fails. - 远程服务器 / 服务令牌 / Server 节点 tables forced a page-level horizontal scrollbar.
.tm-pageuseddisplay: gridwith an implicitautocolumn, so the track grew to the table’s max-content width instead of the viewport. It is nowgrid-template-columns: minmax(0, 1fr), which keeps wide tables scrolling inside their own card while thefixed="right"action column stays reachable.
The administrator menu entry 下载 was also renamed to 发行管理 (Releases in English) and moved to the last position, after 审计日志. Documentation was updated in docs/user-guide/server-admin.md (new 发行管理 section, SSH failure reasons, node-restart session cleanup), docs/README.md, and docs/deployment/binary-release.md.
Round-three verification:
cd web && npx vitest run src/tests/shell.spec.ts src/tests/design-system.spec.ts src/tests/remote-servers-view.spec.ts
cd web && npm test -- --run
cd web && npm run build
rsync -a --delete web/dist/ internal/server/web_dist/
./scripts/verify-web-embed.sh
go test ./internal/storage -run TestWebSSHSessionCloseActiveByNode -count=1
go test ./internal/server -run 'TestWebSSHSessionServiceClosesStale|TestWebSSH' -count=1
go test ./... -count=1
go test -race ./...
go vet ./...
go build ./cmd/...
go build -o ./tunnelmesh-server ./cmd/tunnelmesh-server
git diff --check
Red tests were confirmed failing before implementation: TestWebSSHSessionCloseActiveByNode, TestWebSSHSessionServiceClosesStaleLocalSessionsAfterRestart, TestWebSSHAPIForbiddenReturns403 (create status = 500, want 403: {"code":500,"msg":"Internal Server Error","data":{"error":"resource is forbidden"}}), TestWebSSHAPIActiveSessionLimitReturnsConflict (second create status = 500, want 409), the remote-server-errors.spec.ts domain-mapping suite, the shell.spec.ts menu-order and rename assertions, and the design-system.spec.ts .tm-page assertion.
After implementation the frontend suite reported 26 files and 164 tests passing, go test ./... -count=1 passed every package, and go test -race ./... passed with internal/server completing in 340.095 seconds. go vet ./... and git diff --check are clean, and npx vue-tsc --noEmit stays at the pre-existing 172-error baseline with no new errors in the touched files. scripts/verify-web-embed.sh reported web/dist and internal/server/web_dist match, and the rebuilt binary embeds the new bundle. The final tunnelmesh-server SHA-256 is 70bc83223f589b6b38df4ef8355a1bbbeed34b263322cc7d44b58e71f758649c.
Pre-existing flake, now fixed (test-only): two go test ./... -count=1 runs reported TestServeClientSessionResetsStreamAfterShortRelayWrite: RESET count = 0, want 1. internal/server/ws_client.go is untouched by this branch. The RESET is enqueued from the per-stream relay pump goroutine (ws_client.go:530) while ServeClientSession returns on EOF and drains the fair writer in a defer, so the assertion depended on a goroutine ordering the implementation does not promise. scriptedClientTransport.sent is now guarded by a mutex with a sentFrames() snapshot accessor (the unsynchronized slice was also a latent data race between the writer goroutine and the test), and the assertion uses a new bounded waitForResetCount poll. Verified with 200 repetitions plus two full-package runs; production behavior is unchanged.
A fourth round closed the remaining horizontal-overflow reports and one embedding defect found while verifying in a real browser:
- 服务令牌 still overflowed because it never used the shared page grid.
Tokens.vuewas the only console view whose root element lackedtm-page; its scoped.tokens-page { display: grid; gap: 16px; }duplicated the old unbounded rule. The view now roots intm-page tokens-page, the duplicate rule is deleted, and the same latent pattern was hardened inDownloads.vue(.release),WebSFTP.vue(.file-card,.transfer-panel),WebSSHTerminal.vue(.terminal-card) andTokens.vue(.scope-list) by giving every single-column grid an explicitgrid-template-columns: minmax(0, 1fr). - A new guard test makes the whole class unregressable.
web/src/tests/layout-overflow.spec.tsasserts that every console view roots intm-page, that no view/component/layout style block declaresdisplay: gridwithout explicit columns, thattokens.csskeeps the bounded page grid, and that anyel-tablewhose column widths exceed ~1270px pins its action column withfixed="right". It failed first onTokens.vueandDownloads.vue .release, then passed. //go:embed web_distsilently dropped_-prefixed assets. Go embed excludes files starting with_or., and Vite emits lodash helper chunks such as_baseClone-*.js. The missing chunk made the browser fall back toindex.html(200 +text/html), the ES module load failed, and every route importing it rendered a blank shell.internal/server/web.gonow uses//go:embed all:web_dist, andTestEmbeddedWebDistContainsEverySourceFilediffs the embedded FS against the source directory so any future omission fails the build. Red output before the fix:embedded web_dist is missing 1 source file(s): [assets/_baseClone-CZsh75pW.js].- Layout verified in a real browser, not just by CSS assertions. Headless Chrome at the reported 1521x731 viewport, authenticated against a local SQLite server with 12 seeded tokens, measured all 12 console routes:
documentElement.scrollWidth == clientWidtheverywhere (no page-level horizontal scrollbar),/tokensrendered 12 rows with the table scrolling inside its card (scrollWidth=1410vsclientWidth=1231) and the action column pinned right. A screenshot confirms the rendered result.
A fifth round fixed the quota lockout reported as “活跃 SSH 会话数已达上限,请先断开闲置会话后重试”:
- Completed or dead WebSSH sessions never released their quota.
WebSSHBroker.Handleonly wroteclosedin theagent_unreachablebranch; when the bridge ended normally — browser tab closed, remote sshd exited (including host-side access control dropping the session seconds after the MOTD) — the DB row stayedactiveuntil the 8hsession_ttlsweeper. Withmax_active_sessions_per_user=5, five such connections locked the user out, and no API or UI lists or closes own sessions. The owning node now durably closes the session whenever the bridge returns, with aclose_reasonderived from which side ended first:remote_closed,client_disconnected,server_closed, orbridge_closed. Red before the fix:session status = active, want closed after the remote stream endedand the browser-disconnect twin. - Two new broker tests cover both endings and pin the close reasons; the existing e2e and idle-deadline tests still pass. Documentation in
docs/user-guide/server-admin.mdnow explains quota release, the close reasons, and that a restart reaps legacy rows viawebssh_stale_sessions_closed.
A sixth round implemented the approved plan docs/superpowers/plans/2026-09-12-webssh-session-management.md (T1-T7), giving users a self-service remedy for an exhausted quota:
- Rows left
activeby an older build still had no user-visible escape. Round five stopped new leaks, but legacy rows — or a browser that dies right after consuming its ticket — occupied the quota untilsession_ttl, and nothing listed a caller’s own sessions.WebSSHSessionRepository.ListActivePagenow pages one owner’s active sessions by ID using the shared cursor contract (pageArgs,decodeCursor,ORDER BY id LIMIT limit+1), mirroringcredentialRepo.List. Rows whoseexpires_athas passed are excluded because the sweeper reclaims them without a bridge, so offering them would disconnect nothing. Red first:db.WebSSHSessions().ListActivePage undefined (type WebSSHSessionRepository has no field or method ListActivePage). GET /api/v1/ssh-sessionsis owner-scoped in the service, not in the handler.WebSSHSessionService.ListforcesownerUserID = actor.UserIDand never reads an owner from the request; administrators also list only their own sessions, because cross-user session management needs its own authorization and audit model. The route answers 405 for non-GET on the collection. Red first:post status = 404, want 405.docs/api/openapi.yamlgains the path plusWebSSHSessionPage/WebSSHSessionPageEnvelope.- The console can now release the quota itself. 远程服务器 gains an 活跃会话 card (target server, created, expiry, confirmed 断开) that calls the existing
DELETE /api/v1/ssh-sessions/{id}; no new close path was added. Session rows carry IDs only, so names resolve from the loaded server list first and per-ID second, and an unreachable or deleted server falls back to the raw ID with an 未知服务器 tag. The server list is awaited before sessions resolve, which removes a redundant per-session fetch on first paint. - A closed terminal now explains itself.
TerminalChannel.onExitdelivers{ status, signal }read fromssh2_channel_get_exit_status/ssh2_channel_get_exit_signalbefore the channel handle is freed, reusing the pointer-or-string decoding from the session error path and degrading tonullwhen a build lacks the bindings rather than inventing an exit code. The terminal prints the reason and offers 重新连接, which routes to/remote-servers?ssh=<serverId>; the view opens that server’s auth dialog and clears the query. Only the non-sensitive server ID survives a close, so credentials are still dropped. - Verified in a real browser, not only by source-marker specs. Headless Chrome at 1521x731 against a local SQLite Server with two seeded active sessions (one pointing at a nonexistent server): the card rendered both rows, resolved 验证服务器 from the loaded list, tagged the unknown target, issued no redundant fetch for the known server, and kept
documentElement.scrollWidth == clientWidth(sessions table 1263/1263; the servers table still scrolls inside its own card at 1390/1263). Clicking 断开 and confirming produced200 DELETE /api/v1/ssh-sessions/webssh-live-1, a 会话已断开 toast, a refreshed single-row list, andstatus=closed, close_reason=user_closedin the database./remote-servers?ssh=<id>opened 连接 SSH with the default username prefilled and an empty query string afterwards. No console exception was recorded in any step. Restarting the Server loggedwebssh_stale_sessions_closed count=2 node_id=local, confirming the round-five startup reaper on a live process. - Suite state after the round:
go test ./... -count=1andgo vet ./...clean,go test -race ./...clean,git diff --checkclean, frontend 27 files / 177 tests passing,npm run buildplusscripts/verify-web-embed.shreportingweb/dist and internal/server/web_dist match.npx vue-tsc --noEmitreports 179 errors project-wide; it is not part of the build or test gate and the codebase already carries this class of error. This round adds exactly four: threeDefaultRowtable-slot errors inRemoteServers.vue(lines 23, 24, 37), the same Element Plus slot-typing pattern already present in that file and inWebSFTP.vue, and one untyped-mock error inssh-client.spec.ts:566matching the pre-existing_ssh2_sftp_*assertions in the same spec. - Suite state after the round:
go test ./... -count=1andgo vet ./...clean,go test -race ./...clean,git diff --checkclean, frontend 27 files / 177 tests passing,npm run buildplusscripts/verify-web-embed.shreportingweb/dist and internal/server/web_dist match.npx vue-tsc --noEmitreports 179 errors project-wide; it is not part of the build or test gate and the codebase already carries this class of error. This round adds exactly four: threeDefaultRowtable-slot errors inRemoteServers.vue(lines 23, 24, 37), the same Element Plus slot-typing pattern already present in that file and inWebSFTP.vue, and one untyped-mock error inssh-client.spec.ts:566matching the pre-existing_ssh2_sftp_*assertions in the same spec.
A seventh round fixed the empty-state rendering defect reported as a page-sized black shape on every empty list:
- Root cause: style entries are injected only for template-resolved components.
unplugin-vue-componentswithElementPlusResolverimports a component’s CSS when it resolves the tag from a template.DataState.vueimportsElEmptyandElSkeletonexplicitly in<script setup>, andStatusTag.vueimportsElTag, so the resolver never sees them.el-empty.cssentered the bundle only through lazy route chunks that use<el-empty>in a template (WebSSHTerminal, AgentDetail, Dashboard); on every other route the empty state rendered without.el-empty__imagesizing and without the--el-empty-fill-color-*variables. Measured in a real browser before the fix: image box1263x1375and first SVG path fillrgb(0, 0, 0)— the giant black shape in the report. The same gap left loading skeletons and tags unstyled on routes whose chunk never loads those styles. - The fix follows the convention already used by
AppShell.vue(menu),PageHeader.vue(breadcrumb) andmain.ts(message/message-box): import the style entry next to the explicit component import.DataState.vuenow imports the alert, button, empty and skeleton style entries;StatusTag.vueimports the tag style entry, making both components self-sufficient on every route. - Guard test.
web/src/tests/build.spec.tsgainsloads styles for components imported explicitly in script setup, asserting those five style imports. It failed first on the missingalertimport, then passed. - Verified in a real browser against an empty database.
/credentials,/tokens,/audit-logs,/remote-serversand/agentsnow render the bounded illustration: image box160x174(was1263x1375), first path fillrgb(220, 223, 230)(was black), empty block 294px tall (was 1418px), and no page-level horizontal scroll on any of them. -
Suite state after the round: frontend 27 files / 178 tests passing,
npm run buildplusscripts/verify-web-embed.shmatching, andgo test ./... -count=1,go vet ./...,git diff --checkclean. No Go behavior changed, so the previous-raceresult still applies. An eighth round fixed the report “ssh可用,切到sftp不能用 / 断开连接后应回到主机列表 / 活跃会话列表放到页面下方”. Unlike earlier rounds this one could only be diagnosed against a live SSH target, so a throwaway end-to-end harness (outside the repository, under/tmp/tm-e2e) was built: a local SSH host implemented withgolang.org/x/crypto/sshplusgithub.com/pkg/sftp(read-only, loopback, passwordtmuser/tmpass), the real Server and Agent binaries, and headless Chrome driven over CDP. It reproduced the failure exactly and then proved the fix: six checks pass, including a 23-entry directory listing over the tunnel, the terminal showing the remote prompt after returning from SFTP on the same connection, no second host-key dialog, and disconnect landing back on/remote-servers. - SFTP died on the first non-blocking stall. libssh2 reports “would block” for handle-returning calls (
_ssh2_sftp_init,_ssh2_sftp_open,_ssh2_sftp_opendir) with a NULL handle plusLIBSSH2_ERROR_EAGAINin the session errno; the return value stays 0. The existingretryEAGAIN/retrySFTPloops only recognise the int code-37, so SFTP startup aborted immediately withSFTP initialization failed: Would block starting up channeleven though the tunnel was healthy. A newretryHandleEAGAINretries while the session errno is EAGAIN and the handle is NULL. Red first:promise rejected "Error: SFTP initialization failed: libssh…" instead of resolving. - A successful realpath was reported as an error.
libssh2_sftp_realpathreturns the resolved string length on success, but the code asserted “non-zero means failure”, so/tmp/tm-e2e(length 11) becameSFTP real path failedand the file manager stayed empty. Only negative results are failures now. Red first: the newreads the resolved path when realpath returns its lengthtest rejecting. - Failures were invisible.
WebSFTP.vuesetstatus='error'but notloadError, so a failed connect rendered the disabled toolbar plus “目录为空” — indistinguishable from an empty directory.DataStatenow owns the error surface (:error="loadError", retry re-runs the handshake when the view never connected), andopenSFTPincludes libssh2’s errno/message instead of the bare “SFTP initialization failed”. Red first:expected [Function] to throw error including 'SFTP initialization failed: subsystem…' but got 'SFTP initialization failed'. - The direct SFTP entry never authenticated. Opening SFTP straight from the server list only creates a pending ticket;
WebSFTP.vuereusedstore.activeSessionand threwmissingSession. It now bootstraps the SSH connection itself (same flow as the terminal) and publishes it to the store, so the terminal opened afterwards reuses it. The host-key confirmation moved to a sharedweb/src/webssh/host-key.ts(useHostKeyVerifier) and the WebSocket URL builder towebsshWebSocketURLinbyte-stream.ts, so both entry points present identical prompts and URLs. - Returning from SFTP to the terminal demanded a new one-time ticket.
WebSSHTerminal.vuenow opens another shell channel onstore.activeSessionwhen the session IDs match; the store keeps the non-sensitiveremoteServerIdon the active session so the reconnect button still works after a remount. - 断开连接 now leaves the page. Both views expose
disconnectAndLeave, which closes the session and routes to/remote-servers; staying on a dead session only invites a reload that can never succeed because the ticket is consumed. - 活跃会话 moved below the server table on
/remote-servers: the list is the primary task, the session card is a recovery surface. A view test asserts the card order so it cannot drift back. - Suite state after the round: frontend 27 files / 191 tests passing,
npm run buildplusscripts/verify-web-embed.shmatching,go test ./... -count=1,go test -race ./...,go vet ./...andgit diff --checkclean.npx vue-tsc --noEmitstill reports the pre-existing 177 errors and adds none.
A ninth round implemented the approved plan docs/superpowers/plans/2026-09-12-webssh-zmodem-credential-auto-auth.md (T1-T8): lrzsz/ZMODEM transfers in the browser terminal, and stored credential secrets that let WebSSH/SFTP authenticate without a dialog.
- Schema v12 to v13 is expand-only.
credentialsgains four nullable columns (secret_ciphertext,secret_nonce,secret_key_id,secret_version) inmigrations/ddl.sql,migrations/incremental/v0012_to_v0013/{mysql,sqlite}.sql,SchemaVersion=13anddocs/operations/schema-upgrades.md. Two latent migration-test bugs surfaced while writing the v13 red test and were fixed: the SQLite migration tests mustINSERT INTO schema_meta(id,version)because anUPDATEmatched zero rows and silently skipped the migration, andscanCredentialreadssecret_versionthroughsql.NullInt64so aNULLno longer errors. - One sealed blob, not nine columns. A credential secret is stored as a single AES-256-GCM JSON document (
{"password":…}or{"privateKey":…,"passphrase":…}) through the existingauth.SecretStore, so adding a future secret type needs no DDL. Limits are enforced at the boundary (password 4 KiB, private key 64 KiB, passphrase 1 KiB) and a type change clears the stored secret. WithoutTUNNELMESH_TOKEN_ENCRYPTION_KEYthe service returnsErrCredentialSecretUnavailableand the API answers503; there is no silent plaintext fallback. Reads only ever exposehasSecret. - Decryption is owner-scoped, not admin-scoped.
CredentialService.GetWithSecretrefuses every caller exceptcredential.OwnerUserID, so an administrator cannot read another user’s SSH password or private key even though they can list the credential. - The plaintext never reaches the idempotency store.
createWebSSHSessionbuilds the ticket insidea.mutateand attachesWebSSHAuthPayloadafterwards, so the persisted response body holds no secret and a replay with the sameIdempotency-Keyreturns the ticket withoutauth. The one-time response is sent withCache-Control: no-storeand audited ascredential.revealedwith identifiers only. Missing resolver, missing secret, or a non-owner all fail open to the existing manual password dialog instead of failing the connection. - ZMODEM runs entirely in the browser.
zmodem.js@0.1.10is wrapped by a single module,web/src/webssh/zmodem.ts(createZmodemBridge), whose Sentry splits remote bytes into terminal output and protocol state. Detection is auto-confirmed (a ZMODEM magic sequence can only appear because the user ransz/rz), the header is suppressed from the terminal, lrzsz’s trailingOOis stripped, receive resolves to a Blob download, and send drives a hidden multi-file input throughZmodem.Browser.send_files. Keystrokes are dropped while a transfer is active,abort()sends ZABORT, and the sharedweb/src/webssh/download.tshelper removes the duplicated Blob/anchor code from WebSFTP. Switching the terminal byte path toUint8Arrayplus a streamingTextDecoderalso fixed multi-byte characters split across chunk boundaries. - Auto-authentication is visible before it is used.
RemoteServerDisplayadds nullablecredentialType/credentialHasSecret, rendered as a “自动认证” tag on/remote-servers; clicking SSH or SFTP on such a row skips the dialog and falls back to it when the response carries noauthor the connection fails. - End-to-end proof against real lrzsz. The throwaway harness in
/tmp/tm-e2ewas upgraded so the SSH target is a real pty shell (creack/pty+/bin/zsh) with homebrewsz/rzonPATH. Twelve checks pass under headless Chrome over CDP: the auto-auth tag, no auth dialog, host-key confirmation, terminal connect with the stored password, keystrokes, a byte-for-byte 256 KiBszdownload, panel teardown with the prompt returning, a byte-for-byte 2 KiBrzupload, SFTP reuse listing the uploaded file, disconnect returning to the list, and the manual dialog for a server without a credential. - A pre-existing flake in
internal/agentwas fixed because it broke the gate.TestConnectionControllerRequiresServerAckAndRestartsFailedConnectionslept 20 ms and then assertedrunner.count() >= 2, whilesuperviserestarts after its own 20 ms backoff andSnapshot().Activeis already 1 before the firstRunnercall; the test failed roughly two runs in three. The newwaitConnectionStartspolls the recorded starts under a 2 s deadline. Test-only change, 20 consecutive-count=20runs green. -
Suite state after the round: frontend 30 files / 213 tests passing,
npm run buildplusscripts/verify-web-embed.shmatching,go test ./... -count=1,go test -race ./...,go vet ./...andgit diff --checkclean.npx vue-tsc --noEmitstill reports the pre-existing 177 errors and adds none. A tenth round implemented the approved plandocs/superpowers/plans/2026-09-12-webssh-bulk-transfer-flow-control.md(T1-T8): bulksz/rztransfers died mid-flight, and a browser refresh dead-ended on “credentials missing”. - Root cause: the local Agent relay path had no flow control in either direction. The protocol and the Agent already had the primitives, but
internal/server/agent_relay_transport.gonever used them. Download: the WebSSH broker leftrelay.StreamRequest.InitialWindowat zero, which the Agent reads as “unlimited”, so it never created asendStateand flooded the Server; the Server’s bounded inbound channel overflowed,enqueuereturnedrelay.ErrBackpressure, dispatch treated that as fatal and reset the stream, and the SSH channel hit EOF - which the terminal rendered as a genuine远端退出码:0. Upload: the Agent does enforce a 256 KiB receive window and resets the stream on overrun, but the Server’sWritenever consumed send credit and dropped every inboundWINDOW_UPDATE, so any upload larger than the window was killed. Both directions are now symmetric window accounting. - One shared credit contract. New
internal/protocol/window.goholdsMaxStreamFrame(32 KiB),DefaultServerReceiveWindow(512 KiB),DefaultAgentReceiveWindow(256 KiB) andDefaultWindowUpdateThreshold(128 KiB);internal/agent/session.gonow aliases them instead of keeping private copies, so the two ends cannot drift. No new frame type and no capability negotiation were needed:OPEN_STREAM.WindowandWINDOW_UPDATEalready existed, andDefaultServerReceiveWindowis only advertised when the caller asked for none. - A second-order bug found while fixing the first: the buffer was counted in frames while the window is counted in bytes. After window accounting was correct, transfers still reset between 0.3 and 1.1 MiB. The Agent charges its window per byte but emits one frame per pty read, and a pty read is often far smaller than 32 KiB, so a peer that stayed strictly inside its 512 KiB window could still send thousands of small frames into a 16-slot channel. The inbound path is now a byte-budgeted FIFO whose budget equals the advertised window, with
ErrBackpressureretained only as a guard against a window-ignoring peer. Regression testTestAgentRelayStreamAcceptsManySmallFramesWithinWindowsends 511 KiB in 1 KiB frames to a reader draining one byte at a time; before the fix it failed at exactly 16384 bytes withrelay: backpressure. - Deviation from the plan (recorded, not silent). The plan had the WebSSH broker set the default window. It is applied in
AgentRelayTransport.OpenStreamwhenInitialWindow == 0instead, so every current and future caller of the local relay path is covered by one code path rather than each remembering to set it. - Cross-node relay audited, no change needed (T4). The gRPC relay path already blocks in
Writeand is governed by HTTP/2 flow control, so backpressure propagates to the caller naturally. Adding a second window layer there would have been redundant and would risk deadlock against the transport’s own credit. - The browser converted transient bursts into session kills.
web/src/webssh/byte-stream.tsthrew as soon asbufferedAmountpassed 4 MiB, andweb/src/webssh/ssh-client.tsclosed the stream when the SSH receive queue passed 4 MiB - both trivially reachable by a legitimatesz. Writes now wait for the socket to drain above a 1 MiB high-watermark and only fail past a 64 MiB hard guard, and the receive bound was raised to 64 MiB. Because a waiting writer could otherwise be overtaken by a later one, writes are chained FIFO; reordered bytes would corrupt the SSH stream, and there is a test for the ordering specifically. - A local transport failure no longer masquerades as a remote exit.
ShellExitInfogainstransportError, populated from the client’s transport error when the shell drain loop breaks. The terminal renders it as a red alert with the real reason instead of printing a fabricated exit code, andexitReasonrefuses to report a status or signal when the transport died. - Refresh recovery. The one-time ticket and the password live only in page memory, so a reload always lost them and the view threw
webssh.missingSessionwith no way forward. Three helpers inweb/src/stores/webssh.tsnow remember only the non-sensitive remote-server ID insessionStorage; on reload both the terminal and the SFTP view route to/remote-servers?ssh=<id>, which already re-opens the connect flow and auto-authenticates when a credential secret is stored. An explicit “disconnect” clears the record so a deliberate close is not undone by a later refresh. No credential material is persisted, and the helpers tolerate a browser that blockssessionStorage. - The E2E harness moved into the repository (T7). It previously lived in
/tmp/tm-e2eand was rebuilt by hand each round. It is nowtest/e2e/webssh/: a dependency-free CDP driver (lib/cdp.mjs), stack lifecycle and config resolution (lib/stack.mjs), deterministic generated fixtures instead of committed multi-megabyte blobs (lib/fixtures.mjs), a readable scenario script (run.mjs), and a documented README. Every path, port and payload size is environment-overridable; it builds the three product binaries from the checkout and skips the ZMODEM checks with a clearSKIPwhenlrzszis absent. The throwaway SSH host is a nested Go module (test/e2e/webssh/sshhost/go.mod) socreack/pty,pkg/sftpandx/crypto/sshstay out of the rootgo.mod- the product must never ship an SSH server. It was 14 checks at the end of that round, adding the refresh-recovery assertion and a “browser console stayed clean” gate that fails the run on any console error or uncaught exception; round 11 later added the SFTP upload check, so the current total is 15. - All temporary
[diag]instrumentation was removed fromwebssh_broker.go,ssh-client.ts,byte-stream.tsandWebSSHTerminal.vue, and the web bundle plusinternal/server/web_distwere rebuilt afterwards. - Verification. Go:
go test ./... -count=1,go test -race ./...,go vet ./...,git diff --check. Frontend:npm test -- --run(30 files / 219 tests),npm run build,scripts/verify-web-embed.shmatching. Browser E2E from the repository checkout: 14/14 PASS, including a byte-for-byte 4 MiBszdownload and a byte-for-byte 1.5 MiBrzupload against reallrzsz, with no console errors.npx vue-tsc --noEmitstill reports the pre-existing 177 errors and adds none. All counts in this bullet are as of that round; round 11 raised them to 15 E2E checks and 222 frontend tests.
An eleventh round fixed a production report: WebSFTP uploads failed with the generic “文件传输失败” banner. A new E2E check reproduced it in-repo, and the diagnosis found three defects of one family plus the relay kill switch that actually killed the user’s sessions. This round was unplanned triage, so it is recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-sftp-upload-partial-write.md, with the red-light evidence labelled as observed during implementation rather than re-run.
- The real production root cause: the relay’s WINDOW_UPDATE mirror queue killed bulk uploads.
handleAgentFrameGenerationapplies inbound WINDOW_UPDATE credit tosendStateand then mirrors the frame intocontrolCh(capacity 16) forReadControlconsumers. The WebSSH/SFTP broker only ever callsRead, neverReadControl, so on any upload large enough to earn more than 16 updates the mirror overflowed and the dispatch failed the stream withprotocol: flow-control window exhausted; the bridge closed, sshd saw the connection drop, and the UI reported “SFTP 通道已关闭”. Evidence from a temporary bridge diagnostic:first=protocol: flow-control window exhausted, with the on-disk upload truncated at 768 KiB - sshd’s flush lag behind the ~2 MiB of consumed credit (16 updates x 128 KiB). Downloads never hit this because Server-to-Agent credit travels throughWriteControl, notcontrolCh, which is exactly why only uploads died. The mirror is now best-effort: when the queue is full the frame is dropped, because the authoritative credit already lives insendStateandconsumeSendWindowreads it directly. Regression testTestAgentRelayStreamSurvivesUndrainedWindowUpdatessends 40 undrained updates and then requiresWriteto succeed; before the fix it failed with the exact production error. libssh2_sftp_writepartial returns truncated every upload (browser, pre-existing). The session runs non-blocking, solibssh2_sftp_writereturns the bytes consumed so far - a positive value smaller than the request, notEAGAIN- whenever the channel window or the tunnel stalls.retrySFTPonly retriesEAGAIN, anduploadFiletreated any short write as fatal, which is precisely the generic “文件传输失败” the user saw: the bigger the file or the slower the path, the more certain the failure. The handle write now loops, feeding the unconsumed remainder atpointer + written, with a 5 s zero-progress stall guard so a dead tunnel surfaces as an error instead of spinning the retry loop forever.libssh2_channel_writepartial returns silently dropped terminal bytes (browser, pre-existing, latent). The same non-blocking semantics apply to the shell write path, where the remainder after a partial write was discarded - a long paste could lose keystrokes with no error anywhere. It now uses the same consume-the-remainder loop and stall guard.- The E2E harness gained an SFTP upload check (now 15 checks). It uploads a 3 MiB deterministic payload through the real file picker, waits for the row to appear in the listing, and compares the on-disk sha256 inside the throwaway host’s SFTP root.
sshhostSFTP is writable only whenTM_SSHHOST_SFTP_READONLY=0; the harness sets it because the SFTP root is a temp directory it owns, while manual runs of the throwaway host stay read-only. The check failed at 768 KiB before the relay fix and passes byte-for-byte after it. - Verification. Go:
go test ./... -count=1with all 23 packages ok,go vet ./...,gofmt -l internal/ cmd/ test/andgit diff --checkclean;go test -race ./internal/server ./internal/agent -count=1 -timeout 25mok at 427 s and 4.3 s (the full-suite-racewas validated earlier the same day and needs-timeout 30mas documented). Frontend:npm test -- --run30 files / 222 tests,npm run build,scripts/verify-web-embed.shmatching. Browser E2E re-run in a fresh working directory against binaries rebuilt from the current source: 15/15 PASS including the 3 MiB SFTP upload landing byte-for-byte (on-disk sha256 identical to the fixture), the 4 MiBszdownload and the 1.5 MiBrzupload, with a clean browser console. All temporary[diag]instrumentation removed again after the diagnosis.
A twelfth round fixed a production report that only reproduces on real networks: ZMODEM transfers corrupted in flight, with rz answering Unhandled header: ZRPOS / Peer aborted session and a finished sz dying with PROTOCOL: Only thing after ZFIN should be "OO" when the shell prompt arrived. The fix is browser-only and is recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-zmodem-write-serialization.md.
- Root cause: concurrent outbound writers plus the round-11 partial-write continuation. The terminal view wires ZMODEM’s synchronous
sendercallback as fire-and-forget (void channel.write(bytes)), and zmodem.js calls it several times per consumed chunk (ZACK, ZFIN, data subpackets), so one SSH channel routinely carries several in-flightwrite()promises. Round 11 turned each write into a loop that resends the unconsumed remainder after anawait; under channel-window pressure - routine on a WAN link, absent on localhost - two writers’ continuations interleave and the peer receives torn frames. The red unit test captured the exact wire signature:[1,1,2,2,1,2,1,2]instead of[1,1,1,1,2,2,2,2]. Torn frames explain every reported symptom:rzfails CRC and sendsZRPOS(which zmodem.js does not implement), and the browser’s mangled ZFIN reply makes lrzszszexit without ever sendingOO, so the receive session meets the prompt’s27,93(ESC ]) whereOObelongs. - Fix: one writer at a time per channel. A new
createWriteGate()promise chain serializes every libssh2 write-path call per channel: shellwrite/resize/close(extracted towriteShell/resizeShell/closeShell), and the whole SFTP surface - open, handle read/write/close,readDirectory(held across opendir, every readdir round and the close),unlink,rmdir,rename,realpathandclose. A rejected task does not poison the queue;closequeues behind in-flight operations instead of tearing the channel down mid-continuation;writeShelldrops the remainder whenshellClosedflips while it waits for window credit. The gate is microtask-level, adds no perceptible latency, and has no configuration. - The frozen-terminal symptom was a second defect. When zmodem.js threw,
consumeFromRemotediscarded the session together with the parser’s unconsumed buffer - in the post-ZFIN case that buffer is exactly the shell prompt, so the terminal sat on a blank line until the next keystroke. The catch now replays_input_buffer(only for a receive session past_got_ZFIN; a send session’s leftover is peer protocol bytes) plus any chunks behind the one that threw. - Why E2E never caught it, and what guards it now. On localhost the channel window never tightens, libssh2 always consumes whole buffers, and the continuation loop runs once - the interleaving window does not exist. The invariant is therefore pinned by unit tests:
never interleaves concurrent shell writes when libssh2 consumes partially,never interleaves an SFTP write with a concurrent SFTP request, andreturns the parser leftover to the terminal when a receive session dies after ZFIN. The troubleshooting guide states explicitly that a green E2E is not evidence for this fix. - Verification. Frontend: the two webssh specs 46/46, full suite 30 files / 225 tests,
npm run buildplusscripts/verify-web-embed.shmatching. Browser E2E in a fresh workdir against rebuilt binaries: 15/15 PASS including the byte-for-byte 4 MiBsz, 1.5 MiBrzand 3 MiB SFTP transfers with a clean console. No Go code changed this round, so the same-daygo test ./... -count=1(23 packages ok) andgo test -race ./internal/server ./internal/agent -count=1 -timeout 25m(427 s / 4.3 s) results stand.npx vue-tsc --noEmitremains a non-gate; the project-wide count of pre-existing errors is 194 and none of them sits on a line added this round.
A thirteenth round moved the ZMODEM transfer panel above the terminal and diagnosed a production report of a download whose progress froze at 291 KiB of 17.2 MiB with no error.
- Panel placement. The progress bar, file name and cancel button rendered below the terminal, so a scrolling terminal pushed them out of view mid-transfer; the user guide has promised “above the terminal” since the feature landed, and the implementation had drifted. The fix is a pure template reorder in
WebSSHTerminal.vue(the panel block now precedes the terminal element inside the same grid), guarded by a new source-order assertionkeeps the transfer panel above the terminalinweb/src/tests/zmodem-terminal.spec.ts. No logic, style or i18n changes; the E2E harness locates the panel by selector and is position-independent. - Frozen-progress diagnosis, no code change. A 17 MiB
szpasses end-to-end on localhost (15/15 E2E withTM_E2E_DOWNLOAD_BYTES=17 MiB), and the relay’s inbound FIFO was re-audited for lost wakeups: the stale wake token is drained under the same mutex that guards the append, and the append always signals, so a blockedReadcannot miss bytes. The reported signature instead matches the round-12 torn-outbound defect on a bundle that predates it - a corrupted ZACK makes lrzsz pause mid-send with no error until its own timeout - and the screenshot’s session login (00:39) predates the round-12 bundle (built 00:56). The troubleshooting guide now lists frozen progress as a third manifestation of that defect and tells operators to confirm the deployed bundle contains the single-writer gate before treating a freeze as a link problem; if it recurs after redeploy, Server logs and the browser console are the next evidence. - Verification. Frontend 30 files / 226 tests,
npm run buildplusscripts/verify-web-embed.shmatching, browser E2E 15/15 PASS in a fresh workdir (/tmp/tm-e2e-r13) with a clean console. No Go code changed this round.
A fourteenth round fixed the production sequence “sz freezes, then the whole page shows SSH 通道已关闭”: the freeze and the channel close are one causal chain, and both halves are browser-side. Recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-zmodem-stall-channel-policy.md.
- The freeze amplifier: per-subpacket progress events. The offer
inputhandler emitted oneprogressevent per ZDATA subpacket - tens of thousands of Vue reactive updates andel-progressrepaints for a 17 MiB file. While the main thread churns, the ZACK writes that keep lrzsz sending are delayed,szpauses mid-window, and the panel freezes at an arbitrary percentage. Progress is now reported on a 64 KiB byte step or a 500 ms quiet interval, with the first subpacket always reporting so tiny files leave 0 B immediately (an existing test caught the first version of this, which starved a 34-byte transfer of any event). - The channel close: the write-failure handler overreached.
toPeerclosed the terminal on any write rejection, including the round-11 stall guard’s “SSH shell write stalled after N of M bytes” - which is exactly what a pausedszproduces once the outbound window tightens. The channel was never dead: the read loop would have reported the real end state later. The catch now checks whether a ZMODEM session is active; if so it ends the transfer locally and showsZMODEM 传输失败:…, keeping the shell. Channel liveness stays owned by the read loop. - New bridge primitive
abortLocal(). Ending a transfer because the transport failed must not push a ZABORT into the pipe that just proved undeliverable - that write would reject and recurse into the same handler.abortLocal()clears the session, suppresses the pendingaccept()rejection, emitsabortedso the panel closes, and touches nothing outbound. - Verification. Targeted specs 72/72 including the new
throttles progress events for bulk transfers(200 KiB produced ~200 events before the fix, under 10 after) andabortLocal ends the transfer without sending to the peer; full frontend suite 30 files / 229 tests;npm run buildplusscripts/verify-web-embed.shmatching; browser E2E 15/15 PASS in/tmp/tm-e2e-r14with a clean console. No Go code changed this round.
A fifteenth round fixed two further production reports against the round-14 bundle: a 153 MiB sz that ended before completing (“没传输完,又被关闭了”), and a terminal that stopped echoing after a transfer while the identical PROTOCOL: Only thing after ZFIN should be “OO” banner stacked up once per keystroke. The fix is browser-only and is recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-zmodem-sentry-lifecycle-stall-policy.md.
- The vendored Sentry keeps the session that threw, so the terminal dies permanently.
zsentry.jsroutes every input tothis._zsessionwhile it is non-null and clears it only from_after_session_end(), which is bound to the session’s ownsession_endevent.zsession.js’s_consume_first()throws (a string, not anError) when a post-ZFIN byte is notOO, sosession_endnever fires and the library exposes no reset. Round 12 correctly nulled the bridge’s ownsession, but the Sentry still owned the dead one: every later byte was re-fed and re-threw, producing one identical banner per keystroke and never reaching the terminal - exactly the stacked alerts and the frozen pane in the report.createZmodemBridgenow builds its Sentry through amakeSentry()factory held in aletand recreates it on all three abandonment paths: theconsumeFromRemotecatch,abort()(whosecurrent.abort()can throw before firingsession_end) andabortLocal()(which deliberately sends nothing, sosession_endcan never fire). The vendored library is not patched; keeping the workaround inside the bridge is what makes a future zmodem.js upgrade reproducible. - A missing
OOafter ZFIN is teardown noise, not a failed transfer. ZFIN is only exchanged after ZEOF andOOonly follows ZFIN, so by the time that throw is reachable every file byte is already spooled,completedhas been reported and the browser has saved the file. Reporting it aserrortold operators that a finished 153 MiB download had broken and invited a retry of a file already on disk. The catch now classifies the failure using the very state the leftover replay already read (role === 'receive' && session._got_ZFIN), armsstripNextOverAndOutso a lateOOis still eaten, replays the buffered prompt, and emitsended- after the replay, so the progress panel only closes once the prompt is on screen. Mid-transfer throws still reporterrorunchanged. The round-12 test that asserted this error reached the UI was rewritten to assert the new contract. - The stall deadline killed large transfers, so protocol writes opt out of it. A bulk
szaccumulates megabytes of ZACK backlog in the outbound direction while lrzsz pauses to retransmit; the round-11writeStallLimit = 5000then threwSSH shell write stalled after N of M bytesand the round-14toPeerpolicy ended the transfer, which is the user’s “没传输完,又被关闭了”.TerminalChannel.writenow takesoptions?: { stallLimitMs?: number }with0meaning no deadline, and onlytoPeerpasses it. Keystrokes keep the 5 s bound on purpose: a keystroke that cannot be written in 5 s really does mean a dead channel, and the view relies on that rejection to close the terminal. A genuine tunnel failure still surfaces, because libssh2 returns a negative code,assertResultthrows, and the round-14 “abort the transfer, not the channel” policy applies. - Removing the deadline forced
close()out of the write gate.close()queued behindshellGate, so a single unbounded stalled writer would hangawait channel?.close()incloseTerminal()forever and leak the libssh2 handles - a new failure mode that removing the deadline would otherwise have introduced.close()now callscloseShell()directly, andcloseShell()flipsshellClosedsynchronously, which is what makes the in-flight writer unwind at its next check. To keep that safe,retryEAGAINgained an optionalisCancelledpredicate and returns0(the “no progress” value the write loops already understand) instead of touching a handle the caller may already have freed; without it a retry attempt could land on a channel that_ssh2_channel_freehad just released. - Verification. Red first: four new assertions failed before the implementation -
timed out waiting for ended; got ["detected","receive-offer","progress","completed","error"](twice, for theendedcontract and the passthrough contract),expected Error: SSH shell write stalled after 0 of 3 bytes to be null, and the missingstallLimitMs: 0source marker. Two new tests passed pre-fix and are kept as regression guards:parses shell output normally after abortLocal(the abandoned receive session happens to route invalid bytes back as garbage, so the old code did not error there either) andstill reports a stalled shell write once the default deadline passes(proves the deadline removal is opt-in). Targeted specs 60/60; full frontend suite 30 files / 234 tests, up from 229;npm run buildplusscripts/verify-web-embed.shmatching, withstallLimitMspresent in the embeddedWebSSHTerminalasset; browser E2E 15/15 PASS in/tmp/tm-e2e-r15, includingtransfer panel closes and the prompt returnswith a clean console. No Go code changed this round, but the gates were re-run for completeness:go build ./...,go vet ./...,gofmt -l internal/ cmd/ test/andgit diff --checkclean,go test ./... -count=1with 0 failures across 18 tested packages (23 total).npx vue-tsc --noEmitremains a non-gate; the project-wide count of pre-existing errors is still 194 and none sits on a line added this round.
A sixteenth round fixed a production report of the terminal page collapsing into the “已断开 / SSH 通道已关闭” empty state in the middle of an sz download, together with the operator directive that a channel must not be declared dead before thirty seconds of evidence. The fix is browser-only and is recorded as a back-filled plan in docs/superpowers/plans/2026-09-13-webssh-dead-channel-window.md.
- The only sub-30s “channel is dead” decision in the whole stack was the browser’s 5 s write-stall guard. An audit of every time-based liveness decision found exactly one under thirty seconds:
writeStallLimit = 5000, shared by the shell write loop and the SFTP write loop. On the Server side the only timer isserver.webssh.idle_timeout, defaulted to 5 minutes; the 64 MiB receive-queue and WebSocket backpressure guards are capacity bounds, not timers. The tail of a bulkszkeeps the outbound window congested for longer than five seconds, so a keystroke written right after the transfer - queued behind a protocol write that is still waiting for window credit - hit the guard, threwSSH shell write stalled after N of M bytes, and the keystroke path’s catch closed the whole session. The guard is now30_000ms, with a comment stating that it is the dead-channel decision rather than a latency budget; a genuinely dead tunnel still surfaces earlier through the WebSocket close and the read loop, so thirty seconds is only the fallback bound. - A leftover protocol write could close a living session after the transfer ended. Round 15 lets
toPeerwrites wait for window credit without a deadline, so one can still be inside the write gate when the transfer reportsendedoraborted. Its late rejection - a negative libssh2 result once the peer winds down - reached thetoPeercatch’s inactive branch, which calledcloseTerminal(): a rejection with nothing to do with shell health killed the session and produced exactly the reported empty state (no transport alert, no remote-exit block). That branch now does nothing; channel liveness stays owned by the read loop and by the keystroke path, consistent with the round-14 principle that write failures must not adjudicate liveness. - Verification. Red first: three assertions failed before the implementation -
expected Error: SSH shell write stalled after 0 of 3 bytes to be nullat the 20 s mark,expected SFTPError: SFTP write stalled after 0 of 64 bytes to be nullat the 20 s mark, and the source assertion that thetoPeerslice must not containcloseTerminal(). The two stall tests were rewritten from the round-15 4 s/2 s shape to 20 s silent then 35-36 s failing, and the SFTP stall test’s fake-timer loop was extended past the new window. Targeted specs 60/60; full frontend suite 30 files / 234 tests (unchanged count: three tests rewritten, none added);npm run buildplusscripts/verify-web-embed.shmatching; browser E2E 15/15 PASS in/tmp/tm-e2e-r16with byte-for-bytesz,rzand SFTP transfers and a clean console. No Go code changed this round; gates re-run for completeness:go build,go vet,gofmt -l internal/ cmd/ test/,git diff --checkclean andgo test ./... -count=1with 0 failures across 18 tested packages (23 total).
A seventeenth round made the embed sync impossible to forget and settled the recurring Nginx question. Every round since the admin UI shipped ended with the same release reminder - npm run build, then rsync into internal/server/web_dist/, then rebuild the Server binary - because a build that skipped the middle step silently shipped the previous bundle: the binary rebuilds, the embedded assets do not change, and every frontend fix looks unreleased.
npm run buildnow performs the embed sync itself. The script isvite build && node scripts/sync-web-dist.mjs; the newweb/scripts/sync-web-dist.mjsmirrorsweb/dist/intointernal/server/web_dist/with mirror semantics (clear then copy), because hashed chunk names change on every build and merged leftovers would be embedded forever. It is plain NodermSync/cpSyncrather thanrsyncso the step also works on Windows release hosts, takes optional source/target arguments for tests, and exits non-zero with async-web-dist: source build output missingmessage whenvite buildhas not run - a half-written or empty tree must never reach the embed directory. Manualrsync -a --deleteremains documented as a verification and recovery path only.- Nginx needs no frontend-route configuration in the default deployment. The SPA history fallback is served by the Server’s embedded static handler, so
location /only needsproxy_pass; the/api/and/ws/*locations exist for proxy precedence and WebSocket upgrade, not for frontend routing.try_files $uri $uri/ /index.html;and theapplication/wasmtype addition are required only in the alternative “Nginx serves web/dist statically” mode. Both facts are now stated explicitly indocs/deployment/nginx.mdanddocs/deployment/frontend.md, which previously left operators guessing. - Verification. Red first:
syncs the build output into the Go embed directory as part of buildfailed on the old"build": "vite build"script,mirrors the source tree into the embed directory and deletes stale filesfailed with the script missing (staleapp-old.jssurvived a merge), andfails when the build output is missingfailed until the guard message existed. Green:web/src/tests/build.spec.ts8/8; full frontend suite 30 files / 237 tests;npm run buildnow printssync-web-dist: mirrored … -> …/internal/server/web_distandscripts/verify-web-embed.shreports the trees match. No Go code changed;go build ./...,go vet ./...and the embedded-asset test were re-run clean.
Release Steps
- Back up SQLite or MySQL and verify restore.
- Deploy the new Server with
storage.auto_init=trueso schema v11 advances to v12 and v12 advances to v13. - Keep
server.webssh.enabled=falseduring rollout if a staged release is required. - Verify
/health/ready,schema_meta.version=13, the three WebSSH tables, and the fourcredentials.secret_*columns. - Configure exact
security.allowed_originsand the Nginx/ws/webssh/location. - Inject
TUNNELMESH_TOKEN_ENCRYPTION_KEY(base64 or hex, 16/24/32 bytes) andTUNNELMESH_TOKEN_ENCRYPTION_KEY_IDidentically on every Server node before credential auto-authentication is offered to users. - Enable
server.webssh, sign in to the admin console, and complete one SSH, one SFTP, oneszdownload and onerzupload smoke test.
Rollback Steps
- Disable
server.webssh.enabled. - Close active browser sessions and roll back the application to the previous release.
- Retain the v12 tables and the v13
credentials.secret_*columns; do not issue reverse DDL or manually changeschema_meta.version. A v12 binary ignores the new columns, stored secrets simply become invisible, and browser SSH/SFTP falls back to the manual password dialog. - If exact schema restoration is mandatory, restore the pre-upgrade database backup.
- Keep
TUNNELMESH_TOKEN_ENCRYPTION_KEYunchanged across the rollback window; removing it makes stored secrets undecryptable even after upgrading again.
Reviewer Focus
- Ticket authentication order, one-time consumption, and durable close behavior.
- WebSocket origin checks and
/ws/webssh/precedence over SPA fallback. - Browser memory handling for passwords, private keys, and host-key confirmation.
- SFTP chunking, 1 GiB preflight limit, progress reset, and shared-connection close behavior.
- v11 to v12 migration parity between MySQL and SQLite.
- Nginx timeout and buffering settings for long-lived browser sessions.
- Owner scoping and cursor semantics of
GET /api/v1/ssh-sessions, including that administrators list only their own sessions. - Exit status/signal are read while the libssh2 channel handle is still valid, and the optional-binding fallback returns
nullinstead of a fabricated code. - Owner-only decryption in
CredentialService.GetWithSecret, and the fail-open path that returns noauthinstead of an error. - Ordering of
a.mutateand theauthattachment increateWebSSHSession; the stored idempotency body must never contain plaintext. - Shape validation in
validateCredentialShape: apasswordcredential rejects a private key, anssh_public_keycredential rejects a password, and a passphrase requires a private key. - Type change clearing the stored secret, and the
503fast failure when no encryption key is configured. - v12 to v13 migration parity between MySQL and SQLite, including the corrected SQLite
schema_metaseeding in the migration tests. - ZMODEM lifecycle in
createZmodemBridge: singlefinish,abortedByUsersuppressing spurious errors, keystroke gating, anddispose()on close and unmount. - Relay window symmetry in
agent_relay_transport.go: send credit consumed before every DATA frame, receive credit released only afterReadhands bytes to the consumer, andfinishwaking a blockedWrite. - The byte-budgeted inbound FIFO: budget equals the advertised window,
Readdrains the wake token while still holding the stream lock so no wakeup can be lost, andreadBufis compacted rather than resliced. - FIFO serialization of browser WebSocket writes; a waiting writer must never be overtaken by a later one.
ShellExitInfo.transportErrorpropagation, and thatexitReasonreports nothing when the transport died.sessionStoragetarget memory holds only the remote-server ID; verify no credential material can reach it on any path.- Nested-module isolation of
test/e2e/webssh/sshhost: confirmgo build ./...andgo vet ./...at the root still ignore it and that the rootgo.modgained no SSH server dependency. - WINDOW_UPDATE handling in
handleAgentFrameGeneration: credit must reachsendStatebefore the mirror attempt, and a full mirror queue must neither fail the stream nor lose credit. - The SFTP and shell write loops: remainder offsets, the stall guard bound, and that a negative result still goes through
assertSFTPResult/assertResultunchanged. - That
uploadFile’s short-write assertion remains as a defensive check now that the handle write guarantees full consumption. createWriteGatesemantics: a rejected task must leave the queue usable, the gate must span multi-call operations (the SFTP handle write loop and the wholereadDirectorysequence), and no gated task may call another gated function (deadlock check).- Fire-and-forget callers (
toPeerin the terminal view, keystrokeonData) get strict ordering without awaiting; verify no caller depends onwriteresolving before the bytes are queued. writeShelldropping the remainder onshellClosed, and shell/SFTPclosequeueing behind in-flight operations instead of preempting them.- The zmodem leftover replay: receive role and
_got_ZFINonly, and the tail-chunk arithmetic (offset + ZMODEM_CHUNK_BYTES) for chunks behind the one that threw. - DOM order in
WebSSHTerminal.vue: the transfer panel must stay beforeref="terminalElement"so progress and cancel remain visible while the terminal scrolls; the grid must not let it overlap the transport alert. - The
toPeerfailure policy: transfer-active meansabortLocal()plus an error message and a living shell; transfer-inactive keeps the oldcloseTerminal(); verify no path closes the channel from a write rejection while a session is active. - Progress throttling thresholds (64 KiB / 500 ms) and the unconditional first report;
abortLocalmust never calltoPeer, and must suppress the pendingaccept()rejection exactly once. - Sentry lifecycle in
createZmodemBridge: all three abandonment paths (theconsumeFromRemotecatch,abort(),abortLocal()) must recreate the Sentry, andsuppressed/suppressNextTerminalWritemust be reset together with the session so a fresh Sentry cannot silently swallow the next terminal write. - The
finishedBeforeThrowclassification reads_got_ZFINoff a receive session only. Verify that a send-session protocol failure still reportserror, thatstripNextOverAndOutis armed before the replay, and thatendedis emitted after the leftover reaches the terminal. stallLimitMsis opt-in and defaults towriteStallLimit. Confirm no caller other thantoPeerpasses0, and that the keystroke path still depends on the rejection to close the terminal.-
close()bypassing the write gate:closeShell()must remain synchronous up toshellClosed = true, andretryEAGAIN’sisCancelledmust be evaluated before everyoperation()call so no write can touch a freed channel handle. - The dead-channel window:
writeStallLimitis the policy constant for both the shell and SFTP write loops; confirm nothing else re-introduces a shorter timer and that the keystroke path still closes the terminal once the window elapses. -
The
toPeercatch must have exactly two outcomes: transfer-active aborts the transfer with a banner, transfer-inactive does nothing. Verify no path closes the session from a protocol-write rejection, and that a truly dead tunnel is still closed by the read loop / WebSocket close. - The embed sync in
npm run build: confirm CI and the Dockerfile multi-stage build still produce a matchinginternal/server/web_dist(the Dockerfile performs its own sync and must not double-delete a mounted volume), and thatscripts/verify-web-embed.shstays in the release checklist as the independent check.
Integration Status
The feature is wired into the Server runtime, management API, embedded Vue bundle, and Agent relay path. Generated web/dist and internal/server/web_dist are ignored and are not part of the source diff.