Product review · 4 September 2026
Jarvis voice and conversation pipeline
The ideas improve Jarvis. This implementation does not yet. It raises routing recall at the cost of routing trust, hides mobile outcomes, and adds avoidable latency and privacy exposure.
Twelve P1 defects affect routing authority, canonical instructions, result visibility, privacy, and latency.
Walkthrough
The inclusive range covers commits 5baf144b and b83823ba. It adds direct conversational answers, semantic acknowledgements, descriptive push notifications, mobile barge-in and shorter speech, a match-first project grounder, and a denser mobile Task Desk presentation.
- Conversation, shorter spoken completion, barge-in, and more informative notifications are good product goals.
- Match-first grounding improves recall for novel phrasing and observed ASR variants, but removes the grammatical boundary that made routing safe.
- Mobile now has three competing representations of outcome state: transient message, presentation buffer, and Task Desk cards; the new filter hides the only result-bearing one.
- The range is type-correct and its 87 focused tests pass, yet direct negative probes reproduce routing and clarification failures absent from the suite.
Does it improve Jarvis?
- General questions no longer need a coding task.
- Barge-in gives mobile users immediate control of speech.
- Short completion speech respects voice as a summary channel.
- Project/thread context makes notifications easier to identify when privacy allows.
- Match-first scoring covers real ASR variants and trailing phrasing missed by narrow regexes.
- Any task word can become a project target.
- Confirmation can mutate the instruction.
- Web bypasses confirmation.
- Mobile hides outcomes and keeps stale blockers.
- Conversation adds two model round trips and still requires a project.
- Push enrichment adds privacy and database cost to the event path.
Better product shape
- Ground in two stages: extract one project-bearing span from typed grammar or ASR metadata, then resolve only that span against a precompiled bounded catalog.
- Keep one acknowledgement owner: an action-neutral local cue before the Host responds; contextual speech only from the validated accepted result.
- Use one outcome projection: newest event per task/turn, where terminal events supersede blockers and Task Desk cards never stand in for result text.
- Make converse a separate request kind: no project required, one supervisor call, typed failure, explicit retry semantics.
- Keep notifications generic by default: classify before database reads; descriptive titles require an opt-in.
Files changed (17)
| File | Delta |
|---|---|
| apps/mobile/src/features/jarvis/JarvisMobileProvider.tsx | +14 −9 |
| apps/mobile/src/features/jarvis/JarvisRouteScreen.tsx | +34 −23 |
| apps/mobile/src/features/jarvis/mobilePushToTalk.ts | +3 −5 |
| apps/mobile/src/features/jarvis/mobileSpeechPolicy.test.ts | +16 −0 |
| apps/mobile/src/features/jarvis/mobileSpeechPolicy.ts | +17 −0 |
| apps/mobile/src/features/jarvis/useJarvisVoice.ts | +11 −2 |
| apps/server/src/jarvis/Layers/JarvisController.test.ts | +66 −0 |
| apps/server/src/jarvis/Layers/JarvisController.ts | +43 −0 |
| apps/server/src/jarvis/Services/JarvisController.ts | +13 −0 |
| apps/server/src/jarvis/push/ExpoPushNotifications.test.ts | +22 −0 |
| apps/server/src/jarvis/push/ExpoPushNotifications.ts | +37 −11 |
| packages/contracts/src/jarvis.ts | +5 −0 |
| packages/jarvis-core/src/command.test.ts | +5 −0 |
| packages/jarvis-core/src/command.ts | +28 −1 |
| packages/jarvis-core/src/groundVoiceTurn.test.ts | +28 −0 |
| packages/jarvis-core/src/groundVoiceTurn.ts | +226 −80 |
| packages/jarvis-core/src/semantic.ts | +31 −1 |
Issues table (16)
| # | File | Lines | Priority | Type | Issue |
|---|---|---|---|---|---|
| 1 | packages/jarvis-core/src/groundVoiceTurn.ts | 321–469 | P1 | logic | Locate a project slot before catalog matching |
| 2 | packages/jarvis-core/src/groundVoiceTurn.ts | 402–426 | P1 | logic | Bind confirmation to the selected candidate’s original span |
| 3 | packages/jarvis-core/src/groundVoiceTurn.ts | 398–413 | P1 | logic | Validate stale confirmations before returning not-mentioned |
| 4 | apps/web/src/components/jarvis/JarvisManagerDialog.tsx | 910–947 | P1 | logic | Do not dispatch web voice matches that still need confirmation |
| 5 | packages/jarvis-core/src/groundVoiceTurn.ts | 326–383 | P1 | logic | Bound and index grounding before running it on UI threads |
| 6 | apps/mobile/src/features/jarvis/JarvisRouteScreen.tsx | 90–111 | P1 | logic | Keep terminal outcomes and supersede stale blockers |
| 7 | apps/mobile/src/features/jarvis/JarvisRouteScreen.tsx | 45–54 | P1 | logic | Stop route-owned speech when Jarvis loses focus |
| 8 | apps/mobile/src/features/jarvis/JarvisMobileProvider.tsx | 430–461 | P1 | logic | Do not speculate about project work before validation |
| 9 | apps/server/src/jarvis/Layers/JarvisController.ts | 442–551 | P1 | logic | Make conversation genuinely project-independent |
| 10 | apps/server/src/jarvis/Layers/JarvisController.ts | 68–95, 469–551 | P1 | logic | Answer conversation in one model call and expose failures honestly |
| 11 | apps/server/src/jarvis/Layers/JarvisController.ts | 540–551 | P2 | logic | Apply request idempotency to conversation answers |
| 12 | apps/server/src/jarvis/push/ExpoPushNotifications.ts | 119–125, 181–195 | P1 | logic | Keep push notification content private by default |
| 13 | apps/server/src/jarvis/push/ExpoPushNotifications.ts | 173–196 | P1 | logic | Classify push events before loading full thread details |
| 14 | apps/mobile/src/features/jarvis/mobileSpeechPolicy.ts | 34–51 | P2 | style | Share typed spoken-summary policy across clients |
| 15 | packages/jarvis-core/src/groundVoiceTurn.ts | 98–161, 342–387 | P2 | style | Remove the duplicate dead span scanner |
| 16 | docs/user/jarvis-mobile.md | 1–27 | P2 | info | Update docs for the changed capability and lifecycle |
Request flow
flowchart LR U[Voice or text] --> C[Client grounding] C -->|project ref + utterance| H[Host grounding and semantic supervisor] H -->|control| T[T3 orchestration/provider] H -->|converse| A[Second supervisor answer call] T --> P[Live presentation + Task Desk] P --> S[Mobile/Desktop speech] T --> N[Push reactor] classDef risk fill:#2a0d12,stroke:#fb7185,color:#fff class C,H,A,P,N risk
Prompt for all review comments
Prompt to re-run this review
✓ Remote branch verified: local HEAD and origin/codex/jarvis-orchestrator both b83823ba.
✓ 87 focused tests across seven touched suites passed.
✓ Focused typechecks passed for contracts, jarvis-core, server, and mobile; server emitted existing Effect suggestions only.
! Direct probes reproduced incidental routing, explicit-target ambiguity, stale confirmation, and wrong-span canonicalization.
! Matcher probe: 29 ms / 40 names, 742 ms / 1,200 names, 1,527 ms / 2,400 names.
No browser, simulator, microphone, notification device, or real speech-model acceptance pass was run.
Review comments
packages/jarvis-core/src/groundVoiceTurn.ts · 321–469Locate a project slot before catalog matchinglogicP1correctness
packages/jarvis-core/src/groundVoiceTurn.ts · 402–426Bind confirmation to the selected candidate’s original spanlogicP1data-integrity
The confirmed branch finds the selected candidate by ID but canonicalizes `ranked[0]`, the best span across every candidate. The pending frame stores candidate identity but not the matched span that caused the clarification.
The failure is reproducible: confirming Alertify for ‘Compare Jarvis and Alertify’ produces ‘Compare Alertify and Alertify’. A project clarification can therefore alter the work instruction while still routing to the selected project.
Persist the original slot boundaries with the pending clarification, or recompute only the confirmed candidate within the original slot. Route identity and canonical text must come from the same match.
Prompt for AI agents
packages/jarvis-core/src/groundVoiceTurn.ts · 398–413Validate stale confirmations before returning not-mentionedlogicP1correctness
The function returns not-mentioned when ranking is empty before it validates `confirmedCandidateId`. If the catalog becomes empty between question and answer, the explicit stale-project branch is unreachable.
The direct probe `Open Zivil` with an empty catalog and a confirmed Rivvl ID returned ready/not-mentioned rather than ‘That project is no longer available.’ The semantic path may then use ambient project context.
Check the confirmed identity first, and fail closed when it is absent. Disconnection and deletion must never erase the clarification target and resume elsewhere.
Prompt for AI agents
apps/web/src/components/jarvis/JarvisManagerDialog.tsx · 910–947Do not dispatch web voice matches that still need confirmationlogicP1correctness
The shared grounding result correctly distinguishes resolved matches from needs-confirmation. The web dialog ignores that distinction: at lines 916–918 it assigns the proposed project directly and continues submission.
This was risky before, but match-first grounding makes it routine because ordinary phrases now reach the phonetic threshold. The direct probe ‘Review latest changes’ against Rivvl returned needs-confirmation; web would route it to Rivvl instead of asking.
Use the same typed pause flow as mobile. A needs-confirmation result must become a pending choice and cannot populate `groundedVoiceProject` until the user accepts it.
Prompt for AI agents
packages/jarvis-core/src/groundVoiceTurn.ts · 326–383Bound and index grounding before running it on UI threadslogicP1performance
Every voice turn scans every candidate name against every utterance window and allocates edit-distance rows while repeatedly normalizing and recomputing Soundex. One long alias also raises the window limit for every short name, and the `Math.max(...allNames)` spread is itself unbounded.
A local five-run probe measured about 29 ms for 40 names, 742 ms for 1,200 names, and 1,527 ms for 2,400 names on this machine. Web and mobile invoke this synchronously on their JavaScript UI path, and the Host repeats grounding afterward.
Precompile normalized names and phonetic keys when catalogs change, index exact names, cap tokens/names/window width, and fuzzy-score only the extracted project slot and a small shortlist. This also removes the paired-client CPU-exhaustion path.
Prompt for AI agents
apps/mobile/src/features/jarvis/JarvisRouteScreen.tsx · 90–111Keep terminal outcomes and supersede stale blockerslogicP1correctness
Task cards show state, title, and objective. They do not contain the assistant’s terminal result or failure text, so completed and failed presentations are not duplicates. The new fallback shows a terminal presentation only when there is no focused or recent task, a condition the just-completed task normally prevents.
The inverse state is also wrong. Presentation IDs are per activity, so a later completion does not replace an earlier approval or input presentation. The filter hides the completion and leaves the historical blocker marked as live.
Collapse presentations by task/turn and keep the newest event authoritative. Show the latest terminal outcome until the user dismisses it or a newer outcome supersedes it; derive blocker status from current task state.
Prompt for AI agents
apps/mobile/src/features/jarvis/JarvisRouteScreen.tsx · 45–54Stop route-owned speech when Jarvis loses focuslogicP1security
The blur cleanup now cancels only capture and deliberately lets playback continue. At the same time it detaches the speech sink and leaves the route where the microphone button provides barge-in.
This contradicts both the user guide and internal ownership document, which say leaving Jarvis cancels playback and active synthesis. A user opening another screen can hear private task text with no visible stop control.
Restore `cancelSurface()` on blur. If background speech is a desired product feature, move ownership to an app-level player with a persistent stop control and an explicit preference before changing lifecycle semantics.
Prompt for AI agents
apps/mobile/src/features/jarvis/JarvisMobileProvider.tsx · 430–461Do not speculate about project work before validationlogicP1correctness
Mobile now says ‘Taking a look at Project’ before the Host knows the semantic action or accepts dispatch. It then discards the schema-bounded acknowledgement produced by the semantic layer for actual provider work.
The message is misleading for general questions and control commands, and it creates a client-specific acknowledgement policy that diverges from web. The comment says the server acknowledgement remains visible, but the UI displays `Started objective` instead.
Keep the immediate latency cue action-neutral, then speak the Host-owned acknowledgement only for an accepted started result. One layer should decide contextual wording and speech eligibility.
Prompt for AI agents
apps/server/src/jarvis/Layers/JarvisController.ts · 442–551Make conversation genuinely project-independentlogicP1maintainability
The new converse command says it needs no project or task, but it remains behind `JarvisControllerExecuteInput.projectId`, mobile’s has-project push-to-talk gate, project selection, and project-bound mesh execute routing.
On a fresh Controller installation or a temporarily disconnected execution catalog, the user cannot ask a general question. That is exactly where a project-independent assistant path would be useful.
Classify before choosing an execution project, or add a discriminated project-free conversation request. Keep coding commands on the current node-qualified path.
Prompt for AI agents
apps/server/src/jarvis/Layers/JarvisController.ts · 68–95, 469–551Answer conversation in one model call and expose failures honestlylogicP1performance
A converse request first calls the supervisor to classify the action, then calls the same provider again in a second temporary directory to generate the answer. This doubles the slowest part of the voice path and is likely why the client now tries to mask latency with speculative speech.
The second call converts provider unavailability, schema failure, and execution failure into ‘I couldn’t answer that just now,’ then returns `status: acknowledged`. There is no diagnostic log and no typed unavailable state.
Let the first structured proposal carry a bounded answer when action is converse, or use a deliberate fast classifier before one answer call. Return typed failure and log the cause instead of labelling failure as successful output.
Prompt for AI agents
apps/server/src/jarvis/Layers/JarvisController.ts · 540–551Apply request idempotency to conversation answerslogicP2data-integrity
The controller computes an acceptance key for the request, but the converse early return emits no orchestration command or receipt and never uses the request metadata retained on the converse command.
If the network drops after generation, a retry with the same request ID executes the model again. That breaks the documented idempotency expectation even though no coding task is created.
Either cache the bounded answer by acceptance key for the request lifetime or explicitly exempt conversation from the idempotent execute contract with a separate endpoint. Do not carry unused identity fields.
Prompt for AI agents
apps/server/src/jarvis/push/ExpoPushNotifications.ts · 119–125, 181–195Keep push notification content private by defaultlogicP1security
Push copy now sends thread and project titles to Expo and renders them in the notification. These fields often contain customer names, incident descriptions, repository names, or security findings.
There is no descriptive-notification preference, redaction policy, or lock-screen warning. The project title is also unbounded before it enters the request body.
Use generic notification text by default. Descriptive previews should require an explicit preference and should be bounded and redacted before third-party transmission.
Prompt for AI agents
apps/server/src/jarvis/push/ExpoPushNotifications.ts · 173–196Classify push events before loading full thread detailslogicP1performance
The subscriber calls `getThreadDetailById` before `pushMessageForEvent` determines whether the activity maps to approval, input, completion, or failure. Tool, progress, checkpoint, and ordinary activity events all pay the enrichment cost.
The detail query loads messages, activities, plans, checkpoints, session, and latest turn. On a sequential event consumer this adds growing database work to a high-volume path and can delay actual push events.
Classify the event first. For the four push-worthy kinds, load only a thread shell/title and project ID, then optionally a project shell if descriptive previews are enabled.
Prompt for AI agents
apps/mobile/src/features/jarvis/mobileSpeechPolicy.ts · 34–51Share typed spoken-summary policy across clientsstyleP2maintainability
Mobile takes two raw sentences and 240 characters, while desktop strips Markdown/code and permits 460 characters. The mobile helper accepts `kind: string`, so a future presentation kind silently falls into the speak-in-full branch.
Shorter completion speech is a real product improvement, but arbitrary first-sentence truncation is not a reliable summary and can omit the actual result. Raw Markdown and code punctuation can also sound bad through TTS.
Move normalization and bounded spoken-summary selection into jarvis-core with a closed presentation-kind type. Better still, let the Host provide an explicit bounded spoken summary beside the full durable result.
Prompt for AI agents
packages/jarvis-core/src/groundVoiceTurn.ts · 98–161, 342–387Remove the duplicate dead span scannerstyleP2maintainability
`SpanHit` and `scanSpans` are unused. `groundVoiceTurn` contains a second copy of the same loops, but its tie-breaker prefers longer heard text while the dead helper does not.
This is exactly the kind of duplication that causes a routing fix to land in the wrong implementation. `ProjectMention.explicit` is also now populated but never read.
Delete the abandoned implementation as part of the slot-first redesign, or make one scanner the only source of candidate-associated hits and keep ranking decisions separate.
Prompt for AI agents
docs/user/jarvis-mobile.md · 1–27Update docs for the changed capability and lifecycleinfoP2maintainability
No Markdown changed across a product behavior range that adds conversation, descriptive push copy, barge-in, new speech truncation, and different route-blur playback behavior.
The existing mobile guide explicitly says leaving Jarvis stops playback and synthesis, which the code no longer does. The controller docs also describe the neutral local cue and Host-owned accepted acknowledgement, not the new mobile behavior.
Fix behavior first, then document the shipped conversation boundary, speech-summary rule, barge-in behavior, notification privacy default, and any intentional lifecycle change.
The new matcher treats every token window as a possible project name. The explicit-only mode now means only exact matching, not explicit project syntax, so typed text such as ‘fix auth’ switches to a project named Auth. Voice input can also auto-route an ordinary task word at the 0.8 spelling threshold.
It also flattens exact hits across the whole sentence. ‘In Web, fix the API response’ becomes ambiguous when Web and API are both projects, even though the user supplied an explicit target. A more dangerous variant is an exact project word in the objective beating a fuzzy but syntactically clear trailing project name.
This is a direct regression in Jarvis’s trust contract. Detect one bounded project-bearing slot first, give explicit syntax precedence, and use phonetic or edit-distance matching only to resolve that slot against real catalog entries.
Prompt for AI agents