How JoyPilot works, end to end
One utterance in, one governed, audited DBS payment out.
A model may propose; only deterministic code may decide.
①
UNDERSTAND AND PROPOSE
the AI half
0Authentication
1Audio Input (ASR)
2Evidence Commit
User Input
Voice or typed input
English · Chinese · Bahasa
Agent 1
3Guardrails Agent
Scam + override screening
Backed by a deterministic scam table
Agent 2
4Intent Classifier
Zero-UI multi-intent
transaction parsing
Valid
→ the draft is resolved and checked below
Invalid
→ the turn stops here; nothing downstream is reached
Action Agents
Agent 3
5Payment Agent
Extracts payment details
6Payees and Account Resolution
7Canonical draft and schema gate
Agent 4
8Validation Agent
Checks against original words
Dynamic Ambiguity Resolution and Semantic Discrepancy Auditing
Typed contracts. Agents only return structured JSON proposals:
amount · payee · source · others
Shaped for DBS's APIs. The draft carries the fields a bank transfer needs: source, beneficiary, amount, currency and reference.
No agent holds a key. Without an approval and a signing
key there is no ledger access.
②
DECIDE, AUTHORISE AND EXECUTE
the deterministic half
Banking Policy Check
10Risk / value threshold check
11Velocity throttling check
12Field-by-field confirmation
TTS Voice Output
Cartesia
User Confirmation
Human-in-the-loop
biometric authorisation
13Mock Biometrics
Simulated adapter
14Canonical payload signing
The signed artifact
15Zero-Trust Gateway
Re-checks everything itself
16Payment Complete
The only way a payment is recorded
Zero-trust gateway. The gateway rebuilds
the canonical payload and runs 14 checks: the signature, the draft version, a complete
confirmation, the validation result, the risk policy, freeze and cancel status,
nonce reuse and expiry. Only then is the ledger reachable.
Behind the gateway is a mock ledger. Stage 15 calls one
function, Ledger.execute(), which records the payment in
memory. Connecting DBS's payment API means replacing that one call; that
integration is not built in this prototype.
Reading this against the control chain
The diagram above numbers its boxes 0–16. The sixteen-stage control
chain on the main screen is the same walk with the same names, numbered 0–15, and the
two agree exactly on 0–8. After that:
- Box 8 covers two stages. The diagram draws the Validation Agent and its
“checks against original words” as one box; the chain runs the agent (stage 8) and
the deterministic comparator (stage 9) separately, because one is a model and one
is code.
- The Banking Policy Check is one stage, not four. The dashed block holds
boxes 9–12. The chain runs the policy, risk and velocity checks as one stage
(index 10), because they read the same resolved draft and share one decision. It runs the
field-by-field confirmation as its own stage (index 11).
- So from chain index 11 on, the chain index is the diagram number minus one.
Nothing is missing and nothing is duplicated; the two are the same path at two
different resolutions. The Execution-path page shows it stage by stage.
One sentence in, one checked and audited payment out
AI proposes. Deterministic code decides. Auntie approves. Tencent Cloud
carries the voice. No model ever holds a key or touches the ledger.
①UNDERSTAND AND PROPOSE
the AI half
Auntie speaks
English · Mandarin
Bahasa Melayu · Singlish
Tencent Cloud ASR
Speech → text
asr.cloud.tencent.com
Guardrails
Scam + prompt-injection screen, backed by deterministic rules
5 agents
Guardrails · Intent
Payment · Validation
+ Personalization
Typed contracts
JSON proposals only. Never an account, never an approval
②DECIDE, AUTHORISE AND EXECUTE
the deterministic half
Check the proposal
- Typed-contract schema
- Matches the utterance
- Every intent kept
- Amount + currency
Resolve the details
- Verify beneficiary
- Verify source account
- Retrieve trusted data
- Resolve ambiguity
Apply DBS controls
- Policy + scam decision
- Risk / value threshold
- Velocity throttling
- Field-by-field confirm
Authorise + execute
- Lock the reviewed draft
- Mock FaceID, then signing
- Zero-trust gateway
- Mock ledger records
The five agents
Four agents sit in the payment control chain. The fifth, the
Personalization Agent, runs next to it and only changes how Joy asks. Each agent makes
one decision and returns typed JSON. None of them holds a signing key, a gateway
credential or ledger access.
What keeps the agents in check
- A model's output is data, never an instruction. Every agent returns JSON
that is parsed against a strict schema. Anything out of shape is rejected and the
turn falls back, so it can never be read as a command.
- The Validation Agent works on its own. At stage 8 it never sees the other
agents' reasoning. It reads the saved evidence and the draft, checks each field, and
returns PASS or FREEZE. Stage 9 is plain code that checks that answer again.
- If a model fails, the rules take over. If an agent times out or returns
something malformed, its deterministic mock answers instead. The answer can get
plainer, never less safe.
- Words cannot move money. No prompt, in any language, can produce a
signature, an approval or a ledger write, because those live in code the models
cannot call.
The sixteen-stage control chain, in order
Stages 3, 4, 5 and 8 are the only stages in the chain that call a model.
Everything else is code, and there is code before and after each model stage. The AI
stages can only propose; the code stages decide.
What is simulated and what is real
Two things in this build are mocks, and both are named on the screen.
Everything either side of them is the live system.
LIVE: running for real in this demo
- Speech recognition. Tencent Cloud ASR. Audio goes up, text comes back,
in English, Mandarin and Bahasa Melayu.
- Speech synthesis. Cartesia Sonic by default, with English, Mandarin and
Malay voices; TRTC Mode switches to Tencent Cloud TTS. The top bar shows which
engine is speaking, and says so if it ever has to fall back.
- The five agents. Real model calls over HTTPS when a model key is
configured, each with a typed JSON schema, a retry budget and a hard deadline.
Without a key, or if the model is unreachable, each agent runs its deterministic
mock instead, and its card says engine: mock. It never
silently pretends.
- The sixteen-stage control chain. Plain deterministic code. No model
can call into it, skip a stage, or reorder one.
- The audit ledger. Append-only and hash-chained: every event carries
the hash of the event before it, so editing history breaks the chain visibly.
- The payload signature. An HMAC-SHA256 signature over the canonical
payload, re-verified at the gateway before anything is allowed to execute. The
key is a server-held demo key, not a bank HSM.
- Guardrail screening. The Guardrails Agent's verdict is backed by code:
a scam-typology table (government impersonation, romance, tech support,
secrecy and more), amount, velocity and payee rules. Code can make the verdict
stricter, never looser.
SIMULATED: on purpose, and labelled
- Customer approval. A mock biometric adapter stands in for FaceID. It
returns the same verdict shape the real one would, and it is the only thing in
the system that can authorise a payment.
- The payment itself. A restricted mock capability writes to a synthetic
ledger. No money moves, no real account is touched, no bank system is
contacted.
- The customer. “Mei Ling” and “DBS Savings ending 1234” are synthetic
fixtures, not real customer data.
- Payees and balances. Fixtures, resolved through the same code path a
real directory would use, so the resolution logic is exercised rather than
bypassed.
Why the boundary sits exactly there: the biometric sensor and the
payment ledger are the two places a prototype cannot legally or practically reach.
Everything either side of that line is the real thing.
Requirements coverage 5 OF 5 DEMONSTRATED
The challenge's five criteria, in its own wording, mapped to the
scenario and stage that demonstrate each one. Rows light up as their scenario runs,
so this is live evidence about the turn on screen.
The execution path: from voice to a signed payment
Every stage from 09 onwards is deterministic code, and only stage 15
can record a payment.
The signed artifact is produced at stage 13. The gateway at stage 14
rebuilds it and does not trust anything it is told. The mock ledger at stage 15 is the only place a
payment is recorded.
PROPOSE voice in, typed contracts out
Voice
→
Tencent ASR
→
Orchestrator
→
4 chain agents
→
Typed contracts
may propose · never an approval
STAGES 9–12 deterministic
Resolve · validate · risk · confirm · biometric
no model decides these stages
STAGE 13 the signed artifact
canonical_payload() + sign_payload()
the signed artifact
One canonical serialisation of the reviewed draft, then signed
with HMAC-SHA256 using a server-held demo key (a bank would use an HSM). Everything downstream verifies against this, and nothing
downstream is allowed to trust a claim about it.
STAGE 14 zero-trust gateway
gateway_verify(): 14 checks
no agent may reach this
Re-derives the canonical payload and independently checks the
signature, the draft version, a complete confirmation, the validation result,
the risk policy, freeze and cancel status, nonce reuse and expiry.
A mismatch stops the payment here, after signing and before the ledger.
STAGE 15 restricted mock payment capability
Ledger.execute()
this demo
DBS Fund Transfer API
integration target, not built
OUTCOME after the ledger returns
LEDGER_RECORDED / LEDGER_NOT_INVOKED / SCHEDULE_HELD
Every outcome carries the ledger's transaction reference plus a
hash-chained, append-only audit record. The audit record is the version of
events that counts.
TODAY vs THE INTEGRATION. Stage 15 makes one call, Ledger.execute(),
which records the payment in a simulated in-memory ledger: no money moves, no real
account is touched, and the demo says so on screen. Connecting DBS means replacing
that one call with a DBS API client. It sits behind the same gateway, so
nothing else in the control chain has to change. That integration is not
built in this prototype.
DBS Fund Transfer API: the integration target
- Status
- Not built. From DBS's published API documentation;
this build records payments in the mock ledger and does not call DBS
- Base
- https://www.dbs.com/sandbox/api/sg/v1
- Auth
- a pre-issued token carried in the headers: clientId, accessToken,
uuid, accept-version
- Version
- accept-version: 1.0, to be pinned.
Unset means "latest", which would change behaviour the day DBS publishes a
new version
- Own accounts
- POST /transfers/toOwnAccounts · wrapper
fundTransferDetl. transferType
does not exist on this operation
- Saved payee
- POST /transfers/creditPayeeAccount · wrapper
fundTransferPayee, which carries
transferType, plus
purpose when the destination is another bank
- Amount
- a two-decimal string in every published sample
- Reference
- DBS takes no idempotency key, so the integration would
derive a stable reference from the draft and the gateway's idempotency key,
so that a retry cannot become a second payment
- Response
- 201 {referenceId, status: ACCEPTED};
errors arrive as an array of {code, message, moreInfo}
- Separate product
- bills and card payments are the Bill Payments API
(/billPayments), a different DBS product, not a
parameter on this one
Tencent Cloud: what is live
- Lighthouse
- ap-singapore. Hosts the page, the backend and the audit
DB
- ASR
- asr.cloud.tencent.com · signed WebSocket · two engines,
because no single one covers English + Chinese + Malay:
16k_zh_en (zh + en) and
16k_multi_lang (en + ms). The EN, 中文 and BM pills pick the engine
- TTS
- Cartesia Sonic by default (the only engine with a Malay voice).
TRTC Mode switches to Tencent Cloud TTS: TextToVoice, or TRTC
TextToSpeechSSE streaming when it is configured
- Model
- served from Tencent Cloud MaaS
(tokenhub-intl.tencentcloudmaas.com); /health reports which model is active,
rather than the page naming one that may have changed