Architecture v3 · AI layer · draft for review

AI Context Architecture

Which model calls the worker makes, what goes into each, how a router decides when a new call is worth making, and the checks that protect result quality.

What changed since v2

Data flow

SHARED ROLE LAYER · BUILT ONCE PER ROLE, REUSED BY EVERYONE S1 Gather role evidence Web search · strong model Runs once per role Batch-friendly for refreshes S2 Build role profile Stock duties, ratings, verdict Strong model · no tools Never sees user text Saved role profile MongoDB · dated, versioned Duty vectors stored with it Top roles reviewed by a person builds on the first request for a role, or on a scheduled refresh PERSONAL LAYER · PER USER Find the role Search + vector match Model only if unclear Compare duties Vector search per role No model call Route Per duty, then overall Code, choices logged Assemble result Band, gap, next steps Code over catalogs reads profile and duty vectors ONLY WHEN NEEDED (DASHED) P1 Match title Only if match unclear Mid model P2 Check duty Borderline duties only Mid model P3 Rate duties Far or new duties Strong model P4 Write-up When escalated or custom Strong model
The shared layer costs money per role, not per user. In the personal layer, code does the comparing and routing. A model is called only for the specific duties or situations the saved profile cannot answer. When the router asks questions, P3 asks them once and P4 runs after the answers. If a role has no profile yet, the first user starts the build and waits in the background, and later users get it instantly.

How the router decides

QuestionConditionAction
Is the role right?Search and vector score clearly favor one roleContinue with no call
Weak score or a close tieP1 asks one button question or confirms the match
Is each duty covered?Very close to a stock dutyReuse the saved rating
BorderlineP2 checks whether the two duties are equivalent for AI exposure. Reuse only if yes.
Far from every stock duty, or newP3 rates it with the strong model
Does the whole case need more care?Any trigger: match still unclear, thin evidence, many far or new duties, seniority or industry unlike the profile's typical case, ambiguity flagged by P1 or P3Full personal path: P3 may ask up to 3 questions once, then P4 writes the result
No triggerShared write-up, with the tools gap and next steps added by code

The similarity cutoffs and the "many far duties" limit are not guessed here. They come from the 30-role calibration set described below, and they start strict so that most duties go to a check or a fresh rating until the data shows reuse is safe. Typical paths: 0 calls when the user keeps the stock duties, 1 or 2 when they add their own, up to 4 when escalated. v1 made 6 on every analysis.

Vector search in Atlas

What is storedA role_duties collection with one document per stock duty (role id, text, tag, rating reference, embedding), and an embedding on each occupation for role matching.
How it is queriedDuty comparison uses a vector query filtered to the matched role id, so each user duty is compared with about 8 candidates. Role matching searches across all occupations.
User dutiesEmbedded at request time and stored on the user's own analysis, so deleting the analysis deletes the vectors. They are not compared with other users' text, and they never enter the shared layer.
One embedding modelPick it once, and store the model name and version next to every vector. If the model changes, all vectors are re-embedded, because mixing models gives meaningless scores. It should handle several languages.
Index detailsThe role id field is declared as a filter field in the vector index. Search indexes update a moment after a write, which is fine here because stock duties are written long before anyone queries them.
Embedding providerAtlas can generate embeddings with Voyage AI, and Anthropic points to Voyage for embeddings. Confirm current availability and price before choosing.

What stays in code, with no model

Role suggestionsAtlas Search autocomplete over the stored occupation list.
RoutingEvery decision in the router, with its inputs and the version of its thresholds written to the log.
Risk bandCounted from the duty ratings and shown as a band, not a precise score. The weights are written down and tested, and a single critical duty can raise the band by itself.
Tools gapEach duty has a tag such as writing or data analysis. Code compares tags with the user's tools and usage, and flags exposed duties where they use no AI.
Next stepsCatalog entries matched by duty tags and gaps, free options first, paid ones marked.
Abuse limits and spendPer-user counters, per-address limits and the monthly spend breaker.

What every prompt contains

1
Fixed product rulesThe honest-advisor principles for this product: no flattery and no alarm, state confidence, separate evidence from judgment, never invent statistics, links or course names, and "evidence is thin" is an allowed answer. Identical on every call and kept at the top so provider prompt caching can apply.
2
Task and output schemaWhat this call does and the exact JSON fields it must return, with the allowed values for each.
3
Data blockUser inputs and database records, wrapped and labeled as data. The prompt says nothing inside it is an instruction.
4
Tool results (S1 only)Web pages come back as untrusted data. S1 has no other tools and receives no user text.

What each call entails

S1

Gather role evidence

web search ×3strong modelshared
Runs when
A user picks a role with no fresh evidence, or on a scheduled refresh, most popular roles first. Another request for the same role waits instead of starting a second search.
Context sent
Rules and schema, the role name, today's date. Prefer primary sources such as labor statistics, academic studies and company reports. Report disagreement. Say when evidence is thin. No user text.
Returns
Findings, each tied to a source id. What has already happened in the role. Evidence strength: strong, mixed or thin.
Code checks
Source links are taken from the search tool's own results. A link the model writes that the tool did not return is dropped. Findings without a source are removed or shown as unsourced.
If it fails
Continue with evidence marked thin. The profile is built and says so, and the router treats thin evidence as an escalation trigger.
S2

Build role profile

strong modelno toolsshared
Runs when
Straight after S1 for the same role.
Context sent
Rules and schema, the role's stored tasks, the S1 evidence, the allowed duty tags and the related roles.
Returns
6 to 8 stock duties in plain wording, each with a tag, exposure (low, medium, high), status (not yet, emerging, already happening), what AI does today, what stays human and its evidence. Plus headline, summary, confidence with its reason, unknowns, a hedged timeframe and up to 3 adjacent roles.
Code checks
Schema and allowed values. Every duty has an allowed tag. Confidence cannot be higher than medium when evidence is thin. Adjacent roles must come from the supplied list. The most requested roles are held for a person to read before they are first shown. Each duty is then embedded and saved.
If it fails
Retry once. If it still fails, the user sees "we could not build this role yet" and the job can be resumed.
P1

Match the typed title

mid modelno tools
Runs when
The router finds the role match weak or tied.
Context sent
Rules and schema, the typed title and optional context as data, and the 8 closest candidate roles (id, title, short description).
Returns
match_id or null, confidence, up to 2 alternatives, one disambiguation question if needed.
Code checks
match_id must be one of the candidates. Low confidence shows the alternatives as buttons.
If it fails
Show the candidates as buttons.
P2

Check a borderline duty

mid modelno tools
Runs when
A user duty is neither close to a stock duty nor clearly far from all of them.
Context sent
Rules and schema, the role name, the user's duty as data, and the closest stock duty with its rating and reasoning.
Returns
Equivalent for AI exposure: yes, partly or no, with a one-line reason.
Code checks
Only "yes" reuses the stock rating. "Partly" and "no" go to P3.
If it fails
Send the duty to P3.
P3

Rate duties

strong modelno tools
Runs when
A duty is far from the stock duties, new, or failed the P2 check. It runs once for all such duties together.
Context sent
Rules and schema, the role name and evidence summary, several stock duties with their ratings as calibration examples, and the user's duties as data. When escalated, it also gets the user's tools, usage and context.
Returns
Per duty: tag, exposure, status, a one-line reason. When escalated, also up to 3 clarifying questions, asked once.
Code checks
One rating per duty, values from the allowed lists, tag from the list. At most 3 questions, and none if answers already exist. Code recomputes the risk band.
If it fails
Retry once, then mark the job failed. Retry resumes here.
P4

Personal write-up

strong modelno tools
Runs when
The router escalated the case, or the user's duties differ enough from the stock set that the shared write-up would not describe them. Otherwise the shared write-up is used.
Context sent
Rules and schema, all ratings, evidence strength, the user's answers, a summary of their tools and usage, and related roles.
Returns
A personal summary with confidence reason and unknowns, up to 3 adjacent roles and up to 5 practices to try, each tagged free or paid.
Code checks
Roles must come from the supplied list. Length limits. Low-risk results are framed as "build on your strengths", with no urgency.
If it fails
Retry once, then show the shared result with a note that the personal version is still coming.

Quality safeguards

Calibrate before launchRun about 30 roles with several user profiles each. For every duty pair, compare "reused rating" with "fresh rating" and set the cutoffs so they agree as often as you decide in advance. Choose that target before looking at results.
Start strict, loosen on evidenceBegin with high cutoffs so most duties are checked or rated fresh. Relax them only where the data shows reuse is safe.
Audit a sampleRe-rate a random 5% of reused ratings with the strong model and compare. A rise in disagreement raises an alert and tightens the cutoffs.
Log every routing choiceSimilarity score, outcome, cutoff version and model. This is how a bad result can be traced and fixed.
People review shared contentThe most requested roles are read by a person before they go live, and a profile that gets several "does not match my job" reports is paused.
Test on every changeThe same 30 roles run before any prompt, model or cutoff change. Large swings are read by a person before release.

Keeping calls down without lowering quality

Research per role, not per userThe most expensive call scales with the number of roles. A role nobody picks is never built.
Pre-build the popular rolesBuild the top 100 to 200 before launch so early users never wait. Others build on first request.
Reuse only when the data says it is safeThe router and the audit sample decide, not a fixed rule.
Identical inputs, identical answerA hash of the full input lets a repeated request return its saved result.
Batch for scheduled refreshesRefreshes are not urgent, so they can use the asynchronous batch interface, which is typically discounted. Check current pricing.
Prompt caching and short outputsFixed prompt parts sit at the top so provider caching can apply, and each call has an output size limit.
Abuse limits onlyPer-account and per-address limits stop farming. They are not a way to ration quality.
Spend breaker queues, not degradesAt the monthly limit, new requests that need a model wait in line and finish when budget returns. Cached results keep working.

Scale rules for launch day

Parallel jobsEach worker runs 10 to 20 jobs at once, because a job mostly waits on the model. More workers can be added.
Atomic job claimA job is claimed in one database operation, so two workers never take the same one.
Role build lockThe first request marks the role as "building". Other users with that role wait for it. Fifty people entering "nurse" cause one build.
Retry on rate limitsWhen the provider says slow down, the job waits and retries with a growing delay instead of failing.
Status pollingCheck every 3 seconds at first, then back off to 10 for long jobs.
Mostly cachedLaunch-day load is mainly role builds, and those are capped by the role lock.

Other gaps to close, most important first

  1. One wrong profile reaches everyone. A mistake in a shared result scales to every user of that role. Human review of the top roles, the report button and the pause rule above address this.
  2. Silent wrong reuse. A wrongly reused rating looks like a normal result. The audit sample and the routing log are the only way to see it, so they are not optional.
  3. Cutoffs drift when the model or prompts change. Recalibrate after any change to the embedding model, the rating prompt or the model version.
  4. Embedding model lock-in. Changing it means re-embedding everything. Store the model version with each vector from day one.
  5. Free-tier abuse. Bots could farm calls through new accounts. Use email verification, bot checks on registration, and limits per account and per network address.
  6. Stale results. Serve the saved profile with its date and refresh in the background when it expires. Every result shows "evidence as of" and the date.
  7. No cost visibility. A dashboard for cost per analysis, the share of duties reused, checked and rated, queue length and spend against budget, with an alert before the limit.
  8. Provider outage. Cached results keep working. Anything that needs a model shows "try again later" and does not use up the user's allowance.
  9. Evidence is global. The shared layer ignores country, so the result should say it reflects mostly global sources. Country only shapes the next steps.
  10. One language at first. Each extra language multiplies the saved profiles and the vectors. Start with English and decide on translation later.
  11. Optional growth idea. Saved profiles could become public pages per role at almost no cost, which helps search traffic. They would need the review step first and raise the stakes of a wrong verdict, so I would leave this until after launch.

To confirm before building

  1. Embedding model. Which one, and whether Atlas generates the embeddings or the worker does. It must handle several languages.
  2. Agreement target. How often a reused rating must match a fresh one before you allow reuse. Decide this before the calibration run.
  3. Abuse limits. Analyses per account per day and per network address.
  4. Spend breaker behavior. My default is to queue requests and finish them later. The alternative is to show cached results only.
  5. Pre-build list. How many roles to build before launch, and who reviews the first batch.
  6. Model per call. Strong for S1, S2, P3 and P4, mid for P1 and P2. Only move a call to a smaller model if the calibration run shows no loss.
  7. Evidence expiry. 30 days is my starting guess. Shorter is fresher and costs more.
  8. Search access. Confirm the web search tool is enabled on your Claude API account, and set a monthly spend limit.