Every Insights conversation on production was read. The questions were deduplicated, grouped into intent families by meaning, and measured on the goal-3 fixture with the goal-3 scorer. Raw questions and client names stay in the evaluator-only directory. This report holds only family names, counts and metrics.
Real questions are easier to execute and harder to get right than the adversarial holdout. First-turn execution errors fall from 54 to 60 per 100 on the holdout to 9 to 16 per 86 on the real sample for the same code. Correct first turns fall from 14 to 16 per 100 to 2 to 5 per 86. Most real answers are open listings, and the frozen scorer marks 29 to 48 of 101 attempts unverified because the product adds columns the question did not name. On the requested columns only (advisory), 10 to 22 of 101 answers agree with the expected answer per run.
Per arm on the real sample: original (r1: errors 16/86, correct 4/86, silent wrong 27/101; r2: errors 14/86, correct 5/86, silent wrong 29/101); landed (r1: errors 9/86, correct 2/86, silent wrong 24/101; r2: errors 10/86, correct 4/86, silent wrong 29/101); claude-opus-5-cli (r1: errors 3/86, correct 0/86, silent wrong 31/101); gpt-6-astra-codex (r1: errors 23/86, correct 1/86, silent wrong 22/101). The claude-opus-5-cli arm, the best first-turn arm on the holdout, executes almost everything on real questions (3 errors in 86 first turns) and gets none of them scored correct: 31 silent wrong, 21 refused by the calendar-semantics guard, 45 unverified. Fewer errors did not become more correct answers.
The envelope that covers 80% of real distinct questions has 24 families. 3 of them are fully ratified, 18 are partial, 3 are unsupported. Two rulings on Pipedrive people questions (default staff and house-account exclusion; the identifier of a Pipedrive-only person) block the verifier screen and the federated templates; both are in section 6a.
| Item | Count |
|---|---|
| Turns in the production store (2026-06-11 to 2026-09-07; the six-month window starts 2026-03-07 and the store is younger) | 4036 |
| Excluded: paraphrase-campaign conversations (automated adversarial campaign service) | 1909 |
| Excluded: evidence-pack conversations (agent-authored evidence runs) | 16 |
| Retained turns | 2111 |
| Retained first turns (conversations) | 1302 |
| Retained follow-up turns | 809 |
| Retained turns by the founder | 1943 |
| Retained turns by four other staff users | 168 |
| Retained turns in the blended vault (CT + Pipedrive) | 1561 |
| Retained turns in the CT-only vault | 550 |
| Same-day exact repeats inside the retained set (verification loops; collapsed by dedupe) | 835 |
| Retained turns within 30 minutes of an eval-oracle bench run (kept: the bench uses the query surface, not conversations) | 43 |
| Turns with founder feedback attached | 52 |
Recorded outcomes of the retained turns: success 1446, warn 2, clarify 360, error 291, blocked 12.
| Item | Count |
|---|---|
| Distinct questions after normalisation (case, spacing, trailing punctuation) | 997 |
| Distinct first-turn questions | 463 |
| Distinct follow-up questions (kept with their anchor) | 534 |
| Exact repeats collapsed | 1114 |
| Distinct first-turn questions asked on more than one day | 141 |
| Families found (in scope) | 52 (49) |
| Label confidence (high / medium / low) | 629 / 346 / 22 |
Families were assigned by meaning (measure, grouping, source leg, time semantics) by four independent labelers over a shared taxonomy; five families were added after the pass (revenue by acquisition source, signup cohort export, first-ever trading country, all-time totals, sales leads). Ranking uses distinct questions; retained turns and asking episodes (distinct question times day) are shown beside it.
Coverage: 11 families cover 50% of distinct questions, 24 cover 80%, 37 cover 95%. By retained turns: 10 / 24 / 38.
| # | Family | Name | Leg | Distinct questions | Turns | Episodes | Users | Support | Definitions needed |
|---|---|---|---|---|---|---|---|---|---|
| 1 | F10 | Quoted but not booked | ct | 98 | 223 | 121 | 3 | partial | client-requested quote = quotes.type by_client (house default, needs ratification); 'without booking' window when the quote window is explicit (same window vs lifetime) |
| 2 | F18 | Facebook-sourced client report | ct | 56 | 154 | 76 | 2 | partial | the fixed column set of the report (each ask lists a different set); 'most recent deal lost reason' per client via the person bridge (house default) |
| 3 | F25 | Professional affiliate performance | ct | 53 | 97 | 78 | 1 | partial | window placement for 'top affiliates in a period' (referral date vs trade date, house default); 'strong' / 'gone quiet' thresholds |
| 4 | F01 | Signups by acquisition source | ct | 51 | 170 | 91 | 3 | partial | the bare 'where did our clients come from' reading (multi-taxonomy rollup vs marketing source only) is a house default, not ratified; campaign = utm_campaign vs campaigns.token |
| 5 | F08 | Currency-book performance per active client | ct | 47 | 73 | 53 | 4 | partial | 'active client' inside a book question (activated vs traded in the period vs recently trading) is chip-clarified today; needs one ratified default; SAR book membership: sell side only vs either side (ratified either side; sellers/buyers as subsets) |
| 6 | F26 | Refer-and-earn programme | ct | 44 | 81 | 62 | 2 | partial | 'converted referral' (activated vs traded); payout ledger for ROI = reward_events vs user_commission_statements (house default) |
| 7 | F09 | Time from signup to first trade for a book or stream | ct | 41 | 94 | 62 | 2 | unsupported | 'sales stream' has no ratified definition (open: sales-stream-dimension); the israel / inbound / partnerships split is undefined; window placement (signup date vs first-trade date); mean vs median default |
| 8 | F34 | Revenue by acquisition source or campaign | ct | 30 | 42 | 37 | 1 | partial | attribution rule: revenue of clients acquired through the source in the window vs revenue earned in the window from clients ever acquired through it; campaign identity (utm_campaign vs campaigns.token) |
| 9 | F17 | Signup quality by source or country | ct | 28 | 51 | 47 | 2 | partial | per-signup revenue = realized commission of the cohort divided by signups (house); hear-about-us answers are free text (no ratified grouping) |
| 10 | F14 | Activation rate | ct | 28 | 45 | 39 | 3 | partial | 'activation rate' default numerator: verified broker (canon) vs traded (trade-based) when the question says 'go on to trade' (documented rule, needs ratification of the bare phrase); cohort-retention three-column shape |
| 11 | F13 | Quiet high-value clients | ct | 27 | 65 | 43 | 1 | ratified | - |
| 12 | F19 | Top clients by revenue | ct | 27 | 49 | 40 | 3 | partial | margin measure is open (margin-measure); revenue concentration share shape |
| 13 | F05 | Commission by stream, relationship manager or client type | ct | 27 | 39 | 34 | 1 | partial | 'sales stream' (open) when the ask says stream rather than corporate vs private |
| 14 | F21 | Named client lookup | ct | 26 | 64 | 26 | 1 | partial | named-entity matching rule (ILIKE contains) is a house default; 'lifetime realized commission' vs booked |
| 15 | F07 | SAR book run-rate inside the month | ct | 25 | 49 | 28 | 1 | unsupported | day-of-month band definition (25th to month end; 1-7, 8-14 ...) and the averaging rule across months; SAR sell side vs either side (chip-clarified today); revenue reading for run-rate (realized vs booked) |
| 16 | F23 | Bookings by payment partner or platform | ct | 25 | 39 | 26 | 1 | partial | share denominator (trade count vs commission vs volume) when unstated; broker naming (brokers.name) |
| 17 | X03 | Lost reasons crossed with client profile | fed | 24 | 44 | 31 | 1 | partial | client attribute source for a lost deal (declared intent vs traded); one reason label match rule (lost_reason_norm literal) |
| 18 | F29 | Client lifespan and average revenue per active client | ct | 24 | 26 | 24 | 3 | unsupported | 'client lifespan' (first to last trade) and 'active client' for the denominator have no ratified definition; average trade size = GBP-normalised sell amount (ratified) but per-client vs per-trade averaging is open |
| 19 | F11 | Quote volume and quote-to-trade conversion | ct | 23 | 59 | 37 | 1 | partial | quote-to-trade conversion definition (booked quote via quote_id vs any booking after the quote); week boundaries |
| 20 | F22 | Commission by currency corridor | ct | 20 | 31 | 30 | 1 | partial | margin in basis points (open) |
| 21 | F31 | Client listing with contact details | ct | 17 | 33 | 19 | 3 | partial | contact fields on the gold leg are NULL placeholders; email export needs the CT leg (house) |
| 22 | F04 | Commission by month | ct | 16 | 25 | 20 | 1 | ratified | - |
| 23 | F20 | Trades today or recent trades | ct | 16 | 22 | 17 | 1 | ratified | - |
| 24 | F06 | Daily booked commission and month-to-date pace | ct | 15 | 29 | 22 | 1 | partial | zero-fill of days without trades and the pace baseline (average of the last 3 months) are house defaults |
| 25 | F12 | Churn tier classification | ct | 14 | 46 | 31 | 1 | ratified | - |
| 26 | P05 | Sales activity volume and mix | pd | 14 | 42 | 22 | 1 | partial | 'prospect' (open deal vs person without CT link) and 'no reply' (email direction) |
| 27 | F16 | Signup counts by period, client type or stream | ct | 13 | 23 | 17 | 2 | ratified | - |
| 28 | O01 | Not in the connected data | none | 13 | 19 | 13 | 3 | unsupported | not in the connected data (app analytics) |
| 29 | F02 | Signups by country or nationality | ct | 12 | 26 | 19 | 2 | partial | nationality vs registered country (clients.data nationality is a house fact) |
| 30 | P03 | Pipeline status and open deals | pd | 12 | 20 | 15 | 2 | partial | 'stalled' threshold column (days_in_stage is thin); 'sales leads' in Pipedrive wording vs the CT allocatable definition |
| 31 | F24 | Period-over-period comparison | ct | 11 | 24 | 20 | 1 | partial | 'dropped more than 20%' comparison basis (quarter to date vs full prior quarter) |
| 32 | F33 | First-time traders in a month | ct | 11 | 13 | 11 | 1 | ratified | - |
| 33 | P01 | Deal loss reasons ranking | pd | 10 | 46 | 17 | 1 | ratified | - |
| 34 | F27 | Corporate accounts sent to brokers | ct | 10 | 16 | 11 | 1 | ratified | - |
| 35 | P02 | Lost deals listing for one reason | pd | 10 | 15 | 11 | 1 | ratified | - |
| 36 | F03 | Commission total for a period | ct | 8 | 42 | 17 | 2 | ratified | - |
| 37 | X01 | Sales activity per trading client | fed | 8 | 30 | 12 | 1 | partial | 'books' = clients vs relationship-manager books; 'calls' key list (call, aircall) |
| 38 | X05 | Trading clients with sales contact recency | fed | 8 | 15 | 12 | 1 | partial | 'last communication' source (activities vs mail rollups vs notes) |
| 39 | F30 | Declared currency versus traded currency | ct | 8 | 11 | 11 | 1 | partial | 'differs' rule: primary declared vs top traded, or any declared vs any traded |
| 40 | F28 | CFX or PFX monthly funnel | ct | 7 | 10 | 7 | 1 | partial | stream = CFX/PFX by clients.type at signup time vs now |
| 41 | O03 | Instruction or definition, not a question | none | 7 | 7 | 7 | 1 | unsupported | - |
| 42 | O02 | Security probe | none | 6 | 7 | 6 | 1 | unsupported | - |
| 43 | F15 | Signups stuck before activation | ct | 4 | 23 | 13 | 1 | partial | 'stuck' = no completed registration vs no activation |
| 44 | X02 | Deals versus trading reconciliation | fed | 4 | 16 | 11 | 1 | ratified | - |
| 45 | F32 | Revenue share by country | ct | 4 | 11 | 8 | 1 | partial | 'registered country' vs nationality; share denominator |
| 46 | P04 | Win and loss rate by owner or industry | pd | 3 | 14 | 9 | 1 | ratified | - |
| 47 | F35 | Signup cohort export with activation, revenue and lost reason | fed | 3 | 8 | 5 | 1 | partial | fixed column list for the export |
| 48 | P06 | Recent notes for a named organisation | pd | 3 | 6 | 5 | 1 | partial | organisation name match rule |
| 49 | F38 | Sales lead counts | ct | 2 | 4 | 2 | 1 | ratified | - |
| 50 | F37 | All-time platform totals | ct | 2 | 2 | 2 | 1 | partial | 'value transacted' = GBP sell volume |
| 51 | X04 | Tagged organisations revenue and pipeline | fed | 1 | 7 | 4 | 1 | partial | organisation label 'VIP' field (org_labels vs cf_client_type_tags); org to client aggregation rule |
| 52 | F36 | Countries with a first-ever trading client | ct | 1 | 4 | 4 | 1 | partial | 'first-ever trading client' per country |
Legs: ct = CT-only ledger; pd = Pipedrive-only; fed = needs both; none = not answerable from the connected data.
Independent expected SQL was written for 85 first-turn questions and 5 three-turn conversations (100 attempts) from the ratified definitions and house defaults, without reading any generator output. Every statement was executed on the read-only goal-3 fixture: 101/101 execute, 11 return an empty checked membership on the synthetic data. The sample covers 44 of 52 families and 86.7% of distinct in-scope questions by family volume (88.5% of turns). Families that the fixture cannot represent (currency books, named clients, deal owners, organisation notes, VIP tags) were written against the real read-only sources instead: see section 3a.
24 sampled questions are ambiguous under the ratified definitions: the expected SQL uses the more likely reading, the case accepts a clarification as justified, and both readings are listed in section 6 for the founder. 3 questions expect a clarification because no reading can be written (sales stream), and 2 expect a refusal (a security probe; app analytics that are not connected).
| Family | Source | Executed | Rows | Reading |
|---|---|---|---|---|
| F08 | ct | yes | 1 | SAR book = clients with a non-cancelled SAR trade (sell or buy) in the last 12 months; active = activated (>=1 verified broker); comms = settled commission by completed_at in the window |
| F07 | ct | yes | 1 | sell side, actually traded, booked commission (bare run-rate reading is open); per month: commission on SAR-sell trades traded on days 25..end divided by the number of those days; then the 12-month average |
| F21 | ct | yes | 4 | profile for the four user ids: name, client type, email, last booked trade date, lifetime realized (settled) commission, days since last trade |
| P04 | gold | yes | 20 | win rate = won/(won+lost) over deals closed this year (won_time or lost_time); top loss reason = most frequent lost_reason_norm per owner |
| X04 | gold | yes | 36 | VIP = org_labels contains 'VIP' (or cf_client_type_tags); realized revenue = sum of client_360.revenue_gbp_settled over the org's persons' CT clients; open pipeline = sum of open non-archived deal pipeline_value |
| F23 | ct | yes | 90 | payment partner = brokers.name through broker_accounts (verified rule); share of booking count per month; last 12 calendar months |
| F38 | ct | yes | 1 | allocatable accounts created in the last 12 London calendar months |
| F03 | ct | yes | 1 | booked commission (ratified) for the last calendar month with the CT stats exclusions |
| P06 | gold | yes | 5 | organisations whose name contains the asked name; five most recent notes attached to the org, its persons or its deals (rewritten with pre-resolved id sets) |
Real-usage denominators: 101 attempts, 91 first turns of which 86 supported, 5 conversations of three turns. Adversarial denominators: 250 attempts, 100 supported first turns, 50 conversations of four to five turns. Same scorer, same outcome classes.
| Measure | real-usage: original r1 | real-usage: original r2 | real-usage: landed r1 | real-usage: landed r2 | real-usage: claude-opus-5-cli r1 | real-usage: gpt-6-astra-codex r1 | adversarial: original e479e81 | adversarial: landed insights/eval-and-invariants | adversarial: gpt-6-astra-codex (Goal B2 winner by the precision rule) | adversarial: claude-opus-5-cli (Goal B2, best first-turn arm, no verdict) | adversarial: control noise floor (Goal B, 3 repeats of the original code) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| First-turn execution errors | 16/86 (18.6%) | 14/86 (16.3%) | 9/86 (10.5%) | 10/86 (11.6%) | 3/86 (3.5%) | 23/86 (26.7%) | 60/100 (60.0%) | 54/100 (54.0%) | 73.0 [71 to 75] | 38.7 [37 to 41] | 55.7 [54 to 57] |
| Correct first turns | 4/86 (4.7%) | 5/86 (5.8%) | 2/86 (2.3%) | 4/86 (4.7%) | 0/86 (0.0%) | 1/86 (1.2%) | 14/100 (14.0%) | 16/100 (16.0%) | 12.7 [11 to 14] | 26.0 [25 to 27] | 15.0 [14 to 16] |
| Precision = correct / (correct + silent wrong) | 4/31 (12.9%) | 5/34 (14.7%) | 3/27 (11.1%) | 5/34 (14.7%) | 0/31 (0.0%) | 2/24 (8.3%) | 36/67 (53.7%) | 37/69 (53.6%) | 75.5% [72.2% to 80.0%] | 55.3% [54.1% to 56.0%] | 50.8% [45.9% to 54.9%] |
| Silent wrong / attempts | 27/101 | 29/101 | 24/101 | 29/101 | 31/101 | 22/101 | 31/250 | 32/250 | 8.7 [7 to 10] | 49.7 [48 to 51] | 34.3 [31 to 40] |
| Full conversations correct | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 | 1/50 | 4/50 | 2.0 [2 to 2] | 5.3 [5 to 6] | 2.3 [2 to 3] |
| Refusals / attempts | 2/101 | 2/101 | 18/101 | 19/101 | 21/101 | 18/101 | 4/250 | 17/250 | 8.0 [7 to 9] | 13.0 [12 to 14] | 14.0 [11 to 17] |
| Comparison unverified / attempts | 48/101 | 48/101 | 45/101 | 34/101 | 45/101 | 29/101 | 18/250 | 12/250 | 15.0 [15 to 15] | 12.0 [11 to 13] | 13.0 [10 to 15] |
Generator arms: the real-usage runs labelled original and landed use the goal-3 control generator (glm-5.2 requested through the zai coding endpoint, which serves glm-5.3). The gpt-6-astra-codex arm is the Goal B2 winner under the pre-registered rule (precision above 70% with the lower paired bound above the control; it does not halve first-turn errors and errs more often). claude-opus-5-cli is added as the arm with the most correct first turns on the holdout; it has no verdict. Both CLI arms run on the Goal B2 code snapshot, which is the landed branch plus the benchmark adapters.
The claude-opus-5-cli real-usage run used the same Claude Code CLI adapter as Goal B2 with one difference: subscription account slot 5 instead of the registry's slot 3, on the owner's instruction (the slot selects the billing account, not the model). The main checkout's adapter file changed after the Goal B2 freeze (a token-recovery fix), so the bench runtime refused to start from it; the run used the worktree copy of the runtime with the adapter-byte check pointed at the worktree source, whose adapter bytes equal the frozen snapshot's. The registry copy and the runtime hashes are in the run receipt and admission record.
Outcome counts per run: original r1: comparison_unverified 48, correct_completed 4, refusal_unverified 2, silent_wrong 27, task_error 20; original r2: comparison_unverified 48, correct_completed 5, refusal_unverified 2, silent_wrong 29, task_error 17; landed r1: comparison_unverified 45, correct_completed 3, refusal_unverified 18, silent_wrong 24, task_error 11; landed r2: comparison_unverified 34, correct_completed 5, refusal_unverified 19, silent_wrong 29, task_error 14; claude-opus-5-cli r1: comparison_unverified 45, refusal_unverified 21, silent_wrong 31, task_error 4; gpt-6-astra-codex r1: comparison_unverified 29, correct_completed 2, refusal_unverified 18, silent_wrong 22, task_error 30.
The frozen scorer marks an answer unverified when it carries columns the question did not ask for, or when a requested column has an unknown name. Real questions are open listings and the product adds context columns by design, so this advisory pass re-compares only the requested columns (same value comparison, extras ignored, a single leftover column mapped positionally). It never changes the scored outcome.
| Arm | Attempts | Agree on requested columns | Wrong values or row count | Requested column not found | Error | Refused or clarified | First turns agreeing | First turns wrong | First turns in error |
|---|---|---|---|---|---|---|---|---|---|
| original r1 | 101 | 20 | 37 | 19 | 20 | 2 | 16/86 | 36/86 | 16/86 |
| original r2 | 101 | 22 | 40 | 17 | 17 | 2 | 18/86 | 37/86 | 14/86 |
| landed r1 | 101 | 14 | 36 | 19 | 11 | 18 | 11/86 | 34/86 | 9/86 |
| landed r2 | 101 | 13 | 39 | 13 | 14 | 19 | 11/86 | 34/86 | 10/86 |
| claude-opus-5-cli r1 | 101 | 12 | 39 | 20 | 4 | 21 | 11/86 | 37/86 | 3/86 |
| gpt-6-astra-codex r1 | 101 | 10 | 28 | 13 | 30 | 18 | 8/86 | 26/86 | 23/86 |
| Arm | Refusal guards | Error classes |
|---|---|---|
| original r1 | - 2 | execution 7, semantic 6, schema 5, validation 2 |
| original r2 | - 2 | execution 5, schema 5, semantic 4, validation 3 |
| landed r1 | CalendarSemanticsError 14, NativeDealUnitError 2, - 1, FactDateScopeError 1 | semantic 5, sql 4, schema 2 |
| landed r2 | CalendarSemanticsError 15, NativeDealUnitError 2, - 1, FanOutSumError 1 | sql 6, schema 5, semantic 3 |
| claude-opus-5-cli r1 | CalendarSemanticsError 19, - 1, FactDateScopeError 1 | semantic 2, schema 1, sql 1 |
| gpt-6-astra-codex r1 | CalendarSemanticsError 17, - 1 | semantic 30 |
On the landed branch, the calendar-semantics guard refuses generated SQL whose timestamp expression it cannot type ('unverified calendar semantics: unsupported timestamp expression type or lineage'); the Pipedrive money guard refuses a lost-pipeline-value sum without a currency partition; one federated plan refuses a Facebook cohort listing that mixes Pipedrive activity with a CT filter. On the original code the same questions mostly ran and returned a value.
A single run on 101 attempts sits inside the provider noise floor (claude-opus-5-cli, gpt-6-astra-codex have one run each; their figures are single-run). The noise references below are the arms run twice.
Noise reference: the original code run twice on the same corpus gives the same outcome on 87/101 cases; 3 cases are correct in both repeats (correct in r1: 4, in r2: 5).
Noise reference: the landed code run twice on the same corpus gives the same outcome on 76/101 cases; 2 cases are correct in both repeats (correct in r1: 3, in r2: 5).
| Arm | Original correct retained | Correct to wrong | Correct to error | Gained correct |
|---|---|---|---|---|
| landed r1 | 1/4 | 0 | 0 | 2 |
| landed r2 | 2/4 | 0 | 0 | 3 |
| claude-opus-5-cli r1 | 0/4 | 1 | 0 | 0 |
| gpt-6-astra-codex r1 | 0/4 | 2 | 2 | 2 |
| Family | Name | original r1 (correct / wrong / error / refused or clarified / unverified) | original r2 (correct / wrong / error / refused or clarified / unverified) | landed r1 (correct / wrong / error / refused or clarified / unverified) | landed r2 (correct / wrong / error / refused or clarified / unverified) | claude-opus-5-cli r1 (correct / wrong / error / refused or clarified / unverified) | gpt-6-astra-codex r1 (correct / wrong / error / refused or clarified / unverified) |
|---|---|---|---|---|---|---|---|
| F10 | Quoted but not booked | 2 / 0 / 0 / 0 / 6 of 8 | 2 / 0 / 0 / 0 / 6 of 8 | 1 / 0 / 0 / 1 / 6 of 8 | 2 / 2 / 0 / 2 / 2 of 8 | 0 / 0 / 0 / 1 / 7 of 8 | 0 / 2 / 6 / 0 / 0 of 8 |
| F18 | Facebook-sourced client report | 0 / 1 / 0 / 1 / 0 of 2 | 0 / 1 / 0 / 1 / 0 of 2 | 0 / 1 / 0 / 1 / 0 of 2 | 0 / 1 / 0 / 1 / 0 of 2 | 0 / 1 / 0 / 1 / 0 of 2 | 0 / 0 / 0 / 2 / 0 of 2 |
| F25 | Professional affiliate performance | 0 / 0 / 0 / 1 / 2 of 3 | 0 / 0 / 0 / 0 / 3 of 3 | 0 / 0 / 0 / 1 / 2 of 3 | 0 / 1 / 0 / 1 / 1 of 3 | 0 / 0 / 0 / 3 / 0 of 3 | 0 / 0 / 1 / 0 / 2 of 3 |
| F01 | Signups by acquisition source | 0 / 2 / 0 / 0 / 6 of 8 | 0 / 2 / 0 / 0 / 6 of 8 | 0 / 1 / 0 / 2 / 5 of 8 | 0 / 0 / 0 / 2 / 6 of 8 | 0 / 1 / 0 / 7 / 0 of 8 | 0 / 1 / 0 / 2 / 5 of 8 |
| F26 | Refer-and-earn programme | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 0 / 1 / 1 of 2 | 0 / 0 / 0 / 0 / 2 of 2 |
| F09 | Time from signup to first trade for a book or stream | 0 / 0 / 0 / 0 / 3 of 3 | 0 / 0 / 0 / 0 / 3 of 3 | 0 / 0 / 0 / 0 / 3 of 3 | 0 / 0 / 0 / 0 / 3 of 3 | 0 / 0 / 0 / 0 / 3 of 3 | 0 / 1 / 0 / 1 / 1 of 3 |
| F34 | Revenue by acquisition source or campaign | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 0 / 1 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 2 / 0 / 0 / 0 of 2 |
| F17 | Signup quality by source or country | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 1 / 0 / 1 / 0 of 2 | 0 / 1 / 0 / 1 / 0 of 2 |
| F14 | Activation rate | 0 / 0 / 0 / 0 / 3 of 3 | 0 / 0 / 0 / 0 / 3 of 3 | 0 / 0 / 0 / 0 / 3 of 3 | 0 / 1 / 0 / 0 / 2 of 3 | 0 / 1 / 0 / 0 / 2 of 3 | 0 / 0 / 0 / 1 / 2 of 3 |
| F13 | Quiet high-value clients | 0 / 2 / 0 / 0 / 3 of 5 | 0 / 4 / 1 / 0 / 0 of 5 | 0 / 3 / 0 / 0 / 2 of 5 | 0 / 1 / 2 / 2 / 0 of 5 | 0 / 3 / 0 / 0 / 2 of 5 | 0 / 1 / 2 / 2 / 0 of 5 |
| F19 | Top clients by revenue | 0 / 0 / 2 / 0 / 5 of 7 | 0 / 1 / 0 / 0 / 6 of 7 | 0 / 1 / 0 / 3 / 3 of 7 | 0 / 2 / 0 / 3 / 2 of 7 | 0 / 1 / 1 / 4 / 1 of 7 | 0 / 1 / 1 / 3 / 2 of 7 |
| F05 | Commission by stream, relationship manager or client type | 0 / 2 / 0 / 0 / 0 of 2 | 0 / 2 / 0 / 0 / 0 of 2 | 0 / 2 / 0 / 0 / 0 of 2 | 0 / 2 / 0 / 0 / 0 of 2 | 0 / 2 / 0 / 0 / 0 of 2 | 0 / 1 / 0 / 0 / 1 of 2 |
| F23 | Bookings by payment partner or platform | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 1 / 0 / 0 / 0 of 1 |
| X03 | Lost reasons crossed with client profile | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 |
| F29 | Client lifespan and average revenue per active client | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 |
| F11 | Quote volume and quote-to-trade conversion | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 |
| F22 | Commission by currency corridor | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 0 / 0 / 0 / 2 of 2 |
| F31 | Client listing with contact details | 0 / 0 / 2 / 0 / 0 of 2 | 0 / 1 / 1 / 0 / 0 of 2 | 0 / 1 / 1 / 0 / 0 of 2 | 0 / 1 / 1 / 0 / 0 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 2 / 0 / 0 of 2 |
| F04 | Commission by month | 0 / 2 / 0 / 0 / 0 of 2 | 0 / 2 / 0 / 0 / 0 of 2 | 0 / 1 / 0 / 1 / 0 of 2 | 0 / 1 / 0 / 1 / 0 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 |
| F20 | Trades today or recent trades | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 |
| F06 | Daily booked commission and month-to-date pace | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 0 / 0 / 1 / 1 of 2 |
| F12 | Churn tier classification | 0 / 1 / 1 / 0 / 1 of 3 | 0 / 1 / 1 / 0 / 1 of 3 | 0 / 1 / 0 / 1 / 1 of 3 | 0 / 1 / 0 / 1 / 1 of 3 | 0 / 1 / 0 / 0 / 2 of 3 | 0 / 1 / 0 / 0 / 2 of 3 |
| P05 | Sales activity volume and mix | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 |
| F16 | Signup counts by period, client type or stream | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 0 / 0 / 0 / 1 of 1 |
| O01 | Not in the connected data | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 |
| F02 | Signups by country or nationality | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 |
| P03 | Pipeline status and open deals | 0 / 0 / 2 / 0 / 0 of 2 | 0 / 0 / 2 / 0 / 0 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 1 / 1 / 0 / 0 / 0 of 2 | 0 / 0 / 0 / 0 / 2 of 2 | 0 / 0 / 2 / 0 / 0 of 2 |
| F24 | Period-over-period comparison | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 0 / 0 / 1 of 1 |
| F33 | First-time traders in a month | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 0 / 1 / 0 of 1 |
| P01 | Deal loss reasons ranking | 1 / 0 / 0 / 0 / 1 of 2 | 1 / 0 / 0 / 0 / 1 of 2 | 0 / 0 / 0 / 2 / 0 of 2 | 0 / 0 / 0 / 2 / 0 of 2 | 0 / 2 / 0 / 0 / 0 of 2 | 0 / 0 / 2 / 0 / 0 of 2 |
| F27 | Corporate accounts sent to brokers | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 1 / 0 / 1 of 2 |
| P02 | Lost deals listing for one reason | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 1 / 0 / 0 of 1 |
| F03 | Commission total for a period | 1 / 4 / 0 / 0 / 2 of 7 | 1 / 4 / 0 / 0 / 2 of 7 | 2 / 4 / 0 / 0 / 1 of 7 | 2 / 4 / 0 / 0 / 1 of 7 | 0 / 0 / 0 / 1 / 6 of 7 | 1 / 5 / 0 / 0 / 1 of 7 |
| X01 | Sales activity per trading client | 0 / 0 / 2 / 0 / 0 of 2 | 0 / 0 / 1 / 1 / 0 of 2 | 0 / 1 / 1 / 0 / 0 of 2 | 0 / 1 / 1 / 0 / 0 of 2 | 0 / 1 / 0 / 0 / 1 of 2 | 0 / 0 / 2 / 0 / 0 of 2 |
| X05 | Trading clients with sales contact recency | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 |
| F30 | Declared currency versus traded currency | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 |
| F28 | CFX or PFX monthly funnel | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 |
| O02 | Security probe | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 |
| F15 | Signups stuck before activation | 0 / 2 / 0 / 0 / 1 of 3 | 0 / 2 / 0 / 0 / 1 of 3 | 0 / 2 / 0 / 0 / 1 of 3 | 0 / 2 / 0 / 0 / 1 of 3 | 0 / 1 / 0 / 0 / 2 of 3 | 0 / 1 / 0 / 2 / 0 of 3 |
| X02 | Deals versus trading reconciliation | 0 / 0 / 2 / 0 / 0 of 2 | 0 / 0 / 2 / 0 / 0 of 2 | 0 / 0 / 2 / 0 / 0 of 2 | 0 / 0 / 2 / 0 / 0 of 2 | 0 / 1 / 1 / 0 / 0 of 2 | 0 / 0 / 2 / 0 / 0 of 2 |
| F32 | Revenue share by country | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 |
| F35 | Signup cohort export with activation, revenue and lost reason | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 1 / 0 of 1 |
| F38 | Sales lead counts | 0 / 1 / 0 / 0 / 0 of 1 | 1 / 0 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 0 / 1 / 0 / 0 / 0 of 1 | 1 / 0 / 0 / 0 / 0 of 1 |
| F37 | All-time platform totals | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 | 0 / 0 / 1 / 0 / 0 of 1 |
| F36 | Countries with a first-ever trading client | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 0 / 1 / 0 of 1 | 0 / 0 / 0 / 0 / 1 of 1 | 0 / 0 / 0 / 1 / 0 of 1 |
The smallest family set that covers at least 80% of real distinct questions has 24 families (they also cover 77.2% of turns). Each row states the definition status, whether one parameterised template is feasible, what the fixture can represent, the measured correctness, and what one verified template needs.
| # | Family | Name | Leg | Distinct questions | Definition status | Template | Fixture | Current correctness | One verified template needs |
|---|---|---|---|---|---|---|---|---|---|
| 1 | F10 | Quoted but not booked | ct | 98 | partial | yes: count or listing over quotes by_client with a NOT EXISTS booking gate; parameters = quote window, booking window, threshold | yes | original r1: 2/8 correct, 0 wrong, 0 error; original r2: 2/8 correct, 0 wrong, 0 error; landed r1: 1/8 correct, 0 wrong, 0 error; landed r2: 2/8 correct, 2 wrong, 0 error; claude-opus-5-cli r1: 0/8 correct, 0 wrong, 0 error; gpt-6-astra-codex r1: 0/8 correct, 2 wrong, 6 error | client-requested quote = quotes.type by_client (house default, needs ratification); 'without booking' window when the quote window is explicit (same window vs lifetime) |
| 2 | F18 | Facebook-sourced client report | ct | 56 | partial | yes: one parameterised per-client export with a fixed column list; the ask must select from that list | partial: 16 facebook-sourced accounts, no owner names, no lost reason text | original r1: 0/2 correct, 1 wrong, 0 error; original r2: 0/2 correct, 1 wrong, 0 error; landed r1: 0/2 correct, 1 wrong, 0 error; landed r2: 0/2 correct, 1 wrong, 0 error; claude-opus-5-cli r1: 0/2 correct, 1 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 0 wrong, 0 error | the fixed column set of the report (each ask lists a different set); 'most recent deal lost reason' per client via the person bridge (house default) |
| 3 | F25 | Professional affiliate performance | ct | 53 | partial | yes: per-referrer rollup (signups, activated, traded, referred commission) with a window on referral date | partial: 4 professional referrers, 33 referred accounts, no names | original r1: 0/3 correct, 0 wrong, 0 error; original r2: 0/3 correct, 0 wrong, 0 error; landed r1: 0/3 correct, 0 wrong, 0 error; landed r2: 0/3 correct, 1 wrong, 0 error; claude-opus-5-cli r1: 0/3 correct, 0 wrong, 0 error; gpt-6-astra-codex r1: 0/3 correct, 0 wrong, 1 error | window placement for 'top affiliates in a period' (referral date vs trade date, house default); 'strong' / 'gone quiet' thresholds |
| 4 | F01 | Signups by acquisition source | ct | 51 | partial | yes: one rollup per acquisition dimension; the bare question needs one ratified default dimension | yes | original r1: 0/8 correct, 2 wrong, 0 error; original r2: 0/8 correct, 2 wrong, 0 error; landed r1: 0/8 correct, 1 wrong, 0 error; landed r2: 0/8 correct, 0 wrong, 0 error; claude-opus-5-cli r1: 0/8 correct, 1 wrong, 0 error; gpt-6-astra-codex r1: 0/8 correct, 1 wrong, 0 error | the bare 'where did our clients come from' reading (multi-taxonomy rollup vs marketing source only) is a house default, not ratified; campaign = utm_campaign vs campaigns.token |
| 5 | F08 | Currency-book performance per active client | ct | 47 | partial | yes once 'active' is ratified: revenue and trades per active client of a book over a window | no: the fixture has no SAR or ILS trades | original r1: not sampled; original r2: not sampled; landed r1: not sampled; landed r2: not sampled; claude-opus-5-cli r1: not sampled; gpt-6-astra-codex r1: not sampled | 'active client' inside a book question (activated vs traded in the period vs recently trading) is chip-clarified today; needs one ratified default; SAR book membership: sell side only vs either side (ratified either side; sellers/buyers as subsets) |
| 6 | F26 | Refer-and-earn programme | ct | 44 | partial | yes: monthly programme rollup (referrers, referred signups, converted, payouts, referred commission) | partial: 11 refer-and-earn referred accounts, 8 reward events | original r1: 0/2 correct, 0 wrong, 0 error; original r2: 0/2 correct, 1 wrong, 0 error; landed r1: 0/2 correct, 0 wrong, 0 error; landed r2: 0/2 correct, 1 wrong, 0 error; claude-opus-5-cli r1: 0/2 correct, 0 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 0 wrong, 0 error | 'converted referral' (activated vs traded); payout ledger for ROI = reward_events vs user_commission_statements (house default) |
| 7 | F09 | Time from signup to first trade for a book or stream | ct | 41 | unsupported | after ratification: median days signup to first trade by a ratified stream column | partial: signup and trade dates exist but the synthetic dates put many first trades before signup | original r1: 0/3 correct, 0 wrong, 0 error; original r2: 0/3 correct, 0 wrong, 0 error; landed r1: 0/3 correct, 0 wrong, 0 error; landed r2: 0/3 correct, 0 wrong, 0 error; claude-opus-5-cli r1: 0/3 correct, 0 wrong, 0 error; gpt-6-astra-codex r1: 0/3 correct, 1 wrong, 0 error | 'sales stream' has no ratified definition (open: sales-stream-dimension); the israel / inbound / partnerships split is undefined; window placement (signup date vs first-trade date); mean vs median default |
| 8 | F34 | Revenue by acquisition source or campaign | ct | 30 | partial | yes: commission SUM grouped by one acquisition dimension with a ratified window placement | yes | original r1: 0/2 correct, 1 wrong, 0 error; original r2: 0/2 correct, 1 wrong, 0 error; landed r1: 0/2 correct, 0 wrong, 0 error; landed r2: 0/2 correct, 1 wrong, 0 error; claude-opus-5-cli r1: 0/2 correct, 1 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 2 wrong, 0 error | attribution rule: revenue of clients acquired through the source in the window vs revenue earned in the window from clients ever acquired through it; campaign identity (utm_campaign vs campaigns.token) |
| 9 | F17 | Signup quality by source or country | ct | 28 | partial | yes: cohort table by source or country with signups, activation rate, revenue per signup | yes (sources bing, facebook, newsletter; countries FR GB SA US) | original r1: 0/2 correct, 1 wrong, 0 error; original r2: 0/2 correct, 1 wrong, 0 error; landed r1: 0/2 correct, 1 wrong, 0 error; landed r2: 0/2 correct, 0 wrong, 1 error; claude-opus-5-cli r1: 0/2 correct, 1 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 1 wrong, 0 error | per-signup revenue = realized commission of the cohort divided by signups (house); hear-about-us answers are free text (no ratified grouping) |
| 10 | F14 | Activation rate | ct | 28 | partial | yes: signups per month with activated count and rate; parameters = window, client type, within-N-days | yes | original r1: 0/3 correct, 0 wrong, 0 error; original r2: 0/3 correct, 0 wrong, 0 error; landed r1: 0/3 correct, 0 wrong, 0 error; landed r2: 0/3 correct, 1 wrong, 0 error; claude-opus-5-cli r1: 0/3 correct, 1 wrong, 0 error; gpt-6-astra-codex r1: 0/3 correct, 0 wrong, 0 error | 'activation rate' default numerator: verified broker (canon) vs traded (trade-based) when the question says 'go on to trade' (documented rule, needs ratification of the bare phrase); cohort-retention three-column shape |
| 11 | F13 | Quiet high-value clients | ct | 27 | ratified | yes: per-client listing gated on last booked trade, ranked by 12-month realized commission | yes (15 at-risk clients) | original r1: 0/5 correct, 2 wrong, 0 error; original r2: 0/5 correct, 4 wrong, 1 error; landed r1: 0/5 correct, 3 wrong, 0 error; landed r2: 0/5 correct, 1 wrong, 2 error; claude-opus-5-cli r1: 0/5 correct, 3 wrong, 0 error; gpt-6-astra-codex r1: 0/5 correct, 1 wrong, 2 error | nothing further: definitions are ratified |
| 12 | F19 | Top clients by revenue | ct | 27 | partial | yes: top-N clients by a ratified commission reading with trade count; margin needs ratification | yes | original r1: 0/7 correct, 0 wrong, 2 error; original r2: 0/7 correct, 1 wrong, 0 error; landed r1: 0/7 correct, 1 wrong, 0 error; landed r2: 0/7 correct, 2 wrong, 0 error; claude-opus-5-cli r1: 0/7 correct, 1 wrong, 1 error; gpt-6-astra-codex r1: 0/7 correct, 1 wrong, 1 error | margin measure is open (margin-measure); revenue concentration share shape |
| 13 | F05 | Commission by stream, relationship manager or client type | ct | 27 | partial | yes for RM and client type; stream needs ratification | partial: 2 relationship managers, CFX/PFX streams only | original r1: 0/2 correct, 2 wrong, 0 error; original r2: 0/2 correct, 2 wrong, 0 error; landed r1: 0/2 correct, 2 wrong, 0 error; landed r2: 0/2 correct, 2 wrong, 0 error; claude-opus-5-cli r1: 0/2 correct, 2 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 1 wrong, 0 error | 'sales stream' (open) when the ask says stream rather than corporate vs private |
| 14 | F21 | Named client lookup | ct | 26 | partial | yes: profile card for a resolved client id (last trade, lifetime commission, type, contact) | no: real client ids and names | original r1: not sampled; original r2: not sampled; landed r1: not sampled; landed r2: not sampled; claude-opus-5-cli r1: not sampled; gpt-6-astra-codex r1: not sampled | named-entity matching rule (ILIKE contains) is a house default; 'lifetime realized commission' vs booked |
| 15 | F07 | SAR book run-rate inside the month | ct | 25 | unsupported | after ratification: per-month band revenue and the cross-month average | no: no SAR trades | original r1: not sampled; original r2: not sampled; landed r1: not sampled; landed r2: not sampled; claude-opus-5-cli r1: not sampled; gpt-6-astra-codex r1: not sampled | day-of-month band definition (25th to month end; 1-7, 8-14 ...) and the averaging rule across months; SAR sell side vs either side (chip-clarified today); revenue reading for run-rate (realized vs booked) |
| 16 | F23 | Bookings by payment partner or platform | ct | 25 | partial | yes: monthly share of bookings by broker with a ratified denominator | partial: 2 synthetic brokers | original r1: 0/1 correct, 0 wrong, 0 error; original r2: 0/1 correct, 0 wrong, 0 error; landed r1: 0/1 correct, 0 wrong, 0 error; landed r2: 0/1 correct, 0 wrong, 0 error; claude-opus-5-cli r1: 0/1 correct, 0 wrong, 0 error; gpt-6-astra-codex r1: 0/1 correct, 1 wrong, 0 error | share denominator (trade count vs commission vs volume) when unstated; broker naming (brokers.name) |
| 17 | X03 | Lost reasons crossed with client profile | fed | 24 | partial | yes: lost deals joined to the CT client through the bridge, grouped by a CT attribute | partial: only Price and Timing reasons | original r1: 0/1 correct, 1 wrong, 0 error; original r2: 0/1 correct, 1 wrong, 0 error; landed r1: 0/1 correct, 0 wrong, 0 error; landed r2: 0/1 correct, 1 wrong, 0 error; claude-opus-5-cli r1: 0/1 correct, 1 wrong, 0 error; gpt-6-astra-codex r1: 0/1 correct, 1 wrong, 0 error | client attribute source for a lost deal (declared intent vs traded); one reason label match rule (lost_reason_norm literal) |
| 18 | F29 | Client lifespan and average revenue per active client | ct | 24 | unsupported | after ratification | yes for the mechanics | original r1: 0/1 correct, 0 wrong, 0 error; original r2: 0/1 correct, 0 wrong, 0 error; landed r1: 0/1 correct, 0 wrong, 0 error; landed r2: 0/1 correct, 0 wrong, 0 error; claude-opus-5-cli r1: 0/1 correct, 0 wrong, 0 error; gpt-6-astra-codex r1: 0/1 correct, 0 wrong, 0 error | 'client lifespan' (first to last trade) and 'active client' for the denominator have no ratified definition; average trade size = GBP-normalised sell amount (ratified) but per-client vs per-trade averaging is open |
| 19 | F11 | Quote volume and quote-to-trade conversion | ct | 23 | partial | yes: quotes and booked quotes per period | yes (all client quotes convert in the fixture) | original r1: 0/2 correct, 1 wrong, 0 error; original r2: 0/2 correct, 1 wrong, 0 error; landed r1: 0/2 correct, 0 wrong, 0 error; landed r2: 0/2 correct, 1 wrong, 0 error; claude-opus-5-cli r1: 0/2 correct, 1 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 0 wrong, 1 error | quote-to-trade conversion definition (booked quote via quote_id vs any booking after the quote); week boundaries |
| 20 | F22 | Commission by currency corridor | ct | 20 | partial | yes: commission and GBP volume by corridor | yes (EUR, GBP, USD corridors) | original r1: 0/2 correct, 0 wrong, 0 error; original r2: 0/2 correct, 0 wrong, 0 error; landed r1: 0/2 correct, 0 wrong, 0 error; landed r2: 0/2 correct, 0 wrong, 0 error; claude-opus-5-cli r1: 0/2 correct, 0 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 0 wrong, 0 error | margin in basis points (open) |
| 21 | F31 | Client listing with contact details | ct | 17 | partial | yes: per-client export with email, country, last trade | yes | original r1: 0/2 correct, 0 wrong, 2 error; original r2: 0/2 correct, 1 wrong, 1 error; landed r1: 0/2 correct, 1 wrong, 1 error; landed r2: 0/2 correct, 1 wrong, 1 error; claude-opus-5-cli r1: 0/2 correct, 1 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 0 wrong, 2 error | contact fields on the gold leg are NULL placeholders; email export needs the CT leg (house) |
| 22 | F04 | Commission by month | ct | 16 | ratified | yes: commission per London month for a ratified reading | yes (months March, June to September 2026) | original r1: 0/2 correct, 2 wrong, 0 error; original r2: 0/2 correct, 2 wrong, 0 error; landed r1: 0/2 correct, 1 wrong, 0 error; landed r2: 0/2 correct, 1 wrong, 0 error; claude-opus-5-cli r1: 0/2 correct, 1 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 1 wrong, 0 error | nothing further: definitions are ratified |
| 23 | F20 | Trades today or recent trades | ct | 16 | ratified | yes: trades booked today or yesterday with client, amount, commission | yes (trades up to 5 September 2026) | original r1: 0/1 correct, 0 wrong, 0 error; original r2: 0/1 correct, 0 wrong, 0 error; landed r1: 0/1 correct, 0 wrong, 0 error; landed r2: 0/1 correct, 0 wrong, 0 error; claude-opus-5-cli r1: 0/1 correct, 0 wrong, 0 error; gpt-6-astra-codex r1: 0/1 correct, 0 wrong, 0 error | nothing further: definitions are ratified |
| 24 | F06 | Daily booked commission and month-to-date pace | ct | 15 | partial | yes: daily booked commission with running total and a baseline | yes | original r1: 0/2 correct, 0 wrong, 1 error; original r2: 0/2 correct, 0 wrong, 1 error; landed r1: 0/2 correct, 0 wrong, 0 error; landed r2: 0/2 correct, 0 wrong, 1 error; claude-opus-5-cli r1: 0/2 correct, 0 wrong, 0 error; gpt-6-astra-codex r1: 0/2 correct, 0 wrong, 0 error | zero-fill of days without trades and the pace baseline (average of the last 3 months) are house defaults |
| Order | Family | Name | Leg | Distinct questions | Definition to ratify |
|---|---|---|---|---|---|
| 1 | X03 | Lost reasons crossed with client profile | fed | 24 | client attribute source for a lost deal (declared intent vs traded); one reason label match rule (lost_reason_norm literal) |
| 2 | P05 | Sales activity volume and mix | pd | 14 | 'prospect' (open deal vs person without CT link) and 'no reply' (email direction) |
| 3 | P03 | Pipeline status and open deals | pd | 12 | 'stalled' threshold column (days_in_stage is thin); 'sales leads' in Pipedrive wording vs the CT allocatable definition |
| 4 | X01 | Sales activity per trading client | fed | 8 | 'books' = clients vs relationship-manager books; 'calls' key list (call, aircall) |
| 5 | X05 | Trading clients with sales contact recency | fed | 8 | 'last communication' source (activities vs mail rollups vs notes) |
| 6 | F35 | Signup cohort export with activation, revenue and lost reason | fed | 3 | fixed column list for the export |
| 7 | P06 | Recent notes for a named organisation | pd | 3 | organisation name match rule |
| 8 | X04 | Tagged organisations revenue and pipeline | fed | 1 | organisation label 'VIP' field (org_labels vs cf_client_type_tags); org to client aggregation rule |
| 9 | F10 | Quoted but not booked | ct | 98 | client-requested quote = quotes.type by_client (house default, needs ratification); 'without booking' window when the quote window is explicit (same window vs lifetime) |
| 10 | F18 | Facebook-sourced client report | ct | 56 | the fixed column set of the report (each ask lists a different set); 'most recent deal lost reason' per client via the person bridge (house default) |
| 11 | F25 | Professional affiliate performance | ct | 53 | window placement for 'top affiliates in a period' (referral date vs trade date, house default); 'strong' / 'gone quiet' thresholds |
| 12 | F01 | Signups by acquisition source | ct | 51 | the bare 'where did our clients come from' reading (multi-taxonomy rollup vs marketing source only) is a house default, not ratified; campaign = utm_campaign vs campaigns.token |
| 13 | F08 | Currency-book performance per active client | ct | 47 | 'active client' inside a book question (activated vs traded in the period vs recently trading) is chip-clarified today; needs one ratified default; SAR book membership: sell side only vs either side (ratified either side; sellers/buyers as subsets) |
| 14 | F26 | Refer-and-earn programme | ct | 44 | 'converted referral' (activated vs traded); payout ledger for ROI = reward_events vs user_commission_statements (house default) |
| 15 | F09 | Time from signup to first trade for a book or stream | ct | 41 | 'sales stream' has no ratified definition (open: sales-stream-dimension); the israel / inbound / partnerships split is undefined; window placement (signup date vs first-trade date); mean vs median default |
| 16 | F34 | Revenue by acquisition source or campaign | ct | 30 | attribution rule: revenue of clients acquired through the source in the window vs revenue earned in the window from clients ever acquired through it; campaign identity (utm_campaign vs campaigns.token) |
| 17 | F14 | Activation rate | ct | 28 | 'activation rate' default numerator: verified broker (canon) vs traded (trade-based) when the question says 'go on to trade' (documented rule, needs ratification of the bare phrase); cohort-retention three-column shape |
| 18 | F17 | Signup quality by source or country | ct | 28 | per-signup revenue = realized commission of the cohort divided by signups (house); hear-about-us answers are free text (no ratified grouping) |
| 19 | F05 | Commission by stream, relationship manager or client type | ct | 27 | 'sales stream' (open) when the ask says stream rather than corporate vs private |
| 20 | F19 | Top clients by revenue | ct | 27 | margin measure is open (margin-measure); revenue concentration share shape |
| 21 | F21 | Named client lookup | ct | 26 | named-entity matching rule (ILIKE contains) is a house default; 'lifetime realized commission' vs booked |
| 22 | F07 | SAR book run-rate inside the month | ct | 25 | day-of-month band definition (25th to month end; 1-7, 8-14 ...) and the averaging rule across months; SAR sell side vs either side (chip-clarified today); revenue reading for run-rate (realized vs booked) |
| 23 | F23 | Bookings by payment partner or platform | ct | 25 | share denominator (trade count vs commission vs volume) when unstated; broker naming (brokers.name) |
| 24 | F29 | Client lifespan and average revenue per active client | ct | 24 | 'client lifespan' (first to last trade) and 'active client' for the denominator have no ratified definition; average trade size = GBP-normalised sell amount (ratified) but per-client vs per-trade averaging is open |
| 25 | F11 | Quote volume and quote-to-trade conversion | ct | 23 | quote-to-trade conversion definition (booked quote via quote_id vs any booking after the quote); week boundaries |
| 26 | F22 | Commission by currency corridor | ct | 20 | margin in basis points (open) |
| 27 | F31 | Client listing with contact details | ct | 17 | contact fields on the gold leg are NULL placeholders; email export needs the CT leg (house) |
| 28 | F06 | Daily booked commission and month-to-date pace | ct | 15 | zero-fill of days without trades and the pace baseline (average of the last 3 months) are house defaults |
| 29 | O01 | Not in the connected data | none | 13 | not in the connected data (app analytics) |
| 30 | F02 | Signups by country or nationality | ct | 12 | nationality vs registered country (clients.data nationality is a house fact) |
| 31 | F24 | Period-over-period comparison | ct | 11 | 'dropped more than 20%' comparison basis (quarter to date vs full prior quarter) |
| 32 | F30 | Declared currency versus traded currency | ct | 8 | 'differs' rule: primary declared vs top traded, or any declared vs any traded |
| 33 | F28 | CFX or PFX monthly funnel | ct | 7 | stream = CFX/PFX by clients.type at signup time vs now |
| 34 | F15 | Signups stuck before activation | ct | 4 | 'stuck' = no completed registration vs no activation |
| 35 | F32 | Revenue share by country | ct | 4 | 'registered country' vs nationality; share denominator |
| 36 | F37 | All-time platform totals | ct | 2 | 'value transacted' = GBP sell volume |
| 37 | F36 | Countries with a first-ever trading client | ct | 1 | 'first-ever trading client' per country |
Open definitions recorded in the knowledge docs themselves: margin-measure, sales-stream-dimension, test-account-exclusion, deleted-account-exclusion.
| Family | Name | Distinct questions | Why |
|---|---|---|---|
| F09 | Time from signup to first trade for a book or stream | 41 | 'sales stream' has no ratified definition (open: sales-stream-dimension); the israel / inbound / partnerships split is undefined; window placement (signup date vs first-trade date); mean vs median default |
| F07 | SAR book run-rate inside the month | 25 | day-of-month band definition (25th to month end; 1-7, 8-14 ...) and the averaging rule across months; SAR sell side vs either side (chip-clarified today); revenue reading for run-rate (realized vs booked) |
| F29 | Client lifespan and average revenue per active client | 24 | 'client lifespan' (first to last trade) and 'active client' for the denominator have no ratified definition; average trade size = GBP-normalised sell amount (ratified) but per-client vs per-trade averaging is open |
| O01 | Not in the connected data | 13 | not in the connected data (app analytics) |
| O03 | Instruction or definition, not a question | 7 | not a data question |
| O02 | Security probe | 6 | not a data question |
Each item gives the family, the reading used for the expected SQL, the other reading, and the definition that settles it. Question texts are paraphrased by family; the exact texts are in the evaluator-only directory.
| Family | Asked (turns) | Expected mode | Reading used | Other reading | Definition that settles it |
|---|---|---|---|---|---|
| F10 Quoted but not booked | 20 | rows | 'without booking' = no booked trade in the same 90-day window (used here; 'total historic trades' implies earlier trades are allowed) | 'without booking' = no booked trade ever | ratify: client-requested quote = quotes.type by_client; and the booking window for "without booking" when the quote window is explicit |
| P05 Sales activity volume and mix | 5 | rows | long shape: one row per (deal outcome, activity type) with a count (used here) | wide shape: one row per outcome with calls and emails as columns | ratify the activity-mix shape (long vs wide) and "prospect" |
| F06 Daily booked commission and month-to-date pace | 8 | rows | one row per calendar day of the month so far, zero-filled, with the running total (used here) | one row per day that had a trade | ratify the pace baseline (full prior months vs same day-of-month cumulative) and zero-fill of trade-less days |
| F13 Quiet high-value clients | 7 | rows | 'quiet' = no booked trade in the last 60 days (churn framing → trading recency, ratified 2026-07-25) (used here) | 'quiet' = no Pipedrive activity in the last 60 days | ratified already (churn framing = trading recency); the remaining ambiguity is "quiet" outside churn wording |
| F19 Top clients by revenue | 7 | rows | average margin = commission divided by GBP sell volume, in basis points (used here; trade_details.margin is not an authorized column on the evaluation fixture) | average margin = AVG(trade_details.margin) over the same settled trades (the column has no ratified unit) | ratify the margin measure (basis points on GBP sell volume vs trade_details.margin) |
| F01 Signups by acquisition source | 46 | rows | multi-taxonomy rollup: one row per acquisition source TYPE (professional affiliate, refer-and-earn, paid/UTM, web referral, unknown) (house default acquisition-source-comparison; used here) | marketing source only: one row per canonical utm source | ratify the default reading of a bare acquisition question (multi-taxonomy rollup vs marketing source only) |
| F09 Time from signup to first trade for a book or stream | 5 | scalar | cohort = clients who signed up in the last 12 months and have traded; average of (first trade date - signup date) (used here) | cohort = clients whose first trade fell in the last 12 months | ratify "sales stream" (israel / inbound / partnerships) as a column or predicate, and the signup-to-first-trade window placement |
| F26 Refer-and-earn programme | 5 | rows | new referrer = an ordinary user whose first referred signup happened this quarter (used here) | new referrer = a user account created this quarter that has referred someone | ratify "new referrer" and the refer-and-earn payout ledger (earned vs paid) |
| F26 Refer-and-earn programme | 5 | rows | payouts = statements paid in the window (user_commission_statements.paid_at); revenue = settled commission of refer-and-earn-referred clients completed in the window; roi = revenue / payouts (used here) | payouts = rewards earned (reward_events) in the window regardless of payment | ratify "new referrer" and the refer-and-earn payout ledger (earned vs paid) |
| F09 Time from signup to first trade for a book or stream | 17 | clarification | 'sales stream' is undefined (open: sales-stream-dimension); the question cannot be answered without a ruling | stream = CFX/PFX is meaningless for a corporate-only cohort | ratify "sales stream" (israel / inbound / partnerships) as a column or predicate, and the signup-to-first-trade window placement |
| F09 Time from signup to first trade for a book or stream | 10 | clarification | 'sales stream' is undefined (open) | days from signup to first trade is itself a house default | ratify "sales stream" (israel / inbound / partnerships) as a column or predicate, and the signup-to-first-trade window placement |
| F34 Revenue by acquisition source or campaign | 3 | rows | commission settled in the last 12 months from clients ever acquired through each source (used here) | commission from clients acquired in the last 12 months, grouped by source | ratify revenue attribution to an acquisition source: cohort acquired in the window vs revenue earned in the window |
| F05 Commission by stream, relationship manager or client type | 4 | rows | stream = the CFX/PFX corporate-vs-private stream on the gold leg (used here) | stream = the sales team stream (israel / inbound / partnerships), which is undefined | ratify "stream" in revenue breakdowns (CFX/PFX vs sales team stream) |
| F04 Commission by month | 6 | rows | long shape: one row per month and class (used here) | wide shape: one row per month with corporate and private columns | ratify the default shape for a two-class monthly breakdown (long vs wide) |
| F06 Daily booked commission and month-to-date pace | 5 | rows | compare the month-to-date total with the average of the three full previous months (used here) | compare the month-to-date total with the same day-of-month cumulative average of the three previous months (dom-average-baseline) | ratify the pace baseline (full prior months vs same day-of-month cumulative) and zero-fill of trade-less days |
| F24 Period-over-period comparison | 1 | rows | this quarter to date against the full same quarter last year (used here) | this quarter to date against the same quarter last year to the same day | ratify period-over-period comparison basis (period to date vs full prior period) |
| X01 Sales activity per trading client | 11 | ordered | 'books' = the top 20 clients by realized commission this year; activities = all Pipedrive activities linked to the client through the person bridge (used here) | 'books' = relationship-manager books | ratify "books" (clients vs relationship-manager books) and the activity cohort |
| X01 Sales activity per trading client | 10 | rows | cohort = every house client with a Pipedrive person link, zero counts kept (used here) | cohort = clients with at least one call or email activity | ratify "books" (clients vs relationship-manager books) and the activity cohort |
| X05 Trading clients with sales contact recency | 4 | ordered | high-value = any settled commission in the last 12 months; no logged activity = no call or email event in the last 60 days (used here) | high-value = a top-N or threshold on revenue | ratify "high-value" (threshold or top-N) and the logged-activity source |
| F30 Declared currency versus traded currency | 4 | rows | differs = the most-traded sell currency is not among the declared sell currencies (used here) | differs = any traded sell currency outside the declared set | ratify "differs": most-traded sell currency outside the declared set vs any traded currency outside it |
| F28 CFX or PFX monthly funnel | 2 | rows | one row per month that has any signup, activation or trading client (used here) | one row per calendar month over the whole 10-year span, zero-filled | ratify zero-fill for monthly funnel series |
| F15 Signups stuck before activation | 10 | clarification | 'sales stream' is undefined (open) | the listing itself (signed up, registered, not activated) is answerable | ratify "sales stream" |
| F32 Revenue share by country | 6 | rows | long shape: one row per country and class, signups in the window, settled revenue of those signups (used here) | wide shape: one row per country with private and corporate columns | ratify the default shape for a two-class country breakdown (long vs wide) and the revenue basis of a signup cohort |
| F38 Sales lead counts | 3 | scalar | allocatable accounts (ratified sales-lead canon) created in the last 12 London calendar months (used here, matching the admin page's monthly buckets) | allocatable accounts created in the rolling 12 months from the as-of date | ratify the sales-lead window (London calendar months vs rolling 12 months) |
| O02 Security probe | 2 | refusal | permission guard; an answer is scored as wrong | - | see family-support.json |
| O01 Not in the connected data | 3 | refusal | app crash rate is not in the connected data; an answer is scored as wrong | - | see family-support.json |
| F29 Client lifespan and average revenue per active client | 1 | scalar | average over every client with a booked trade of (last trade date - first trade date), single-trade clients count as 0 days (used here) | average over clients with at least two trades only | ratify client lifespan (population and single-trade clients) and "active client" |
| F01 Signups by acquisition source | follow-up | rows | multi-taxonomy rollup (house default) | marketing source only | ratify the default reading of a bare acquisition question (multi-taxonomy rollup vs marketing source only) |
| F19 Top clients by revenue | follow-up | rows | average margin = commission divided by GBP sell volume, in basis points (used here; trade_details.margin is not an authorized column on the evaluation fixture) | average margin = AVG(trade_details.margin) over the same settled trades (the column has no ratified unit) | ratify the margin measure (basis points on GBP sell volume vs trade_details.margin) |
| F19 Top clients by revenue | follow-up | rows | client details = type, email and country for the same 20 clients (used here) | client details = the full profile record | ratify the margin measure (basis points on GBP sell volume vs trade_details.margin) |
| F19 Top clients by revenue | follow-up | rows | adds a client-quote count over the rolling last 3 months to the previous rows (used here) | - | ratify the margin measure (basis points on GBP sell volume vs trade_details.margin) |
The blinded verifier screen (v1 and v2) and the sealed goal-3 answers resolve these two points differently, so any replay of the screen and the templates for the Pipedrive and federated families wait on them. Each is a family-level question with the two readings and what each reading changes.
Ruling pd-people-default-exclusions. Families: P05 Sales activity volume and mix; P03 Pipeline status and open deals; P02 Lost deals listing for one reason; X01 Sales activity per trading client; X05 Trading clients with sales contact recency; X03 Lost reasons crossed with client profile.
Question: On a Pipedrive people question that does not mention staff or house accounts, do the staff exclusion (persons_enriched.pd_is_ct_employee IS NOT TRUE) and the house-account exclusion (CT account 1, staff e-mail domain) apply by default?
| Reading | Rule | What it changes | Who holds it today |
|---|---|---|---|
| A: apply by default | Every Pipedrive people listing or count excludes CT employees and house accounts unless the question asks to include them. | The verifier's v1 behaviour and the ratified entry pd-person-table-key become the contract. The sealed goal-3 answers that keep staff (13 cases in the verifier screen) must be re-sealed, and every real-usage expected answer on a pd or fed family must add the predicate. | ratified entry pd-person-table-key (verified fact, 2026-07-27); the CT-side house rule in period-over-period (house default). |
| B: apply only when asked | Staff and house-account exclusions are opt-in wording ('excluding staff', 'real clients only'); a bare people question returns every person. | The sealed goal-3 answers stand. The verifier screen's v2 instruction becomes the contract, and the 16 accepted-correct verdicts that carried a would-apply-default signal in v2 are correct as sealed. The knowledge entry pd-person-table-key must be re-worded from 'exclude staff via ...' to 'staff can be excluded via ...'. | the sealed goal-3 holdout answers; verifier screen v2 instruction. |
What it settles: 13 of 61 correct proposals rejected in verifier v1 (staff exclusion) and 4 (house exclusion); the default predicate for six real-usage families (P02, P03, P05, X01, X03, X05: 66 distinct questions).
Evaluator's recommendation (not a ruling): Reading A for listings that name people (a staff member in a 'clients to call' list is a wrong answer for the sales team), reading B for counts and rates that the question scopes to deals or activities; state which one in one sentence in the definition so both the generator and the verifier read the same text.
Ruling pd-only-person-identifier. Families: P05 Sales activity volume and mix; P03 Pipeline status and open deals; X01 Sales activity per trading client; X05 Trading clients with sales contact recency; F18 Facebook-sourced client report; F35 Signup cohort export with activation, revenue and lost reason.
Question: When an answer must return one identifier per person and the person may or may not have a CT link (persons_enriched.ct_id), what is the identifier column, and what does the clause 'use the person ID for a Pipedrive-only person' mean?
| Reading | Rule | What it changes | Who holds it today |
|---|---|---|---|
| A: person id for everyone | The identifier is persons_enriched.id for every row; ct_id is an extra column when the question asks for the CT link. 'Use the person ID for a Pipedrive-only person' only reminds that such a person has no CT id. | The sealed goal-3 answers stand (they return pe.id). Verifier verdicts that rejected pe.id-for-everyone (11 cases in v2) were verifier errors. One identifier space per answer; a CT-linked person is identified by a Pipedrive id, so joining the answer to CT tables needs a second lookup. | the sealed goal-3 holdout answers. |
| B: CT user id first, person id as fallback | The identifier is COALESCE(ct_id, id) per row (CT user id when linked, Pipedrive person id otherwise), with a type column saying which one it is. | The verifier's reading becomes the contract; 11 sealed answers must be re-sealed and the real-usage expected answers on federated families (X01, X05, F18, F35) must switch to the coalesced key. Mixed identifier spaces in one column need the type column or downstream joins break. | verifier screen v1 and v2 (all passes). |
What it settles: 11 of 61 correct proposals still rejected in verifier v2 (identifier mapping); the identifier column of every federated per-person export in the real-usage corpus.
Evaluator's recommendation (not a ruling): Reading A, with the clause re-worded in the holdout contract to 'return persons_enriched.id; add ct_id as a separate column when the question asks for the CT link'; a coalesced key mixes two id spaces in one column and is the class of defect the fabricated-cohort guard exists to catch.