NEERAJ VENNACV
← Project Alice
Evaluated synthetic prototype

Recon

Recon is a financial reconciliation app I built with Go, PostgreSQL, and React. It processes transaction files, keeps an audit history, and helps investigate records that disagree. I also use it to test how agents handle conflicting financial information.

Explore the experimentHeld-out results reportHeld-out aggregate data (CSV)Distribution-shift results reportDistribution-shift aggregate data (CSV)
  • Go
  • PostgreSQL
  • Python
  • React
  • TypeScript
  • Docker Compose

Implementation

The Go REST API imports purchases, refunds, and settlements from CSV files. PostgreSQL stores each transaction as balanced journal entries: the amounts on both sides must match. Entries are append-only, so corrections leave an audit history.

A journal update and its event are saved in the same database transaction. A background worker reads these events and updates derived balances and case records. Duplicate requests cannot post the same transaction twice; retries wait for missing dependencies, while failed events can be quarantined for review.

The React console lets an operator import files and examine balances, reconciliation cases, and audit records. Research policies use a separate observation interface. They can update their own remembered state, but cannot change the underlying journal.

Recon architecture connecting synthetic sources, Go ingestion, PostgreSQL journal and outbox, asynchronous processing, and the research console.
Journal writes and research belief updates use separate paths.

Results

Agents received conflicting reports about synthetic transactions and could spend a limited budget on extra evidence. I compared rules for checking reports, changing memory, and deciding what to do with a disputed transaction.

  • In held-out cases where the evidence could identify the cause, conventional reconciliation completed 95.3% and Alice completed 57.8% at budget 10. Alice made no harmful memory changes in this comparison. The completion difference was −37.5 percentage points (95% family-bootstrap interval: −43.8 to −31.2).
  • In the separate distribution-shift comparison at the same budget, completion fell to 33.3% for conventional reconciliation and 32.8% for Alice. Alice’s weighted harmful-revision rate was 62.5%.
  • At budget 10, Alice also made unsafe commitments in every tested case where the evidence could not identify the cause. More accurate explanations and fewer memory errors did not consistently lead to more completed tasks.
Evaluation details

The study compares five policies, four ablations, and fixed/responsive planner variants through the public Go API. Every evidence request has a cost. Completion requires the correct case decision, transaction state, and supporting evidence.

The held-out and distribution-shift evaluations each use 32 independent scenario families and 10,752 episode-policy-budget records at budgets 0, 3, 6, and 10. Repeated records within a family are not independent scenarios.

The reported comparisons use budget 10 and exclude controls and cases where the available evidence cannot identify the cause. Episode scores are averaged within each condition; conditions receive equal weight within each family, then families receive equal weight.

A harmful revision is an episode in which the policy changes an initially correct remembered balance. Its reported rate uses the same condition and family weighting; it is not the percentage of all 10,752 records or of individual memory edits.

Limitations

The experiments use synthetic records and a fixed set of reporting failures. Sources can share faults, and the scores assigned to evidence are not calibrated probabilities.

The planned 128-family evaluations have not been run.

Memory revisionProject Alice