Evaluation
Run history
Every run of our reporting suite, unedited, including the early failures. Ten portfolio-reporting questions (occupancy, delinquency, lease expirations, NOI, expenses) answered by operating a property-management application through its UI, each answer cross-checked before it is returned. Newest first.
| Date | Run | Score | Cost | Duration |
|---|---|---|---|---|
| Jul 14, 2026 | run 11: qwen3.7-plus with auto-screenshot per action (demo replay) portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
9 / 10 | $0.26 | 20.2 min |
| Jul 14, 2026 | run 10b: qwen3.7-plus, tool_choice fallback, 300s timeout portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
10 / 10 | $0.22 | 18.9 min |
| Jul 14, 2026 | run 10: qwen3.7-plus, 300s timeout guard · aborted portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026 |
0 / 3 | - | - |
| Jul 13, 2026 | run 9: sereen-agent-qwen (qwen3-vl-32b-instruct via Model Studio), first full Qwen suite portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
3 / 10 | $0.03 | 15.8 min |
| Jul 13, 2026 | run 8: string-typed structured_data parse + 2-retry bounce portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
10 / 10 | $2.05 | 9.8 min |
| Jul 13, 2026 | run 7: unpaid_rent pinned to May 2026, capitalization + prose-echo conventions portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
9 / 10 | $1.89 | 8.7 min |
| Jul 13, 2026 | run 6: sonnet-5, vacancy ruling + href ids + final_answer key validation portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
7 / 10 | $1.51 | 7.7 min |
| Jul 13, 2026 | run 5: claude-sonnet-5, timeout disabled, caching off portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
6 / 10 | $4.65 | 16.8 min |
| Jul 12, 2026 | run 4: fable-5, caching OFF, cap off, max_tokens=16000 + prompt fixes portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
7 / 10 | $10.67 | 15.3 min |
| Jul 12, 2026 | run 3: claude-fable-5, prompt caching, dollar cap disabled, refreshed expected answers portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
6 / 10 | $3.57 | 13.3 min |
| Jul 12, 2026 | sereen v1 live run 2 (warm world model, triangulation on, verifier gate fix) portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
4 / 10 | $4.28 | 7.7 min |
| Jul 12, 2026 | sereen v1 live run 1 (cold world model, triangulation on) portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
4 / 10 | $3.24 | 9.9 min |
| Jul 12, 2026 | dry run with placeholder API key portfolio_count_propertiesportfolio_count_unitsrent_due_may_2026unpaid_rent_current_monthoccupancy_rate_by_propertymaintenance_open_over_30leases_expiring_q3_2026vacancy_units_todaynoi_property_x_q1_2026expense_breakdown_by_category_2026 |
0 / 10 | - | - |
Autonomous exploration sessions
Sessions in which the agent learned the application with no documentation, no source code, and no human explanation, producing the semantic model.
| Date | Session | Cost |
|---|---|---|
| Jul 20, 2026 | autonomous exploration (claude-sonnet-5) | $0.93 |
| Jul 20, 2026 | autonomous exploration (claude-sonnet-5) | $0.61 |
| Jul 20, 2026 | autonomous exploration (claude-sonnet-5) | $0.61 |
| Jul 20, 2026 | autonomous exploration (claude-sonnet-5) · aborted | - |
| Jul 20, 2026 | autonomous exploration (claude-sonnet-5) · aborted | - |
| Jul 20, 2026 | autonomous exploration (claude-sonnet-5) | $0.27 |
| Jul 20, 2026 | autonomous exploration (claude-sonnet-5) · aborted | - |
| Jul 14, 2026 | autonomous exploration (claude-sonnet-5) | $0.49 |
| Jul 14, 2026 | autonomous exploration (claude-sonnet-5) | $0.55 |
| Jul 14, 2026 | autonomous exploration (claude-sonnet-5) | $0.18 |
| Jul 14, 2026 | autonomous exploration (claude-sonnet-5) | $0.11 |
| Jul 14, 2026 | autonomous exploration (claude-sonnet-5) | $0.20 |
Validation runs
| Date | Run | Score | Cost |
|---|---|---|---|
| Jul 14, 2026 | unpaid-ruling-validation: unpaid-definition ruling validation | 1 / 1 | $0.03 |
| Jul 14, 2026 | screenshot-validation: auto-screenshot validation (flag on) | 1 / 1 | $0.00 |
Suite runs executed against a purpose-built property-management test application. Harness smoke tests using a mock agent are excluded. Internally, every run is replayable step by step: each answer traces back to the exact screens the agent saw and the database state that confirmed it.