From d055e03bf44e1b2b7e13f591c03878c9be7bdb09 Mon Sep 17 00:00:00 2001 From: Pradip Nichite <9526749+PradipNichite@users.noreply.github.com> Date: Thu, 22 Oct 2026 16:54:39 +0530 Subject: [PATCH] docs: tell one research improvement story and index the outcomes --- README.md | 53 ++++++++---------- docs/improvements/README.md | 104 +++++++++++++++++++++--------------- docs/source-export.json | 4 +- 3 files changed, 84 insertions(+), 77 deletions(-) diff --git a/README.md b/README.md index cd16f71..1252fdf 100644 --- a/README.md +++ b/README.md @@ -4,45 +4,36 @@ This is a working Database Agent that lets you connect a SQL database, ask questions in natural language, inspect generated SQL, explore results, and create visualizations. -**Research → product change: [S64 — unsupported data was reported as zero](docs/improvements/S64-unsupported-data.md).** -[Original published finding](https://aidemos.com/benchmarks/ai-database-agents/results/futuresmart-ai-database-agent/the-connected-data-cannot-answer-the-question): **Fail, unchanged** · Engineering correction: **released** · Independently accepted benchmark retest: **pending**. -This retrospective connects the finding to the code, regression controls and recorded validation; the correction predates this repository's public history. +## How AI Demos research improves Database Agent -But the purpose of this project goes beyond providing another text-to-SQL application. +We use findings from [AI Demos](https://aidemos.com/) to investigate product +failures and improve this working, open-source reference application. Our +[improvement history](docs/improvements/README.md) connects published findings, +engineering decisions, code and verification, while preserving the failures. -## Why we're building this publicly +### Example: an answer using the wrong data -At [AI Demos](https://aidemos.com/), a research and benchmarking platform developed by FutureSmart AI, we evaluate AI products against real-world scenarios, capture evidence, and publish what works and what fails. +**The research finding — S64.** A user asked how many five-star reviews had been +received. The connected database had no review data, but the agent queried an +unrelated activity log and returned zero. The [published benchmark result was +Fail](https://aidemos.com/benchmarks/ai-database-agents/results/futuresmart-ai-database-agent/the-connected-data-cannot-answer-the-question). -We wanted to demonstrate what happens when a product team actually uses those research findings to improve its own product. +**What we improved.** We added an explicit answerability decision: does the +available schema support the requested information? Unsupported questions should +produce an unavailable answer, while a valid query that genuinely returns zero +remains answerable. [Read the investigation, code and regression controls](docs/improvements/S64-unsupported-data.md). -So we're using FutureSmart Database Agent as a **public, ongoing case study**. +**What we verified.** Existing engineering regressions use mocked model/database +boundaries. Separate historical browser checks covered missing information, +genuine zero results and valid positive counts. These are engineering checks, +not a new independently graded benchmark run. -We take findings from AI Demos research, investigate reported failures, make engineering decisions, implement fixes, and document what actually changed. +**Engineering correction released · Original benchmark Fail preserved · Independent benchmark retest pending.** -You can follow that process here—from the original benchmark result to the implementation and subsequent verification. +[Explore all research-driven findings and engineering outcomes →](docs/improvements/README.md) -**We're not trying to build another commercial Database Agent SaaS or achieve the highest benchmark score.** The objective is to maintain a complete, deployable, open-source reference implementation and demonstrate how systematic research can guide product development. - -## From benchmark findings to product improvements - -Our improvement history connects: - -**Published benchmark result → Identified issue → Engineering investigation → Implementation → Verification** - -And we publish the failures too. - -For example: - -- **Unsupported questions:** AI Demos identified a case where the agent returned zero using unrelated database information. We introduced explicit answerability checks and verified the behavior against missing, zero, and positive data cases. -- **Changing the question mid-conversation:** Research found cases where the agent changed the meaning of a metric during follow-ups. This remains an ongoing engineering investigation. -- **Excluded database tables:** A published test found that the agent accessed an excluded table despite refusing the request. We reproduced gaps with synthetic data, implemented fail-closed enforcement, and independently verified the source fix. It has not been deployed, and the original benchmark Fail is unchanged. - -These are examples of actual findings, not claims that every issue has been resolved. - -[Explore the published AI Demos benchmark results](https://aidemos.com/benchmarks/ai-database-agents/tools/futuresmart-ai-database-agent) - -[Explore our engineering decisions and improvement history](docs/improvements/README.md) +We prioritize a dependable, deployable product and reproducible evidence over +commercial feature parity or a higher benchmark score. ## A complete reference application, not just a library diff --git a/docs/improvements/README.md b/docs/improvements/README.md index 305e3f0..986a838 100644 --- a/docs/improvements/README.md +++ b/docs/improvements/README.md @@ -1,44 +1,60 @@ -# Database Agent: research-driven improvement history - -Recorded on 21 October 2026 (IST). This is a public-facing history of our own -reference application. It preserves failures, decisions and limited validation -alongside successful fixes. It is not a leaderboard or a current accuracy score. - -- [S64 retrospective: unsupported data was reported as zero](S64-unsupported-data.md) — original Fail unchanged; correction released; independent benchmark retest pending. -- [Original evaluation baseline and published sources](baseline.md) -- [Dated 17-result documented baseline and unchanged verdicts](published-results.md) -- [Engineering history, including unfinished and declined work](history.md) -- [Structured public records](records.json) -- [How maintainers must update the history](MAINTAINING.md) - -Published research verdicts stay as published. Private case-level verdicts are -preserved separately and are not disclosed without publication approval. A fix, -new passing fixture, green CI job or deployment does not rewrite a benchmark. -Implementation identifiers below are recorded provenance; future fresh-history -source distributions may not include the original private commits. Current -source/test links remain the practical way to inspect the released behavior. - -Current release qualification (22 October 2026 IST) covers the centralized model -profile, one-key installation, ordinary UI/API journey and a clean source export. -The documented 17-result baseline is sufficient; additional research belongs to -later public iterations. PR77/PR79 are deferred unless an ordinary-flow failure -requires them. Earlier issue queues in dated records are historical context. -This history does not independently authorize model calls or deployment. Candidates -are explicitly labelled and are not features of the current accepted release. - -S60/DA-008 has scoped independent source/synthetic acceptance on562db70 with -all required CI checks passed (historical private identifier 39605744198; original link omitted). -This is accepted source for integration, not a production deployment or an upgrade -to the original published Fail. See the detailed history for preserved initial -candidate and CI failures, accepted controls and remaining limits. - - -The 22 October 2026 [one-key qualification receipt](evidence/2026-10-22-one-key-qualification.json) -records the accepted fresh UI/API installation, preserved first-run failure, -restart/reopen evidence and remaining release limits. It does not replace or -upgrade any published benchmark verdict. - - -Apache-2.0 and the new public source repository were approved after qualification; -see [publication notes](../publication.md). Earlier pending-release wording in -dated records describes its original phase and is not a benchmark verdict. +# Research findings and engineering outcomes + +This is the navigation hub for how AI Demos research and maintainer investigation +shape FutureSmart Database Agent. Start with a finding, inspect the engineering +record, then check what was actually verified. The index is not a leaderboard or +a claim that every reported problem has been fixed. + +## Benchmark findings + +| Scenario and original evidence | What research found | Engineering outcome | Details and verification limits | +|---|---|---|---| +| [S64 — original Fail](https://aidemos.com/benchmarks/ai-database-agents/results/futuresmart-ai-database-agent/the-connected-data-cannot-answer-the-question) | Unsupported review information was reported as zero using an unrelated activity log. | **Correction released; benchmark retest pending.** | [Complete case study](S64-unsupported-data.md): answerability code, mocked regressions and recorded separate historical browser checks. [Public PR #1](https://github.com/FutureSmart-AI/database-agent/pull/1) is retrospective documentation, not the historical fix PR. | +| [S60 — original Fail](https://aidemos.com/benchmarks/ai-database-agents/results/futuresmart-ai-database-agent/an-excluded-table-is-needed-to-answer-the-question) | The agent refused an answer but still read an excluded table and returned a result. | **Scoped source fix accepted; production deployment not claimed.** | [DA-008 engineering record](history.md#da-008-verify-table-access-before-reading-source-data): fail-closed access controls and synthetic/independent source validation. No accepted benchmark retest. | +| [S43 — original Fail](https://aidemos.com/benchmarks/ai-database-agents/results/futuresmart-ai-database-agent/the-user-changes-direction-mid-conversation) | A request for one overall cancellation rate became a by-state result using refund events. | **Investigation/candidate; semantic correction not independently accepted for release.** | [DA-003 engineering record](history.md#da-003-preserve-metric-meaning-when-changing-scope). Candidate work is separate from the accepted application; no new implementation or benchmark retest is claimed here. | +| [S53 — original Fail](https://aidemos.com/benchmarks/ai-database-agents/results/futuresmart-ai-database-agent/same-answer-different-format) | A chart-only follow-up lost a refund filter, changed values and denied chart rendering. | **Semantic correction not accepted.** | [Related DA-003 metric/scope work](history.md#da-003-preserve-metric-meaning-when-changing-scope). The separate [released DA-002 renderer fix](history.md#da-002-render-nested-stacked-chart-results-without-losing-series) does not establish a fix or benchmark Pass for S53. | + +These original results remain **Fail**. Engineering release, source acceptance, +production deployment and an independently accepted benchmark retest are different +states. No independently accepted retest is recorded for the four findings above. +The [dated 17-result baseline](published-results.md) covers the other published +scenarios too, including original Pass results; it is not a new accuracy score. + +## Other engineering records + +Some work responds to an installation or maintainer finding rather than a single +published benchmark case. Follow the linked record for its original failures, +code/test references, independent checks and remaining limits. + +| Record | Engineering response | Current evidence boundary | +|---|---|---| +| [DA-002 — chart rendering](history.md#da-002-render-nested-stacked-chart-results-without-losing-series) | Released normalization of nested and flat chart results. | Separate renderer defect and live/saved display checks; not acceptance of S53 metric/filter continuity. | +| [DA-004 — answer delivery and persistence](history.md#da-004-deliver-and-save-the-primary-answer-before-optional-work) | Candidate delivery and saved-result work. | Independent acceptance remains incomplete; no production promotion claimed. | +| [DA-005 — installable reference application](history.md#da-005-preserve-a-complete-installable-reference-application) | Source accepted for one-repository installation. | Installation evidence does not establish general answer quality or load capacity. | +| [DA-006 — speed and correctness](history.md#da-006-evaluate-speed-together-with-correctness) | Evaluation-only investigation. | No accepted quality winner or general performance claim. | +| [DA-007 — evaluator reliability](history.md#da-007-reject-false-positive-evaluation-results) | Independently accepted evaluator source controls. | Synthetic comparator/fixture verification, not product or model accuracy. | +| [DA-009 — earlier one-provider attempt](history.md#da-009--one-provider-for-default-model-operations) and [DA-010 — selection policy](history.md#da-010--selection-uses-the-verified-table-catalog) | Failed provider/selection attempts retained; later selector source accepted. | Preserve the failures and source-review limits. Later successful installation evidence is recorded separately below. | +| [DA-011–DA-013 — model profiles and setup](history.md#one-key-fresh-install-qualification-closeout-2026-10-22-ist) | Central model configuration, canonical table flags and template inheritance. | [Bounded one-key installation qualified](evidence/2026-10-22-one-key-qualification.json): two questions on four synthetic rows, UI/chart, restart/saved reopen and direct API override. Not a benchmark retest. | +| [DA-DEC-001 — product scope](history.md#da-dec-001-core-capability-quality-takes-precedence-over-commercial-feature-parity) | Core quality takes priority over speculative feature parity. | Deferred capabilities remain explicit; this is a scope decision, not an implemented feature. | + +## Explore the evidence + +- [Detailed engineering history](history.md) — decisions, implementation, validation and limits. +- [Structured engineering records](records.json) — the current public status source for this index. +- [Published results and original verdicts](published-results.md), [machine-readable results](published-results.json), and [source baseline](baseline.md). +- [Maintainer summaries](evidence/2026-10-22-one-key-qualification.json) and [S64's historical validation boundaries](S64-unsupported-data.md#regression-controls-and-recorded-independent-validation). +- [History-maintenance rules](MAINTAINING.md) and [publication/source boundaries](../publication.md). + +## How future improvements are tracked + +New work should connect a **public issue → implementation PR → independent product +acceptance → history/index update → appropriate separate research retest**. Each +step records its actual outcome; a green CI job does not upgrade a benchmark. +A retest, when appropriate and authorized, produces new evidence while retaining +the original result. This index does not schedule or authorize any new run. + +Existing corrections can predate the public repository. Their retrospective +records identify the actual public code and already approved validation without +inventing historical public issues or PRs. Create a detailed case study when +meaningful engineering work warrants one; keep the main README to one useful +example and this index as the place to explore the rest. diff --git a/docs/source-export.json b/docs/source-export.json index 9919b21..17622c4 100644 --- a/docs/source-export.json +++ b/docs/source-export.json @@ -320,7 +320,7 @@ "docs/improvements/published-observations.json": "0546f15602447774cc7b08e7e3c6a331aba996afd849abc1f146799203ffbe6a", "docs/improvements/published-results.json": "6951d846a5f8a968264e948f4a8001779f5a709a50aafa799049e60316cb807b", "docs/improvements/published-results.md": "71648bfb948cd190b4a46322ccc45c3e9da7bfe78e275f41434a0de0bfe4fae1", - "docs/improvements/README.md": "4748c623602ad259667e7b11b095fb00e58ae25dae3c084cc5dfed0abc484764", + "docs/improvements/README.md": "32d20a0881ce7a391eb0c6c5616cf101abd2d554dfa8c0be7c161bf3fc92b4a1", "docs/improvements/records.json": "feadc6540f7632b6471bdcc2708bf4b487e8930542ca0cc26e1fb818f5425670", "docs/model-configuration.md": "0caab9561ad7c4fc8a7f340113cb23c2169e740271b0e9c299f8e5d954df5455", "docs/publication.md": "c354454f30da314d6837dc8a717eda323ec1182bdbde964654c8bd40f96a55b5", @@ -789,7 +789,7 @@ "frontend/vitest.setup.ts": "b9fc49f524436b712003dfaf8c89659a986f8a08acde4eb2b21922253d0067fb", "LICENSE": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30", "NOTICE": "3c880ad6a14f915c63684d5cf8a085dccbfc1c0d3cabd65015be2382ab3012f8", - "README.md": "ce80e546f99cbb92bbb228905b4e80860ba5b708d16a6425675e1bf49b23a620", + "README.md": "7548ce60af88dd0f69ef828c5bd019567604e8f48d8fc03538513d46bc4b5dd5", "tests/accuracy/fixtures/answerability.sql": "124ef5ae989bea644445dd9f2115a5a9f15bfb58395d80cc2d0a41e64e59fa13", "tests/accuracy/fixtures/reviews-positive.sql": "759dda31bcdc4ec460df6ebba2e06bcbd047b164327600841c70f6f98dc60d2e", "tests/accuracy/fixtures/reviews-zero.sql": "2a7736327df80803b096d5eb5de92fe5379c9b0d76c77acbbee261201076e6e1",