Fudan's Open-Source Health Agent Posts 45.7% Accuracy on Its Own Synthetic Benchmark

Fudan University researchers have released HealthClaw, an open-source personal health agent. The system achieved 45.7% answer accuracy on a synthetic benchmark created by the team and spanning a simulated year, according to the project’s GitHub repository and an arXiv paper.

The benchmark tested whether HealthClaw’s governed memory could retain facts from simulated daily health histories and use them in later interactions, while withholding information that the system should neither store nor disclose. The researchers present the results as evidence that the system could support users over extended periods without relying on cloud-based memory. They caution, however, that its clinical effectiveness still requires prospective evaluation.

HealthClaw’s March release

According to the repository, HealthClaw was publicly released on March 22, 2026. Most of the authors are affiliated with the School of Data Science at Fudan University in Shanghai. Haoran Li and Jiebi Deng contributed equally, while Hongcheng Guo conceived, led and supervised the study.

Other authors came from Fudan’s Institute of Science and Technology for Brain-Inspired Intelligence, Huashan Hospital, Beijing University of Chinese Medicine and Huazhong University of Science and Technology.

HealthClaw uses a memory-processing loop that runs after each interaction. The system keeps behavioral rules and safety restrictions separate from general medical knowledge, personal profile information, reusable procedures and records of individual episodes.

After an interaction ends, an induction process determines whether the information should update the user’s profile, revise a reusable procedure, remain as a time-stamped record or be excluded from future use.

The README describes how this architecture is presented to users. HealthClaw includes a Streamlit web interface, a Feishu bot and a command-line interface. It supports several model backends and can switch to fallback models when necessary.

The repository also states a clear medical limitation: “This system is for assistance only; it is not medical advice and does not replace professional care.”

64,493 characters in full-history prompting

The researchers evaluated HealthClaw on 900 longitudinal-support questions that did not involve privacy restrictions. Answer accuracy rose from 0.2% when the model received only the current interaction to 45.7% when it used HealthClaw’s memory system.

The automated rubric score increased from 0.182 to 0.568, while coverage of the reference facts rose from 0.027 to 0.524.

A Qwen-3.7-based evaluator graded the responses using predetermined rubrics. The paper notes that no human evaluators participated in this part of the analysis.

Full-history prompting, which received the complete visible dialogue before the query day, reached 0.612 answer accuracy, a 0.752 automated rubric score and 0.759 reference-fact coverage, beating HealthClaw on all three. It averaged 64,492.64 prompt characters after one simulated year, compared with 18,273.73 for HealthClaw. The paper calculates that as a 71.7% reduction in prompt-side context exposure.

Privacy probes produced a different ordering. In 100 probes, HealthClaw scored 0.640 answer accuracy and 0.696 on the automated rubric, ahead of current-only prompting and full-history prompting. Constraint violations appeared in 24% of HealthClaw responses, compared with 41% for current-only prompting and 53% for full-history prompting. Unauthorized disclosure was 5% for HealthClaw, 18% for current-only prompting and 15% for full-history prompting.

Nine tasks and Supplementary Table 1

The paper also evaluated HealthClaw on nine biomedical tasks, each with 200 cases. The tasks covered CT nodules, ultrasound breast images, skin lesions, fundus images, ICU SOFA prediction, diabetes readmission, protein localization, genomic question answering and multi-omics classification. The main text reports a mean absolute primary-metric gain of 27.0 percentage points, with seven gains remaining significant after Benjamini-Hochberg false-discovery-rate correction.

The largest primary-metric gains came from NoduleMNIST3D CT, GeneTuring and PAD-UFES-20. NoduleMNIST3D rose from 0.2150 to 0.8300, a gain of 61.5 percentage points. GeneTuring exact-match accuracy rose from 0.0650 to 0.5900. PAD-UFES-20 rose from 0.3900 to 0.8050. BreastMNIST, DeepLoc, ODIR5K and MLOmics also improved on the primary metric.

For NoduleMNIST3D, the task with the largest gain, the tool the agent called was a MedMNIST-derived ResNet18-3D classifier run against public MedMNIST benchmark volumes. The evaluation note in Supplementary Table 1 says, "Potential same-source overlap; not a zero-shot comparison." PAD-UFES-20 used an RG-DermNet-based classifier with PAD-UFES-20 label mapping. The diabetes readmission task used a UCI-derived XGBoost model on UCI data.

The authors write that the nine-task comparison "supports agentic routing and evidence integration rather than zero-shot clinical generalization." The paper also states that the conditions used different model backbones and evaluation pipelines, making the biomedical results "a matched case-level comparison rather than a strict same-model ablation."

The failures in the table

Two primary-metric gains were small and non-significant. PhysioNet ICU SOFA prediction rose from 0.4700 to 0.5200, a 5.0 percentage-point gain. Diabetes readmission rose from 0.4050 to 0.4500, a 4.5 percentage-point gain. Neither remained significant on the primary metric.

Macro-F1 added another caveat. MLOmics accuracy rose from 0.2650 to 0.5000, but its macro-F1 gain was 2.6 percentage points and was not significant. Diabetes readmission moved the other way across metrics: its accuracy gain was non-significant, while its macro-F1 rose from 0.2950 to 0.4498 and remained significant after correction.

The paper's BEHSOF fatty-liver screening analysis gives a concrete failure case. The base model already favored NAFLD, and weak or poorly calibrated evidence retrieved from memory and tools reinforced that prior, narrowing the prediction distribution. The authors point to future checks on class distributions, tool-derived evidence calibration and explicit counter-evidence before recurrent memory is allowed to strengthen an existing label.

Google's comparator and the FDA document

Google Research evaluated its own Personal Health Agent on data from about 1,200 users who gave informed consent in an IRB-reviewed study to share Fitbit wearables data, a health questionnaire and blood test results. That evaluation ran across 10 benchmark tasks, Google Research reported, "involving more than 7,000 annotations and 1,100 hours of effort from health experts and end-users."

The Fudan authors position HealthClaw against that literature, citing work on sleep and fitness coaching published in Nature Medicine in 2025 and on wearable-data insights published in Nature Communications in 2026.

HealthClaw's longitudinal benchmark used 20 synthetic users, 7,300 daily turns and 1,000 evaluation queries, all of them generated. The authors state the limit themselves: "The year-long trajectories were simulated and graded by a Qwen-3.7-based evaluator without human ratings." They write that prospective studies should test sustained use, clinician and user judgement, privacy and security, and stability under sparse feedback, contradiction and distribution shift.

The authors cite the FDA's Clinical Decision Support Software guidance, final guidance issued January 29, 2026, and state that diagnosis, treatment recommendation or autonomous intervention would require separate regulatory, ethics and security review followed by prospective clinical evaluation. The paper says a sanitized HealthClaw-YearLong benchmark will be released with publication of the article, including the schema, synthetic trajectories, evaluation queries, category annotations and representative examples after strong identifiers and non-public operational metadata are removed.

Claude-Mem Hit 80K Stars By Giving Your Second Session a Memory

San Francisco | Thursday, June 4, 2026 Claude-Mem, an open-source memory layer for coding agents, has drawn 80,253 GitHub stars since launch. Hooks intercept Claude Code tool use, compress sessions i

The Implicator

Repo Radar: 5 GitHub Projects Worth Your Week

The open-source projects gaining the most stars on GitHub this week were tools for running AI on a user's own hardware. openhuman, a Rust desktop agent, added more than 17,000 stars over seven days, p

The Implicator

Anthropic Opens Claude Security Beta as Mythos Access Fight Deepens

Anthropic has published Claude Security today for Claude Enterprise customers globally, according to company materials shared with The Implicator. The public beta turns the February Claude Code Securi

The Implicator

Marcus Schuler

San Francisco

Editor-in-Chief and founder of Implicator.ai. Former ARD correspondent and senior broadcast journalist with 10+ years covering tech. Writes daily briefings on policy and market developments. Based in San Francisco. E-mail: editor@implicator.ai