Getting the Source Right, Not Just the Fact: Source-Aware Verification for MCP Agents

Tool-using AI agents that pull information from multiple sources, such as patient records, databases, and documents, face a verification challenge: a fact can be true somewhere in the evidence but attributed to the wrong source, which matters especially in sensitive domains like healthcare or finance. ProvenanceGuard is a verification system that checks not just whether claims are factual, but whether they are supported by the specific source an agent names or implies, rather than pooling all evidence together anonymously. The system breaks answers into individual claims, finds the most relevant source for each one, checks if that source supports it, and compares the source to what the answer states. Testing on medical agent responses showed ProvenanceGuard caught 138 of 139 unsupported claims while identifying the correct source for claims about 86 percent of the time, outperforming existing verification methods that do not track which tool output supports each claim.