--- title: "How we built Agent Fill" description: "The story behind Agent Fill - an AI agent that automatically fills out forms by understanding context, extracting data, and navigating complex workflows." date: "2025-08-27" authors: "Ramp Labs" tag: "Agents" slug: "how-we-built-agent-fill" canonical: "https://labs.ramp.com/research/how-we-built-agent-fill/" --- > [Figure: how-we-built-agent-fill/ascii-hero — ascii-logo] *The story behind Agent Fill - an AI agent that automatically fills out forms by understanding context, extracting data, and navigating complex workflows.* Ramp Labs is the home for AI experiments from Ramp (@tryramp). Our focus is shipping AI tools for the finance community and sharing learnings on building with AI. Today, we launched Agent Fill, our agentic PDF form filler, in alpha. Read about how we built it below. ## Finance teams waste time filling out PDFs No one dreads filling out PDF forms more than finance teams. Countless hours are wasted on repetitive data entry that requires painstaking attention to detail. That's why we started to build Agent Fill to help our internal finance team move faster. Now, we're sharing the tool with the community. ## How we built this technically When filling out a PDF form, humans visually navigate complex pages, interpret questions, and draw from their knowledge to input answers accurately. A traditional approach to automate this process parses and engages with a form’s file structure directly. While these systems can get the job done in certain ways, they tend to be brittle, and lack the breadth or context required to handle the high structural variance and nuanced content found in financial documents. To accomplish this in an AI-native way, Agent Fill is equipped with a diverse arsenal of tools. > [Figure: how-we-built-agent-fill/knowledge-extraction — agent-flow] We start with knowledge extraction from previously filled forms to give our agent the relevant context it needs to fill blank forms. Throwing raw PDFs at Gemini 2.5 Pro (@googleaidevs) takes advantage of its efficacy in producing rich, nuanced output from multimodal data when schema constraints are not enforced. > [Figure: how-we-built-agent-fill/preprocessing — agent-flow] To help our agent understand the form we want to fill in, we transform it to text with OCR and extract any present PDF widgets with pymupdf. This pool of data is fed to Claude Sonnet 4 (@AnthropicAI) for structured extraction. Claude excels with schema fidelity, making it the perfect choice in mapping our data pool to form fields, interpreting the structure of complex financial documents in a format that can be handed off to the agent. > [Figure: how-we-built-agent-fill/pipeline — agent-flow] When a human fills out a form on a computer, they click, type, and scroll in their document editor. With Anthropic’s experimental yet formidable [computer use tool](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool), Claude Sonnet 4 can do the same. Combining the layered context we generated earlier with screenshots of the form featuring bounding boxes over fillable fields, the agent has everything it needs to fill out the diverse array of PDF forms encountered by finance teams. Agents are clearly capable, but financial documents can be nebulous and mistakes are inevitable. Our goal is not to squeeze perfection from non-deterministic AI models, but speedy completion of tedious forms. So, we built a simple PDF editor for a human to quickly review and edit the filled form before downloading. Agent Fill pairs AI automation with human in the loop UX to fill out PDFs in minutes, not hours. ## Learnings 1. The power of multimodal context. When we provided the AI agent with partially redundant context in multiple modalities (e.g. screenshot of the form, OCR text, PDF widget data), its effectiveness in executing complex tasks and reasoning improved significantly. 2. Our flexible agent architecture allowed us to optimize each step of the agent's workflow. We were able to quickly experiment and swap out tools to increase accuracy in results. 3. Despite the effectiveness of AI agents, for critical financial workflows, a human in the loop is still necessary for the end-to-end experience. We built a PDF editor for the human to review the agent's results. Try out Agent Fill today: https://labs.ramp.com/agent-fill Thank you to @hshaiyon, @shevchenkoaalex, @alexstauffer_, @thetkkong, and @zackpizackpi for contributing to this project. Want to get early access to future AI experiments? Sign up on our [newsletter](https://forms.gle/vVz5Rw8Zemr5Sdmv7) and follow us on X @RampLabs. ## Citation ## Citation See article metadata for citation details.