Receipt OCR Pipeline

A Replicate pipeline model that extracts line items from receipt images using dual OCR methods with intelligent reconciliation.

Overview

This pipeline processes receipt images to extract detailed line item information including item names, unit prices, quantities, and totals. It uses two different OCR approaches and applies consensus analysis with AI-powered reconciliation to provide the most accurate results possible.

How It Works

The pipeline follows a 5-step process:

1. Claude Vision Analysis: Direct image analysis using Claude-4-Sonnet’s vision capabilities to extract structured line items
2. Text Extract OCR: Raw text extraction using a specialized OCR model, then structured parsing with Claude
3. Consensus Detection: Compares results from both methods to identify agreements and differences
4. Difference Analysis: Catalogs discrepancies between the two OCR approaches
5. AI Reconciliation: Uses Claude to analyze all results and create a final, authoritative list of line items

Input

• receipt_image: A receipt image file (JPG, PNG, etc.)

Output

A JSON structure containing:

{
  "consensus": [
    {
      "item": "Product Name",
      "price": "$X.XX",
      "quantity": 2,
      "total": "$Y.YY", 
      "confidence": "high|medium|low",
      "agreement": "both_models|claude_only|ocr_only"
    }
  ],
  "differences": [
    {
      "type": "details_mismatch|claude_only|ocr_only",
      "item": "Product Name",
      "claude": {"price": "$X.XX", "quantity": 2, "total": "$Y.YY"},
      "ocr": {"price": "$X.XX", "quantity": 2, "total": "$Y.YY"}
    }
  ],
  "reconciled": [
    {
      "item": "Final Product Name",
      "price": "$X.XX",
      "quantity": 2, 
      "total": "$Y.YY",
      "confidence": "high|medium|low",
      "reasoning": "Explanation of reconciliation decision"
    }
  ],
  "summary": {
    "consensus_count": 5,
    "differences_count": 2,
    "reconciled_count": 7,
    "reconciliation_needed": true
  }
}

Key Features

• Dual OCR Approach: Combines vision-based and text-extraction OCR for maximum accuracy
• Intelligent Consensus: Automatically identifies where both methods agree vs. disagree
• AI Reconciliation: Uses advanced language model reasoning to resolve discrepancies
• Detailed Structure: Extracts price, quantity, and total for each line item
• Confidence Scoring: Provides confidence levels for each extracted item
• Comprehensive Output: Returns raw data, consensus analysis, differences, and final reconciled results

Use Cases

• Expense tracking and accounting
• Receipt digitization for bookkeeping
• Inventory management
• Financial record automation
• Audit trail creation

Accuracy

The dual-method approach with AI reconciliation significantly improves accuracy over single-method OCR, especially for:

• Handwritten or poor-quality receipts
• Complex receipt layouts
• Items with quantities and calculations
• Receipts with various fonts or formatting

The consensus analysis helps identify high-confidence extractions while flagging areas that need human review.