🏆 MIT Policy Hackathon Submission (2025)

DocScope Copilot

Automated auditing for AI documentation governance

22
Frameworks Audited
1,303
Chunks Processed
81%
Governance Gap Detected
0.537
Avg. Quality Score

The Challenge: An Opaque Industry

Voluntary AI documentation standards are currently failing. While frameworks like the NIST AI RMF exist, my analysis of 22 major model cards (including GPT-4o) revealed a systematic lack of substantive governance data.

We call this the "Framework-Practice Gap": the massive divergence between what governance frameworks recommend (e.g., detailed equity audits) and what companies actually disclose (marketing fluff).

The Solution: DocScope Copilot

I built DocScope Copilot, an automated auditing tool that ingests technical documentation (PDF/Markdown) and scores it against 8 weighted governance pillars.

Critical Findings

Equity & Bias Disclosure
0.5% (Actual) vs 2.7% (Rec.)

Critical Gap: Only 54.5% of documents contained any equity coverage.

Governance & Oversight
0.3% (Actual) vs 1.9% (Rec.)

Technical Implementation

The tool uses a three-stage pipeline:

  • Ingestion: Section-aware extraction using PyPDF2 and Regex, separating text from tables (tables weighted 1.3x for quantitative value).
  • NLP Analysis: Custom keyword scoring (343 keywords) combined with Negation Detection to distinguish "we tested for bias" from "no bias testing was performed."
  • Scoring Engine: Calculates an "Artifact Quality Score" (0-1.0) based on density of substantive technical disclosures vs. promotional language.

Built With

Python NLP (Spacy) Regex JSON Schema Streamlit
View Project on GitHub ← Back to Projects