The Problem
Iran’s stock exchange requires public companies to publish annual sustainability and financial reports. These documents arrive as unstructured, scanned Persian-language PDFs — inconsistent formatting, mixed tabular and narrative content, and no standardized schema. Extracting ESG compliance data manually required enormous effort and produced inconsistent results. No automated tooling existed for Persian financial NLP at this scale.
The Approach
I designed a modular, asynchronous pipeline with five stages:
- OCR layer — extracted raw text from scanned PDFs using optimized OCR tuned for Persian script
- NLP pre-processing — cleaned, segmented, and normalized the extracted text
- LLM extraction — structured prompts guided an LLM to identify ESG fields, values, and references
- Graph RAG — enabled cross-document reasoning to reconcile conflicting data across report sections
- Pydantic validation — enforced strict schema contracts on every extracted record, rejecting malformed outputs automatically
The backend was built with FastAPI for async processing of 150+ documents in parallel. LangChain orchestrated the LLM calls and retrieval logic.
The Outcome
- Structured dataset extracted from 150+ public companies within days
- Achieved a field extraction accuracy that passed manual spot-check validation
- Dataset enabled a university research paper analyzing social responsibility compliance across Iran’s stock exchange
- Pipeline architecture is reusable for any Persian-language document corpus
Tools Used
Python · LangChain · FastAPI · Pydantic · OCR · LLMs (GPT / local models) · Graph RAG · Pandas · AsyncIO