PDF-First Extraction Pipeline Setup¶

This guide covers installation and configuration of the PDF-first content extraction pipeline for bioRxiv papers.

Overview¶

The extraction pipeline uses a multi-strategy approach with graceful fallback:

Nougat - Learning-based PDF parser (best for complex papers)
MarkItDown - Fast document-to-markdown extraction (included by default)
pdftotext - Command-line PDF tool (system package)
langextract - Robust HTML extraction (Google's library)
Regex - Final HTML fallback (always available)

All strategies except regex are optional - the system gracefully degrades when dependencies are unavailable.

Installation¶

Core Dependencies (Included)¶

These are already in pyproject.toml:

markitdown>=0.1.5  # MarkItDown for fast document-to-markdown extraction
aiohttp>=3.13.3    # Async HTTP for PDF downloads

Optional: Nougat (Recommended for Scientific Papers)¶

Nougat is Facebook Research's learning-based PDF parser trained on scientific documents. It excels at: - Mathematical equations and notation - Complex tables and figures - Structured document layout - Scientific paper formatting

Installation:

pip install nougat-ocr

Requirements: - Python 3.8+ - CUDA (optional, for GPU acceleration) - ~2GB disk space for model weights

First Run: The model will be downloaded automatically on first use (~1.5GB). This happens once and is cached locally.

Performance: - CPU: ~60-120 seconds per paper - GPU: ~10-30 seconds per paper

Optional: langextract (Recommended for HTML)¶

Google's langextract provides robust HTML content extraction, better than regex-based parsing.

Installation:

pip install langextract

Benefits: - Identifies main content vs navigation/ads - Handles complex HTML structures - More reliable than regex patterns

Optional: pdftotext (Alternative PDF Tool)¶

Command-line PDF extraction tool from Poppler.

Ubuntu/Debian:

sudo apt-get install poppler-utils

macOS:

brew install poppler

Windows: Download from Poppler for Windows

Configuration¶

Basic Usage¶

from src.ingest_pipeline.content_extractor import ContentExtractor

# Use default configuration (all available strategies enabled)
extractor = ContentExtractor()
text = await extractor.extract_from_url("https://www.biorxiv.org/content/...")

Custom Configuration¶

from src.ingest_pipeline.content_extractor import ContentExtractor
from src.ingest_pipeline.extraction_models import ExtractionConfig

# Customize extraction behavior
config = ExtractionConfig(
    # Strategy enablement
    enable_nougat=True,          # Use Nougat if available
    enable_markitdown=True,      # Use MarkItDown
    enable_pdftotext=True,       # Use pdftotext if available
    enable_langextract=True,     # Use langextract if available

    # Timeouts
    nougat_timeout=120,          # Nougat timeout (seconds)
    pdf_timeout=10,              # MarkItDown/pdftotext timeout

    # Post-processing
    remove_references=True,      # Remove References section
    remove_acknowledgements=True,# Remove Acknowledgements section

    # Quality thresholds
    quality_min_length=100,      # Minimum text length
    quality_max_nonalpha_ratio=0.5  # Max non-alphanumeric ratio
)

extractor = ContentExtractor(config=config)

Disable Specific Strategies¶

# Fast extraction only (skip Nougat)
config = ExtractionConfig(
    enable_nougat=False,  # Skip slow Nougat processing
    enable_markitdown=True,  # Use fast MarkItDown
)

# HTML-only extraction
config = ExtractionConfig(
    enable_nougat=False,
    enable_markitdown=False,
    enable_pdftotext=False,
    enable_langextract=True,  # HTML only
)

Verification¶

Check which strategies are available:

from src.ingest_pipeline.content_extractor import ContentExtractor

extractor = ContentExtractor()

# Check available strategies
for strategy in extractor.strategies:
    print(f"{strategy.name}: {'✓' if strategy.is_available() else '✗'}")

Expected output with all dependencies:

nougat: ✓
markitdown: ✓
pdftotext: ✓
langextract: ✓
regex: ✓

Troubleshooting¶

Nougat Model Download Fails¶

If the model download fails, you can manually download it:

# Download model manually
python -c "from nougat import NougatModel; NougatModel.from_pretrained()"

pdftotext Not Found¶

Verify installation:

which pdftotext  # Should show path to binary
pdftotext -v     # Should show version

langextract Import Error¶

Verify installation:

python -c "import langextract; print('OK')"

Memory Issues with Nougat¶

Nougat requires significant memory (~2-4GB per process). If you encounter OOM errors:

Reduce concurrent extractions
Disable Nougat for batch processing: enable_nougat=False
Use GPU if available (much faster and more memory efficient)

Performance Considerations¶

Strategy Performance¶

Strategy	Speed	Quality	Memory	Use Case
Nougat	Slow (60-120s)	Excellent	High (2-4GB)	Complex papers with equations
MarkItDown	Fast (1-2s)	Good	Low (50MB)	General PDFs, markdown output
pdftotext	Fast (1-2s)	Good	Low (50MB)	Alternative to MarkItDown
langextract	Fast (<1s)	Good	Low (50MB)	HTML fallback
Regex	Fast (<1s)	Fair	Low (10MB)	Final fallback

Recommendations¶

For Production (High Volume):

config = ExtractionConfig(
    enable_nougat=False,      # Skip for speed
    enable_markitdown=True,   # Fast and reliable
    enable_langextract=True,  # Good HTML fallback
)

For Research (High Quality):

config = ExtractionConfig(
    enable_nougat=True,       # Best quality
    nougat_timeout=180,       # Allow more time
    enable_markitdown=True,   # Fallback
)

For Development (Fast Iteration):

config = ExtractionConfig(
    enable_nougat=False,      # Skip slow processing
    enable_markitdown=True,   # Fast enough
)

Testing¶

Run the property-based tests to verify the extraction pipeline:

# Run all PDF extraction property tests
python -m pytest tests/properties/test_pdf_first_priority_properties.py -m properties -v

# Run with more examples for thorough testing
python -m pytest tests/properties/ -m properties --hypothesis-max-examples=1000

References¶

Nougat Paper - "Nougat: Neural Optical Understanding for Academic Documents"
langextract GitHub - Google's HTML content extraction library
MarkItDown GitHub - Microsoft's document-to-markdown library
Design Document - Architecture and implementation details