Extract text for an LLM

You’ll turn a PDF into three outputs an LLM or a search index can use: Markdown in reading order, JSON giving every paragraph and table its page and box, and 400-token chunks.

In TypeScript the function returns the three outputs and needs the extract module, init({ extract: true }). In Rust, the program instead extracts once and writes the three files. Both give the same JSON:

TypeScript
Rust

// Extract a PDF's text for an LLM: Markdown in reading order, JSON with a box for every block,
// and chunks sized for retrieval.
//
// Tables drawn with lines come out as Markdown tables, and text a reader can't see (white,
// invisible or off the page) is left out of all three.
import { type Chunk, Document, type Extraction } from "@laminapdf/sdk";

/** What a PDF gives an LLM or a search index. */
export interface ForAnLlm {
  /** The text in reading order, with a `<!-- page N -->` comment before each page. */
  readonly markdown: string;
  /** Every block with its page and box, the text left out, the file's warnings and the chunks. */
  readonly json: Extraction;
  /** About 400 tokens each, a new one at each heading. */
  readonly chunks: readonly Chunk[];
}

/** The Markdown, the JSON and the chunks of `bytes`. Needs `init({ extract: true })`. */
export function extractForAnLlm(bytes: Uint8Array): ForAnLlm {
  using doc = Document.open(bytes);
  // Page markers let a reader cite the page a passage came from.
  const markdown = doc.extract({ format: "markdown", pageMarkers: true });
  // The JSON carries the same chunks, and each block's place in the Markdown.
  const json = doc.extract({ format: "json", pageMarkers: true, chunk: { maxTokens: 400 } });
  return { markdown, json, chunks: json.chunks ?? [] };
}

//! Extract a PDF's text for an LLM: Markdown in reading order, JSON with a box for every
//! block, and chunks sized for retrieval.
//!
//! Tables drawn with lines come out as Markdown tables, and text a reader can't see (white,
//! invisible or off the page) is left out of all three.
//!
//! ```sh
//! cargo run -p laminapdf --example extract_markdown -- <file.pdf> [out-dir]
//! # writes <out-dir>/<file>.md, <file>.json and <file>.chunks.jsonl
//! ```

use std::error::Error;
use std::path::{Path, PathBuf};

use laminapdf::extract::{ChunkOptions, ExtractOptions, MarkdownOptions, chunks_jsonl};
use laminapdf::{Document, OpenOptions};

fn main() -> Result<(), Box<dyn Error>> {
    let mut args = std::env::args().skip(1);
    let input = PathBuf::from(args.next().ok_or("usage: extract_markdown <file.pdf> [out-dir]")?);
    let out = PathBuf::from(args.next().unwrap_or_else(|| "target/examples-out".into()));
    std::fs::create_dir_all(&out)?;
    let stem = input.file_stem().and_then(|s| s.to_str()).unwrap_or("document");
    write_outputs(&std::fs::read(&input)?, &out, stem)
}

/// Writes the Markdown, the JSON and the chunks of `bytes` into `out`.
pub(crate) fn write_outputs(bytes: &[u8], out: &Path, stem: &str) -> Result<(), Box<dyn Error>> {
    let doc = Document::open_bytes(bytes.to_vec(), OpenOptions::new())?;
    let extraction = doc.extract(&ExtractOptions::new())?;

    // Page markers let a reader cite the page a passage came from.
    let markdown = extraction.to_markdown(&MarkdownOptions::new().with_page_markers(true));
    std::fs::write(out.join(format!("{stem}.md")), &markdown.text)?;

    // About 400 tokens a chunk, starting a new one at each heading.
    let chunks = extraction.chunks(&markdown, &ChunkOptions::new().with_max_tokens(400));
    std::fs::write(out.join(format!("{stem}.chunks.jsonl")), chunks_jsonl(&chunks))?;

    let json = doc.extract_json(&extraction, &markdown, Some(chunks));
    std::fs::write(out.join(format!("{stem}.json")), json)?;
    Ok(())
}

Run the Rust program on any PDF: cargo run -p laminapdf --example extract_markdown -- report.pdf out.

How it works

extract (doc.extract in TypeScript) reads every page’s text, joins it into lines and paragraphs, and puts the paragraphs in reading order. A two-column page reads down the left column, then the right; a title across both columns comes first.
Tables drawn with lines become table blocks with their cells, and Markdown writes them as tables. A cell that spans columns or rows makes the table HTML, since Markdown tables can’t span.
Text a reader can’t see never reaches the outputs: invisible text (such as the hidden layer of a scanned page), text off the page and text too small to read. The JSON lists each such run under dropped, with the reason and how many characters it held.
to_markdown (format: "markdown" in TypeScript) writes headings, paragraphs and tables, with a  comment before each page, so a passage can be traced to its page.
chunks (format: "chunks", or the chunk option with the JSON) groups whole paragraphs and tables up to the budget, starting a new chunk at each heading. Each chunk records the blocks, pages and boxes it came from, and where its text sits in the Markdown.
extract_json (format: "json") writes the laminapdf.extract/1 document: the pages, every block with its box in points from the top-left corner, the dropped text, the file’s warnings and the chunks.

What to change

Only some pages: pages: [0, 1] in TypeScript, ExtractOptions::new().with_pages(vec![0, 1]) in Rust (page indexes start at 0).
Tables without lines: tables: "lattice-and-stream" (.with_tables(TableMode::LatticeAndStream)) also looks for text lined up in columns. It finds more tables, and now and then a pair of columns that isn’t one.
Bigger or smaller chunks: change maxTokens (with_max_tokens). A token here is about four characters, a rough count that runs low for code, numbers and scripts other than Latin, so leave room.
Overlap between chunks: overlap: 80 (.with_overlap(80)) repeats up to about 80 tokens of whole paragraphs from the end of the previous chunk.
One output (TypeScript): set format to "markdown", "text", "json" or "chunks".
A long file in steps (Rust): doc.extract_job(&options) and step do a page per unit of work, so a long file can report progress or stop between steps.

Next: all recipes.