Verifying LLM-extracted text with token alignment

Large language models can extract text effectively but may hallucinate content absent from their source. Exact matching verifies contiguous spans, whereas discontiguous extraction depends strongly on tokenization and ordered alignment. LLM-specific subword tokenization with ordered alignment improves performance by about 50% over word-level tokenization, and taln implements this practical verification approach.