---
name: pageindex
description: "Vectorless, reasoning-based document indexing and RAG. Builds a hierarchical tree index from PDF or Markdown files using LLM reasoning (no vector DB, no chunking). Use when the user wants to: index a document, create a tree index, do reasoning-based retrieval, query/search a PDF or markdown, analyze document structure, or ask questions about a document. Triggers on: 'index this PDF', 'pageindex', 'tree index', 'query this document', 'search in this PDF', 'what does this document say about X', 'analyze this document'."
---

# PageIndex — Vectorless Reasoning-Based Document Indexing & RAG

Build hierarchical tree indexes from documents and query them with LLM reasoning. No vector DB, no chunking — human-like retrieval that achieves 98.7% accuracy on FinanceBench.

## When to Use

- User wants to index a PDF or Markdown file
- User wants to query/search a document intelligently
- User asks "what does this document say about X?"
- User wants reasoning-based RAG without vector DB
- User mentions "pageindex", "tree index", "document index"

## Core Philosophy

**Similarity ≠ Relevance.** Vector search finds similar text, but relevance requires reasoning. PageIndex simulates how a human expert navigates a document: look at the table of contents, reason about which section likely has the answer, go there, read it.

## Architecture: Two Phases

### Phase 1: INDEX — Build the Tree
```
Document → Extract pages → Detect TOC → Generate hierarchical structure → Verify → Fix → Add summaries → Save JSON
```

### Phase 2: RETRIEVE — Tree Search RAG
```
Query → Load tree index → LLM reasons over tree → Select relevant nodes → Extract page text → Generate answer with citations
```

---

## PHASE 1: INDEXING A DOCUMENT

### Step 1: Extract Pages

For PDF — write a Python script using PyPDF2:

```python
import PyPDF2
import json

def extract_pages(pdf_path):
    reader = PyPDF2.PdfReader(pdf_path)
    pages = []
    for i, page in enumerate(reader.pages):
        text = page.extract_text() or ""
        tokens = len(text) // 4  # ~4 chars per token
        pages.append({"page_num": i + 1, "text": text, "tokens": tokens})
    return pages
```

For Markdown — parse headers directly:

```python
import re

def extract_md_structure(md_path):
    with open(md_path, 'r') as f:
        content = f.read()
        lines = content.split('\n')

    nodes = []
    in_code_block = False
    for i, line in enumerate(lines):
        if line.strip().startswith('```'):
            in_code_block = not in_code_block
            continue
        if in_code_block:
            continue
        match = re.match(r'^(#{1,6})\s+(.+)$', line)
        if match:
            level = len(match.group(1))
            title = match.group(2).strip()
            nodes.append({"level": level, "title": title, "line_num": i + 1})
    return nodes, lines
```

### Step 2: Detect Table of Contents (PDF only)

Check the first 20 pages. For each page, use this prompt:

```
Your job is to detect if there is a table of content provided in the given text.

Given text: {page_text}

return the following JSON format:
{
    "thinking": <why do you think there is a table of content in the given text>,
    "toc_detected": "<yes or no>"
}

Directly return the final JSON structure. Do not output anything else.
Please note: abstract, summary, notation list, figure list, table list, etc. are not table of contents.
```

Algorithm: Check pages sequentially. Once TOC is found, keep collecting contiguous TOC pages. Stop when a non-TOC page appears after finding TOC pages.

### Step 3: Generate Structure

Three strategies depending on TOC detection result:

#### Strategy A: TOC with page numbers found

1. Extract TOC text, replace dots (`....`) with colons
2. Transform to JSON:

```
You are given a table of contents. Transform the whole table of content into a JSON format.

structure is the numeric system which represents the index of the hierarchy section. For example, the first section has structure index 1, the first subsection has structure index 1.1, etc.

Response format:
{
  "table_of_contents": [
    {"structure": "<x.x.x>", "title": "<section title>", "page": <page number or null>}
  ]
}

You should transform the full table of contents in one go.
Directly return the final JSON structure, do not output anything else.
```

3. Calculate page offset: Find where sections actually start in the physical document. `offset = physical_page - toc_page`. Apply to all items.

#### Strategy B: TOC without page numbers

Same as A, but then search the document with tagged pages to find each section:

```
You are given a table of contents in JSON format and several pages of a document.
Add the physical_index to the table of contents.

Pages contain tags like <physical_index_X> to indicate the physical location of page X.

Response format:
[
    {"structure": "<x.x.x>", "title": "<title>", "physical_index": "<physical_index_X>"}
]

Only add physical_index to sections found in the provided pages.
Directly return the final JSON structure. Do not output anything else.
```

#### Strategy C: No TOC — generate from scratch

Process document text in windows with page tags:

```
You are an expert in extracting hierarchical tree structure. Generate the tree structure of the document.

structure is the numeric system (e.g., "1", "1.1", "1.2").
Extract original titles from text, only fix space inconsistency.
Text contains tags like <physical_index_X> indicating page boundaries.

Response format:
[
    {"structure": "<x.x.x>", "title": "<section title>", "physical_index": "<physical_index_X>"}
]

Directly return the final JSON structure. Do not output anything else.
```

For continuation across page windows:

```
You are an expert in extracting hierarchical tree structure.
You are given a tree structure of the previous part and the text of the current part.
Continue the tree structure from the previous part to include the current part.

Previous structure: {previous_json}
Current text: {current_text_with_tags}

Return only the ADDITIONAL part of the structure. Do not output anything else.
```

### Step 4: Verify Accuracy (PDF only)

For each TOC item, verify the title appears at the assigned page:

```
Your job is to check if the given section appears or starts in the given page_text.
Note: do fuzzy matching, ignore any space inconsistency.

Section title: {title}
Page text: {page_text}

Reply format:
{
    "thinking": <reasoning>,
    "answer": "yes or no"
}
Directly return the final JSON structure. Do not output anything else.
```

Run concurrently for all items. Track accuracy percentage.

### Step 5: Fix Incorrect Assignments

For items that failed verification, search surrounding pages:

```
You are given a section title and several pages of a document. Find the physical index of the start page.

Pages contain tags like <physical_index_X> indicating page boundaries.

Section title: {title}
Document pages: {surrounding_pages_text}

Reply in JSON:
{
    "thinking": <explain which page contains the section start>,
    "physical_index": "<physical_index_X>"
}
Directly return the final JSON structure. Do not output anything else.
```

Retry up to 3 times. **Fallback cascade**: If accuracy < 60%, try next strategy (A → B → C).

### Step 6: Build Tree & Post-Process

Convert flat list `[{structure: "1.2.3", ...}]` to nested tree:

```python
def build_tree(flat_items):
    nodes = {}
    roots = []
    for item in flat_items:
        structure = item['structure']
        node = {
            'title': item['title'],
            'start_index': item.get('start_index'),
            'end_index': item.get('end_index'),
            'nodes': []
        }
        nodes[structure] = node
        parts = structure.split('.')
        parent_key = '.'.join(parts[:-1]) if len(parts) > 1 else None
        if parent_key and parent_key in nodes:
            nodes[parent_key]['nodes'].append(node)
        else:
            roots.append(node)
    return roots
```

Add `end_index` calculation: each node's end_index = next sibling's start_index (or parent's end_index for last child).

Split large nodes recursively if they exceed 10 pages or 20,000 tokens — re-run structure generation on the node's page range.

### Step 7: Add Node IDs and Summaries

Assign sequential 4-digit IDs (0000, 0001, ...):

```python
def add_node_ids(tree, counter=[0]):
    for node in tree:
        node['node_id'] = str(counter[0]).zfill(4)
        counter[0] += 1
        if node.get('nodes'):
            add_node_ids(node['nodes'], counter)
```

Generate summaries for each node (skip if text < 200 tokens — use text directly):

```
You are given a part of a document. Generate a description of the main points covered.

Partial Document Text: {node_text}

Directly return the description, do not include any other text.
```

### Step 8: Save Index (PERSISTENCE)

**CRITICAL**: Always save the tree index JSON for reuse across sessions.

```python
import os, json

def save_index(tree_data, doc_path, index_dir=".pageindex"):
    os.makedirs(index_dir, exist_ok=True)
    doc_name = os.path.splitext(os.path.basename(doc_path))[0]
    index_path = os.path.join(index_dir, f"{doc_name}.json")

    with open(index_path, 'w', encoding='utf-8') as f:
        json.dump(tree_data, f, indent=2, ensure_ascii=False)

    # Save manifest for multi-doc lookup
    manifest_path = os.path.join(index_dir, "manifest.json")
    manifest = {}
    if os.path.exists(manifest_path):
        with open(manifest_path, 'r') as f:
            manifest = json.load(f)

    manifest[doc_name] = {
        "index_file": f"{doc_name}.json",
        "source_path": os.path.abspath(doc_path),
        "doc_description": tree_data.get("doc_description", ""),
        "indexed_at": datetime.now().isoformat()
    }

    with open(manifest_path, 'w') as f:
        json.dump(manifest, f, indent=2)
```

Save location priority:
1. Project directory: `./.pageindex/`
2. Home config: `~/.pageindex/`

### Output Schema

```json
{
  "doc_name": "report.pdf",
  "doc_description": "Annual financial report for Company X covering FY2025.",
  "structure": [
    {
      "title": "Executive Summary",
      "node_id": "0000",
      "start_index": 1,
      "end_index": 5,
      "summary": "Overview of financial performance, key metrics, and strategic outlook...",
      "nodes": [
        {
          "title": "Financial Highlights",
          "node_id": "0001",
          "start_index": 2,
          "end_index": 3,
          "summary": "Revenue grew 15% YoY to $2.3B...",
          "nodes": []
        }
      ]
    }
  ]
}
```

---

## PHASE 2: TREE SEARCH RETRIEVAL (THE KEY DIFFERENTIATOR)

This is what makes PageIndex powerful. Instead of vector similarity, the LLM **reasons** through the tree like a human expert navigating a book.

### Step 1: Load Index

Check if an index already exists before re-indexing:

```python
def load_index(doc_path, index_dir=".pageindex"):
    doc_name = os.path.splitext(os.path.basename(doc_path))[0]
    index_path = os.path.join(index_dir, f"{doc_name}.json")
    if os.path.exists(index_path):
        with open(index_path, 'r') as f:
            return json.load(f)
    return None
```

### Step 2: Tree Search — LLM Reasons Over the Tree

This is the CORE prompt. Present the full tree structure (without text content, only titles + summaries + node_ids) and let the LLM reason about which nodes contain the answer:

```
You are given a question and a tree structure of a document.
Each node contains a node_id, title, page range, and a summary describing its content.
Your task is to find all nodes that are most likely to contain the answer to the question.

IMPORTANT: Reason step by step:
1. Read the question carefully
2. Scan the tree structure top-down
3. For each branch, consider whether its title and summary suggest relevance
4. Select the most specific (deepest) nodes that are relevant
5. Prefer leaf nodes over parent nodes when the leaf clearly matches

Question: {query}

Document tree structure:
{tree_json_without_text}

Reply in the following JSON format:
{
    "thinking": "<Your step-by-step reasoning about which nodes are relevant, referencing specific titles and summaries>",
    "node_list": ["node_id_1", "node_id_2", ..., "node_id_n"]
}
Directly return the final JSON structure. Do not output anything else.
```

**Why this works**: The LLM sees the ENTIRE document structure at once — titles, summaries, hierarchy — and can reason about WHERE information is likely to be. This is fundamentally different from vector search which matches text chunks in isolation.

### Step 3: Extract Content from Selected Nodes

Map the selected node_ids back to page ranges and extract text:

```python
def extract_node_content(tree_data, node_ids, pdf_path):
    node_map = build_node_map(tree_data['structure'])
    content_parts = []

    for nid in node_ids:
        node = node_map.get(nid)
        if not node:
            continue

        start_page = node['start_index']
        end_page = node['end_index']

        reader = PyPDF2.PdfReader(pdf_path)
        text = ""
        for p in range(start_page - 1, min(end_page, len(reader.pages))):
            text += f"\n--- Page {p + 1} ---\n"
            text += reader.pages[p].extract_text() or ""

        content_parts.append({
            "node_id": nid,
            "title": node['title'],
            "pages": f"{start_page}-{end_page}",
            "text": text
        })

    return content_parts


def build_node_map(structure):
    node_map = {}
    def traverse(nodes):
        for node in nodes:
            if 'node_id' in node:
                node_map[node['node_id']] = node
            if node.get('nodes'):
                traverse(node['nodes'])
    if isinstance(structure, list):
        traverse(structure)
    else:
        traverse([structure])
    return node_map
```

### Step 4: Generate Answer with Citations

```
You are a document analysis expert. Answer the question using ONLY the provided document sections.

RULES:
- Base your answer strictly on the provided content
- Cite specific page numbers for every claim: (p.X) or (pp.X-Y)
- If the content doesn't contain the answer, say so explicitly
- Quote key passages when relevant
- Structure your answer clearly with headers if the answer is complex

Question: {query}

Retrieved Sections:
{formatted_sections}

Provide a comprehensive, well-cited answer.
```

### Step 5: Display Results

Present to the user:
1. Which nodes were selected and why (the "thinking" from tree search)
2. The answer with page citations
3. The node_ids and titles used as sources

---

## PHASE 2B: MULTI-DOCUMENT RETRIEVAL

When the user has multiple indexed documents and asks a question:

### Step 1: Document Selection

Load the manifest and present all indexed documents:

```
You are given a list of indexed documents. Select which documents are likely to contain information relevant to the user's query.

Query: {query}

Documents:
{manifest_entries_with_descriptions}

Reply in JSON:
{
    "thinking": "<reasoning about which documents are relevant>",
    "selected_docs": ["doc_name_1", "doc_name_2"]
}
```

### Step 2: Tree Search Per Document

Run tree search on each selected document, then merge results.

### Step 3: Cross-Document Answer

```
Answer the question using content from multiple documents. Cite both the document name and page number for each claim.

Question: {query}

Document 1 ({doc1_name}) — Retrieved Sections:
{doc1_sections}

Document 2 ({doc2_name}) — Retrieved Sections:
{doc2_sections}

Provide a comprehensive answer citing sources as (Document Name, p.X).
```

---

## PHASE 2C: EXPERT KNOWLEDGE INJECTION

For specialized domains, inject expert preferences into tree search:

```
You are given a question and a tree structure of a document.
Each node contains a node_id, title, page range, and a summary.
Find all nodes likely to contain the answer.

Question: {query}

Document tree structure:
{tree_json_without_text}

Expert Knowledge of relevant sections: {expert_preference}

Reply in JSON:
{
    "thinking": "<reasoning referencing expert knowledge>",
    "node_list": ["node_id_1", ...]
}
```

Example expert preferences:
- "For EBITDA adjustments, prioritize Item 7 (MD&A) and footnotes in Item 8 (Financial Statements) in 10-K reports."
- "For methodology questions in research papers, check both the Methods section AND supplementary materials."
- "For legal clauses, check both the main agreement body AND schedules/exhibits."

---

## WORKFLOW SUMMARY

When the user invokes this skill:

### If indexing a new document:
1. Check if index already exists in `.pageindex/` — if yes, skip to retrieval
2. Extract pages (PDF: PyPDF2/pymupdf, MD: regex header parsing)
3. Detect TOC (first 20 pages for PDF)
4. Generate structure (Strategy A/B/C based on TOC detection)
5. Verify and fix page assignments
6. Build hierarchical tree
7. Split large nodes (>10 pages or >20k tokens)
8. Add node IDs (4-digit zero-padded)
9. Add summaries (LLM-generated, skip if <200 tokens)
10. Save index JSON to `.pageindex/{doc_name}.json` + update manifest
11. Show tree overview to user

### If querying a document:
1. Load existing index (or create if not exists)
2. Extract page text from PDF source (kept separate from index for size)
3. Run Tree Search: LLM reasons over tree to select relevant nodes
4. Extract text from selected nodes' page ranges
5. Generate answer with page citations
6. Show: reasoning trace + answer + sources

### If querying across multiple documents:
1. Load manifest from `.pageindex/manifest.json`
2. Document selection: LLM picks relevant docs by description
3. Tree search per selected document
4. Cross-document answer generation with dual citations

## Configuration

| Parameter | Default | Description |
|-----------|---------|-------------|
| toc_check_pages | 20 | Pages to scan for TOC |
| max_pages_per_node | 10 | Split threshold (pages) |
| max_tokens_per_node | 20000 | Split threshold (tokens) |
| summary_token_threshold | 200 | Min tokens to trigger LLM summary |

## Tips

- Always save indexes — re-indexing costs time and tokens
- For large PDFs (100+ pages), the indexing may take a few minutes
- The tree search itself is fast (single LLM call)
- Include summaries during indexing — they dramatically improve retrieval accuracy
- For multi-document RAG, always generate doc_descriptions during indexing
- Markdown files are instant to index (no verification needed)
- The fallback cascade (A→B→C) ensures robustness even with messy PDFs
