Collections with metadata
Put related documents in a collection to search them together. Add metadata to each document, such as its company and year, then filter by it when you search.
You can use collections with PageIndex Agent, Page Search, and Page Extract. Set up your client as in the quick start first. For HTTP requests, see the API reference.
Collections
When you upload a file, pass the collection's ID as collection_id in Python or collectionId in JavaScript. You can only assign a document to a collection at upload time.
This example creates a collection named Annual reports, uploads your PDF to it, and lists the documents in that collection.
reports = client.create_collection("Annual reports")["collection"]["id"]
doc_id = client.submit_document("3M_2018_10K.pdf", collection_id=reports, wait=True)["doc_id"]
for doc in client.list_documents(collection_id=reports)["documents"]:
print(doc["id"], doc["name"])
import { readFileSync } from "node:fs";
const reports = (await client.api.createCollection({ name: "Annual reports" })).collection.id;
const { doc_id } = await client.api.submitDocument(readFileSync("3M_2018_10K.pdf"), "3M_2018_10K.pdf", {
collectionId: reports,
});
await client.api.waitDocument(doc_id);
// A collection's documents, by the collection's name
const { documents } = await client.tools.browseDocuments({ collection: "Annual reports" });
for (const doc of documents) {
console.log(doc.name, doc.status);
}
This upload preprocesses the document for PageIndex Agent, and costs $1 per 1,000 pages. If you only use Page Search or Page Extract, upload with the Page Search client instead, at $0.10 per 1,000 pages:
from pageindex import PageSearchClient
search = PageSearchClient() # reads PAGEINDEX_API_KEY
doc_id = search.upload("3M_2018_10K.pdf", collection=reports)["doc_id"]
import { PageSearchClient } from "@pageindex/sdk";
const search = new PageSearchClient(); // reads PAGEINDEX_API_KEY
const { doc_id } = await search.upload({ path: "3M_2018_10K.pdf" }, { collection: reports });
If you upload the same file again:
- Same file, same collection. PageIndex returns the existing document ID and keeps the saved metadata, even if you send different metadata with the second upload.
- Same file, different collection. PageIndex creates a separate document with its own ID and metadata. The original stays in its collection.
List your collections
List each collection's ID, name, and document count.
for c in client.list_collections()["collections"]: # newest first
print(c["id"], c["name"], c["document_count"])
client.get_collection(reports) # Look up one collection by ID
const { collections } = await client.api.listCollections(); // newest first
for (const c of collections) {
console.log(c.id, c.name, c.document_count);
}
Ask within a collection
Use reports, the collection ID from the upload example, to ask about just those documents.
With PageIndex Agent
The agent can use your metadata to choose which documents to read for its answer.
answer = client.chat("What was 3M's revenue in 2018?", collection_id=reports)
With Page Search
Page Search has its own client. Create it, find the relevant pages, then give their text to your LLM. See the Page Search quick start.
from pageindex import PageSearchClient
search = PageSearchClient() # reads PAGEINDEX_API_KEY
result = search.search("What was 3M's revenue in 2018?", collection=reports, include_text=True)
import { PageSearchClient } from "@pageindex/sdk";
const search = new PageSearchClient(); // reads PAGEINDEX_API_KEY
const result = await search.search("What was 3M's revenue in 2018?", { collection: reports, includeText: true });
Metadata
You can add metadata when you upload a document, change it later, and use it to filter which documents you search.
Add metadata on upload
Pass metadata when you upload the document. The examples use two fields, ticker and year. Choose fields that fit your documents.
client.submit_document(
"3M_2019_10K.pdf",
collection_id=reports,
metadata={"ticker": "MMM", "year": 2019},
)
await client.api.submitDocument(readFileSync("3M_2019_10K.pdf"), "3M_2019_10K.pdf", {
collectionId: reports,
metadata: { ticker: "MMM", year: 2019 },
});
search.upload() takes metadata the same way.
Each document can have up to 16 fields. A field's value can be a string, a number, or a boolean. Within a collection, the first value you give a field sets its type.
Edit metadata after upload
Use the document ID from the upload to change its metadata.
Pass the fields you want to change. Set a field to None to remove it.
client.update_document(doc_id, metadata={"ticker": "MMM", "year": 2018})
Use the REST API from Node.js. It replaces all the metadata, so read the current fields first and include the ones you want to keep.
const { metadata } = await client.api.getDocument(doc_id);
const response = await fetch(`https://api.pageindex.ai/doc/${doc_id}/metadata`, {
method: "PUT",
headers: {
Authorization: `Bearer ${process.env.PAGEINDEX_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ ...metadata, ticker: "MMM", year: 2018 }),
});
if (!response.ok) throw new Error(await response.text());
Collection schema
By default, PageIndex learns the field names and types from the metadata you add. These fields and their types make up the collection's schema.
The code below lists each field in Annual reports, its type, and how many documents use it. For a text field like ticker, it shows the most common values. For a number field like year, it shows the lowest and highest values.
schema = client.get_metadata_schema(collection_id=reports)["schema"]
for key, field in (schema or {}).get("fields", {}).items():
print(key, field["type"], field["documents"], field.get("values") or (field.get("min"), field.get("max")))
Declare the fields
Declare fields when you need everyone uploading to a collection to use the same field names and types. For example, if your team filters annual reports by year, declaring year as a number keeps that filter consistent across uploads. List the fields the collection should accept, and give each a type and an optional description. A document can leave out any of them.
client.create_metadata_schema(collection_id=reports, fields={
"ticker": {"type": "string", "description": "Stock ticker"},
"year": {"type": "number", "description": "Fiscal year"},
})
Use the schema endpoint from Node.js.
const response = await fetch(`https://api.pageindex.ai/collection/${reports}/metadata/schema`, {
method: "PUT",
headers: {
Authorization: `Bearer ${process.env.PAGEINDEX_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ fields: {
ticker: { type: "string", description: "Stock ticker" },
year: { type: "number", description: "Fiscal year" },
} }),
});
if (!response.ok) throw new Error(await response.text());
Change a declared schema
Once you declare a collection's fields, that declaration is fixed. Changing even one field requires replacing the schema.
- Delete the current schema. Every document keeps its metadata. PageIndex goes back to learning field names and types from it.
- Declare the replacement schema. List every field you want to keep, with its type. PageIndex keeps the existing fields that match, and deletes any field whose name is missing or whose value has the wrong type.
Filters
To find 3M reports from 2019 onward in Annual reports, filter by both ticker and year. The ticker must be MMM, and $gte means the year must be greater than or equal to 2019.
reports_3m = client.list_documents(
collection_id=reports,
filter={"ticker": "MMM", "year": {"$gte": 2019}},
)["documents"]
- Use
$andwhen every condition must match. Different fields in the same filter already work this way. Use an explicit$andto put two conditions on the same field, such as a start and end year. - Use
$orwhen any condition can match. Joining the same ticker and year conditions with$orincludes 3M reports from any year and reports from any company from 2019 onward.
| Condition | Holds when the key's value | Example |
|---|---|---|
$eq | equals it; a value alone means the same | {"ticker": "MMM"} |
$ne | does not equal it, or the key is missing | {"ticker": {"$ne": "MMM"}} |
$in | is one of them | {"ticker": {"$in": ["MMM", "AAPL"]}} |
$gt, $gte | is greater than (or equal to) it | {"year": {"$gte": 2020}} |
$lt, $lte | is less than (or equal to) it | {"year": {"$lt": 2025}} |
$and | meets every filter in the list | {"$and": [{"year": {"$gte": 2019}}, {"year": {"$lt": 2025}}]} |
$or | meets any filter in the list | {"$or": [{"ticker": "MMM"}, {"year": {"$gte": 2019}}]} |
- Values compare exactly:
"2018"is not2018. - Store dates as numbers, such as
20240301, to compare them with$gt,$gte,$ltand$lte.
To search with a filter, pass the same filter to Page Search, with the search client from above:
result = search.search(
"What was capital expenditure?",
collection=reports,
filter={"ticker": "MMM", "year": {"$gte": 2019}},
)
const result = await search.search("What was capital expenditure?", {
collection: reports,
filter: { ticker: "MMM", year: { $gte: 2019 } },
});
Delete a collection
client.delete_collection(reports)
await client.api.deleteCollection(reports);