Collections with metadata

Put related documents in a collection to search them together. Add metadata to each document, such as its company and year, then filter by it when you search.

You can use collections with PageIndex Agent, Page Search, and Page Extract. Set up your client as in the quick start first. For HTTP requests, see the API reference.

Collections

When you upload a file, pass the collection's ID as collection_id in Python or collectionId in JavaScript. You can only assign a document to a collection at upload time.

This example creates a collection named Annual reports, uploads your PDF to it, and lists the documents in that collection.

reports = client.create_collection("Annual reports")["collection"]["id"]

doc_id = client.submit_document("3M_2018_10K.pdf", collection_id=reports, wait=True)["doc_id"]

for doc in client.list_documents(collection_id=reports)["documents"]:
    print(doc["id"], doc["name"])

This upload preprocesses the document for PageIndex Agent, and costs $1 per 1,000 pages. If you only use Page Search or Page Extract, upload with the Page Search client instead, at $0.10 per 1,000 pages:

from pageindex import PageSearchClient

search = PageSearchClient()  # reads PAGEINDEX_API_KEY
doc_id = search.upload("3M_2018_10K.pdf", collection=reports)["doc_id"]

If you upload the same file again:

  • Same file, same collection. PageIndex returns the existing document ID and keeps the saved metadata, even if you send different metadata with the second upload.
  • Same file, different collection. PageIndex creates a separate document with its own ID and metadata. The original stays in its collection.

List your collections

List each collection's ID, name, and document count.

for c in client.list_collections()["collections"]:    # newest first
    print(c["id"], c["name"], c["document_count"])

client.get_collection(reports)       # Look up one collection by ID

Ask within a collection

Use reports, the collection ID from the upload example, to ask about just those documents.

With PageIndex Agent

The agent can use your metadata to choose which documents to read for its answer.

answer = client.chat("What was 3M's revenue in 2018?", collection_id=reports)

Page Search has its own client. Create it, find the relevant pages, then give their text to your LLM. See the Page Search quick start.

from pageindex import PageSearchClient

search = PageSearchClient()  # reads PAGEINDEX_API_KEY
result = search.search("What was 3M's revenue in 2018?", collection=reports, include_text=True)

Metadata

You can add metadata when you upload a document, change it later, and use it to filter which documents you search.

Add metadata on upload

Pass metadata when you upload the document. The examples use two fields, ticker and year. Choose fields that fit your documents.

client.submit_document(
    "3M_2019_10K.pdf",
    collection_id=reports,
    metadata={"ticker": "MMM", "year": 2019},
)

search.upload() takes metadata the same way.

Each document can have up to 16 fields. A field's value can be a string, a number, or a boolean. Within a collection, the first value you give a field sets its type.

Edit metadata after upload

Use the document ID from the upload to change its metadata.

Pass the fields you want to change. Set a field to None to remove it.

client.update_document(doc_id, metadata={"ticker": "MMM", "year": 2018})

Collection schema

By default, PageIndex learns the field names and types from the metadata you add. These fields and their types make up the collection's schema.

The code below lists each field in Annual reports, its type, and how many documents use it. For a text field like ticker, it shows the most common values. For a number field like year, it shows the lowest and highest values.

schema = client.get_metadata_schema(collection_id=reports)["schema"]
for key, field in (schema or {}).get("fields", {}).items():
    print(key, field["type"], field["documents"], field.get("values") or (field.get("min"), field.get("max")))

Declare the fields

Declare fields when you need everyone uploading to a collection to use the same field names and types. For example, if your team filters annual reports by year, declaring year as a number keeps that filter consistent across uploads. List the fields the collection should accept, and give each a type and an optional description. A document can leave out any of them.

client.create_metadata_schema(collection_id=reports, fields={
    "ticker": {"type": "string", "description": "Stock ticker"},
    "year": {"type": "number", "description": "Fiscal year"},
})

Change a declared schema

Once you declare a collection's fields, that declaration is fixed. Changing even one field requires replacing the schema.

  1. Delete the current schema. Every document keeps its metadata. PageIndex goes back to learning field names and types from it.
  2. Declare the replacement schema. List every field you want to keep, with its type. PageIndex keeps the existing fields that match, and deletes any field whose name is missing or whose value has the wrong type.

Filters

To find 3M reports from 2019 onward in Annual reports, filter by both ticker and year. The ticker must be MMM, and $gte means the year must be greater than or equal to 2019.

reports_3m = client.list_documents(
    collection_id=reports,
    filter={"ticker": "MMM", "year": {"$gte": 2019}},
)["documents"]
  • Use $and when every condition must match. Different fields in the same filter already work this way. Use an explicit $and to put two conditions on the same field, such as a start and end year.
  • Use $or when any condition can match. Joining the same ticker and year conditions with $or includes 3M reports from any year and reports from any company from 2019 onward.
ConditionHolds when the key's valueExample
$eqequals it; a value alone means the same{"ticker": "MMM"}
$nedoes not equal it, or the key is missing{"ticker": {"$ne": "MMM"}}
$inis one of them{"ticker": {"$in": ["MMM", "AAPL"]}}
$gt, $gteis greater than (or equal to) it{"year": {"$gte": 2020}}
$lt, $lteis less than (or equal to) it{"year": {"$lt": 2025}}
$andmeets every filter in the list{"$and": [{"year": {"$gte": 2019}}, {"year": {"$lt": 2025}}]}
$ormeets any filter in the list{"$or": [{"ticker": "MMM"}, {"year": {"$gte": 2019}}]}
  • Values compare exactly: "2018" is not 2018.
  • Store dates as numbers, such as 20240301, to compare them with $gt, $gte, $lt and $lte.

To search with a filter, pass the same filter to Page Search, with the search client from above:

result = search.search(
    "What was capital expenditure?",
    collection=reports,
    filter={"ticker": "MMM", "year": {"$gte": 2019}},
)

Delete a collection

client.delete_collection(reports)