Collections with metadata
Group documents into collections and tag them with metadata. Collections work with all three products: PageIndex Agent, Page Search and Page Extract. PageIndex Agent can also use folders.
The examples use the Python SDK (pip install -U pageindex), with the client set up as in the Quick start. In JavaScript, the agent tools can list collections and their documents; see Use with your agent.
Collections
A collection is a named group of documents, as a folder holds files: a project, a customer, a knowledge base, which you search whole.
- A document is in at most one collection: the one it is uploaded into. It cannot be moved into a collection later.
- The same file uploaded into another collection is another document there, with its own metadata; uploaded again into the same one, it is the document already there, and metadata sent with that upload is not applied.
- Deleting a collection deletes its documents.
- Metadata describes each document: what it is, to filter by. A collection says which set it belongs to.
reports = client.create_collection("Annual reports")["collection"]["id"]
doc_id = client.submit_document("3M_2018_10K.pdf", collection_id=reports, wait=True)["doc_id"] # a new document, in it
for doc in client.list_documents(collection_id=reports)["documents"]:
print(doc["id"], doc["name"])
To find and remove collections:
for c in client.list_collections()["collections"]: # newest first
print(c["id"], c["name"], c["document_count"])
client.get_collection(reports) # one collection, the same fields
client.delete_collection(reports) # the collection and every document in it
delete_collection() refuses while one of its documents is still processing; call it again once that document is done.
Ask within a collection
With PageIndex Agent, pass collection_id= to keep a question to one collection's documents. The agent can pick them out by their metadata:
answer = client.chat("How did 3M's revenue change from 2018 to 2019?", collection_id=reports)
Page Search can search a whole collection too: see Search pages.
Metadata
Give each document in a collection your own keys, on upload or later: up to 16, each a string, number or boolean. Setting a key to None removes it; keys you do not name stay. A key keeps the type of its first value in the collection: a value of another type is refused.
client.submit_document("3M_2019_10K.pdf", collection_id=reports,
metadata={"ticker": "MMM", "year": 2019}) # on upload
client.update_document(doc_id, metadata={"ticker": "MMM", "year": 2018}) # or later; None removes a key
recent = client.list_documents(collection_id=reports, filter={"year": {"$gte": 2019}})["documents"]
To see which keys a collection's documents have, with their types and values:
schema = client.get_metadata_schema(collection_id=reports)["schema"] # None until a document has metadata
for key, field in (schema or {}).get("fields", {}).items():
print(key, field["type"], field["documents"], field.get("values") or (field.get("min"), field.get("max")))
Declare the fields
A collection learns its keys from its documents' metadata. To fix them up front instead, declare them, each with a type and an optional description that tells the agent what it means:
client.create_metadata_schema(collection_id=reports, fields={
"ticker": {"type": "string", "description": "Stock ticker"},
"year": {"type": "number", "description": "Fiscal year"},
})
- Documents may then use only these keys, with their types; a document may leave any of them out.
- Declaring removes, from documents already in the collection, the keys it does not declare or declares with another type.
- Fields are declared once. To change them, call
client.delete_metadata_schema(collection_id=reports), then declare them again.
Filters
A filter picks documents by metadata; a document must meet every condition.
| Condition | Holds when the key's value | Example |
|---|---|---|
$eq | equals it; a value alone means the same | {"ticker": "MMM"} |
$ne | does not equal it, or the key is missing | {"ticker": {"$ne": "MMM"}} |
$in | is one of them | {"ticker": {"$in": ["MMM", "AAPL"]}} |
$gt, $gte | is greater than (or equal to) it | {"year": {"$gte": 2020}} |
$lt, $lte | is less than (or equal to) it | {"year": {"$lt": 2025}} |
$and | meets every filter in the list | {"$and": [{"year": 2018}, {"audited": true}]} |
$or | meets any filter in the list | {"$or": [{"ticker": "MMM"}, {"region": "US"}]} |
- Values compare exactly:
"2018"is not2018. - Store dates as numbers, such as
20240301, to compare them with$gt,$gte,$ltand$lte. - A key takes one condition; for a range, join two with
$and:{"$and": [{"year": {"$gte": 2020}}, {"year": {"$lt": 2025}}]}. - A document without the key meets only
$ne. $andand$ormay nest, up to five deep.