Collections with metadata

Group documents into collections and tag them with metadata. Collections work with all three products: PageIndex Agent, Page Search and Page Extract. PageIndex Agent can also use folders.

The examples use the Python SDK (pip install -U pageindex), with the client set up as in the Quick start. In JavaScript, the agent tools can list collections and their documents; see Use with your agent.

Collections

A collection is a named group of documents, as a folder holds files: a project, a customer, a knowledge base, which you search whole.

  • A document is in at most one collection: the one it is uploaded into. It cannot be moved into a collection later.
  • The same file uploaded into another collection is another document there, with its own metadata; uploaded again into the same one, it is the document already there, and metadata sent with that upload is not applied.
  • Deleting a collection deletes its documents.
  • Metadata describes each document: what it is, to filter by. A collection says which set it belongs to.
reports = client.create_collection("Annual reports")["collection"]["id"]

doc_id = client.submit_document("3M_2018_10K.pdf", collection_id=reports, wait=True)["doc_id"]  # a new document, in it

for doc in client.list_documents(collection_id=reports)["documents"]:
    print(doc["id"], doc["name"])

To find and remove collections:

for c in client.list_collections()["collections"]:    # newest first
    print(c["id"], c["name"], c["document_count"])

client.get_collection(reports)       # one collection, the same fields
client.delete_collection(reports)    # the collection and every document in it

delete_collection() refuses while one of its documents is still processing; call it again once that document is done.

Ask within a collection

With PageIndex Agent, pass collection_id= to keep a question to one collection's documents. The agent can pick them out by their metadata:

answer = client.chat("How did 3M's revenue change from 2018 to 2019?", collection_id=reports)

Page Search can search a whole collection too: see Search pages.

Metadata

Give each document in a collection your own keys, on upload or later: up to 16, each a string, number or boolean. Setting a key to None removes it; keys you do not name stay. A key keeps the type of its first value in the collection: a value of another type is refused.

client.submit_document("3M_2019_10K.pdf", collection_id=reports,
                       metadata={"ticker": "MMM", "year": 2019})                  # on upload
client.update_document(doc_id, metadata={"ticker": "MMM", "year": 2018})  # or later; None removes a key

recent = client.list_documents(collection_id=reports, filter={"year": {"$gte": 2019}})["documents"]

To see which keys a collection's documents have, with their types and values:

schema = client.get_metadata_schema(collection_id=reports)["schema"]  # None until a document has metadata
for key, field in (schema or {}).get("fields", {}).items():
    print(key, field["type"], field["documents"], field.get("values") or (field.get("min"), field.get("max")))

Declare the fields

A collection learns its keys from its documents' metadata. To fix them up front instead, declare them, each with a type and an optional description that tells the agent what it means:

client.create_metadata_schema(collection_id=reports, fields={
    "ticker": {"type": "string", "description": "Stock ticker"},
    "year": {"type": "number", "description": "Fiscal year"},
})
  • Documents may then use only these keys, with their types; a document may leave any of them out.
  • Declaring removes, from documents already in the collection, the keys it does not declare or declares with another type.
  • Fields are declared once. To change them, call client.delete_metadata_schema(collection_id=reports), then declare them again.

Filters

A filter picks documents by metadata; a document must meet every condition.

ConditionHolds when the key's valueExample
$eqequals it; a value alone means the same{"ticker": "MMM"}
$nedoes not equal it, or the key is missing{"ticker": {"$ne": "MMM"}}
$inis one of them{"ticker": {"$in": ["MMM", "AAPL"]}}
$gt, $gteis greater than (or equal to) it{"year": {"$gte": 2020}}
$lt, $lteis less than (or equal to) it{"year": {"$lt": 2025}}
$andmeets every filter in the list{"$and": [{"year": 2018}, {"audited": true}]}
$ormeets any filter in the list{"$or": [{"ticker": "MMM"}, {"region": "US"}]}
  • Values compare exactly: "2018" is not 2018.
  • Store dates as numbers, such as 20240301, to compare them with $gt, $gte, $lt and $lte.
  • A key takes one condition; for a range, join two with $and: {"$and": [{"year": {"$gte": 2020}}, {"year": {"$lt": 2025}}]}.
  • A document without the key meets only $ne.
  • $and and $or may nest, up to five deep.