Search pages

Find the pages that answer your questions, each with its relevance, in one document or across many. Send them to your LLM, in place of whole documents.

Questions

A search takes 1 to 10 questions, each up to 2,000 characters. Ten questions take about as long as one, and cost less than ten searches. Ask the questions as you would ask a person: "What was capital expenditure in 2018?", "Which pages describe the PFAS litigation?".

Relevance

A page's relevance is the chance it helps answer the question, from 0 to 1, given to one decimal: 1.0 means it almost certainly does, 0.1 that it almost certainly does not.

  • By default each question lists the pages likely to help, most relevant first. An answer spread over several sections gets more pages, and a question the document does not answer may get none.
  • To get a set number of pages, pass top_k: the most relevant pages, exactly that many.
  • When no page is listed, the document probably does not answer the question; pass top_k to see the closest pages anyway.

The answer to a question is often on more than one page, such as a figure in a summary table and again in a note.

In one document

Pass the document's id, as in Quick start: up to 10 questions at once, each with the pages likely to help, and more_pages, the next most relevant, to read if those do not answer. The document's pages are priced as pages.

  • start_page and end_page search only those pages, and only they are charged. One search covers up to 5,000 pages.
  • include_text=True gives each listed page with its text, to pass straight to your LLM.
  • Each page says how it is best read: "text", or "image" for a page that is a picture, such as a scan or a chart, whose text may be incomplete; read that one as a picture.
  • A document still being processed cannot be searched yet: upload it with wait=True.
result = client.search("What was capital expenditure in 2018?", document=doc_id,
                       start_page=30, end_page=60, include_text=True)

for page in result["results"][0]["documents"][0]["pages"]:
    print(page["page"], page["relevance"], page["read"], page["text"][:80])

Across documents

Leave the document out and give a scope: a collection, the documents whose metadata meets a filter, a list of documents, or all of yours. The documents likely to answer are found first, then the pages in them that do.

Every search has a scope: the documents it searches.

ScopePass
One documentdocument=doc_id
A list of documentsdocuments=[a, b]
A collectioncollection=reports
Documents whose metadata meets a filterfilter={...}
A collection, narrowed by a filtercollection=reports, filter={...}
All your documentsleave out
  • A filter works alone, with a collection, or across all your documents; not with a list of documents.
  • Send a list of documents or a collection, not both.
  • Across documents, a search covers up to 1,000 documents.
  • Documents kept in folders are searched the same way: folder=folder_id, with recursive=True for its subfolders too.
result = client.search(
    "What was capital expenditure in 2018?",
    collection=reports,      # a collection's id; or documents=[...]; leave both out for all your documents
    filter={"year": 2018},
)

for doc in result["results"][0]["documents"]:  # most relevant first
    print(doc["name"], doc["relevance"], [(p["page"], p["relevance"]) for p in doc["pages"]])

What search() returns, in Python:

{
  "id": "srch_…",
  "results": [
    {"question": "What was capital expenditure in 2018?",
     "documents": [{"id": "pi-cmwi3zsxk000109p6gspztvzb", "name": "3M_2018_10K.pdf", "pageNum": 160,
                    "folderId": null, "collectionId": "cmwi3zsj5000009p64s38ue3v",
                    "metadata": {"ticker": "MMM", "year": 2018}, "relevance": 1.0,
                    "pages": [{"page": 39, "relevance": 1.0, "read": "text"}, …],
                    "more_pages": […], "failed_pages": []}, …]}
  ],
  "failed_documents": [],
  "processing_documents": [],
  "usage": {"documents": …, "billable_documents": …, "pages": …, "billable_pages": …, "questions": 1, "credits": …, "balance": …},
  "seconds": …
}
  • max_documents: open at most this many documents a question (default 3, up to 10).
  • document_top_k: open exactly this many instead, the most relevant.
  • A document several questions open is searched once.
  • failed_pages lists pages that could not be read, and failed_documents documents that could not be searched; they are not charged. Documents still being processed are left out and listed in processing_documents.
  • Each document searched is priced as a document, and each page of the documents opened as a page (see Pricing). usage.credits is what the search cost; a credit is $0.01.

To see which documents answer before searching their pages, see Search documents.