Search pages
Find the pages that answer your questions, each with its relevance, in one document or across many. Send them to your LLM, in place of whole documents.
Questions
A search takes 1 to 10 questions, each up to 2,000 characters. Ten questions take about as long as one, and cost less than ten searches. Ask the questions as you would ask a person: "What was capital expenditure in 2018?", "Which pages describe the PFAS litigation?".
Relevance
A page's relevance is the chance it helps answer the question, from 0 to 1, given to one decimal: 1.0 means it almost certainly does, 0.1 that it almost certainly does not.
- By default each question lists the pages likely to help, most relevant first. An answer spread over several sections gets more pages, and a question the document does not answer may get none.
- To get a set number of pages, pass
top_k: the most relevant pages, exactly that many. - When no page is listed, the document probably does not answer the question; pass
top_kto see the closest pages anyway.
The answer to a question is often on more than one page, such as a figure in a summary table and again in a note.
In one document
Pass the document's id, as in Quick start: up to 10 questions at once, each with the pages likely to help, and more_pages, the next most relevant, to read if those do not answer. The document's pages are priced as pages.
start_pageandend_pagesearch only those pages, and only they are charged. One search covers up to 5,000 pages.include_text=Truegives each listed page with its text, to pass straight to your LLM.- Each page says how it is best
read:"text", or"image"for a page that is a picture, such as a scan or a chart, whose text may be incomplete; read that one as a picture. - A document still being processed cannot be searched yet: upload it with
wait=True.
result = client.search("What was capital expenditure in 2018?", document=doc_id,
start_page=30, end_page=60, include_text=True)
for page in result["results"][0]["documents"][0]["pages"]:
print(page["page"], page["relevance"], page["read"], page["text"][:80])
Across documents
Leave the document out and give a scope: a collection, the documents whose metadata meets a filter, a list of documents, or all of yours. The documents likely to answer are found first, then the pages in them that do.
Every search has a scope: the documents it searches.
| Scope | Pass |
|---|---|
| One document | document=doc_id |
| A list of documents | documents=[a, b] |
| A collection | collection=reports |
| Documents whose metadata meets a filter | filter={...} |
| A collection, narrowed by a filter | collection=reports, filter={...} |
| All your documents | leave out |
- A filter works alone, with a collection, or across all your documents; not with a list of documents.
- Send a list of documents or a collection, not both.
- Across documents, a search covers up to 1,000 documents.
- Documents kept in folders are searched the same way:
folder=folder_id, withrecursive=Truefor its subfolders too.
result = client.search(
"What was capital expenditure in 2018?",
collection=reports, # a collection's id; or documents=[...]; leave both out for all your documents
filter={"year": 2018},
)
for doc in result["results"][0]["documents"]: # most relevant first
print(doc["name"], doc["relevance"], [(p["page"], p["relevance"]) for p in doc["pages"]])
What search() returns, in Python:
{
"id": "srch_…",
"results": [
{"question": "What was capital expenditure in 2018?",
"documents": [{"id": "pi-cmwi3zsxk000109p6gspztvzb", "name": "3M_2018_10K.pdf", "pageNum": 160,
"folderId": null, "collectionId": "cmwi3zsj5000009p64s38ue3v",
"metadata": {"ticker": "MMM", "year": 2018}, "relevance": 1.0,
"pages": [{"page": 39, "relevance": 1.0, "read": "text"}, …],
"more_pages": […], "failed_pages": []}, …]}
],
"failed_documents": [],
"processing_documents": [],
"usage": {"documents": …, "billable_documents": …, "pages": …, "billable_pages": …, "questions": 1, "credits": …, "balance": …},
"seconds": …
}
max_documents: open at most this many documents a question (default 3, up to 10).document_top_k: open exactly this many instead, the most relevant.- A document several questions open is searched once.
failed_pageslists pages that could not be read, andfailed_documentsdocuments that could not be searched; they are not charged. Documents still being processed are left out and listed inprocessing_documents.- Each document searched is priced as a document, and each page of the documents opened as a page (see Pricing).
usage.creditsis what the search cost; a credit is $0.01.
To see which documents answer before searching their pages, see Search documents.