3fiAddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z ddlmZddlmZddlmZddlmZddlmZerdd lmZGd d eZGd d eZy)N)Path) TYPE_CHECKINGAnyDictIterableListOptionalTupleUnion)Document) Embeddings) VectorStore) ConfigDict)Self neural_dbc *eZdZUdZd)dZdZded< edZe d*d e e ddfd Z e d*d e e d edefd Ze d*d ee efd e e defdZe d*dee dede eededdf dZ d*dee de eededee fdZ d+deee dfdedededee f dZdeee dfdedfdZde deee fddfdZdee e efddfdZ!d e d!e ddfd"Z"d#ee e e fddfd$Z# d,de d%ededee$fd&Z%d'e ddfd(Z&y)-NeuralDBVectorStoreaVectorstore that uses ThirdAI's NeuralDB. To use, you should have the ``thirdai[neural_db]`` python package installed. Example: .. code-block:: python from langchain_community.vectorstores import NeuralDBVectorStore from thirdai import neural_db as ndb db = ndb.NeuralDB() vectorstore = NeuralDBVectorStore(db=db) db ndb.NeuralDBreturnNc||_yNrselfrs o/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_community/vectorstores/thirdai_neuraldb.py__init__zNeuralDBVectorStore.__init__ forbidextra thirdai_keyc ddlm}tjj d|j |xst jdy#t$r tdwxYw)Nr) licensingzthirdai.neural_db THIRDAI_KEYz{Could not import thirdai python package and neuraldb dependencies. Please install it with `pip install thirdai[neural_db]`.) thirdair& importlibutil find_specactivateosgetenv ImportError)r$r&s r_verify_thirdai_libraryz+NeuralDBVectorStore._verify_thirdai_library*s[  ) NN $ $%8 9   {Fbii .F G K  s A AA% model_kwargsc jtj|ddlm}||jdi|S)a Create a NeuralDBVectorStore from scratch. To use, set the ``THIRDAI_KEY`` environment variable with your ThirdAI API key, or pass ``thirdai_key`` as a named parameter. Example: .. code-block:: python from langchain_community.vectorstores import NeuralDBVectorStore vectorstore = NeuralDBVectorStore.from_scratch( thirdai_key="your-thirdai-key", ) vectorstore.insert([ "/path/to/doc.pdf", "/path/to/doc.docx", "/path/to/doc.csv", ]) documents = vectorstore.similarity_search("AI-driven music therapy") rrr)rr0r(rNeuralDB)clsr$r1ndbs r from_scratchz NeuralDBVectorStore.from_scratch8s.: 33K@,lcll2\233r checkpointc|tj|ddlm}||jj |S)a: Create a NeuralDBVectorStore with a base model from a saved checkpoint To use, set the ``THIRDAI_KEY`` environment variable with your ThirdAI API key, or pass ``thirdai_key`` as a named parameter. Example: .. code-block:: python from langchain_community.vectorstores import NeuralDBVectorStore vectorstore = NeuralDBVectorStore.from_checkpoint( checkpoint="/path/to/checkpoint.ndb", thirdai_key="your-thirdai-key", ) vectorstore.insert([ "/path/to/doc.pdf", "/path/to/doc.docx", "/path/to/doc.csv", ]) documents = vectorstore.similarity_search("AI-driven music therapy") rrr)rr0r(rr4from_checkpoint)r5r8r$r6s rr:z#NeuralDBVectorStore.from_checkpointZs/< 33K@,cll22:>??r texts embedding metadataskwargsc ti}d|vr |d|d<|d=|jdi|}|j||fi||S)z9Return VectorStore initialized from texts and embeddings.r$r3)r7 add_texts)r5r;r<r=r>r1 vectorstores r from_textszNeuralDBVectorStore.from_texts}sZ F "*0*?L '}%&c&&66  eY9&9r c ddl}ddlm}|jd|i}|r.|j ||jj |gd}t jddd }|j||j|j|jgfi|d}|jjjj|d} t!t#|D cgc]} t%| | zc} Scc} w) arRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. kwargs: vectorstore specific parameters Returns: List of ids from adding the texts into the vectorstore. rNrr;)axiswF)deletedelete_on_close)pandasr(r DataFrameconcat from_recordstempfileNamedTemporaryFileto_csvinsertCSVnamer_savable_state documentsget_source_by_idrangelenstr) rr;r=r>pdr6dftemp source_idoffsetis rr@zNeuralDBVectorStore.add_textss , \\7E* + B 9 9) DEANB**3ueT $DKK!3 4??B ''11BB9MaP).s5z):;AFQJ;;;s0Dsourcesz ndb.Documenttrain fast_modec d|j|}|jjd|||d|S)auInserts files / document sources into the vectorstore. Args: train: When True this means that the underlying model in the NeuralDB will undergo unsupervised pretraining on the inserted files. Defaults to True. fast_mode: Much faster insertion with a slight drop in performance. Defaults to True. )r_r`fast_approximationr3)_preprocess_sourcesrrP)rr_r`rar>s rrPzNeuralDBVectorStore.insertsC **73tww~~ (    r cddlm}|s|Sg}|D]}t|ts|j |%|j j dr"|j |j|f|j j dr"|j |j||j j dr"|j |j|td|d|S)zChecks if the provided sources are string paths. If they are, convert to NeuralDB document objects. Args: sources: list of either string paths to PDF, DOCX or CSV files, or NeuralDB document objects. rrz.pdfz.docxz.csvzCould not automatically load z. Only files with .pdf, .docx, or .csv extensions can be loaded automatically. For other formats, please use the appropriate document object from the ThirdAI library.) r(r isinstancerXappendlowerendswithPDFDOCXrQ RuntimeError)rr_r6preprocessed_sourcesdocs rrdz'NeuralDBVectorStore._preprocess_sourcess -N! Cc3'$++C099;''/(// =YY[))'2(// >YY[))&1(// =&7u=PP "$#r query document_idcN|jj|t|y)a%The vectorstore upweights the score of a document for a specific query. This is useful for fine-tuning the vectorstore to user behavior. Args: query: text to associate with `document_id` document_id: id of the document to associate query with. N)rtext_to_resultint)rrorps rupvotezNeuralDBVectorStore.upvotes uc+&67r query_id_pairsc |jj|Dcgc]\}}|t|fc}}ycc}}w)aGiven a batch of (query, document id) pairs, the vectorstore upweights the scores of the document for the corresponding queries. This is useful for fine-tuning the vectorstore to user behavior. Args: query_id_pairs: list of (query, document id) pairs. For each pair in this list, the model will upweight the document id for the query. N)rtext_to_result_batchrs)rrurodoc_ids r upvote_batchz NeuralDBVectorStore.upvote_batchs4 $$7E FmeVeS[ ! F Fs; sourcetargetc<|jj||y)aAThe vectorstore associates a source phrase with a target phrase. When the vectorstore sees the source phrase, it will also consider results that are relevant to the target phrase. Args: source: text to associate to `target`. target: text to associate `source` to. N)r associate)rrzr{s rr}zNeuralDBVectorStore.associates &&)r text_pairsc:|jj|y)a0Given a batch of (source, target) pairs, the vectorstore associates each source phrase with the corresponding target phrase. Args: text_pairs: list of (source, target) text pairs. For each pair in this list, the source will be associated with the target. N)rassociate_batch)rr~s rrz#NeuralDBVectorStore.associate_batchs  +r kc f |jjd||d|}|Dcgc]a}t|j|j|j |j |j|j|jddcc}Scc}w#t$r}td||d}~wwxYw)Retrieve {k} contexts with for a given query Args: query: Query to submit to the model k: The max number of context results to retrieve. Defaults to 10. rotop_krD)id upvote_idsrzmetadatascorecontext page_contentr"Error while retrieving documents: Nr3) rsearchr textrrrzrrr Exception ValueErrorrrorr> referencesrefes rsimilarity_searchz%NeuralDBVectorStore.similarity_searchs N'Ge1GGJ& !$!ff&)nn"%**$'LL!$#&;;q>      NA!EFA M Ns)#BA&B BB B0B++B0pathc:|jj|y)zSaves a NeuralDB instance to disk. Can be loaded into memory by calling NeuralDB.from_checkpoint(path) Args: path: path on disk to save the NeuralDB instance to. N)rsave)rrs rrzNeuralDBVectorStore.save,s  Tr )rrrNr)TT )'__name__ __module__ __qualname____doc__rr__annotations__r model_config staticmethodr rXr0 classmethodrrr7r rr:rr dictrBrr@listboolrPrdrsrtr ryr}rr rrr3r rrrs BL Xc] d  &*4c]44  44B&* @#t)$ @c] @  @ @D +/ CyDJ'     &+/<}<DJ'< < c <>  eC/01      c  0 $E#~"567 $ n  $D8C8eCHo8$8  4c3h+@  T   * *S *T *,$uS#X*?,D,$&NN N14N hN8r rc eZdZUdZddZdZded< edZ dd e d e d e de e fd Zd e ee e fde fdZdee ddfdZy)NeuralDBClientVectorStoreasVectorstore that uses ThirdAI's NeuralDB Enterprise Python Client for NeuralDBs. To use, you should have the ``thirdai[neural_db]`` python package installed. Example: .. code-block:: python from langchain_community.vectorstores import NeuralDBClientVectorStore from thirdai.neural_db import ModelBazaar, NeuralDBClient bazaar = ModelBazaar(base_url="http://{NEURAL_DB_ENTERPRISE_IP}/api/") bazaar.log_in(email="user@thirdai.com", password="1234") ndb_client = NeuralDBClient( deployment_identifier="user/model-0:user/deployment-0", base_url="http://{NEURAL_DB_ENTERPRISE_IP}/api/", bazaar=bazaar ) vectorstore = NeuralDBClientVectorStore(db=ndb_client) retriever = vectorstore.as_retriever(search_kwargs={'k':5}) rndb.NeuralDBClientrNc||_yrrrs rrz"NeuralDBClientVectorStore.__init__Nrr r!r"rorr>c |jjd ||d|d}|Dcgc]'}t|d|d|d|d|d|dd )c}Scc}w#t$r}t d ||d }~wwxYw) rrrrrrzrr)rrzrrrrrNr3)rrr rrrs rrz+NeuralDBClientVectorStore.similarity_searchXs N'Ge1GG UJ& !$V!$i"%h-$' O!$X#&y>      NA!EFA M Ns(&A,AAA A9%A44A9rTc8|jj|S)a Inserts documents into the VectorStore and return the corresponding Sources. Args: documents (List[Dict[str, Any]]): A list of dictionaries that represent documents to be inserted to the VectorStores. The document dictionaries must be in the following format: {"document_type": "DOCUMENT_TYPE", **kwargs} where "DOCUMENT_TYPE" is one of the following: "PDF", "CSV", "DOCX", "URL", "SentenceLevelPDF", "SentenceLevelDOCX", "Unstructured", "InMemoryText". The kwargs for each document type are shown below: class PDF(Document): document_type: Literal["PDF"] path: str metadata: Optional[dict[str, Any]] = None on_disk: bool = False version: str = "v1" chunk_size: int = 100 stride: int = 40 emphasize_first_words: int = 0 ignore_header_footer: bool = True ignore_nonstandard_orientation: bool = True class CSV(Document): document_type: Literal["CSV"] path: str id_column: Optional[str] = None strong_columns: Optional[List[str]] = None weak_columns: Optional[List[str]] = None reference_columns: Optional[List[str]] = None save_extra_info: bool = True metadata: Optional[dict[str, Any]] = None has_offset: bool = False on_disk: bool = False class DOCX(Document): document_type: Literal["DOCX"] path: str metadata: Optional[dict[str, Any]] = None on_disk: bool = False class URL(Document): document_type: Literal["URL"] url: str save_extra_info: bool = True title_is_strong: bool = False metadata: Optional[dict[str, Any]] = None on_disk: bool = False class SentenceLevelPDF(Document): document_type: Literal["SentenceLevelPDF"] path: str metadata: Optional[dict[str, Any]] = None on_disk: bool = False class SentenceLevelDOCX(Document): document_type: Literal["SentenceLevelDOCX"] path: str metadata: Optional[dict[str, Any]] = None on_disk: bool = False class Unstructured(Document): document_type: Literal["Unstructured"] path: str save_extra_info: bool = True metadata: Optional[dict[str, Any]] = None on_disk: bool = False class InMemoryText(Document): document_type: Literal["InMemoryText"] name: str texts: list[str] metadatas: Optional[list[dict[str, Any]]] = None global_metadata: Optional[dict[str, Any]] = None on_disk: bool = False For Document types with the arg "path", ensure that the path exists on your local machine. )rrP)rrTs rrPz NeuralDBClientVectorStore.insertssdww~~i((r source_idsc:|jj|y)z Deletes documents from the VectorStore using source ids. Args: files (List[str]): A list of source ids to delete from the VectorStore. N)rrG)rrs rremove_documentsz*NeuralDBClientVectorStore.remove_documentss z"r )rrrNr)rrrrrrrrrrXrsrrr rrrPrrr3r rrr6s. $B#"L $&NN N14N hN6R)T#s(^ 4R)R)h#49##r r)r)r-rMpathlibrtypingrrrrrr r r langchain_core.documentsr langchain_core.embeddingsr langchain_core.vectorstoresrpydanticrtyping_extensionsrr(rr6rrr3r rrsL SSS-03"(b+bJ Y# Y#r