3fiT*ddlmZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZddlmZddlmZddlmZer dd lmZdd lmZmZdd lmZdd lm Z dd l!m"Z"ejFe$Z%edddGddeZ&y)) annotationsN) TYPE_CHECKINGAnyIterableListOptionalTupleType) deprecated)Document) Embeddings) VectorStore)get_client_infostorage)MatchingEngineIndexMatchingEngineIndexEndpoint) Namespace) CredentialsTensorflowHubEmbeddingsz0.0.12z1.0z1langchain_google_vertexai.VectorSearchVectorStore)sinceremovalalternative_importcHeZdZdZ ddd dfdZeddZddZ d ddZddZ d dd Z d d d Z d d!d Z d d"d Z d#d Zd$dZe d d%dZe d& d'dZed(dZe d)dZe d*dZe d+dZe d,dZe d-dZed.dZxZS)/MatchingEnginea@`Google Vertex AI Vector Search` (previously Matching Engine) vector store. While the embeddings are stored in the Matching Engine, the embedded documents will be stored in GCS. An existing Index and corresponding Endpoint are preconditions for using this module. See usage in docs/integrations/vectorstores/google_vertex_ai_vector_search.ipynb Note that this implementation is mostly meant for reading if you are planning to do a real time implementation. While reading is a real time operation, updating the index takes close to one hour.N)document_id_keyct ||j||_||_||_||_||_||_||_ ||_ y)aGoogle Vertex AI Vector Search (previously Matching Engine) implementation of the vector store. While the embeddings are stored in the Matching Engine, the embedded documents will be stored in GCS. An existing Index and corresponding Endpoint are preconditions for using this module. See usage in docs/integrations/vectorstores/google_vertex_ai_vector_search.ipynb. Note that this implementation is mostly meant for reading if you are planning to do a real time implementation. While reading is a real time operation, updating the index takes close to one hour. Attributes: project_id: The GCS project id. index: The created index class. See ~:func:`MatchingEngine.from_components`. endpoint: The created endpoint class. See ~:func:`MatchingEngine.from_components`. embedding: A :class:`Embeddings` that will be used for embedding the text sent. If none is sent, then the multilingual Tensorflow Universal Sentence Encoder will be used. gcs_client: The GCS client. gcs_bucket_name: The GCS bucket name. credentials (Optional): Created GCP credentials. document_id_key (Optional): Key for storing document ID in document metadata. If None, document ID will not be returned in document metadata. N) super__init__'_validate_google_libraries_installation project_idindexendpoint embedding gcs_client credentialsgcs_bucket_namer) selfr"r#r$r%r&r(r'r __class__s n/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_community/vectorstores/matching_engine.pyr zMatchingEngine.__init__1sXX  446$   "$&..c|jSN)r%)r)s r+ embeddingszMatchingEngine.embeddingsis ~~r,cR ddlm}m}ddlm}y#t $r t dwxYw)z>Validates that Google libraries that are needed are installed.r) aiplatformrservice_accountzwYou must run `pip install --upgrade google-cloud-aiplatform google-cloud-storage`to use the MatchingEngine Vectorstore.N) google.cloudr1r google.oauth2r3 ImportError)r)r1rr3s r+r!z6MatchingEngine._validate_google_libraries_installationms.  8 5 9  s&c t|}|;t|t|k7r$tdt|dt|dtj d|j j |}g}g}tt||D]k\}\}} ttj} |j| | |d} |||| d<|j| |j| d| mtj dt|d d j|D cgc]} tj | c} } d tj}|d t#j"d }|j| |tj d|j$d |d|j&j)d|j$d |d |_tj d|Scc} w)asRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. kwargs: vectorstore specific parameters. Returns: List of ids from adding the texts into the vectorstore. z:texts and metadatas do not have the same length. Received z texts and z metadatas.zEmbedding documents.)idr%metadata documents/z Uploaded z documents to GCS. zindexes//z.jsonz)Uploaded updated json with embeddings to .gs://)contents_delta_uriz%Updated index with new configuration.)listlen ValueErrorloggerdebugr%embed_documents enumeratezipstruuiduuid4append_upload_to_gcsjoinjsondumpstimer(r#update_embeddings)r)texts metadataskwargsr/jsonsidsidxr%textr8json_x result_strfilename_prefixfilenames r+ add_textszMatchingEngine.add_textsys U   SZ3y>%ALu:,k#i.)9F   +,^^33E: &/J0F&G 9 "C")TTZZ\"B JJrN!#)YYu=! 1 => $TZZ\N3%&a }E: J1 7##$AhZq 2 ZZ11!&t';';&s1Hc|jj|j}|j|}|j |y)zUploads data to gcs_location. Args: data: The data that will be stored. gcs_location: The location where the data will be stored. N)r& get_bucketr(blobupload_from_string)r)data gcs_locationbucketras r+rLzMatchingEngine._upload_to_gcss:++D,@,@A{{<( %r,ctjd|d|jj|}|j |||S)aReturn docs most similar to query and their cosine distance from the query. Args: query: String query look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: Optional. A list of Namespaces for filtering the matching results. For example: [Namespace("color", ["red"], []), Namespace("shape", [], ["squared"])] will match datapoints that satisfy "red color" but not include datapoints with "squared shape". Please refer to https://cloud.google.com/vertex-ai/docs/matching-engine/filtering#json for more detail. Returns: List[Tuple[Document, float]]: List of documents most similar to the query text and cosine distance in float for each. Lower score represents more similarity. zEmbedding query r=kfilter)rCrDr% embed_query&similarity_search_by_vector_with_score)r)queryrhriembedding_querys r+similarity_search_with_scorez+MatchingEngine.similarity_search_with_scoresM2  'wa01..44U;:: q;  r,c|xsg}t|jdrE|jjr/|jj|j |g||}n.|jj |j |g||}t jdt|dt|dk(rgSg}|dD]q}|jd|j}i}|j|j||j<t||} |j| |jfst jd|S) aReturn docs most similar to the embedding and their cosine distance. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: Optional. A list of Namespaces for filtering the matching results. For example: [Namespace("color", ["red"], []), Namespace("shape", [], ["squared"])] will match datapoints that satisfy "red color" but not include datapoints with "squared shape". Please refer to https://cloud.google.com/vertex-ai/docs/matching-engine/filtering#json for more detail. Returns: List[Tuple[Document, float]]: List of documents most similar to the query text and cosine distance in float for each. Lower score represents more similarity. _public_match_client)deployed_index_idqueries num_neighborsrizFound z matches.rr:) page_contentr9zDownloaded documents for query.)hasattrr$rpfind_neighbors _get_index_idmatchrCrDrA_download_from_gcsr8rr rKdistance) r)r%rhriresponsedocsresultrtr9documents r+rkz5MatchingEngine.similarity_search_by_vector_with_scoresN42 4=="8 9 MM . .}}33"&"4"4"6"  4H}}**"&"4"4"6"  +H  vc(m_I67 x=A I-/ qk 5F22Z {3KLLH##/17--.)!H KK6??3 4 5  67 r,c d|j|f||d|}|Dcgc]\}}| c}}Scc}}w)aReturn docs most similar to query. Args: query: The string that will be used to search for similar documents. k: The amount of neighbors that will be retrieved. filter: Optional. A list of Namespaces for filtering the matching results. For example: [Namespace("color", ["red"], []), Namespace("shape", [], ["squared"])] will match datapoints that satisfy "red color" but not include datapoints with "squared shape". Please refer to https://cloud.google.com/vertex-ai/docs/matching-engine/filtering#json for more detail. Returns: A list of k matching documents. rg)rn)r)rlrhrirTdocs_and_scoresdoc_s r+similarity_searchz MatchingEngine.similarity_search%sE.<$;;  v )/ #22Q222 ,c d|j|f||d|}|Dcgc]\}}| c}}Scc}}w)aReturn docs most similar to the embedding. Args: embedding: Embedding to look up documents similar to. k: The amount of neighbors that will be retrieved. filter: Optional. A list of Namespaces for filtering the matching results. For example: [Namespace("color", ["red"], []), Namespace("shape", [], ["squared"])] will match datapoints that satisfy "red color" but not include datapoints with "squared shape". Please refer to https://cloud.google.com/vertex-ai/docs/matching-engine/filtering#json for more detail. Returns: A list of k matching documents. rg)rk)r)r%rhrirTrrrs r+similarity_search_by_vectorz*MatchingEngine.similarity_search_by_vectorBsF.F$EE  6 -3 #22Q222rc|jjD]3}|j|jjk(s'|jcSt d|jjd|jj d)zGets the correct index id for the endpoint. Returns: The index id if found (which should be found) or throws ValueError otherwise. zNo index with id z deployed on endpoint r=)r$deployed_indexesr# resource_namer8rB display_name)r)r#s r+rwzMatchingEngine._get_index_id_sy]]33 E{{djj666xx  8 89:$}}))*! -  r,c|jj|j}|j|}|j S)zDownloads from GCS in text format. Args: gcs_location: The location where the file is located. Returns: The string contents of the file. )r&r`r(radownload_as_string)r)rdreras r+ryz!MatchingEngine._download_from_gcsps;++D,@,@A{{<(&&((r,c td)zUse from components instead.zThis method is not implemented. Instead, you should initialize the class with `MatchingEngine.from_components(...)` and then call `add_texts`)NotImplementedError)clsrRr%rSrTs r+ from_textszMatchingEngine.from_texts}s"   r,c &|j|}|j|} |j|||| } |j|||| } |j | |} |j |||| |d|| | |xs|j | | |d|S)a|Takes the object creation out of the constructor. Args: project_id: The GCP project id. region: The default location making the API calls. It must have the same location as the GCS bucket and must be regional. gcs_bucket_name: The location where the vectors will be stored in order for the index to be created. index_id: The id of the created index. endpoint_id: The id of the created endpoint. credentials_path: (Optional) The path of the Google credentials on the local file system. embedding: The :class:`Embeddings` that will be used for embedding the texts. kwargs: Additional keyword arguments to pass to MatchingEngine.__init__(). Returns: A configured MatchingEngine with the texts added to the index. )r"r#r$r%r&r'r()_validate_gcs_bucket_create_credentials_from_file_create_index_by_id_create_endpoint_by_id_get_gcs_client_init_aiplatform_get_default_embeddings) rr"regionr(index_id endpoint_idcredentials_pathr%rTr'r#r$r&s r+from_componentszMatchingEngine.from_componentss>22?C778HI ''*fkR--      ((jA  Z+N  !@3#>#>#@!#+    r,cN|jdd}d|vrtd||S)zValidates the gcs_bucket_name as a bucket name. Args: gcs_bucket_name: The received bucket uri. Returns: A valid gcs_bucket_name or throws ValueError if full path is provided. r>r<zFThe argument gcs_bucket_name should only be the bucket name. Received )replacerB)rr(s r+rz#MatchingEngine._validate_gcs_bucketsC*11'2> / !--<,=? r,cPddlm}d}||jj|}|S)aCreates credentials for GCP. Args: json_credentials_path: The path on the file system where the credentials are stored. Returns: An optional of Credentials or None, in which case the default will be used. rr2N)r5r3rfrom_service_account_file)rjson_credentials_pathr3r's r+rz,MatchingEngine._create_credentials_from_files3 2 ,)55OO%Kr,cjddlm}tjd|d|j ||||S)aJCreates a MatchingEngineIndex object by id. Args: index_id: The created index id. project_id: The project to retrieve index from. region: Location to retrieve index from. credentials: GCS credentials. Returns: A configured MatchingEngineIndex. rr1z'Creating matching engine index with id r=) index_nameprojectlocationr')r4r1rCrDr)rrr"rr'r1s r+rz"MatchingEngine._create_index_by_idsA , >xjJK--# .  r,cjddlm}tjd|d|j ||||S)a`Creates a MatchingEngineIndexEndpoint object by id. Args: endpoint_id: The created endpoint id. project_id: The project to retrieve index from. region: Location to retrieve index from. credentials: GCS credentials. Returns: A configured MatchingEngineIndexEndpoint. rrzCreating endpoint with id r=)index_endpoint_namerrr')r4r1rCrDr)rrr"rr'r1s r+rz%MatchingEngine._create_endpoint_by_id sA , 1+a@A55 +# 6  r,cLddlm}|j||tdS)z\Lazily creates a GCS client. Returns: A configured GCS client. rrzvertex-ai-matching-engine)module)r'r client_info)r4rClientr)rr'r"rs r+rzMatchingEngine._get_gcs_client#s) )w~~#'/JK  r,c xddlm}tjd|d|d|d|j ||||y) a`Configures the aiplatform library. Args: project_id: The GCP project id. region: The default location making the API calls. It must have the same location as the GCS bucket and must be regional. gcs_bucket_name: GCS staging location. credentials: The GCS Credentials object. rrz%Initializing AI Platform for project z on z and for r=)rrstaging_bucketr'N)r4r1rCrDinit)rr"rr(r'r1s r+rzMatchingEngine._init_aiplatform5sN$ , 3J,.26/*.6/6/#6/. 6/  6/ # 6/6/+6/'6/p +/77(7 7  7r &,0     *  &  D,0 GG G* G & GX,0 33 3* 3  3  3@,0 33 3* 3  3  3: " ) +/  #      (        +/*.4 #4 4 4  4  4  4 (4 (4 4  4 4 l$$1 0  (+ 58 GT   2  +. 8; JW $  2 ' 58   "     #     < ) )r,r)' __future__rrNloggingrPrItypingrrrrrr r langchain_core._api.deprecationr langchain_core.documentsr langchain_core.embeddingsr langchain_core.vectorstoresr&langchain_community.utilities.vertexairr4rgoogle.cloud.aiplatformrrFgoogle.cloud.aiplatform.matching_engine.matching_engine_index_endpointrgoogle.oauth2.service_accountrrr getLoggerrrCrrr,r+rs"  LLL6-03B$X:F   8 $  J |)[|)  |)r,