f3fiUdZddlmZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z mZmZddlmZddlmZddlmZdd lmZmZdd lmZe rdd lmZmZmZmZm Z m!Z!ejDd Z#edZ$d+dZ%da&de'd<d,dZ( d-dZ) d.dZ*d/dZ+d0dZ, d1dZ- d2dZ.GddeZ/ d3dZ0 d4dZ1GddZ2 d5d Z3Gd!d"e Z4d#ddd$dd%dd& d6d'Z5d7d(Z6 d5d)Z7d#ddd$dd%dd& d8d*Z8y)9z@Module contains logic for indexing documents into vector stores.) annotationsN)islice) TYPE_CHECKINGAnyLiteral TypedDictTypeVarcast) BaseLoader)Document)LangChainException) DocumentIndex RecordManager) VectorStore) AsyncIterable AsyncIteratorCallableIterableIteratorSequencei)intTctj|jddj}t t j t|S)z3Hashes a string and returns the corresponding UUID.utf-8Fusedforsecurity)hashlibsha1encode hexdigeststruuiduuid5NAMESPACE_UUID) input_string hash_values Y/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_core/indexing/api.py_hash_string_to_uuidr(,sBG$eik tzz.*5 66Fbool_WARNED_ABOUT_SHA1cNtstjdtddayy)z9Emit a one-time warning about SHA-1 collision weaknesses.aUUsing SHA-1 for document hashing. SHA-1 is *not* collision-resistant; a motivated attacker can construct distinct inputs that map to the same fingerprint. If this matters in your threat model, switch to a stronger algorithm such as 'blake2b', 'sha256', or 'sha512' by specifying `key_encoder` parameter in the `index` or `aindex` function. )category stacklevelTN)r+warningswarn UserWarningr)r'_warn_about_sha1r47s.   M ! " r)cl|dk(r tt||}tjt|S)zKHash *input_string* to a deterministic UUID using the configured algorithm.r)r4_calculate_hashr"r#r$)r% algorithmr&s r' _hash_stringr8Is/F y9J ::nj 11r)cJtj|d}t||S)zBHash a nested dictionary to a UUID using the configured algorithm.T sort_keysr7)jsondumpsr8)datar7serialized_datas r'_hash_nested_dictrASs!jj6O 9 ==r)c#\Kt|} tt||}|sy|wzUtility batching function.N)iterlistr)sizeiterableitchunks r'_batchrJ[s3 hB VB%&  s*,cKg}|23d{}t||kr|j|t||k\s7|g}?7:6|r|yywrC)lenappend)rFrGbatchelements r'_abatchrPes_E!g u:  LL ! u: KE    s*AAAA.AAA AcdSttrfdStrSddtd}t |)z$Get the source id from the document.cyNr3)_docs r'z)_get_source_id_assigner..ysr)c"|jSrS)metadata)doc source_id_keys r'rUz)_get_source_id_assigner..{s3<< 6r)zAsource_id_key should be either None, a string or a callable. Got z of type .) isinstancer!callabletype ValueError)rYmsgs` r'_get_source_id_assignerr`ts]  -%66  oYtM':&;1 > S/r)c#Kt}|D]:}|j|vs|jtd|j|<yw)z>Deduplicate a list of hashed documents while preserving order.r!N)setidaddr )hashed_documentsseen hashed_docs r'_deduplicate_in_orderrhsIUD& == $ HHT%/ 0  s A ,A ceZdZdZy)IndexingExceptionz(Raised when an indexing operation fails.N)__name__ __module__ __qualname____doc__r3r)r'rjrjs2r)rjc$|dk(rWtj|jddj}t t j t|S|dk(r2tj|jdjS|dk(r2tj|jdjS|dk(r2tj|jdjSd|}t|) z8Return a hexadecimal digest of *text* using *algorithm*.rrFrblake2bsha256sha512zUnsupported hashing algorithm: ) rrrr r!r"r#r$rprqrrr^)textr7digestr_s r'r6r6sFdkk'2EJTTV4::nf566It{{734>>@@H~~dkk'23==??H~~dkk'23==?? +I; 7C S/r)c~t|jxsi}t|r ||}nLt|j|} t j |d}t||}t||z|}t||j|jS#t$r}d|d}t||d}~wwxYw)a Calculate a hash of the document, and assign it to the uid. When using one of the predefined hashing algorithms, the hash is calculated by hashing the content and the metadata of the document. Args: document: Document to hash. key_encoder: Hashing algorithm to use for hashing the document. If not provided, a default encoder using SHA-1 will be used. SHA-1 is not collision-resistant, and a motivated attacker could craft two different texts that hash to the same cache key. New applications should use one of the alternative encoders or provide a custom and strong key encoder function to avoid this risk. When changing the key encoder, you must change the index as well to avoid duplicated documents in the cache. Raises: ValueError: If the metadata cannot be serialized using json. Returns: Document with a unique identifier based on the hash of the content and metadata. r<Tr:zFailed to hash metadata: z6. Please use a dict that can be serialized using json.N)rc page_contentrW) dictrWr\r6rvr=r> Exceptionr^r ) document key_encoderrWhash_ content_hashserialized_metaer_ metadata_hashs r'_get_document_with_hashrs> $H$5$5$;t3d|}|j5t3d|Dcgc]}|j6c}}g}g}g} t!}!t-||dD]e\}}"t3d|j6}#|"r&|r|!j1|#n| j9|#D|j9|#|j9|g| r!|j;| ||t%| z }|rzt| tr| j<|f||d| xsin't| tr| j>|fi| xsi|t%|t%|!z z }|t%|!z }|j;t3d|Dcgc]}|j6c}|||dk(sT|D]}|d} tA| t3d|}$|jC|$||x}%stE| |%|jG|%|t%|%z }|jC|$||x}%rB|dk(s|dk(rl|rjd}&|dk(r t'|}&|jC|&||x}%rBtE| |%|jG|%|t%|%z }|jC|&||x}%rB||||dS#t$rt|j}YwxYwcc}wcc}wcc}wcc}w) ajIndex data from the loader into the vector store. Indexing functionality uses a manager to keep track of which documents are in the vector store. This allows us to keep track of which documents were updated, and which documents were deleted, which documents should be skipped. For the time being, documents are indexed using their hashes, and users are not able to specify the uid of the document. !!! warning "Behavior changed in `langchain-core` 0.3.25" Added `scoped_full` cleanup mode. !!! warning * In full mode, the loader should be returning the entire dataset, and not just a subset of the dataset. Otherwise, the auto_cleanup will remove documents that it is not supposed to. * In incremental mode, if documents associated with a particular source id appear across different batches, the indexing API will do some redundant work. This will still result in the correct end state of the index, but will unfortunately not be 100% efficient. For example, if a given document is split into 15 chunks, and we index them using a batch size of 5, we'll have 3 batches all with the same source id. In general, to avoid doing too much redundant work select as big a batch size as possible. * The `scoped_full` mode is suitable if determining an appropriate batch size is challenging or if your data loader cannot return the entire dataset at once. This mode keeps track of source IDs in memory, which should be fine for most use cases. If your dataset is large (10M+ docs), you will likely need to parallelize the indexing process regardless. Args: docs_source: Data loader or iterable of documents to index. record_manager: Timestamped set to keep track of which documents were updated. vector_store: `VectorStore` or DocumentIndex to index the documents into. batch_size: Batch size to use when indexing. cleanup: How to handle clean up of documents. - incremental: Cleans up all documents that haven't been updated AND that are associated with source IDs that were seen during indexing. Clean up is done continuously during indexing helping to minimize the probability of users seeing duplicated content. - full: Delete all documents that have not been returned by the loader during this run of indexing. Clean up runs after all documents have been indexed. This means that users may see duplicated content during indexing. - scoped_full: Similar to Full, but only deletes all documents that haven't been updated AND that are associated with source IDs that were seen during indexing. - None: Do not delete any documents. source_id_key: Optional key that helps identify the original source of the document. cleanup_batch_size: Batch size to use when cleaning up documents. force_update: Force update documents even if they are present in the record manager. Useful if you are re-indexing with updated embeddings. key_encoder: Hashing algorithm to use for hashing the document content and metadata. Options include "blake2b", "sha256", and "sha512". !!! version-added "Added in `langchain-core` 0.3.66" key_encoder: Hashing algorithm to use for hashing the document. If not provided, a default encoder using SHA-1 will be used. SHA-1 is not collision-resistant, and a motivated attacker could craft two different texts that hash to the same cache key. New applications should use one of the alternative encoders or provide a custom and strong key encoder function to avoid this risk. When changing the key encoder, you must change the index as well to avoid duplicated documents in the cache. upsert_kwargs: Additional keyword arguments to pass to the add_documents method of the `VectorStore` or the upsert method of the DocumentIndex. For example, you can use this to specify a custom vector_field: upsert_kwargs={"vector_field": "embedding"} !!! version-added "Added in `langchain-core` 0.3.10" Returns: Indexing result which contains information about how many documents were added, updated, deleted, or skipped. Raises: ValueError: If cleanup mode is not one of 'incremental', 'full' or None ValueError: If cleanup mode is incremental and source_id_key is None. ValueError: If `VectorStore` does not have "delete" and "add_documents" required methods. ValueError: If source_id_key is not None, but is not a string or callable. TypeError: If `vectorstore` is not a `VectorStore` or a DocumentIndex. AssertionError: If `source_id` is None when cleanup mode is incremental. (should be unreachable code). r>Nfull incremental scoped_fullKcleanup should be one of 'incremental', 'full', 'scoped_full' or None. Got rZ>rrNJSource id key is required when cleanup mode is incremental or scoped_full.r add_documents Vectorstore  does not have required method z1Vectorstore has not implemented the delete methodrrrzFstrictlSource IDs are required when cleanup mode is incremental or scoped_full. Document that starts with content: r was not assigned as source id.r Sequence[str]r! time_at_leastrr group_idsrrAsource_id cannot be None at this point. Reached unreachable code.rbeforelimitrrrrr)$r4r^r[rhasattrr]rrrr lazy_loadrrDloadr`get_timerbrJrLrErhrziprvrdr existsrcrMupdaterupsertAssertionError list_keysr delete_keys)' docs_sourcerecord_managerrrrrYrrrzrr_ destinationmethodsmethod doc_iteratorsource_id_assignerindex_start_dtrrrrscoped_full_cleanup_source_ids doc_batchoriginal_batch_sizerX hashed_docsrg source_ids source_id exists_batchuids docs_to_indexuids_to_refresh seen_docs doc_exists hashed_id source_ids_uids_to_deletedelete_group_idss' r'indexr"s`fBB)1  o11}7L X oK+{+_- &F;/";-/NvhW!o%  &   # #{'9'9 9FCS/ ! K /  $%Q ( n+z* 4&002LK( 0?$,,.NIKKK/2u"J 5o3 !)n ! ),C[I   *S-=== >I, /9 z *, ,  4 4),ZU)S B% :$$%/$;$;DS$A#BC9:%S/)m+266yA Boz:J%,, ["Ac366"A B  !e &)+|E&R - "J UJMM2IMM),#**95 KK "   , -   ! !/ ! P 3/ /K +{3) ))!)%*  K7" ""!$* ]+c)n< )K  ["Ac366"A B (   m #( . $4)--  . ;K$2$<$<%nDV%=%. ^4**>:s>22 %3$<$<%nDV%=%.Qo3b&= %C15 m ##$BC  . 8 8&~EW!9!  n  K 0  & &~ 6 3~. .K!/ 8 8&~EW!9!  n """  _# 4 0 0 23L 4*, ,#B^#Bs*#RR1 R6 R;$S"R.-R.c$K|D]}| yw)z)Convert an iterable to an async iterator.Nr3)iteratoritems r'_to_async_iteratorrYs scJKt|tr.|j|d{}||dur d}t|yyt|tr4|j|d{}d|vr|ddkDr d}t|yydt |d}t |7w7;w)NFrrrrrrZ)r[radeleterjrr]rrs r'_adeleter_s, ,&..s33  Y%%7?C#C( (&8  L- 0 , 4 4S 99 ? *|/Lq/PAC#C( (0Q *  %&a ) n4 :s!%B#B=B#%B!&:B#!B#c & K|dk(r t|dvrd|d} t| |dvr| d} t| |} t| trdd g} | D]!} t | | rd | d | } t| t | j tj k(r^t | jtjk(r8d } t| t| trnd t | d} t| t|tr |j}nt |dr|}n t|}t|}|j!d{}d}d}d}d}t#}t%||23d{}t'|}t)t+|Dcgc]}t-||c}}||t'|z z }|Dcgc] }|| }}|dvrYt/||dD]<\}}|d|j0ddd} t| |dk(s,|j3|>t5d|}|j7t5d|Dcgc]}|j8c}d{}g}g}g} t#}!t/||dD]e\}}"t5d|j8}#|"r&|r|!j3|#n| j;|#D|j;|#|j;|g| r)|j=| |d{|t'| z }|rt| tr#| j>|f||d| xsid{n/t| tr| j@|fi| xsid{|t'|t'|!z z }|t'|!z }|j=t5d|Dcgc]}|j8c}||d{|dk(s|D]}|d} tC| t5d|}$|jE|$||d{x}%stG| |%d{|jI|%d{|t'|%z }|jE|$||d{x}%rZ#t$rt|j}YwxYw7k7Gcc}wcc}wcc}w7c777Wcc}w77777`6|dk(s|dk(r|rd}&|dk(r t)|}&|jE|&||d{7x}%r]tG| |%d{7|jI|%d{7|t'|%z }|jE|&||d{7x}%r]||||d Sw)!aAsync index data from the loader into the vector store. Indexing functionality uses a manager to keep track of which documents are in the vector store. This allows us to keep track of which documents were updated, and which documents were deleted, which documents should be skipped. For the time being, documents are indexed using their hashes, and users are not able to specify the uid of the document. !!! warning "Behavior changed in `langchain-core` 0.3.25" Added `scoped_full` cleanup mode. !!! warning * In full mode, the loader should be returning the entire dataset, and not just a subset of the dataset. Otherwise, the auto_cleanup will remove documents that it is not supposed to. * In incremental mode, if documents associated with a particular source id appear across different batches, the indexing API will do some redundant work. This will still result in the correct end state of the index, but will unfortunately not be 100% efficient. For example, if a given document is split into 15 chunks, and we index them using a batch size of 5, we'll have 3 batches all with the same source id. In general, to avoid doing too much redundant work select as big a batch size as possible. * The `scoped_full` mode is suitable if determining an appropriate batch size is challenging or if your data loader cannot return the entire dataset at once. This mode keeps track of source IDs in memory, which should be fine for most use cases. If your dataset is large (10M+ docs), you will likely need to parallelize the indexing process regardless. Args: docs_source: Data loader or iterable of documents to index. record_manager: Timestamped set to keep track of which documents were updated. vector_store: `VectorStore` or DocumentIndex to index the documents into. batch_size: Batch size to use when indexing. cleanup: How to handle clean up of documents. - incremental: Cleans up all documents that haven't been updated AND that are associated with source IDs that were seen during indexing. Clean up is done continuously during indexing helping to minimize the probability of users seeing duplicated content. - full: Delete all documents that have not been returned by the loader during this run of indexing. Clean up runs after all documents have been indexed. This means that users may see duplicated content during indexing. - scoped_full: Similar to Full, but only deletes all documents that haven't been updated AND that are associated with source IDs that were seen during indexing. - None: Do not delete any documents. source_id_key: Optional key that helps identify the original source of the document. cleanup_batch_size: Batch size to use when cleaning up documents. force_update: Force update documents even if they are present in the record manager. Useful if you are re-indexing with updated embeddings. key_encoder: Hashing algorithm to use for hashing the document content and metadata. Options include "blake2b", "sha256", and "sha512". !!! version-added "Added in `langchain-core` 0.3.66" key_encoder: Hashing algorithm to use for hashing the document. If not provided, a default encoder using SHA-1 will be used. SHA-1 is not collision-resistant, and a motivated attacker could craft two different texts that hash to the same cache key. New applications should use one of the alternative encoders or provide a custom and strong key encoder function to avoid this risk. When changing the key encoder, you must change the index as well to avoid duplicated documents in the cache. upsert_kwargs: Additional keyword arguments to pass to the add_documents method of the `VectorStore` or the upsert method of the DocumentIndex. For example, you can use this to specify a custom vector_field: upsert_kwargs={"vector_field": "embedding"} !!! version-added "Added in `langchain-core` 0.3.10" Returns: Indexing result which contains information about how many documents were added, updated, deleted, or skipped. Raises: ValueError: If cleanup mode is not one of 'incremental', 'full' or None ValueError: If cleanup mode is incremental and source_id_key is None. ValueError: If `VectorStore` does not have "adelete" and "aadd_documents" required methods. ValueError: If source_id_key is not None, but is not a string or callable. TypeError: If `vector_store` is not a `VectorStore` or DocumentIndex. AssertionError: If `source_id_key` is None when cleanup mode is incremental or `scoped_full` (should be unreachable). r>NrrrrrZ>rrNrraadd_documentsrrz,>>QCS/ ! K /  $%Q ( n+z* H!,!7!7!9  k *(/ <0?*3355NIKKK/2u"":/ABp3p3i!)n ! ),C[I   *S-=== 0;, (+ s #, ,  4 4),ZU)S B% :$$%/$;$;DS$A#BC9:%S/)m+266yA Boz:J+33 ["Ac366"A B  (* !e &)+|E&R - "J UJMM2IMM),#**95 KK "   , -  (((W W W 3/ /K +{30k00!)%*  K7)k))!$* ]+c)n< )K $$ ["Ac366"A B (%    m # ( . $4)--  . ;K*8*C*C%nDV+D+%.{N;;;$11.AAAs>22 +9*C*C%nDV+D+%.# H "4K4D4D4F!G   H6p3 , ,#B , X#B .%<A %SCd&= %C15 m ##$BC &4&?&?&~EW'@' ! !  n ;7 7 7 --n= = = 3~. .K'5&?&?&~EW'@' ! !  n """  sAV"B(V R9VR?"V7S/;S<S/?VS- V S ?V4VS#V1S2B"VSAVS/VSAVS V,S%- V8V1V1S'2V:V S) V"S+#)V S- VV"R<8V;R<<VS/VVVV V'V)V+V-V/6V%T(&V<T?=VU*VVV V)r%r!rr!)rr)r%r!r7.Literal['sha1', 'sha256', 'sha512', 'blake2b']r uuid.UUID)r?zdict[Any, Any]r7rrr)rFrrG Iterable[T]rzIterator[list[T]])rFrrGzAsyncIterable[T]rzAsyncIterator[list[T]])rY&str | Callable[[Document], str] | Nonerz Callable[[Document], str | None])rezIterable[Document]rzIterator[Document])rsr!r7rrr!)ryr rzzJCallable[[Document], str] | Literal['sha1', 'sha256', 'sha512', 'blake2b']rr )rVectorStore | DocumentIndexrz list[str]rr)rzBaseLoader | Iterable[Document]rrrrrrr4Literal['incremental', 'full', 'scoped_full'] | NonerYrrrrr*rzJLiteral['sha1', 'sha256', 'sha512', 'blake2b'] | Callable[[Document], str]rdict[str, Any] | Nonerr)rrrzAsyncIterator[T])rz9BaseLoader | Iterable[Document] | AsyncIterator[Document]rrrrrrrrrYrrrrr*rzrrrrr)9rn __future__rrr=r"r0 itertoolsrtypingrrrrr r $langchain_core.document_loaders.baser langchain_core.documentsr langchain_core.exceptionsr langchain_core.indexing.baserrlangchain_core.vectorstoresrcollections.abcrrrrrrUUIDr$rr(r+rr4r8rArJrPr`rhrjr6rrrrrrrrr3r)r'rsF" <-8E3t$ CL7!D "$22%S22> >(V>> 9%" (  3*3 H$7757  7x ' '-  H LY L$DH<@#"(+/s0s!s.s  s B s:sss s)ssn  -  6DH<@#"(+/J!.   B : )r)