3fizdZddlmZddlmZmZddlmZmZm Z  ddl m Z dZ Gdd eZy #e $rdZ YwxYw) z$Sentence transformers text splitter.) annotations)Anycast) TextSplitter Tokenizersplit_text_on_tokens)SentenceTransformerTFcneZdZUdZ d d fd Zd dZd dZddZdZde d<dd Z xZ S)%SentenceTransformersTokenTextSplitterz8Splitting text to tokens using sentence model tokenizer.intc t|di|d|its d}t|||_t |j|_|j j|_|j|y)zCreate a new TextSplitter. chunk_overlapzCould not import sentence_transformers python package. This is needed in order to use SentenceTransformersTokenTextSplitter. Please install it with `pip install sentence-transformers`.)tokens_per_chunkN) super__init___HAS_SENTENCE_TRANSFORMERS ImportError model_namer _model tokenizer_initialize_chunk_configuration)selfrrrkwargsmsg __class__s l/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_text_splitters/sentence_transformers.pyrz.SentenceTransformersTokenTextSplitter.__init__so ?6??)N  c" "$)$//: .. ,,>N,Oc |jj|_||j|_n||_|j|jkDr5d|jd|jd|jd}t |y)NzThe token limit of the models 'z' is: z. Argument tokens_per_chunk=z > maximum token limit.)rmax_seq_lengthmaximum_tokens_per_chunkrr ValueError)rrrs rrzESentenceTransformersTokenTextSplitter._initialize_chunk_configuration.s(, (B(B%  #$($A$AD !$4D !  4#@#@ @1$//1BC5567..2.C.C-D)+  S/ ! Arcdfd }tjjjj|}t ||S)aSplits the input text into smaller components by splitting text on tokens. This method encodes the input text using a private `_encode` method, then strips the start and stop token IDs from the encoded result. It returns the processed segments as a list of strings. Args: text: The input text to be split. Returns: A list of string components derived from the input text after encoding and processing. c,j|ddS)N)_encode)textrs r%encode_strip_start_and_stop_token_idsz_SentenceTransformersTokenTextSplitter.split_text..encode_strip_start_and_stop_token_idsNs<<%a+ +r)rrdecodeencode)r(rr(strreturn list[int])r_chunk_overlaprrr*r)rr(r)rs` r split_textz0SentenceTransformersTokenTextSplitter.split_text?sE ,--!22>>((8  $CCrc6t|j|S)asCounts the number of tokens in the given text. This method encodes the input text using a private `_encode` method and calculates the total number of tokens in the encoded result. Args: text: The input text for which the token count is calculated. Returns: int: The number of tokens in the encoded text. )lenr')rr(s r count_tokensz2SentenceTransformersTokenTextSplitter.count_tokensZs4<<%&&rl _max_length_equal_32_bit_integercj|jj||jd}td|S)Ndo_not_truncate) max_length truncationr/)rr+r5r)rr(&token_ids_with_start_and_end_token_idss rr'z-SentenceTransformersTokenTextSplitter._encodejs<151F1F <<(2G2 . K!GHHr)2z'sentence-transformers/all-mpnet-base-v2N) rr rr-r int | Nonerrr.None)rr<r.r=)r(r-r.z list[str])r(r-r.r r,) __name__ __module__ __qualname____doc__rrr1r4r5__annotations__r' __classcell__)rs@rr r soB C'+ PPP% P  P  P.""D6 '-2$c1Irr N)rA __future__rtypingrrlangchain_text_splitters.baserrrsentence_transformersr rrr rrrrHsM*"WW'"& \IL\I '!&'s 0::