3fihv&dZddlmZddlZddlZddlmZmZmZm Z m Z ddl m Z m Z mZddlZddlZddlmZddlmZddlmZmZmZdd lmZmZmZmZmZdd lm Z dd l!m"Z"ejFe$Z%d Z& dd Z'GddeeZ(y)z#Base classes for OpenAI embeddings.) annotationsN) AwaitableCallableIterableMappingSequence)AnyLiteralcast) Embeddings)run_in_executor)from_envget_pydantic_field_namessecret_from_env) BaseModel ConfigDictField SecretStrmodel_validator)Self) _resolve_sync_and_async_api_keysic t|Dcgc]}g}}t|Dcgc]}g}}tt|D]S}|rt||dk(r|||j|||||jt||Ug} t|D]}||} t| dk(r| jd(t| dk(r| j| dKt||} t | ddiD cgc]&} tdt | ||dD| z (} } td| Ddz}| j| Dcgc]}||z  c}| Scc}wcc}wcc} wcc}w) NrstrictFc3,K|] \}}||zywN).0valweights ^/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_openai/embeddings/base.py z6_process_batched_chunked_embeddings..Fs Cf s)rc3&K|] }|dz yw)Nr)rrs r!r"z6_process_batched_chunked_embeddings..Ps23Q2g?)rangelenappendsumzip) num_textstokensbatched_embeddingsindices skip_empty_resultsnum_tokens_in_batchi embeddings_result total_weight embeddingaverage magnituders r!#_process_batched_chunked_embeddingsr:s5:)4D'Eq'EG'E 9>i8H+I1B+I+I 3w< ? #0349  ""#5a#89GAJ'..s6!9~> ?,.J 9 @%,QZ w<1    d #  w<1    gaj )  .q12 !'8%8   #&y2Ea2HQV#W     2'22c9 g>s3?>??@B a(F ,J: ?s E6 E;+F F ceZdZUdZeddZded<eddZded<dZd ed <dZ d ed < eZ d ed<ee dddZ d ed< ede ddZ d ed< ee ddZd ed<ee ddZd ed<dZded< ed ed!dZd"ed#< ed$e d%d&gdZd ed'< dZd(ed)<dZd*ed+<d,Zded-< d.Zded/< edd01Zd2ed3< dZded4<dZd5ed6< dZd ed7< d8Zd5ed9< eeZd:ed;< d8Zd5ed<< dZ d=ed><dZ!d?ed@<dAZ"dedB< dCZ#dedD< dZ$dEedF< dZ%dEedG< dZ&d5edH< e'dIddJKZ(e)dLMe*d[dNZ+e)dOMd\dPZ,e-d]dQZ.d^dRZ/ d_dSZ0ddT d`dUZ1ddT d`dVZ2 da dbdWZ3 da dbdXZ4dcdYZ5dcdZZ6y)dOpenAIEmbeddingsuOpenAI embedding model integration. Setup: Install `langchain_openai` and set environment variable `OPENAI_API_KEY`. ```bash pip install -U langchain_openai export OPENAI_API_KEY="your-api-key" ``` Key init args — embedding params: model: Name of OpenAI model to use. dimensions: The number of dimensions the resulting output embeddings should have. Only supported in `'text-embedding-3'` and later models. Key init args — client params: api_key: OpenAI API key. organization: OpenAI organization ID. If not passed in will be read from env var `OPENAI_ORG_ID`. max_retries: Maximum number of retries to make when generating. request_timeout: Timeout for requests to OpenAI completion API See full list of supported init args and their descriptions in the params section. Instantiate: ```python from langchain_openai import OpenAIEmbeddings embed = OpenAIEmbeddings( model="text-embedding-3-large" # With the `text-embedding-3` class # of models, you can specify the size # of the embeddings you want returned. # dimensions=1024 ) ``` Embed single text: ```python input_text = "The meaning of life is 42" vector = embeddings.embed_query("hello") print(vector[:3]) ``` ```python [-0.024603435769677162, -0.007543657906353474, 0.0039630369283258915] ``` Embed multiple texts: ```python vectors = embeddings.embed_documents(["hello", "goodbye"]) # Showing only the first 3 coordinates print(len(vectors)) print(vectors[0][:3]) ``` ```python 2 [-0.024603435769677162, -0.007543657906353474, 0.0039630369283258915] ``` Async: ```python await embed.aembed_query(input_text) print(vector[:3]) # multiple: # await embed.aembed_documents(input_texts) ``` ```python [-0.009100092574954033, 0.005071679595857859, -0.0029193938244134188] ``` NT)defaultexcluder client async_clientztext-embedding-ada-002strmodel int | None dimensionsz str | None deploymentOPENAI_API_VERSION)r= api_version)default_factoryaliasopenai_api_versionbase_urlOPENAI_API_BASE)rIrHopenai_api_baseOPENAI_API_TYPE)rHopenai_api_type OPENAI_PROXY openai_proxyiintembedding_ctx_lengthapi_keyOPENAI_API_KEYzCSecretStr | None | Callable[[], str] | Callable[[], Awaitable[str]]openai_api_key organization OPENAI_ORG_IDOPENAI_ORGANIZATIONopenai_organizationz Literal['all'] | set[str] | Noneallowed_specialz0Literal['all'] | set[str] | Sequence[str] | Nonedisallowed_speciali chunk_sizer$ max_retriestimeout)r=rIz(float | tuple[float, float] | Any | Nonerequest_timeoutheadersbooltiktoken_enabledtiktoken_model_nameFshow_progress_bardict[str, Any] model_kwargsr/zMapping[str, str] | Nonedefault_headerszMapping[str, object] | None default_queryretry_min_secondsretry_max_secondsz Any | None http_clienthttp_async_clientcheck_embedding_ctx_lengthforbidr)extrapopulate_by_nameprotected_namespacesbefore)modec ht|}|jdi}t|D]O}||vrd|d}t|||vst j d|d|d|d|j |||<Q|j|j}|rd|d }t|||d<|S) z>Build extra kwargs from additional params that were passed in.rgzFound z supplied twice.z WARNING! z/ is not default parameter. zJ was transferred to model_kwargs. Please confirm that z is what you intended.z Parameters za should be specified explicitly. Instead they were passed in as part of `model_kwargs` parameter.) rgetlist ValueErrorwarningswarnpop intersectionkeys)clsvaluesall_required_field_namesrr field_namemsginvalid_model_kwargss r! build_extrazOpenAIEmbeddings.build_extra.s$2.v, ;JU"zl*:; o%!99 !*.L!))3 4JN %+JJz$:j! ; 8DDUZZ\R 234ST S/ !!&~ afterc^|jdvr d}t|d}d}|jt|j\}}|j|j |j |j|j|jd}|jrR|js |jr:|j}|j}|j}d|d|d|}t||js}|d|_ ns|jr2|js& ddl}|j!|j |_ |j|d } t#j$d i|| j&|_ |j(ss|jr2|js& ddl}|j+|j |_ |j|d } t#j,d i|| j&|_|S#t$r} d }t|| d} ~ wwxYw#t$r} d }t|| d} ~ wwxYw) z?Validate that api key and python package exists in environment.)azureazure_adazureadzEIf you are using Azure, please use the `AzureOpenAIEmbeddings` class.N)rWrKr_r^rhrizwCannot specify 'openai_proxy' if one of 'http_client'/'http_async_client' is already specified. Received: openai_proxy=z http_client=z http_async_client=rzRCould not import httpx python package. Please install it with `pip install httpx`.)proxy)rnrTr)rOrzrVrrZrMr`r^rhrirQrnror?httpx ImportErrorClientopenaiOpenAIr4r@ AsyncClient AsyncOpenAI) selfrsync_api_key_valueasync_api_key_value client_paramsrQrnrore sync_specificasync_specifics r!validate_environmentz%OpenAIEmbeddings.validate_environmentKsh   #C CW S/ !>BIM    *7W##7 3  3 !44,,++++#33!//     $"2"2d6L6L,,L**K $ 6 6 !/K>1F4E3GI  S/ !{{!)# $$T-=-=6$(-||$:K:K|'LD$#'#3#31! %mmMmM}MXX     )?)?2 */):):ARAR):)S&#55.N!' 2 2!! !j   ?'6J*#.A5 6 #2F&c*1 2s0G1H1 H :HH  H,H''H,cpd|ji|j}|j|j|d<|S)NrBrD)rBrgrD)rparamss r!_invocation_paramsz#OpenAIEmbeddings._invocation_paramss8At/@/@A ?? &#'??F<  rc6|j d}t|y)z8Check that sync client is available, raise error if not.NzSync client is not available. This happens when an async callable was provided for the API key. Use async methods (ainvoke, astream) instead, or provide a string or sync callable for the API key.)r?rz)rrs r!_ensure_sync_client_availablez.OpenAIEmbeddings._ensure_sync_client_availables( ;; Q  S/ ! rc g}g}g}|jxs |j}|js ddlm}|j|} t|D]\} } | j| d} tdt| |jD]a} | | | |jz}| j|}|j||j| |jt|cn: tj |}|j&|j(dj+Dcic] \}}||| }}}t|D]\} } |jj-d r| j/d d } |r|j| fi|}n|j1| }tdt||jD]^} |j|| | |jz|j| |jt|| | |jz`|j2r$ dd lm}|tdt||}ntdt||}||||fS#t $rd}t |wxYw#t"$rtj$d}YwxYwcc}}w#t $rtdt||}YlwxYw) aTokenize and batch input texts. Splits texts based on `embedding_ctx_length` and groups them into batches of size `chunk_size`. Args: texts: The list of texts to tokenize. chunk_size: The maximum number of texts to include in a single batch. Returns: A tuple containing: 1. An iterable of starting indices in the token list for each batch. 2. A list of tokenized texts (token arrays for tiktoken, strings for HuggingFace). 3. An iterable mapping each token array to the index of the original text. Same length as the token list. 4. A list of token counts for each tokenized text. r) AutoTokenizerzCould not import transformers python package. This is needed for OpenAIEmbeddings to work without `tiktoken`. Please install it with `pip install transformers`. )pretrained_model_name_or_pathF)add_special_tokens cl100k_base)r[r\001  )tqdm)rdrBrc transformersrrrzfrom_pretrained enumerateencoder&r'rSdecoder(tiktokenencoding_for_modelKeyError get_encodingr[r\itemsendswithreplaceencode_ordinaryre tqdm.autor)rtextsr]r,r. token_counts model_namerr tokenizerr3text tokenizedj token_chunk chunk_textencodingkvencoder_kwargstokenr_iters r! _tokenizezOpenAIEmbeddings._tokenizes*)+"$ --; $$ &6&55.86I%U+ :4'0'7'7QV'7'W q#i.$2K2KL :A-6A 9 99.K '0&6&6{&CJMM*-NN1% ''K(89 :  :  @#66zB (,';';*.*A*A%' .Aq= 1.N.%U+ W4::&&u- <<c2D!+HOODCNCE$44T:Eq#e*d.G.GHWAMM%A0I0I,I"JKNN1% ''E!a$:S:S6S,T(UVW W$  ! ! :*"&uQF Z'H"I!S[*5Efg|33E &V !o%  &: @#00? @.> :aVj9 :s5I83J2J79"J=8JJ43J4=KK)r]c |xs j}ij|j||\}}}} g} d} | t|krd} | } t | t | |zt|D])}| |}| |zt kDr | | k(r|dz} n | |z } |dz} +|| | }jjdd|i}t|ts|j}| jd|dD| } | t|krtt||| |j}dd fd }|Dcgc] }||n|c}Scc}w) a2Generate length-safe embeddings for a list of texts. This method handles tokenization and embedding generation, respecting the `embedding_ctx_length` and `chunk_size`. Supports both `tiktoken` and HuggingFace `transformers` based on the `tiktoken_enabled` flag. Args: texts: The list of texts to embed. engine: The engine or model to use for embeddings. chunk_size: The size of chunks for processing embeddings. Returns: A list of embeddings for each input text. rrinputc3&K|] }|d ywr7Nrrrs r!r"z.C%Oa n%Or%dataNcIjjdddi}t|ts|j }|dddSNrrrr7r)r?create isinstancedict model_dumpaverage_embedded_cached_empty_embedding client_kwargsrs r!empty_embeddingzBOpenAIEmbeddings._get_len_safe_embeddings..empty_embeddingLs]&.#54;;#5#5#PB#P-#P !"2D9'7'B'B'D$*:6*B1*Ek*R'* *rrreturn list[float])r]rrr'r&minMAX_TOKENS_PER_REQUESTr?rrrrextendr:r/rrenginer]kwargs _chunk_sizerr,r.rr-r3batch_token_count batch_endr chunk_tokens batch_tokensresponser4rrrrs` @@r!_get_len_safe_embeddingsz)OpenAIEmbeddings._get_len_safe_embeddingss,!3DOO =422=f= /3~~e[/Q,vw 02 #f+o ! I1c!k/3v;?@ "+A $|36LL A~$%E !\1!E  ""!I.L)t{{))N N NHh-#..0  % %%Ohv>N%O OA/#f+o29 J 2GT__ 7; +DNNaQ](99NNNs>Ec K|xs j}ij|tdj||d{\}}}} g} d} | t |krd} | } t | t | |zt |D])}| |}| |ztkDr | | k(r|dz} n | |z } |dz} +|| | }jjdd|id{}t|ts|j}| jd|dD| } | t |krtt ||| |j}dd fd }|Dcgc]}||n|d{c}S7277cc}ww) aAAsynchronously generate length-safe embeddings for a list of texts. This method handles tokenization and embedding generation, respecting the `embedding_ctx_length` and `chunk_size`. Supports both `tiktoken` and HuggingFace `transformers` based on the `tiktoken_enabled` flag. Args: texts: The list of texts to embed. engine: The engine or model to use for embeddings. chunk_size: The size of chunks for processing embeddings. Returns: A list of embeddings for each input text. Nrrrc3&K|] }|d ywrrrs r!r"z=OpenAIEmbeddings._aget_len_safe_embeddings..rr%rcKQjjdddid{}t|ts|j }|dddS71wr)r@rrrrrs r!rzCOpenAIEmbeddings._aget_len_safe_embeddings..empty_embeddingsw&.)A):):)A)A** -*$ ""2D9'7'B'B'D$*:6*B1*Ek*R'* * $s$AA2Arr)r]rr rr'r&rrr@rrrrrr:r/rs` @@r!_aget_len_safe_embeddingsz*OpenAIEmbeddings._aget_len_safe_embeddingsYs,!3DOO =422=f= 5D $..%6 0 ,vw 13 #f+o ! I1c!k/3v;?@ "+A $|36LL A~$%E !\1!E  ""!I.L5T..55"&3Hh-#..0  % %%Ohv>N%O OA3#f+o69 J 2GT__ 7; +JTTAQ]o.?(??TTe0 22)@TsO.!KQ!K.!Kr%rrr]r)rr]rrpr&r'r?rrrrrr rArEr rrr]r chunk_size_rr4r3rrs r!embed_documentsz OpenAIEmbeddings.embed_documentss **, 3DOO =422=f= ..,.J1c%j+6 L-4;;--A O48E"(D1'224H!!!K(6:J!KK  L c4??+,t,,   Z ;A  rc K|xs |j}i|j|}|jsg}tdt ||D]i}|j j dd||||zi|d{}t|ts|j}|jd|dDk|Stt|j} |j|f| |d|d{S7{7w)aAsynchronously call OpenAI's embedding endpoint to embed search docs. Args: texts: The list of texts to embed. chunk_size: The chunk size of embeddings. If `None`, will use the chunk size specified by the class. kwargs: Additional keyword arguments to pass to the embedding API. Returns: List of embeddings, one for each text. rrNc3&K|] }|d ywrrrs r!r"z4OpenAIEmbeddings.aembed_documents..rr%rrr)r]rrpr&r'r@rrrrrr rArErrs r!aembed_documentsz!OpenAIEmbeddings.aembed_documentss!3DOO =422=f= ..,.J1c%j+6 L!9!2!2!9!9"A O4"8E""(D1'224H!!!K(6:J!KK  L c4??+3T33   Z ;A    s%A/C11C-2A6C1(C/)C1/C1c P|j|j|gfi|dS)aCall out to OpenAI's embedding endpoint for embedding query text. Args: text: The text to embed. kwargs: Additional keyword arguments to pass to the embedding API. Returns: Embedding for the text. r)rr)rrrs r! embed_queryzOpenAIEmbeddings.embed_querys/ **,#t##TF5f5a88rcPK|j|gfi|d{}|dS7 w)a Call out to OpenAI's embedding endpoint async for embedding query text. Args: text: The text to embed. kwargs: Additional keyword arguments to pass to the embedding API. Returns: Embedding for the text. Nr)r)rrrr4s r! aembed_queryzOpenAIEmbeddings.aembed_querys41400$B6BB !}Cs &$ &)rrfrr )rr)rrf)rNone)r list[str]r]rRrzAtuple[Iterable[int], list[list[int] | str], list[int], list[int]]) rrrrAr]rCrr rlist[list[float]]r)rrr]rCrr rr)rrArr rr)7__name__ __module__ __qualname____doc__rr?__annotations__r@rBrDrErrJrMrOrQrSrrVrZr[r\r]r^r`rarcrdrerrgr/rhrirkrmrnrorpr model_configr classmethodrrpropertyrrrrrrrrrrrr!r<r<VsL\d3FC3dD9L#9)E3)!J ! #J "&+ !5tD& T#((3Dd*S#OZ#( !2DA#OZ % > L*!%#$8 9ISW)X K P&+ 3 4d ' O8L.>VJI04O-415M.5 s7s7"K" %)z( (,+-BL(#$6'"N#N` "`4`4,/`4 J`4R"& DODO DO  DO  DO DOZ"& JUJU JU  JU  JU JUZ:>" " ,6" IL" " J:>! ! ,6! IL! ! F 9 rr<) r+rRr,zlist[list[int] | str]r-rr.z list[int]r/rbrzlist[list[float] | None]))r __future__rloggingr{collections.abcrrrrrtypingr r r rrlangchain_core.embeddingsr langchain_core.runnables.configr langchain_core.utilsrrrpydanticrrrrrtyping_extensionsr*langchain_openai.chat_models._client_utilsr getLoggerrloggerrr:r<rrr!rs)"LL%% 0;TTMM"W   8 $199 !9*9 9  9  9xn y*n r