3fi'~dZddlmZmZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZmZdd lmZmZdd lmZmZmZdd lmZdd lmZdd lmZddlm Z m!Z!ddl"m#Z$eeedzgeeze%zfZ&eeeeedzgeeze%zfZ'GddeZ(Gdde(Z)e&e ze!zZ*e e+ze(zZ,GddeZ-y)z!Configuration for run evaluators.)CallableSequence)Any) Embeddings)BaseLanguageModel)BasePromptTemplate) RunEvaluator)EvaluationResultEvaluationResults)ExampleRun) BaseModel ConfigDictField)override) CRITERIA_TYPE)EmbeddingDistance) EvaluatorTypeStringEvaluator)StringDistanceNc2eZdZUdZeed<deeeffdZ y) EvalConfigztConfiguration for a given run evaluator. Attributes: evaluator_type: The type of evaluator to use. evaluator_typereturnc8i}|D]\}}|dk(s||||<|S)zGet the keyword arguments for the `load_evaluator` call. Returns: The keyword arguments for the `load_evaluator` call. r)selfkwargsfieldvals g/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_classic/smith/evaluation/config.py get_kwargszEvalConfig.get_kwargs+s<  JE3((CKF5M  N) __name__ __module__ __qualname____doc__r__annotations__dictstrrr"rr#r!rr"s# "! DcN r#rceZdZUdZdZedzed< dZedzed< dZedzed< e de ee fffd Z xZ S)SingleKeyEvalConfigzBConfiguration for a run evaluator that only requires a single key.N reference_keyprediction_key input_keyrcVt|}dD]}|j|d|S)N)r-r.r/)superr"pop)rrkey __class__s r!r"zSingleKeyEvalConfig.get_kwargsGs2#%C "C JJsD ! " r#)r$r%r&r'r-r*r(r.r/rr)rr" __classcell__)r4s@r!r,r,9sfL $M3:$@!%NC$J%!IsTz BDcNr#r,ceZdZUdZeeZeeeze d< dZ eedze d< dZ ee dze d< dZ edze d< dZedze d< dZedze d < dZedze d < ed ZGd deZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGdd eZ Gd!d"eZ!Gd#d$eZ"Gd%d&e"Z#y)' RunEvalConfigz#Configuration for a run evaluation.)default_factory evaluatorsNcustom_evaluatorsbatch_evaluatorsr-r.r/eval_llmTarbitrary_types_allowedc\eZdZUdZdZedzed<dZedzed<e jZ e ed<y)RunEvalConfig.CriteriazConfiguration for a reference-free criteria evaluator. Attributes: criteria: The criteria to evaluate. llm: The language model to use for the evaluation chain. Ncriteriallmr) r$r%r&r'rArr(rBrrCRITERIArrr#r!Criteriar@{s8 *.-$&-(,  %,(5(>(> >r#rDc\eZdZUdZdZedzed<dZedzed<e jZ e ed<y)RunEvalConfig.LabeledCriteriazConfiguration for a labeled (with references) criteria evaluator. Attributes: criteria: The criteria to evaluate. llm: The language model to use for the evaluation chain. NrArBr) r$r%r&r'rArr(rBrrLABELED_CRITERIArrr#r!LabeledCriteriarFs8 *.-$&-(,  %,(5(F(F Fr#rHcneZdZUdZej Zeed<dZe dzed<dZ e dzed<e dZ y)RunEvalConfig.EmbeddingDistancezConfiguration for an embedding distance evaluator. Attributes: embeddings: The embeddings to use for computing the distance. distance_metric: The distance metric to use for computing the distance. rN embeddingsdistance_metricTr=)r$r%r&r'rEMBEDDING_DISTANCErr(rKrrLEmbeddingDistanceEnumr model_configrr#r!rrJsE )6(H(H H(, J%,8<.5<!$( r#rcVeZdZUdZej Zeed<dZe dzed<dZ e ed<y)RunEvalConfig.StringDistancea}Configuration for a string distance evaluator. Attributes: distance: The string distance metric to use (`damerau_levenshtein`, `levenshtein`, `jaro`, or `jaro_winkler`). normalize_score: Whether to normalize the distance to between 0 and 1. Applies only to the Levenshtein and Damerau-Levenshtein distances. rNdistanceTnormalize_score) r$r%r&r'rSTRING_DISTANCErr(rRStringDistanceEnumrSboolrr#r!rrQs3 )6(E(E E.2$t+2 $$r#rc\eZdZUdZej Zeed<dZe dzed<dZ e dzed<y)RunEvalConfig.QAzConfiguration for a QA evaluator. Attributes: prompt: The prompt template to use for generating the question. llm: The language model to use for the evaluation chain. rNrBprompt) r$r%r&r'rQArr(rBrrYrrr#r!rZrXs9 )6(8(8 8(,  %,,0"T)0r#rZc\eZdZUdZej Zeed<dZe dzed<dZ e dzed<y)RunEvalConfig.ContextQAConfiguration for a context-based QA evaluator. Attributes: prompt: The prompt template to use for generating the question. llm: The language model to use for the evaluation chain. rNrBrY r$r%r&r'r CONTEXT_QArr(rBrrYrrr#r! ContextQAr\9 )6(@(@ @(,  %,,0"T)0r#r`c\eZdZUdZej Zeed<dZe dzed<dZ e dzed<y)RunEvalConfig.CoTQAr]rNrBrYr^rr#r!CoTQArcrar#rdc4eZdZUdZej Zeed<y)RunEvalConfig.JsonValidityz,Configuration for a json validity evaluator.rN)r$r%r&r'r JSON_VALIDITYrr(rr#r! JsonValidityrf:(5(C(C Cr#rhc4eZdZUdZej Zeed<y)#RunEvalConfig.JsonEqualityEvaluatorz,Configuration for a json equality evaluator.rN)r$r%r&r'r JSON_EQUALITYrr(rr#r!JsonEqualityEvaluatorrkrir#rmc^eZdZUdZej Zeed<dZe ed<dZ e ed<dZ e ed<y)RunEvalConfig.ExactMatcha<Configuration for an exact match string evaluator. Attributes: ignore_case: Whether to ignore case when comparing strings. ignore_punctuation: Whether to ignore punctuation when comparing strings. ignore_numbers: Whether to ignore numbers when comparing strings. rF ignore_caseignore_punctuationignore_numbersN) r$r%r&r'r EXACT_MATCHrr(rprVrqrrrr#r! ExactMatchros8 )6(A(A A! T!#(D($$r#rtcBeZdZUdZej Zeed<dZe ed<y)RunEvalConfig.RegexMatchzConfiguration for a regex match string evaluator. Attributes: flags: The flags to pass to the regex. Example: `re.IGNORECASE`. rrflagsN) r$r%r&r'r REGEX_MATCHrr(rwintrr#r! RegexMatchrvs" )6(A(A Asr#rzceZdZUdZej Zeed<dZe dzed<dZ e dzed<dZ e dzed<dZedzed<y)RunEvalConfig.ScoreStringauConfiguration for a score string evaluator. This is like the criteria evaluator but it is configured by default to return a score on the scale from 1-10. It is recommended to normalize these scores by setting `normalize_by` to 10. Attributes: criteria: The criteria to evaluate. llm: The language model to use for the evaluation chain. normalize_by: If you want to normalize the score, the denominator to use. If not provided, the score will be between 1 and 10. prompt: The prompt template to use for evaluation. rNrArB normalize_byrY)r$r%r&r'r SCORE_STRINGrr(rArrBrr}floatrYrrr#r! ScoreStringr|sV )6(B(B B)--$&-(,  %,%) edl),0"T)0r#rc4eZdZUdZej Zeed<y) RunEvalConfig.LabeledScoreStringz3Configuration for a labeled score string evaluator.rN)r$r%r&r'rLABELED_SCORE_STRINGrr(rr#r!LabeledScoreStringrsA(5(J(J Jr#r)$r$r%r&r'rlistr9SINGLE_EVAL_CONFIG_TYPECUSTOM_EVALUATOR_TYPEr(r:r;BATCH_EVALUATOR_LIKEr-r*r.r/r<rrrOr,rDrHrrrZr`rdrhrrmrtrzrrrr#r!r7r7Tsq-HMIJ,/DDEQ=At12T9@8:>d/047> !%M3:$@!%NC$J%!IsTz B)-H$&-H $L ?& ? G- G / %, % 1 1 1' 1 1# 1D*D D D %( %(1)1.K[Kr#r7).r'collections.abcrrtypingrlangchain_core.embeddingsrlangchain_core.language_modelsrlangchain_core.promptsr langsmithr langsmith.evaluation.evaluatorr r langsmith.schemasr r pydanticrrrtyping_extensionsr0langchain_classic.evaluation.criteria.eval_chainr4langchain_classic.evaluation.embedding_distance.baserrN#langchain_classic.evaluation.schemarr1langchain_classic.evaluation.string_distance.baserrUr)RUN_EVALUATOR_LIKErrr,rr*rr7rr#r!rs'.0<5"N*11&JO'D.((4/1  c]HW%,-((4/1 .*.+\9OK'#- :}KI}Kr#