3fiQE:dZddlmZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZdd lmZdd lmZej.eZGd dee ZGddeZGddZGddeeZGddeeZGddeeZ y)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Sequence)Enum)Any)warn) AgentAction)BaseLanguageModel)run_in_executor)ChainceZdZdZdZ dZ dZ dZ dZ dZ dZ d Z d Z d Z d Z d Z dZ dZ dZ dZ dZ dZ dZ dZy) EvaluatorTypezThe types of the evaluators.qacot_qa context_qapairwise_string score_stringlabeled_pairwise_stringlabeled_score_string trajectorycriterialabeled_criteriastring_distance exact_match regex_matchpairwise_string_distanceembedding_distancepairwise_embedding_distance json_validity json_equalityjson_edit_distancejson_schema_validationN)__name__ __module__ __qualname____doc__QACOT_QA CONTEXT_QAPAIRWISE_STRING SCORE_STRINGLABELED_PAIRWISE_STRINGLABELED_SCORE_STRINGAGENT_TRAJECTORYCRITERIALABELED_CRITERIASTRING_DISTANCE EXACT_MATCH REGEX_MATCHPAIRWISE_STRING_DISTANCEEMBEDDING_DISTANCEPAIRWISE_EMBEDDING_DISTANCE JSON_VALIDITY JSON_EQUALITYJSON_EDIT_DISTANCEJSON_SCHEMA_VALIDATIONa/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_classic/evaluation/schema.pyrrs& B F%JS'O!L7H1@#VH<)7'OPKIKN9=-M"?;#M.#M=-T5Ir=rc,eZdZdZeeddZy) LLMEvalChainz,A base class for evaluators that use an LLM.c y)z#Create a new evaluator from an LLM.Nr<)clsllmkwargss r>from_llmzLLMEvalChain.from_llmNr=N)rCr rDrreturnr@)r$r%r&r' classmethodrrEr<r=r>r@r@Ks622r=r@cpeZdZdZeddZeddZed dZed dZ d d dZ y) _EvalArgsMixinz(Mixin for checking evaluation arguments.cyz2Whether this evaluator requires a reference label.Fr<selfs r>requires_referencez!_EvalArgsMixin.requires_referenceWr=cy)0Whether this evaluator requires an input string.Fr<rMs r>requires_inputz_EvalArgsMixin.requires_input\rPr=c6d|jjdS)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__r$rMs r>_skip_input_warningz"_EvalArgsMixin._skip_input_warningas$DNN$;$;#<_skip_reference_warningz&_EvalArgsMixin._skip_reference_warningfs!%T^^%<%<$==U V r=Ncd|jr&|$|jjd}t||#|jst |j d|j r&|$|jjd}t||%|j st |jdyyy)aTCheck if the evaluation arguments are valid. Args: reference: The reference label. input_: The input string. Raises: ValueError: If the evaluator requires an input string but none is provided, or if the evaluator requires a reference label but none is provided. Nz requires an input string.) stacklevelz requires a reference string.)rSrWr$ ValueErrorr rXrOrZ)rN referenceinput_msgs r>_check_evaluation_argsz%_EvalArgsMixin._check_evaluation_argsms   6>^^,,--GHCS/ !  d&9&9 ))a 8  " "y'8^^,,--JKCS/ !  )@)@ --! <*A r=rGboolrGstr)NN)r_ str | Noner`rgrGNone) r$r%r&r'propertyrOrSrXrZrbr<r=r>rJrJTs~2 VV  !%!===  =r=rJceZdZdZed dZed dZeddd d dZddd d dZ ddd d dZ ddd d d Z y)StringEvaluatorzString evaluator interface. Grade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels. c.|jjS)zThe name of the evaluation.rVrMs r>evaluation_namezStringEvaluator.evaluation_names~~&&&r=cyrLr<rMs r>rOz"StringEvaluator.requires_referencerPr=Nr_inputc y)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr<rN predictionr_rprDs r>_evaluate_stringsz!StringEvaluator._evaluate_stringsrFr=cTKtd|jf|||d|d{S7w)aAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nrsr_rp)r rtrrs r>_aevaluate_stringsz"StringEvaluator._aevaluate_stringssE.%   " " "       s (&(c T|j|||jd|||d|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. r_r`rvr<)rbrtrrs r>evaluate_stringsz StringEvaluator.evaluate_stringssD& ##i#F%t%% !    r=cpK|j|||jd|||d|d{S7w)aAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction: The LLM or chain prediction to evaluate. reference: The reference label to evaluate against. input: The input to consider during evaluation. **kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: The evaluation results containing the score or value. ryrvNr<)rbrwrrs r>aevaluate_stringsz!StringEvaluator.aevaluate_stringssR& ##i#F,T,, !      s -646rerc) rsz str | Anyr_str | Any | Nonerpr}rDrrGdict) rsrfr_rgrprgrDrrGr~) r$r%r&r'rirmrOrrtrwrzr|r<r=r>rkrks; '' '+"&   $         8'+"&   $        H!%           >!%           r=rkceZdZdZeddd ddZddd ddZddd ddZddd ddZy) PairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).Nroc y)Evaluate the output string pairs. Args: prediction: The output string from the first model. prediction_b: The output string from the second model. reference: The expected output / reference string. input: The input string. **kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: `dict` containing the preference, scores, and/or other information. Nr<rNrs prediction_br_rprDs r>_evaluate_string_pairsz.PairwiseStringEvaluator._evaluate_string_pairs rFr=cVKtd|jf||||d|d{S7w)Asynchronously evaluate the output string pairs. Args: prediction: The output string from the first model. prediction_b: The output string from the second model. reference: The expected output / reference string. input: The input string. **kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: `dict` containing the preference, scores, and/or other information. Nrsrr_rp)r rrs r>_aevaluate_string_pairsz/PairwiseStringEvaluator._aevaluate_string_pairs#sH*%   ' ' "%      )')c V|j|||jd||||d|S)rryrr<)rbrrs r>evaluate_string_pairsz-PairwiseStringEvaluator.evaluate_string_pairsBsG* ##i#F*t** !%     r=crK|j|||jd||||d|d{S7w)rryrNr<)rbrrs r>aevaluate_string_pairsz.PairwiseStringEvaluator.aevaluate_string_pairs`sU* ##i#F1T11 !%       .757) rsrfrrfr_rgrprgrDrrGr~) r$r%r&r'rrrrrr<r=r>rr s8N !%             6!%            H!%            F!%            r=rceZdZdZed dZedd d dZdd d dZdd d dZ dd d dZ y) AgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.cy)rRTr<rMs r>rSz'AgentTrajectoryEvaluator.requires_inputsr=N)r_c y)Evaluate a trajectory. Args: prediction: The final predicted response. agent_trajectory: The intermediate steps forming the agent trajectory. input: The input to the agent. reference: The reference answer. **kwargs: Additional keyword arguments. Returns: The evaluation result. Nr<rNrsagent_trajectoryrpr_rDs r>_evaluate_agent_trajectoryz3AgentTrajectoryEvaluator._evaluate_agent_trajectoryrFr=cVKtd|jf||||d|d{S7w)Asynchronously evaluate a trajectory. Args: prediction: The final predicted response. agent_trajectory: The intermediate steps forming the agent trajectory. input: The input to the agent. reference: The reference answer. **kwargs: Additional keyword arguments. Returns: The evaluation result. N)rsrr_rp)r rrs r>_aevaluate_agent_trajectoryz4AgentTrajectoryEvaluator._aevaluate_agent_trajectorysH,%   + + "-      rc V|j|||jd||||d|S)rryrsrprr_r<)rbrrs r>evaluate_agent_trajectoryz2AgentTrajectoryEvaluator.evaluate_agent_trajectorysG, ##i#F.t.. !-     r=crK|j|||jd||||d|d{S7w)rryrNr<)rbrrs r>aevaluate_agent_trajectoryz3AgentTrajectoryEvaluator.aevaluate_agent_trajectorysU, ##i#F5T55 !-       rrc) rsrfrz!Sequence[tuple[AgentAction, str]]rprfr_rgrDrrGr~) r$r%r&r'rirSrrrrrr<r=r>rrs@6 !%   <          :!%   <         L!%   <         J!%   <         r=r)!r' __future__rloggingabcrrcollections.abcrenumrtypingrwarningsr langchain_core.agentsr langchain_core.language_modelsr langchain_core.runnables.configr langchain_classic.chains.baser getLoggerr$loggerrfrr@rJrkrrr<r=r>rs9"#$-<;/   8 $3JC3Jl2521=1=h~ nc~ Bs ncs l| ~s| r=