3fi UdZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZdd lmZmZdd lmZdd lmZmZdd lmZddlmZddlmZm Z m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,m-Z-de.de/e0fdZ1ie%jde!e%jfe e%jhee%jje e%jle*e%jnee%jpe)e%jre e%jtee%jvee%jxe-e%jze,e%j|ee%j~ee%jee%jee%jee%jee%je#e%jeiZFe0e%eGe&eGezeGe'zfeHd<ddde%dedzdedee'zfdZIddddee%dedzd e0dzdede/ee'zf d!ZJy)"z Loading datasets and evaluators.)Sequence)Any)BaseLanguageModel)Chain)TrajectoryEvalChain)PairwiseStringEvalChain)LabeledPairwiseStringEvalChain)CriteriaEvalChainLabeledCriteriaEvalChain)EmbeddingDistanceEvalChain"PairwiseEmbeddingDistanceEvalChain)ExactMatchStringEvaluator)JsonEqualityEvaluatorJsonValidityEvaluator)JsonEditDistanceEvaluator)JsonSchemaEvaluator)ContextQAEvalChainCotQAEvalChain QAEvalChain)RegexMatchStringEvaluator) EvaluatorType LLMEvalChainStringEvaluator)LabeledScoreStringEvalChainScoreStringEvalChain)PairwiseStringDistanceEvalChainStringDistanceEvalChainurireturnc ddlm}|d|}t|dS#t$r}d}t||d}~wwxYw)aLoad a dataset from the [LangChainDatasets on HuggingFace](https://huggingface.co/LangChainDatasets). Args: uri: The uri of the dataset to load. Returns: A list of dictionaries, each representing a row in the dataset. **Prerequisites** ```bash pip install datasets ``` Examples: -------- ```python from langchain_classic.evaluation import load_dataset ds = load_dataset("llm-math") ``` r) load_datasetzXload_dataset requires the `datasets` package. Please install with `pip install datasets`NzLangChainDatasets/train)datasetsr! ImportErrorlist)rr!emsgdatasets b/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_classic/evaluation/loading.pyr!r!4sY.&)/u56G  !! & : #A% &s! =8=_EVALUATOR_MAPN)llm evaluatorr+kwargsc |tvr.d|dttj}t|t|}t |t r+ ddlm}|xs |ddd }|jd d |i|S|d i|S#t$r* ddl m}n#t$r}d}t||d}~wwxYwY]wxYw#t$r}d |d }t||d}~wwxYw)aFLoad the requested evaluation chain specified by a string. Parameters ---------- evaluator : EvaluatorType The type of evaluator to load. llm : BaseLanguageModel, optional The language model to use for evaluation, by default None **kwargs : Any Additional keyword arguments to pass to the evaluator. Returns: ------- Chain The loaded evaluation chain. Examples: -------- >>> from langchain_classic.evaluation import load_evaluator, EvaluatorType >>> evaluator = load_evaluator(EvaluatorType.QA) zUnknown evaluator type: z Valid types are: r) ChatOpenAIzCould not import langchain_openai or fallback onto langchain_community. Please install langchain_openai or specify a language model explicitly. It's recommended to install langchain_openai AND specify a language model explicitly.Nzgpt-4*)modelseed temperaturezEvaluation with the z requires a language model to function. Failed to create the default 'gpt-4' model. Please manually provide an evaluation LLM or check your openai credentials.r+) r*r%keys ValueError issubclassrlangchain_openair/r$&langchain_community.chat_models.openai Exceptionfrom_llm)r,r+r-r' evaluator_clsr/r&s r)load_evaluatorr=ss!6&&yk!$~':':'<"=!> @ o"9-M-. ) 27 J'JC&}%%8#888  "6 ""5 2 2#2?&c*12 2  )&}o655 S/q ( )sZBB9 B6 BB6 B0B++B00B63B95B66B99 CCC)r+config evaluatorsr>c g}|D]9}|r|j|ini}|jt|fd|ii||;|S)aoLoad evaluators specified by a list of evaluator types. Parameters ---------- evaluators : Sequence[EvaluatorType] The list of evaluator types to load. llm : BaseLanguageModel, optional The language model to use for evaluation, if none is provided, a default ChatOpenAI gpt-4 model will be used. config : dict, optional A dictionary mapping evaluator types to additional keyword arguments, by default None **kwargs : Any Additional keyword arguments to pass to all evaluators. Returns: ------- List[Chain] The loaded evaluators. Examples: -------- >>> from langchain_classic.evaluation import load_evaluators, EvaluatorType >>> evaluators = [EvaluatorType.QA, EvaluatorType.CRITERIA] >>> loaded_evaluators = load_evaluators(evaluators, criteria="helpfulness") r+)getappendr=)r?r+r>r-loadedr,_kwargss r)load_evaluatorsrEs\BFS /5&**Y+2 nYQCQ;Pf;P;PQRS M)K__doc__collections.abcrtypingrlangchain_core.language_modelsrlangchain_classic.chains.baser9langchain_classic.evaluation.agents.trajectory_eval_chainr'langchain_classic.evaluation.comparisonr2langchain_classic.evaluation.comparison.eval_chainr 0langchain_classic.evaluation.criteria.eval_chainr r 4langchain_classic.evaluation.embedding_distance.baser r -langchain_classic.evaluation.exact_match.baser)langchain_classic.evaluation.parsing.baserr2langchain_classic.evaluation.parsing.json_distancer0langchain_classic.evaluation.parsing.json_schemarlangchain_classic.evaluation.qarrr-langchain_classic.evaluation.regex_match.baser#langchain_classic.evaluation.schemarrr/langchain_classic.evaluation.scoring.eval_chainrr1langchain_classic.evaluation.string_distance.baserrstrr%dictr!QACOT_QA CONTEXT_QAPAIRWISE_STRING SCORE_STRINGLABELED_PAIRWISE_STRINGLABELED_SCORE_STRINGAGENT_TRAJECTORYCRITERIALABELED_CRITERIASTRING_DISTANCEPAIRWISE_STRING_DISTANCEEMBEDDING_DISTANCEPAIRWISE_EMBEDDING_DISTANCE JSON_VALIDITY JSON_EQUALITYJSON_EDIT_DISTANCEJSON_SCHEMA_VALIDATION REGEX_MATCH EXACT_MATCHr*type__annotations__r=rEr4rFr)rrs&$</LTYP T  !"c!"d4j!"Nk.0!!#:   4  ))+I &&(C""$7-""$<!!#:**,K$$&@--/Q!6 !6!"$$&?#$((*=88)e$tO'<<><%)@#@# T !@# @#  _ @#L%) %'% T !% 4K %  %  %/ !" %rF