3fi ddlZddlZddlmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZdZdZd Zej(eZdd d ed ed edeeededef dZeeddfdedededeedef dZGddeZy)N)AnyCallableListMappingOptional)CallbackManagerForLLMRun) ConfigDict)SelfHostedPipeline)enforce_stop_tokensgpt2text-generation)text2text-generationr summarization)stoppipelinepromptargsrkwargsreturnc"||g|i|}|jdk(r|ddt|d}nP|jdk(r |dd}n8|jdk(r |dd}n td|jd td | t ||}|S) zInference function to send to the remote hardware. Accepts a Hugging Face pipeline (or more likely, a key pointing to such a pipeline on the cluster's object store) and returns generated text. r rgenerated_textNrr summary_textGot invalid task , currently only  are supported)tasklen ValueError VALID_TASKSr )rrrrrresponsetexts o/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_community/llms/self_hosted_hugging_face.py_generate_textr#s000H}})){+,S[]; 0 0{+, / ){>* /)]. :   "4. Kmodel_idrdevice model_kwargscddlm}m}m}ddlm}|xsi}|j |fi|} |dk(r|j |fi|} n.|dvr|j |fi|} nt d|dtdtjjd Zdd l } | jj} |d ks|| k\rt d|d| d|dkr| dkDrtj!d| ||| | ||}|j"tvr t d|j"dtd|S#t$r} td |d | d } ~ wwxYw)zInference function to send to the remote hardware. Accepts a huggingface model_id and returns a pipeline for the task. r)AutoModelForCausalLMAutoModelForSeq2SeqLM AutoTokenizer)rr )rrrrrzCould not load the z# model due to missing dependencies.Ntorchz Got device==z', device is required to be within [-1, )zDevice has %d GPUs available. Provide device={deviceId} to `from_model_id` to use availableGPUs for execution. deviceId is -1 for CPU and can be a positive integer associated with CUDA device id.)rmodel tokenizerr&r') transformersr)r*r+rfrom_pretrainedrr ImportError importlibutil find_specr,cuda device_countloggerwarningr)r%rr&r'r)r*r+ hf_pipeline _model_kwargsr0r/er,cuda_device_countrs r"_load_transformerr?1sXW4 &BM- --hH-HI  $ $8(88S]SE > >9)99(TmTE#D6):;-~V ~~(4!JJ335 B;6%66vh'88I7J!M  A:+a/ NNL"   " H}}K' /)]. :   OG !$'J K  sAD,, E 5EE c 6eZdZUdZeZeed< eZ eed< dZ e ed< dZ e eed< dZeed< gd Zeeed < eZeed < eZeed < ed Zdeffd ZedeeeffdZedefdZ ddede eede ededef dZ xZ!S)SelfHostedHuggingFaceLLMayHuggingFace Pipeline API to run on self-hosted remote hardware. Supported hardware includes auto-launched instances on AWS, GCP, Azure, and Lambda, as well as servers specified by IP address and SSH credentials (such as on-prem, or another cloud like Paperspace, Coreweave, etc.). To use, you should have the ``runhouse`` python package installed. Only supports `text-generation`, `text2text-generation` and `summarization` for now. Example using from_model_id: .. code-block:: python from langchain_community.llms import SelfHostedHuggingFaceLLM import runhouse as rh gpu = rh.cluster(name="rh-a10x", instance_type="A100:1") hf = SelfHostedHuggingFaceLLM( model_id="google/flan-t5-large", task="text2text-generation", hardware=gpu ) Example passing fn that generates a pipeline (bc the pipeline is not serializable): .. code-block:: python from langchain_community.llms import SelfHostedHuggingFaceLLM from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline import runhouse as rh def get_pipeline(): model_id = "gpt2" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id) pipe = pipeline( "text-generation", model=model, tokenizer=tokenizer ) return pipe hf = SelfHostedHuggingFaceLLM( model_load_fn=get_pipeline, model_id="gpt2", hardware=gpu) r%rrr&Nr'hardware)z./r1r, model_reqs model_load_fn inference_fnforbid)extrarc |jdt|jdt|jdd|jddd}t|d d|i|y) a Construct the pipeline remotely using an auxiliary function. The load function needs to be importable to be imported and run on the server, i.e. in a module and not a REPL or closure. Then, initialize the remote inference function. r%rr&rr'N)r%rr&r'load_fn_kwargs)getDEFAULT_MODEL_ID DEFAULT_TASKsuper__init__)selfrrI __class__s r"rOz!SelfHostedHuggingFaceLLM.__init__s\ :/?@JJv|4jj1-"JJ~t<   AA&Ar$rc>id|jid|jiS)zGet the identifying parameters.r%r')r%r'rPs r"_identifying_paramsz,SelfHostedHuggingFaceLLM._identifying_paramss0 4==) t001  r$cy)Nselfhosted_huggingface_pipelinerJrSs r" _llm_typez"SelfHostedHuggingFaceLLM._llm_types0r$rr run_managerc B|jd|j||d|S)N)rrrrJ)client pipeline_ref)rPrrrXrs r"_callzSelfHostedHuggingFaceLLM._calls1t{{ &&vD DJ  r$)NN)"__name__ __module__ __qualname____doc__rLr%str__annotations__rMrr&intr'rdictrBrrCrr?rDrr#rEr model_configrOpropertyrrTrWrr\ __classcell__)rQs@r"rArAps"&P%Hc$2D#FCOT#'L(4.'1Hc<;JS ;E/M8/<+L(+<L B B WS#X%6  1311 %):>    tCy!  67     r$rA)importlib.utilr4loggingtypingrrrrrlangchain_core.callbacksrpydanticr $langchain_community.llms.self_hostedr langchain_community.llms.utilsr rLrMr getLoggerr]r9rar#rcrdr?rArJr$r"rps99=C> J   8 $!%    49      @%#' << < <4. <  <~c 1c r$