3fiKdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZdd lmZerdd lmZdd lmZdd lmZej:eZeGd dZ GddeZ!y)zModule contains a PDF parser based on Document AI from Google Cloud. You need to install two libraries to use this parser: pip install google-cloud-documentai pip install google-cloud-documentai-toolbox N) dataclass) TYPE_CHECKINGAnyIteratorListOptionalSequence)BaseBlobParser)Blob)Document) batch_iterate)get_client_info) OperationDocumentProcessorServiceClient)ProcessOptionsc&eZdZUdZeed<eed<y)DocAIParsingResultsz1A dataclass to store Document AI parsing results. source_path parsed_pathN)__name__ __module__ __qualname____doc__str__annotations__^/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_google_community/docai.pyrrs;rrceZdZdZdddddddeddeedeedeed eef d Zd ed ee fd Z d'dee dee e dee dee d df dZ d(d edeeded ee fdZ d)deedeede de ded ee f dZde ed ee fdZde ed e dfdZd e dd e fd!Zddd"dd#deedeed eed$e deeded e dfd%Zd e dd e efd&Zy)* DocAIParseraGoogle Cloud Document AI parser. Inherits from [`BaseBlobParser`][langchain_core.document_loaders.BaseBlobParser]. Parses documents using Google Cloud Document AI for text extraction and layout analysis. See [Document AI documentation](https://cloud.google.com/document-ai/docs/overview) for detailed information. !!! note "Installation" Requires additional dependencies: ```bash pip install langchain-google-community[docai] ``` N)client project_idlocationgcs_output_pathprocessor_namer"rr#r$r%r&ct|t|k(r tdd}|r%tj||std|d||_||_|r||_y ddlm}ddl m }|||d  } || td |_|j j|j|_|jdk(rd|_y d|_y #t$r} td| d } ~ wwxYw)aInitializes the parser. Args: client: A `DocumentProcessorServiceClient` to use location: A Google Cloud location where a Document AI processor is located gcs_output_path: A path on Google Cloud Storage to store parsing results processor_name: Full resource name of a Document AI processor or processor version You should provide either a client or location (and then a client would be instantiated). zGYou must specify either a client or a location to instantiate a client.z?projects\/[0-9]+\/locations\/[a-z\-0-9]+\/processors\/[a-z0-9]+zProcessor name z has the wrong format. If your prediction endpoint looks like https://us-documentai.googleapis.com/v1/projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID:process, use only projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID part.r) ClientOptionsrCould not import google-cloud-documentai python package. Please, install docai dependency group: `pip install langchain-google-community[docai]`Nz-documentai.googleapis.com)quota_project_id api_endpointz document-ai)module)client_options client_infonameLAYOUT_PARSER_PROCESSORTF)bool ValueErrorre fullmatch_gcs_output_path_processor_name_clientgoogle.api_core.client_optionsr(google.cloud.documentair ImportErrorr get_processortype_processor_type_use_layout_parser) selfr"r#r$r%r&patternr(rexcoptionss r__init__zDocAIParser.__init__<s", <4> )  U ",,w"G!.!12 !0- !DL HR$!+ (z)CDG:&+=ADL $(<<#=#=>#=#R#W#WD ##'@@*.'*/'' !F  s% C!! C;* C66C;blobreturnc#\K|j|g|jEd{y7w)zParses a blob lazily. Args: blob: a `Blob` to parse This is a long-running operation. A recommended way is to batch documents together and use the `batch_parse` method. )r%N) batch_parser6)r@rEs r lazy_parsezDocAIParser.lazy_parses)##TFDz-DocAIParser.online_process..s? !&$)NN"&)) s8;c3K|]M}t|jjj|jj dOyw)pagermrnN)r layoutdocumenttext page_numberrr)rsrxrbrEresponses rruz-DocAIParser.online_process..sT !24;;@Q@Q@V@V!W $ 0 0"&)) AAr) google.cloudr`r;-google.cloud.documentai_toolbox.wrappers.pagerbr[r8process_documentProcessRequestr7 GcsDocumentrrmimetyper?rzchunked_documentchunksrR) r@rEr\r]r`rBrZrbr}s ` @@ronline_processzDocAIParser.online_processsM  /  8$77Q:PQ<<00  % %))'33 II"mm@/@4!0"&% &     " " &..??FF    %--33   Y B    B  .  scD.C0D B,D.D*)D.*D,+D.0 D 9 DD  D. D' D""D''D.,D.blobs timeout_seccheck_in_interval_secc+K|xs |j}|s td|j|fd|i|}|Dcgc]}|jj} }t j d| d} |j|rUtj|| |z } | |kDrtd| dt j d|j|rU|j|} |j| Ed {y cc}w7 w) ahParses a list of `Blob` lazily. Args: blobs: A list of `Blob` to parse. gcs_output_path: Path on Google Cloud Storage to store parsing results. timeout_sec: Timeout to wait for Document AI to complete, in seconds. check_in_interval_sec: Interval to wait until next check whether parsing operations have been completed, in seconds. process_options_kwargs: Optional parameters to pass to the Document AI processors ??? example "Long-running operation" ```python # Submit async jobs operations = parser.docai_parse(blobs, gcs_path) # Optionally poll until finished parser.is_running(operations) # Save operation names operation_names = [op.operation.name for op in operations] # Later, load results and build Documents operations = parser.operations_from_names(operation_names) results = parser.get_results(operations) docs = parser.parse_from_results(results) ``` :An output path on Google Cloud Storage should be provided.r%z9Started parsing with Document AI, submitted operations %srz#Timeout exceeded! Check operations z later!.) operationsN) r6r3 docai_parse operationr0loggerdebug is_runningtimesleep TimeoutError get_resultsparse_from_results) r@rr%rrr] output_pathropoperation_names time_elapsedresultss rrHzDocAIParser.batch_parse s!J&>)>)> L &T%%  #. 2H 8BB2<<,,BB G  ooj) JJ, - 1 1Lk)"9/9J'R LL ooj)""j"9**7333C 4s#7D DA?D &D <D= D rc#0 K ddlm}ddlm}ddlm|D]W | j\}}|||dz}|jr fd|DEd{D fd|DEd{Yy#t $r}t d|d}~wwxYw797&w) Nr) split_gcs_uri) _get_shardsrarc/c3K|]M}|jjD]2}t|j|jj d4Oywrk)rrr rqrlr)rsshardrtresults rruz1DocAIParser.parse_from_results..^sd !&!7!7!>!> %*]](-&,&8&8"  r~c3K|]T}|jD]C}t|j|j|jj dEVywrw)rRr ryr{r|r)rsrrxrbrs rruz1DocAIParser.parse_from_results..jsi  %  %6t{{EJJ%O$($4$4&,&8&8"  sAA) 7google.cloud.documentai_toolbox.utilities.gcs_utilitiesr1google.cloud.documentai_toolbox.wrappers.documentrrrbr;rr?) r@rrrrBgcs_bucket_name gcs_prefixshardsrbrs @@rrzDocAIParser.parse_from_resultsJs    X  F*78J8J*K 'OZ *s2BCF&& "(    "(   !   B    sFBA5ABBB-B.B5 B> B  BBBrrc ddlm}|Dcgc]%}|jj ||'c}S#t$r}td|d}~wwxYwcc}w)z5Initializes Long-Running Operations from their names.r)GetOperationRequestzplong running operations package not found, please install it with`pip install langchain-google-community[docai]`Nr/)request)!google.longrunning.operations_pb2rr;r8 get_operation)r@rrrBr0s roperations_from_namesz!DocAIParser.operations_from_namesvso  (  LL & &/B/M & N   B    s9*A A AArc&td|DS)Nc3>K|]}|j ywN)done)rsrs rruz)DocAIParser.is_running..s6Rrwwy=6s)any)r@rs rrzDocAIParser.is_runnings6:666ri)r%r& batch_sizer\rc  ddlm}|xs |j} | t d|xs |j }| t dg} t ||D]} |j|j| D cgc]-} |j| j| jxsd /c}  } |j|jj| |  }|jdi|}| j|j j#|j%|| ||d| S#t$r}td|d}~wwxYwcc} w)aJRuns Google Document AI PDF Batch Processing on a list of `Blob`. Args: blobs: List of `Blob` to be parsed gcs_output_path: Path (folder) on GCS to store results processor_name: Name of a Document AI processor. batch_size: Amount of documents per batch field_mask: Comma-separated list of which fields to include in the Document AI response. Suggested: `'text,pages.pageNumber,pages.layout'` process_options_kwargs: Optional parameters to pass to the Document AI processors Document AI has a 1000 file limit per batch, so batches larger than that need to be split into multiple requests. Batch processing is an async long-running operation and results are stored in a output GCS bucket. rr_rPNrz0A Document AI processor name should be provided.)sizeiterablerdre) documents) gcs_documents)rfr\)gcs_output_configT)r0input_documentsdocument_output_configrZrir)rr`r;r6r3r7r BatchDocumentsInputConfig GcsDocumentsrrrrDocumentOutputConfigGcsOutputConfigr[appendr8batch_process_documentsBatchProcessRequest)r@rr%r&rr\r]r`rBrrbatchrE input_config output_configrZs rrzDocAIParser.docai_parses8  /&>)>)>  L (?4+?+?  !OP P " UC E%??(55 %*  ! #..$(II&*mm&H7H/6@ L';;","A"A"Q"Q'J#R#<M TUO    4422+(4/<(7*. 3 + @a B  &sD..2E . E7 EEc x ddlm}|Dcgc]}t|j|r|jj n.|j |jjj D]#}t|j|j%c}}S#t$r}td|d}~wwxYwcc}}w)Nr)BatchProcessMetadatarP)rr) google.cloud.documentai_v1rr; isinstancerpindividual_process_statuses deserializevaluerinput_gcs_sourceoutput_gcs_destination)r@rrrBrstatuss rrzDocAIParser.get_resultss  !  bkk+?@ 77)55KK%%--   "33"99       B    sBBB6 B3" B..B3)TNiTr)Ni<)rrrrrrrDr rr rIr2rintr[rrr rHrrrrrrrrrr!r!(s`,>B$("&)-(,A09:A0SM A0 3- A0 "# A0! A0F St S(: S59*.$'48 8#+D>8T#Y'8SM 8 $,D> 8  8z%)FFSMF#& F (  FV*.%' <4~<4"#<4 <4 # <4 #& <4 ( <4|*/0* ( *X T#Y 4 CT "7T+%6747*.(,$(N~N"# N ! N  NSMN#&N k N` d;&7 DATrsw !II:=-3=3@   8 $  J .J r