g3fidZddlZddlZddlmZddlmZmZeje Z dZ dZ ddje Dcgc]}ej|d zc}zd zZddje Dcgc]}ej|c}zd zZd ed ed Zdddedeej&zdzdeefdZdddddddedededzdeej&zdzdedeededeefdZycc}wcc}w)z Utilities for working with HTML.N)Sequence)urljoinurlparse)z javascript:zmailto:#)z.cssz.jsz.icoz.pngz.jpgz.jpegz.gifz.svgz.csvz.bz2z.zipz.epubz.webpz.pdfz.docxz.xlsxz.pptxz.pptmz(?!|z[\#'\"])z href=[\"']z((?:z .)*?)[\#'\"]patternraw_htmlr returncf|xst}tttj||S)zExtract all links from a raw HTML string. Args: raw_html: original HTML. pattern: Regex to use for extracting links from raw HTML. Returns: A list of all links found in the HTML. )DEFAULT_LINK_REGEXlistsetrefindall)r r s W/var/www/auto_recruiter/arenv/lib/python3.12/site-packages/langchain_core/utils/html.pyfind_all_linksr*s*++G BJJw12 33TF)base_urlr prevent_outsideexclude_prefixescontinue_on_failureurlrrrrc||n|}t|}t|} t||} t} | D]} t| } | jdvr| }nW| j dr| jd| }n4t || j }| jr|d| jz }| j|g}| D]atfd|Dr|r7t}|j|jk7r?j |sQ|jc|S#t$r%}|rtjd| |Yd}~d}~wwxYw) aExtract all links from a raw HTML string and convert into absolute paths. Args: raw_html: original HTML. url: the url of the HTML. base_url: the base URL to check for outside links against. pattern: Regex to use for extracting links from raw HTML. prevent_outside: If `True`, ignore external links which are not children of the base URL. exclude_prefixes: Exclude any URLs that start with one of these prefixes. continue_on_failure: If `True`, continue if parsing a specific link raises an exception. Otherwise, raise the exception. Returns: A list of absolute paths to sub links. Nr >httphttpsz//:?z-Unable to load link %s. Raised exception: %sc3@K|]}j|yw)N) startswith).0exclude_prefixpaths r z$extract_sub_links..qsV>t~.Vs)rrrschemer"rr%queryadd Exceptionloggerwarninganynetlocappend)r rrr rrrbase_url_to_useparsed_base_url parsed_url all_linksabsolute_pathslink parsed_link absolute_patheresults parsed_pathr%s @rextract_sub_linksr;:sk4#+"6hCO/O#Jx9IUN "4.K!!%66 $ &#-#4#4"5Qtf = '[-=-= > $$!q):):(;%<rLs& $*   8 $4* CHH9KLAibiilZ/L MMPSS CHH,>?qibiil? @@3F*+40H/IV :> 4 4"RZZ/$6 4 #Y 4( '+ &( %FF FDj F 2::  $ F  FsmFF #YF5M@s C: 7C?