Mj^iDddlmZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZGd d e ZGd de ZGddZy))genaiN)Part)storage)transfer_manager)Path) AIEditLLM)settings)service_accountc2eZdZdZddZddZd dZd dZy) AIEditLLMGCScBtjjtjj t jdt jdg}tjdt jd||_ d|_ t|_y)ag Initialize the AIEditLLMGCS client with Vertex AI credentials and GCS helper. Detailed Description: Loads a Google Cloud service account JSON from Django BASE_DIR/ai_settings, constructs scoped credentials for Cloud Platform, and initializes a [`genai.Client`](ai_settings/ai_gen_gcs_ext.py) configured for Vertex AI with a specific project and location. Sets up a chat handle placeholder (None) and instantiates a [`GCS`](ai_settings/ai_gen_gcs_ext.py) helper for Google Cloud Storage operations. This prepares the instance for token counting and file uploads via GCS-backed Parts, extending the base [`AIEditLLM`](ai_settings/ai_gen.py) behavior to use external storage. Parameters (Args): - None Returns: - None Raises: - FileNotFoundError: If the service account JSON file does not exist under BASE_DIR/ai_settings. - Exception: If genai.Client initialization fails due to invalid credentials, project, or location configuration. Side Effects: - Reads service account credentials from disk. - Creates a genai client bound to the specified project and region. - Instantiates a GCS helper that configures storage client and paths. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCS llm = AIEditLLMGCS() # Ready to create chats, count tokens, and upload files via GCS-backed Parts llm.create_chat(aiedit_temperature=0.3, aiedit_budget=16384) ai_settingsz.https://www.googleapis.com/auth/cloud-platform)scopesTus-east1)vertexaiprojectlocation credentialsN)r Credentialsfrom_service_account_fileospathjoinr BASE_DIRSECRET_KEY_LLM_API_KEYrClientSECRET_PROJECT_NAMEclientchatGCS gcs_client)selfrs 5/var/www/auto_recruiter/ai_settings/ai_gen_gcs_ext.py__init__zAIEditLLMGCS.__init__s}:&11KK GGLL**M8;Z;Z [DEL llT8;W;Wbl{FH  %Nc|sP|jd}|dkDr:||dzdj}|dk(rd}n|dk(s|dk(rd }n|d k(rd }n|d k(rd }|j|||}td||S)uf Upload a local file to GCS and return a Gemini Part, with MIME auto-detection and optional cache bypass. Detailed Description: Determines the MIME type from the file extension when not provided (pdf → application/pdf; txt/json → text/plain; mp4 → video/mp4; mov → video/quicktime), then delegates the actual upload/URI resolution to [`AIEditLLMGCS.upload_gcs_files`](ai_settings/ai_gen_gcs_ext.py). If force=True, bypasses cached GCS lookups and re-uploads. Prints a completion message with the source file name and returns the constructed Part for use in Gemini message contents. Parameters (Args): - file_name (str): Absolute or project-relative path to the local file (e.g., "/data/1001/SCENE_3A/script_scene_3A.pdf"). - force (bool): If True, forces a fresh GCS upload even if a cached entry exists; otherwise reuses existing GCS URI when available. - mime_type (str or None): Optional explicit MIME type. If None, inferred from file extension. Returns: - object: A Gemini [`Part`](ai_settings/ai_gen_gcs_ext.py) representing the uploaded file (created via [`AIEditLLMGCS.upload_gcs_files`](ai_settings/ai_gen_gcs_ext.py)). Raises: - Exception: Any error propagated from the underlying GCS upload helper ([`AIEditLLMGCS.upload_gcs_files`](ai_settings/ai_gen_gcs_ext.py)). Side Effects: - Prints "Upload completed. File Name: " to stdout. - Performs network I/O to Google Cloud Storage via the delegated upload method. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCS llm = AIEditLLMGCS() # Auto-detect MIME from extension and reuse cached GCS URI if present part = llm.upload_file("/data/1001/SCENE_3A/20240115_153045/videos/A001C002_240115_R2A3.mov", force=False) # Force re-upload with explicit MIME type part_forced = llm.upload_file("/data/1001/scripts/script_scene_3A.pdf", force=True, mime_type="application/pdf") .rrNpdfapplication/pdftxtjson text/plainmp4 video/mp4movvideo/quicktimeUpload completed. File Name: )rfindlowerupload_gcs_filesprintr# file_nameforce mime_type ext_indexext file_parts r$ upload_filezAIEditLLMGCS.upload_file6s@!,I1} ! -335%< 1IE\SF] ,IE\ +IE\ 1I)))YF  -i[9:r&c||d}|r||d<|jjj||}|jS)a Count prompt tokens for a given content payload using Vertex AI Gemini. Detailed Description: Invokes the Vertex AI token counting endpoint via [`genai.Client.models.count_tokens`](ai_settings/ai_gen_gcs_ext.py) to estimate the total tokens for a prospective request. Accepts optional `system_instruction` (not currently passed to the SDK in this implementation) and a configurable `model_name` (default "gemini-2.5-pro"). Returns the `total_tokens` from the SDK response, enabling preflight budgeting and rate-limit planning before sending messages. Parameters (Args): | Parameter | Type | Description | | :----------------- | :------ | :-------------------------------------------------------------------------- | | contents | Any | Request contents compatible with Gemini (text, Parts, or SDK-structured payload). | | system_instruction | Any | Optional system instruction to contextualize the request (currently not forwarded). | | model_name | str | Gemini model identifier to use for counting (default: "gemini-2.5-pro"). | Returns: | Type | Description | | :--- | :-------------------------------------------------- | | int | Total token count reported by the SDK for contents. | Raises: | Exception Type | Condition | | :------------- | :------------------------------------------------------------------------ | | Exception | Propagated if the SDK call fails (network errors, invalid inputs, etc.). | Side Effects: - Performs a network request to the Vertex AI Models API. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCS from google.genai.types import Part llm = AIEditLLMGCS() text_part = Part.from_text("Summarize the following scene transcript.") tokens = llm.count_prompt_tokens([text_part], model_name="gemini-2.5-pro") print("Estimated tokens:", tokens) modelcontentssystem_instructionrmodels count_tokens total_tokens)r#rBrC model_nameargsresponses r$count_prompt_tokensz AIEditLLMGCS.count_prompt_tokensgsUJ    );D% &;;%%223 $$$r&ch|jjj||}|jS)a Count session tokens for a prepared Gemini chat message payload. Detailed Description: Calls the Vertex AI token counting endpoint via [`genai.Client.models.count_tokens`](ai_settings/ai_gen_gcs_ext.py) to estimate total tokens for a list of chat messages (contents) intended for a session. Useful for budgeting, preflight checks, and rate-limit planning before sending streaming or non-streaming requests. This function is a thin wrapper: it forwards the messages and model_name to the SDK and returns `total_tokens` from the response without additional validation or transformation. Parameters (Args): | Parameter | Type | Description | | :--------- | :--- | :---------- | | messages | Any | Message payload compatible with Gemini (list of dicts, Parts, or SDK-structured contents) representing the chat session input. | | model_name | str | Gemini model identifier to use for counting (default: "gemini-2.5-pro"). | Returns: | Type | Description | | :--- | :---------- | | int | Total token count reported by the SDK for the provided messages. | Raises: | Exception Type | Condition | | :------------- | :-------- | | Exception | Propagated if the SDK call fails (e.g., network errors, invalid payloads, authentication issues). | Side Effects: - Performs a network request to the Vertex AI Models API. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCS from google.genai.types import Part llm = AIEditLLMGCS() msgs = [ Part.from_text("System: You are an assistant."), Part.from_text("User: Count tokens for this prompt.") ] tokens = llm.count_session_tokens(msgs, model_name="gemini-2.5-pro") print("Session tokens:", tokens) r@rD)r#messagesrHrJs r$count_session_tokensz!AIEditLLMGCS.count_session_tokenss8L;;%%223 $$$r&c |jj|||}|S#t$r}td|d|dd}~wwxYw)u Upload a local file to Google Cloud Storage and return a Gemini Part from its GCS URI. Detailed Description: Delegates to [`GCS.get_gcs_uri_part`](ai_settings/ai_gen_gcs_ext.py) to resolve or upload the given file to GCS and construct a [`Part`](ai_settings/ai_gen_gcs_ext.py) using its GCS URI and MIME type. If force=True, bypasses cached entries and re-uploads the file. On success, returns the constructed Part; on failure, prints an error and re-raises the exception to let callers handle it (e.g., retries, user feedback). Parameters (Args): | Parameter | Type | Description | | :-------- | :------ | :---------------------------------------------------------------------------------------------- | | file_name | str | Absolute path under BASE_DIR/data or project-relative path to the local file to upload. | | mime_type | str | MIME type for the uploaded file (e.g., "video/mp4", "application/pdf"). Defaults to "video/mp4".| | force | bool | If True, forces re-upload even if a cached GCS URI exists. Defaults to False. | Returns: | Type | Description | | :---- | :---------------------------------------------------------------------------------------------------- | | object| A Gemini [`Part`](ai_settings/ai_gen_gcs_ext.py) created from the file’s GCS URI (via Part.from_uri). | Raises: | Exception Type | Condition | | :------------- | :------------------------------------------------------------------ | | Exception | Any error encountered during GCS URI resolution or upload is re-raised. | Side Effects: - Performs network I/O to Google Cloud Storage (via [`GCS.get_gcs_uri_part`](ai_settings/ai_gen_gcs_ext.py)). - Prints an error message to stdout on failure. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCS llm = AIEditLLMGCS() part = llm.upload_gcs_files( file_name="/data/1001/SCENE_3A/20240115_153045/videos/A001C002_240115_R2A3.mov", mime_type="video/quicktime", force=False ) print(part) %An error occurred during File upload: TdebugNr"get_gcs_uri_part Exceptionr6r#r8r:r9 video_partes r$r5zAIEditLLMGCS.upload_gcs_filessSN 999iQVXJ   9)CsKSW X  s! A>AFN)Ngemini-2.5-pro)r[Nr/F)__name__ __module__ __qualname__r%r>rKrNr5r&r$r r s% N/b.%`*%Z,r&r c2eZdZdZddZddZddZddZy) AIEditLLMGCSHctjdtjd|_d|_t |_y)a Initialize the AIEditLLMGCSH client with Vertex AI configuration and GCS helper. Detailed Description: Creates a [`genai.Client`](ai_settings/ai_gen_gcs_ext.py) instance configured for Vertex AI with a fixed project and location, sets an empty chat handle, and instantiates a [`GCS`](ai_settings/ai_gen_gcs_ext.py) helper for Google Cloud Storage operations. Unlike [`AIEditLLMGCS.__init__`](ai_settings/ai_gen_gcs_ext.py), this variant does not load explicit service account credentials and relies on ambient/default authentication. Intended for quick setup of GCS-backed Parts and chat sessions without credential bootstrapping code. Parameters (Args): - None Returns: - None Raises: - Exception: If `genai.Client` initialization fails due to environment or configuration issues. Side Effects: - Initializes a Vertex AI client bound to the specified project and region. - Creates a GCS helper instance for subsequent storage interactions. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCSH llm = AIEditLLMGCSH() # Ready to create chats and upload files via GCS-backed Parts llm.create_chat(aiedit_temperature=0.3, aiedit_budget=16384) Tr)rrrN)rrr rrr r!r"r#s r$r%zAIEditLLMGCSH.__init__s/8llT8;W;Wblm  %r&Nc|sh|jd}|dkDrF||dzdj}td||dk(rd}n|dk(s|d k(rd }n|d k(rd }n|d k(rd}td||j|||}td||S)u& Upload a local file to GCS and return a Gemini Part, with MIME auto-detection and optional cache bypass. Detailed Description: Infers MIME type from the file extension when mime_type is not provided (pdf → application/pdf; txt/json → text/plain; mp4 → video/mp4; mov → video/quicktime). Delegates upload and URI resolution to [`AIEditLLMGCS.upload_gcs_files`](ai_settings/ai_gen_gcs_ext.py), which returns a Gemini [`Part`](ai_settings/ai_gen_gcs_ext.py) constructed from the file’s GCS URI. Prints a completion message including the source file name and returns the Part. Use force=True to bypass cached GCS entries and re-upload. Parameters (Args): - file_name (str): Absolute or project-relative path to the local file. - force (bool): If True, forces re-upload even if a cached GCS URI exists; otherwise reuses the cached URI when available. - mime_type (str or None): Explicit MIME type; if None, inferred from the file extension. Returns: - object or None: A Gemini Part representing the uploaded file (created via [`AIEditLLMGCS.upload_gcs_files`](ai_settings/ai_gen_gcs_ext.py)). Returns None if upload fails. Raises: - Exception: Propagated from the underlying GCS upload helper ([`AIEditLLMGCS.upload_gcs_files`](ai_settings/ai_gen_gcs_ext.py)). Side Effects: - Performs network I/O to Google Cloud Storage. - Prints "Upload completed. File Name: " to stdout. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCSH llm = AIEditLLMGCSH() # Auto-detect MIME from extension and reuse cached GCS URI if present part = llm.upload_file("/data/1001/SCENE_3A/videos/A001C002_240115_R2A3.mov", force=False) # Force re-upload with explicit MIME type part_forced = llm.upload_file("/data/1001/scripts/script_scene_3A.pdf", force=True, mime_type="application/pdf") r(rrNzEXT :r)r*r+r,r-r.r/r0r1z Mime Type :r2)r3r4r6r5r7s r$r>zAIEditLLMGCSH.upload_files@!,I1} ! -335gs#%< 1IE\SF] ,IE\ +IE\ 1I - +)))YF  -i[9:r&cd}t|d5}|j}dddtjd}t d||S#1swY0xYw)u3 Upload a local text file and return a Gemini Part from its raw bytes. Detailed Description: Reads the specified file from disk as bytes and constructs a [`Part`](ai_settings/ai_gen_gcs_ext.py) using `Part.from_bytes` with MIME type "text/plain". This utility is intended for lightweight text uploads (e.g., prompts, JSON, transcripts) without involving Google Cloud Storage or caching logic. It simply loads the file, wraps it as a Part, prints a completion message, and returns the Part for inclusion in Gemini chat/message contents. Parameters (Args): | Parameter | Type | Description | | :-------- | :--- | :---------- | | file_name | str | Path to the local text file to read and convert into a Part. | | force | bool | Unused flag reserved for parity with other upload methods; currently ignored. | | type | Any | Unused metadata parameter; reserved for future behavior. | Returns: | Type | Description | | :----- | :---------- | | object | A [`Part`](ai_settings/ai_gen_gcs_ext.py) created from the file’s raw bytes with MIME "text/plain". | Raises: | Exception Type | Condition | | :----------------- | :-------- | | FileNotFoundError | When the file_name path does not exist. | | PermissionError | When the file cannot be opened due to insufficient permissions. | | Exception | Any unexpected error during file I/O or Part construction. | Side Effects: - Reads the specified file from local disk. - Prints "Upload completed. File Name: " to stdout. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCSH llm = AIEditLLMGCSH() part = llm.upload_file_text("/data/1001/SCENE_3A/merged_shot_scores.txt") # Use 'part' in a Gemini chat request llm.create_chat(aiedit_temperature=0.3, aiedit_budget=16384) resp = llm.send_message([part]) print(resp) Nrbr-)datar:r2)openreadr from_bytesr6)r#r8r9typer:f blog_bytesr=s r$upload_file_textzAIEditLLMGCSH.upload_file_textGs^P )T " "aJ "OO|L  -i[9:  " "s AAc< d}tjj|}d|i}j r|j jdk7rwt d tjd |jjj|j }|j s]|j jdk7rwt d |j|S#t$rt dddYyt $r}t d|dYd}~yd}~wwxYw) u Upload a local file to Gemini Files and wait until it becomes ACTIVE. Detailed Description: Attempts to upload a file and then polls the Gemini Files API until the uploaded file’s state transitions to ACTIVE. It prepares a config map with the file’s display_name derived from the local path. If an exception occurs during upload or polling, it prints an error and returns None. This helper is intended for synchronous workflows that require a ready-to-use file handle before proceeding. Note: The current implementation assumes a prior upload step populates `my_file`; callers should ensure the initial upload is performed and returns a valid file object. Parameters (Args): | Parameter | Type | Description | | :--------- | :--- | :-------------------------------------------------------------------------- | | file_name | str | Path to the local file intended for upload. Used to derive display_name. | | force | bool | Reserved flag for future cache-bypass behavior. Currently unused. | | type | Any | Reserved metadata parameter for future behavior. Currently unused. | Returns: | Type | Description | | :----- | :-------------------------------------------------------------------------- | | object | The ACTIVE Gemini File object (with attributes like name and display_name). | | None | Returned on failure or when exceptions are caught. | Raises: | Exception Type | Condition | | :---------------- | :----------------------------------------------------- | | FileNotFoundError | When the local file path does not exist. | | Exception | Any unexpected error during polling or client access. | Side Effects: - Prints error and progress messages to stdout. - Performs network I/O to poll file status via client.files.get. - Blocks execution using time.sleep while waiting for ACTIVE state. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCSH llm = AIEditLLMGCSH() llm.create_chat(aiedit_temperature=0.3, aiedit_budget=16384) # Upload a file using your own initial step to obtain my_file (not shown here), # then finalize with ACTIVE polling using upload_filex result = llm.upload_filex("/data/1001/SCENE_3A/videos/A001C002_240115_R2A3.mov") if result: print("File is ACTIVE:", result.display_name) else: print("Upload or polling failed") N display_namezError: file not found at 'z'.TrRz&An error occurred during File upload: ACTIVEz2File is processing, checking again in 5 seconds...)namer2)rrbasenameFileNotFoundErrorr6rVstaterttimesleeprfilesgetrq) r#r8r9rlr:rq config_mapmy_filerYs r$ upload_filexzAIEditLLMGCSH.upload_filexxsZ I77++I6L(,7J--7==#5#5#A F G JJqMkk''+++>G--7==#5#5#A -g.B.B-CDE!  .wir:$ G   :1#>d K  s%CD9DDDc |jj|||}|S#t$r}td|d|dYd}~yd}~wwxYw)u Upload a local file to Google Cloud Storage and return a Gemini Part from its GCS URI. Detailed Description: Resolves or uploads the provided file to GCS using [`GCS.get_gcs_uri_part`](ai_settings/ai_gen_gcs_ext.py) and constructs a Gemini [`Part`](ai_settings/ai_gen_gcs_ext.py) with the correct MIME type. When force=False, existing cached entries in the DB (TblGCSFileManager) may be reused to avoid redundant uploads; when force=True, the file is re-uploaded and the cached mapping updated. Returns the constructed Part on success. Errors are printed and re-raised to allow upstream retry/handling in callers like [`AIEditLLMGCS.upload_file`](ai_settings/ai_gen_gcs_ext.py). Parameters (Args): | Parameter | Type | Description | | :-------- | :--- | :---------- | | file_name | str | Absolute path under BASE_DIR/data or a project-relative path to the local file. | | mime_type | str | Explicit MIME type (e.g., "video/mp4", "application/pdf"). Defaults to "video/mp4". | | force | bool | If True, bypasses cache and forces re-upload; otherwise reuses cached GCS URI when available. | Returns: | Type | Description | | :----- | :-------------------------------------------------------------------------- | | object | A Gemini [`Part`](ai_settings/ai_gen_gcs_ext.py) created from the file’s GCS URI via Part.from_uri. | Raises: | Exception Type | Condition | | :------------- | :------------------------------------------------------------------------ | | Exception | Any error encountered during GCS URI resolution or upload is re-raised after logging. | Side Effects: - Performs network I/O to Google Cloud Storage via [`GCS.get_gcs_uri_part`](ai_settings/ai_gen_gcs_ext.py). - May update the local-to-GCS URI cache in TblGCSFileManager. Usage Example: from ai_settings.ai_gen_gcs_ext import AIEditLLMGCS llm = AIEditLLMGCS() part = llm.upload_gcs_files( file_name="/data/1001/SCENE_3A/20240115_153045/videos/A001C002_240115_R2A3.mov", mime_type="video/quicktime", force=False ) print("Part ready:", part) rPrQTrRNrTrWs r$r5zAIEditLLMGCSH.upload_gcs_filessVN Y999iQVXJ  Y 9)CsKSW X X Ys! AAArZr\)r]r^r_r%r>ror~r5r`r&r$rbrbs  B0d/b>@+Yr&rbceZdZdZej ZeZejjejdejZ eZ dZdZdZdZd dZd dZd Zd Zd d Zy)r!Nrrhc|tjj|j|_t t jj|_ tjjtjd|_tjj|jd|_y)a# Initialize the GCS helper with service account credentials and default paths. Detailed Description: Creates a Google Cloud Storage client from a service account JSON, and initializes path variables used throughout uploads: - BASE_DIR: Module directory of ai_gen_gcs_ext.py - MEDIA_ROOT: Django BASE_DIR/data - LOCAL_VIDEO_FOLDER: MEDIA_ROOT/videos This setup enables methods like get_uri_by_file_name and upload_to_gcs to resolve local file locations and construct GCS destinations consistently. Parameters (Args): - None Returns: - None Raises: - FileNotFoundError: If the JSON_API credential file path is invalid. - Exception: If storage.Client initialization fails due to credential or environment issues. Side Effects: - Reads service account credentials from disk. - Initializes a GCS client bound to the configured project. Usage Example: from ai_settings.ai_gen_gcs_ext import GCS gcs = GCS() print(gcs.MEDIA_ROOT) # e.g., "/path/to/project/data" print(gcs.LOCAL_VIDEO_FOLDER) # e.g., "/path/to/project/data/videos" rhvideosN)rrfrom_service_account_jsonJSON_APIstorage_clientr__file__resolveparentrrrrr MEDIA_ROOTLOCAL_VIDEO_FOLDERrds r$r%z GCS.__init__sq@&nnFFt}}UX..077 '',,x'8'8&A"$'',,t"Ir&c||_y)a Set the local video directory used for GCS uploads and path resolution. Detailed Description: Updates the [`GCS.LOCAL_VIDEO_FOLDER`](ai_settings/ai_gen_gcs_ext.py) path at runtime to point to a custom directory containing local media files. This is useful when the default path (BASE_DIR/data/videos) needs to be overridden for batch uploads or environment-specific locations. The new directory is used by methods that construct local file paths before uploading to Google Cloud Storage. Parameters (Args): | Parameter | Type | Description | | :-------- | :--- | :---------- | | dir_path | str | Absolute path to the local video directory to set (e.g., "/data/1001/video/footage"). | Returns: | Type | Description | | :--- | :---------- | | None | No return value; updates internal state only. | Raises: | Exception Type | Condition | | :------------- | :-------- | | ValueError | If dir_path is empty or not a string. | | FileNotFoundError | If dir_path does not exist on the filesystem. | Side Effects: - Mutates `self.LOCAL_VIDEO_FOLDER` to the provided directory path. - Subsequent uploads and path lookups use the new directory. Usage Example: from ai_settings.ai_gen_gcs_ext import GCS gcs = GCS() # Override the default local video folder gcs.set_local_dir("/data/1001/video/footage") # Upload a specific file using the new base directory part = gcs.get_gcs_uri_part("/data/1001/video/footage/A001C002_240115_R2A3.mov", "video/quicktime") print(part) N)r)r#dir_paths r$ set_local_dirzGCS.set_local_dirs L#+r&ctjjtjd}tjj |}tjj |}tjj ||g|k(r"tjj||}|Std|)a Resolve a path under Django BASE_DIR/data to a project-relative path. Detailed Description: Normalizes the provided absolute file path and the Django data root (BASE_DIR/data), then verifies that the file resides within this root using os.path.commonpath. If valid, returns the relative path from BASE_DIR/data to the file (suitable for GCS keying, e.g., "1001/SCENE_3A/videos/A001C002_240115_R2A3.mov"). If the file is outside the expected data root, prints an error and raises a generic exception. Parameters (Args): | Parameter | Type | Description | | :------------- | :--- | :------------------------------------------------------------------------------ | | full_file_name | str | Absolute path to a file that should be located under BASE_DIR/data. | Returns: | Type | Description | | :--- | :----------------------------------------------------------------------------------------------------------- | | str | Relative path from BASE_DIR/data to the file (using OS-specific separators normalized by os.path.relpath). | Raises: | Exception Type | Condition | | :------------- | :--------------------------------------------------------------------------------------------- | | Exception | Raised when full_file_name is not under BASE_DIR/data (after normalization and common path check). | Side Effects: - Prints an error message to stdout if the file is not under the expected BASE_DIR/data root. Usage Example: from ai_settings.ai_gen_gcs_ext import GCS from django.conf import settings import os gcs = GCS() abs_path = os.path.join(settings.BASE_DIR, "data", "1001", "SCENE_3A", "videos", "A001C002_240115_R2A3.mov") rel = gcs.get_relative_path(abs_path) print(rel) # e.g., "1001/SCENE_3A/videos/A001C002_240115_R2A3.mov" rhz4Error: The file does't have the relative path 'data') rrrr rnormpath commonpathrelpathr6)r#full_file_name GCS_DATA_ROOT rel_file_names r$get_relative_pathzGCS.get_relative_path>sF X%6%6? )).9((7 77  ~}= >- OGGOONMJM  H. Y r&cX|j||}tj||}|S)u$ Create a Gemini Part from a file’s Google Cloud Storage URI. Detailed Description: Resolves the GCS URI for the provided local file path using [`GCS.get_uri_by_file_name`](ai_settings/ai_gen_gcs_ext.py) and constructs a [`Part`](ai_settings/ai_gen_gcs_ext.py) with the specified MIME type via Part.from_uri. When force=True, bypasses cached mappings and re-uploads the file to GCS before building the Part. This helper centralizes the “local path → GCS URI → Part” flow for downstream Gemini requests. Parameters (Args): | Parameter | Type | Description | | :-------- | :--- | :---------- | | file_name | str | Absolute path under BASE_DIR/data to the local file whose GCS URI should be resolved. | | mime_type | str | MIME type to associate with the Part (e.g., "video/mp4", "application/pdf", "text/plain"). | | force | bool | If True, forces re-upload to GCS instead of using cached URI; defaults to False. | Returns: | Type | Description | | :----- | :---------- | | object | A Gemini [`Part`](ai_settings/ai_gen_gcs_ext.py) constructed from the file’s GCS URI and the provided MIME type. | Raises: | Exception Type | Condition | | :------------- | :-------- | | Exception | Propagated if URI resolution or Part construction fails (e.g., missing file, invalid MIME type, GCS access issues). | Side Effects: - May perform network I/O to upload the file to GCS (when force=True or no cache entry exists). - Reads/updates local-to-GCS URI cache via [`GCS.get_uri_by_file_name`](ai_settings/ai_gen_gcs_ext.py). Usage Example: from ai_settings.ai_gen_gcs_ext import GCS gcs = GCS() part = gcs.get_gcs_uri_part( file_name="/data/1001/SCENE_3A/videos/A001C002_240115_R2A3.mov", mime_type="video/quicktime", force=False ) # Use 'part' in a Gemini request )file_urir:)get_uri_by_file_namerfrom_uri)r#r8r:r9 gcs_video_urirXs r$rUzGCS.get_gcs_uri_partns5P11)UC ]]" r&cNd}d}|r|j|}tjj|j }|r>t |}|j }|s%|r#|jd}|r|}td||std||j|}|S)a Resolve or upload a local file to GCS and return its GCS URI, with optional cache bypass. Detailed Description: Converts an absolute path under BASE_DIR/data to a project-relative path using [`GCS.get_relative_path`](ai_settings/ai_gen_gcs_ext.py), then checks the cache table TblGCSFileManager for an existing GCS URI. On a cache hit and when is_force=False, returns the cached URI immediately. Otherwise, uploads the file to Google Cloud Storage via [`GCS.upload_to_gcs`](ai_settings/ai_gen_gcs_ext.py), persists the mapping in TblGCSFileManager, and returns the newly created URI. Prints diagnostic messages indicating cache hits and upload initiation. Parameters (Args): | Parameter | Type | Description | | :--------- | :--- | :-------------------------------------------------------------------------- | | file_name | str | Absolute path under BASE_DIR/data for the local file to resolve/upload. | | is_force | bool | If True, bypasses cache and forces upload to GCS; defaults to False. | Returns: | Type | Description | | :---- | :--------------------------------------------------------- | | str | GCS URI string (e.g., "gs://bucket/data/.../file.mov"). | | None | Returned if file_name is falsy or an unrecoverable error occurs. | Raises: | Exception Type | Condition | | :------------- | :------------------------------------------------------------------------ | | Exception | Propagated if path normalization fails or upload_to_gcs raises an error. | Side Effects: - Queries TblGCSFileManager for cached URI entries. - Prints status messages (cache hit, upload start). - Performs network I/O on cache miss to upload the file to GCS. - Persists/updates cache mapping in TblGCSFileManager after upload. Usage Example: from ai_settings.ai_gen_gcs_ext import GCS gcs = GCS() # Try to reuse cached URI uri = gcs.get_uri_by_file_name("/data/1001/SCENE_3A/videos/A001C002_240115_R2A3.mov", is_force=False) print(uri) # Force re-upload and refresh cache uri_forced = gcs.get_uri_by_file_name("/data/1001/SCENE_3A/videos/A001C002_240115_R2A3.mov", is_force=True) print(uri_forced) N)key_name value_namezGCS File exists File Name: zUploading File to GCS:) rTblGCSFileManagerobjectsfilterfirstTblGCSFileManagerSerializerrhr{r6 upload_to_gcs) r#r8is_forcerr}gcs_file_valueserializer_gcs_valuegcs_data gcs_file_names r$rzGCS.get_uri_by_file_namesT   229=M/66==}=U[[]N'B>'R$/44H$,LL$>M$"/ ;G9EF. >,,];r&c|j}|j}|jjtj }t jj||}|jdd}|d|}dtj d|}td|d|dtdtjj|d|i |S) a Upload a local project file to Google Cloud Storage and persist its GCS URI mapping. Detailed Description: Constructs a source file path under Django BASE_DIR/data, normalizes the relative Windows-style path to Unix separators, and builds the destination blob path under the configured GCS destination folder (e.g., "data/"). It then logs the upload intent and (in a production setup) would upload the file to the configured bucket. After upload, it persists or updates the local-to-GCS URI mapping in TblGCSFileManager using the relative key (lfile_name) and returns the finalized GCS URI string. This function centralizes path normalization and cache persistence for consistent downstream lookups via [`GCS.get_uri_by_file_name`](ai_settings/ai_gen_gcs_ext.py). Parameters (Args): | Parameter | Type | Description | | :---------- | :--- | :---------------------------------------------------------------------------------------------- | | lfile_name | str | Relative path under BASE_DIR/data to the file being uploaded (e.g., "1001/SCENE_3A/videos/A.mov"). | Returns: | Type | Description | | :--- | :----------------------------------------------------------------------------- | | str | The GCS URI of the uploaded file (e.g., "gs:///data/1001/.../A.mov"). | Raises: | Exception Type | Condition | | :------------- | :------------------------------------------------------------------------ | | FileNotFoundError | If the constructed source_file_path does not exist on disk. | | Exception | Propagated for storage client or DB update errors (e.g., permission issues). | Side Effects: - Performs or simulates a file upload to Google Cloud Storage. - Writes/updates a URI mapping row in TblGCSFileManager with key_name=lfile_name. Usage Example: from ai_settings.ai_gen_gcs_ext import GCS gcs = GCS() rel_path = "1001/SCENE_3A/videos/A001C002_240115_R2A3.mov" uri = gcs.upload_to_gcs(rel_path) print("Uploaded URI:", uri) # gs:///data/1001/SCENE_3A/videos/A001C002_240115_R2A3.mov \/zgs:// Uploading z to z...zUpload complete.r)rdefaults)rGCS_DESTINATION_FOLDERrbucketr!MY_BUCKET_NAMErrrreplacer6rrupdate_or_create) r# lfile_name source_folderdestination_folderrsource_file_pathrdestination_blob_name uploaded_uris r$rzGCS.upload_to_gcssFoo !88$$++C,>,>?77<< zB"**45 #5"6a  Gs112!4I3JK  +,DcBC  !!!22"L1 3 r&c |jjtj}j|}t j |Dcgc]_}t jjt jj||r t jj||a}}tdt|d|dtjdtjj||d|d}|D]!}t|t std|#td y cc}w) u Upload a local directory of files to a Google Cloud Storage bucket in parallel. Detailed Description: Scans a local directory for regular files and uploads them concurrently to the configured GCS bucket (`GCS.MY_BUCKET_NAME`) using Google Cloud Storage’s transfer manager. Each uploaded object is placed under the provided `gcs_folder_prefix` as a blob name prefix. The function prints progress (file count, start/end) and iterates over the transfer results to report any exceptions. This is intended for fast bulk uploads of footage or artifacts where parallelism significantly reduces total time. It does not update TblGCSFileManager cache mappings—use [`GCS.upload_to_gcs`](ai_settings/ai_gen_gcs_ext.py) or [`GCS.get_uri_by_file_name`](ai_settings/ai_gen_gcs_ext.py) for per-file cache persistence. Parameters (Args): | Parameter | Type | Description | | :------------------- | :--- | :-------------------------------------------------------------------------- | | local_directory_path | str | Absolute path to the local directory to scan and upload (files only). | | gcs_folder_prefix | str | Prefix to prepend to GCS blob names (e.g., "data/1001/SCENE_3A/videos/"). | Returns: | Type | Description | | :--- | :----------------------------------------------------------- | | None | No return value; prints status and exception messages only. | Raises: | Exception Type | Condition | | :------------- | :------------------------------------------------------------------------------------------ | | FileNotFoundError | If `local_directory_path` does not exist or is not accessible. | | Exception | Propagated from Google Cloud Storage transfer manager if upload initialization fails. | Side Effects: - Performs network I/O to upload files to Google Cloud Storage. - Prints diagnostic messages for progress and failed uploads. - Creates objects in the configured GCS bucket under `gcs_folder_prefix`. Usage Example: from ai_settings.ai_gen_gcs_ext import GCS gcs = GCS() # Upload all files in a local footage folder to GCS under the given prefix local_dir = "/data/1001/SCENE_3A/20240115_153045/videos" prefix = "data/1001/SCENE_3A/20240115_153045/videos/" gcs.upload_directory_to_gcs(local_dir, prefix) rz files to 'z ' in bucket 'z'...T)skip_if_existsblob_name_prefix max_workerszFailed to upload: z All files uploaded successfully.N)rrr!rTransferManagerrlistdirrisfilerr6lenrrupload_many_files isinstancerV) r#local_directory_pathgcs_folder_prefixrrrmlocal_file_pathsresultsresults r$upload_directory_to_gcszGCS.upload_directory_to_gcss)L$$++C,>,>?<+;;FC ZZ 45 ww~~bggll+?CD GGLL-q 1   3/01=N Upload multiple files from a local directory to Google Cloud Storage concurrently. Detailed Description: Scans the provided source_directory recursively, builds a list of local file paths with corresponding GCS blob names (preserving subdirectory structure), and uploads them in parallel using Google Cloud Storage’s transfer manager. The gcs_destination_prefix is prepended to each blob name, allowing uploads under a common folder/prefix (e.g., "data/1001/SCENE_3A/videos/"). Prints progress, concurrency details, and result summaries for each file. Returns None; intended for bulk uploads where parallelism improves throughput. Parameters (Args): | Parameter | Type | Description | | :------------------- | :--- | :--------------------------------------------------------------------------------------- | | source_directory | str | Absolute path to the local directory to walk recursively for file uploads. | | gcs_destination_prefix | str | Optional GCS blob name prefix to prepend (e.g., "data/1001/SCENE_3A/videos/"). Defaults to empty string. | Returns: | Type | Description | | :--- | :---------------------------------------------------------- | | None | No return value; prints status and per-file upload results. | Raises: | Exception Type | Condition | | :------------- | :------------------------------------------------------------------------ | | Exception | Propagated from transfer manager if any upload futures resolve to errors. | Side Effects: - Performs network I/O to upload files to the configured GCS bucket. - Prints diagnostic messages (scanned files, concurrency settings, success/failure per file). Usage Example: from ai_settings.ai_gen_gcs_ext import GCS gcs = GCS() src_dir = "/data/1001/SCENE_3A/20240115_153045/videos" prefix = "data/1001/SCENE_3A/20240115_153045/videos/" gcs.upload_multiple_files_to_gcs(src_dir, prefix) zScanning local directory: z relative pathrrz Will upload 'z ' to 'gs://'zNo files found in 'z ' to upload.Nz Starting concurrent upload of z files using transfer_manager...zUsing default concurrency: z workers. )r filenamesrz Processing upload results:u ❌ Failed: u → z ()u✅ Uploaded: u → gs://u🎉 Upload completed.z Upload process completed.)rrr!rr6rwalkrrrrappendrrrDEFAULT_MAX_WORKERSdirupload_many_from_filenameszipindexsplitrrV)r#source_directorygcs_destination_prefixrr blob_namesroot_rzfilenamelocal_filepath relative_path gcs_blob_namerr blob_names r$upload_multiple_files_to_gcsz GCS.upload_multiple_files_to_gcs]seF$$++C,>,>?  *+;*<=> gg&67 jND!U! j!#dH!= !#@P Q 'HXY " -C] S [ [\`be f  ''7!!-0'7{3CUCUBVVWXeWffghi j j '(8'9F G  05E1F0GGghi +G,D,D,X,X+YYbcd +C0H0H,I+JKL**EE&F  ,g6 #$4g > ] HfHR #3#9#9(#CDX`XfXfgjXklnXoI&), XJeI;bJKxj 3;M;M:NaPY{[\  ] &' +,r&)F))r]r^r_rr SECRET_BUCKET_NAMErrrrrrrrrr%rrrUrrrrr`r&r$r!r!s| H00NJww||=(*I*IH"##JJ&+P.`/b>@5nD2NT-r&r!)googlerrxrgoogle.genai.typesr google.cloudrgoogle.cloud.storagerpathlibrai_genr django.confr google.oauth2r r rbr!r`r&r$rsQ # 1 *b9bJpYIpYdL-L-r&