diff --git a/README.md b/README.md index 7f19f5ff3..3f5dcb762 100644 --- a/README.md +++ b/README.md @@ -14,10 +14,111 @@ The application utilizes **Azure Cosmos DB** for storing conversations, metadata ## Quick Deploy -Use azd up [MORE DETAILS TO COME] +[Detailed deployment Guide](./deployers/bicep/README.md) +### Pre-Configuration: + +The following procedure must be completed with a user that has permissions to create an application registration in the users Entra tenant. + +#### Create the application registration: + +```powershell +cd ./deployers +``` + +Define your application name and your environment: + +``` +appName = +``` + +``` +environment = +``` + +The following script will create an Entra Enterprise Application, with an App Registration named *\*-*\*-ar for the web service called *\*-*\*-app. + +> [!TIP] +> +> The web service name may be overriden with the `-AppServceName` parameter. + +> [!TIP] +> +> A different expiration date for the secret which defaults to 180 days with the `-SecretExpirationDays` parameter. + +```powershell +.\Initialize-EntraApplication.ps1 -AppName "" -Environment "" -AppRolesJsonPath "./azurecli/appRegistrationRoles.json" ``` -azd up + +> [!NOTE] +> +> Be sure to save this information as it will not be available after the window is closed.* + +```======================================== +App Registration Created Successfully! +Application Name: +Client ID: +Tenant ID: +Service Principal ID: +Client Secret: +Secret Expiration: +``` + +In addition, the script will note additional steps that must be taken for the app registration step to be completed. + +1. Grant Admin Consent for API Permissions: + + - Navigate to Azure Portal > Entra ID > App registrations + - Find app: *\* + - Go to API permissions + - Click 'Grant admin consent for [Tenant]' + +2. Assign Users/Groups to Enterprise Application: + - Navigate to Azure Portal > Entra ID > Enterprise applications + - Find app: *\* + - Go to Users and groups + - Add user/group assignments with appropriate app roles + +3. Store the Client Secret Securely: + - Save the client secret in Azure Key Vault or secure credential store + - The secret value is shown above and will not be displayed again + +#### Configure AZD Environment + +Using the bash terminal in Visual Studio Code + +```powershell +cd ./deployers +``` + +If you work with other Azure clouds, you may need to update your cloud like `azd config set cloud.name AzureUSGovernment` - more information here - [Use Azure Developer CLI in sovereign clouds | Microsoft Learn](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/sovereign-clouds) + +```powershell +azd config set cloud.name AzureCloud +``` + +This will open a browser window that the user with Owner level permissions to the target subscription will need to authenticate with. + +```powershell +azd auth login +``` + +Use the same value for the \ that was used in the application registration. + +```powershell +azd env new +``` + +Select the new environment + +```powershell +azd env select +``` + +This step will begin the deployment process. + +```powershell +Use azd up ``` ## Architecture @@ -27,50 +128,27 @@ azd up ## Features - **Chat with AI**: Interact with an AI model based on Azure OpenAI’s GPT and Thinking models. - - **RAG with Hybrid Search**: Upload documents and perform hybrid searches (vector + keyword), retrieving relevant information from your files to augment AI responses. - - **Document Management**: Upload, store, and manage multiple versions of documents—personal ("Your Workspace") or group-level ("Group Workspaces"). - - **Group Management**: Create and join groups to share access to group-specific documents, enabling collaboration with Role-Based Access Control (RBAC). - - **Ephemeral (Single-Convo) Documents**: Upload temporary documents available only during the current chat session, without persistent storage in Azure AI Search. - - **Conversation Archiving (Optional)**: Retain copies of user conversations—even after deletion from the UI—in a dedicated Cosmos DB container for audit, compliance, or legal requirements. - - **Content Safety (Optional)**: Integrate Azure AI Content Safety to review every user message *before* it reaches AI models, search indexes, or image generation services. Enforce custom filters and compliance policies, with an optional `SafetyAdmin` role for viewing violations. - - **Feedback System (Optional)**: Allow users to rate AI responses (thumbs up/down) and provide contextual comments on negative feedback. Includes user and admin dashboards, governed by an optional `FeedbackAdmin` role. - - **Bing Web Search (Optional)**: Augment AI responses with live Bing search results, providing up-to-date information. Configurable via Admin Settings. - - **Image Generation (Optional)**: Enable on-demand image creation using Azure OpenAI's DALL-E models, controlled via Admin Settings. - - **Video Extraction (Optional)**: Utilize Azure Video Indexer to transcribe speech and perform Optical Character Recognition (OCR) on video frames. Segments are timestamp-chunked for precise retrieval and enhanced citations linking back to the video timecode. - - **Audio Extraction (Optional)**: Leverage Azure Speech Service to transcribe audio files into timestamped text chunks, making audio content searchable and enabling enhanced citations linked to audio timecodes. - - **Document Classification (Optional)**: Admins define custom classification types and associated colors. Users tag uploaded documents with these labels, which flow through to AI conversations, providing lineage and insight into data sensitivity or type. - - **Enhanced Citation (Optional)**: Store processed, chunked files in Azure Storage (organized into user- and document-scoped folders). Display interactive citations in the UI—showing page numbers or timestamps—that link directly to the source document preview. - - **Metadata Extraction (Optional)**: Apply an AI model (configurable GPT model via Admin Settings) to automatically generate keywords, two-sentence summaries, and infer author/date for uploaded documents. Allows manual override for richer search context. - - **File Processing Logs (Optional)**: Enable verbose logging for all ingestion pipelines (workspaces and ephemeral chat uploads) to aid in debugging, monitoring, and auditing file processing steps. - - **Redis Cache (Optional)**: Integrate Azure Cache for Redis to provide a distributed, high-performance session store. This enables true horizontal scaling and high availability by decoupling user sessions from individual app instances. - - **Authentication & RBAC**: Secure access via Azure Active Directory (Entra ID) using MSAL. Supports Managed Identities for Azure service authentication, group-based controls, and custom application roles (`Admin`, `User`, `CreateGroup`, `SafetyAdmin`, `FeedbackAdmin`). - - **Supported File Types**: - - Text: `txt`, `md`, `html`, `json` - - * Documents: `pdf`, `docx`, `pptx`, `xlsx`, `xlsm`, `xls`, `csv` - * Images: `jpg`, `jpeg`, `png`, `bmp`, `tiff`, `tif`, `heif` - * Video: `mp4`, `mov`, `avi`, `wmv`, `mkv`, `webm` - * Audio: `mp3`, `wav`, `ogg`, `aac`, `flac`, `m4a` - -## Demos - -ADD DEMOS HERE \ No newline at end of file + - **Text**: `txt`, `md`, `html`, `json`, `xml`, `yaml`, `yml`, `log` + - **Documents**: `pdf`, `doc`, `docm`, `docx`, `pptx`, `xlsx`, `xlsm`, `xls`, `csv` + - **Images**: `jpg`, `jpeg`, `png`, `bmp`, `tiff`, `tif`, `heif` + - **Video**: `mp4`, `mov`, `avi`, `wmv`, `mkv`, `flv`, `mxf`, `gxf`, `ts`, `ps`, `3gp`, `3gpp`, `mpg`, `asf`, `m4v`, `isma`, `ismv`, `dvr-ms` + - **Audio**: `wav`, `m4a` \ No newline at end of file diff --git a/application/single_app/functions_documents.py b/application/single_app/functions_documents.py index e48c01310..017b819fa 100644 --- a/application/single_app/functions_documents.py +++ b/application/single_app/functions_documents.py @@ -3087,13 +3087,8 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): 'analysis': 'detailed analysis' } or None if vision analysis is disabled or fails """ -<<<<<<< HEAD debug_print(f"[VISION_ANALYSIS_V2] Function entry - document_id: {document_id}, user_id: {user_id}") -======= - if not settings.get('enable_multimodal_vision', False): - return None ->>>>>>> origin/main try: # Convert image to base64 @@ -3101,7 +3096,6 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): image_bytes = img_file.read() base64_image = base64.b64encode(image_bytes).decode('utf-8') -<<<<<<< HEAD image_size = len(image_bytes) base64_size = len(base64_image) debug_print(f"[VISION_ANALYSIS] Image conversion for {document_id}:") @@ -3116,13 +3110,6 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): # Get vision model settings vision_model = settings.get('multimodal_vision_model', 'gpt-4o') debug_print(f"[VISION_ANALYSIS] Vision model selected: {vision_model}") -======= - # Determine image mime type - mime_type = mimetypes.guess_type(image_path)[0] or 'image/jpeg' - - # Get vision model settings - vision_model = settings.get('multimodal_vision_model', 'gpt-4o') ->>>>>>> origin/main if not vision_model: print(f"Warning: Multi-modal vision enabled but no model selected") @@ -3130,7 +3117,6 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): # Initialize client (reuse GPT configuration) enable_gpt_apim = settings.get('enable_gpt_apim', False) -<<<<<<< HEAD debug_print(f"[VISION_ANALYSIS] Using APIM: {enable_gpt_apim}") if enable_gpt_apim: @@ -3143,19 +3129,11 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): gpt_client = AzureOpenAI( api_version=api_version, azure_endpoint=endpoint, -======= - - if enable_gpt_apim: - gpt_client = AzureOpenAI( - api_version=settings.get('azure_apim_gpt_api_version'), - azure_endpoint=settings.get('azure_apim_gpt_endpoint'), ->>>>>>> origin/main api_key=settings.get('azure_apim_gpt_subscription_key') ) else: # Use managed identity or key auth_type = settings.get('azure_openai_gpt_authentication_type', 'key') -<<<<<<< HEAD api_version = settings.get('azure_openai_gpt_api_version') endpoint = settings.get('azure_openai_gpt_endpoint') @@ -3164,39 +3142,26 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): debug_print(f" API Version: {api_version}") debug_print(f" Auth Type: {auth_type}") -======= ->>>>>>> origin/main if auth_type == 'managed_identity': token_provider = get_bearer_token_provider( DefaultAzureCredential(), cognitive_services_scope ) gpt_client = AzureOpenAI( -<<<<<<< HEAD api_version=api_version, azure_endpoint=endpoint, -======= - api_version=settings.get('azure_openai_gpt_api_version'), - azure_endpoint=settings.get('azure_openai_gpt_endpoint'), ->>>>>>> origin/main azure_ad_token_provider=token_provider ) else: gpt_client = AzureOpenAI( -<<<<<<< HEAD api_version=api_version, azure_endpoint=endpoint, -======= - api_version=settings.get('azure_openai_gpt_api_version'), - azure_endpoint=settings.get('azure_openai_gpt_endpoint'), ->>>>>>> origin/main api_key=settings.get('azure_openai_gpt_key') ) # Create vision prompt print(f"Analyzing image with vision model: {vision_model}") -<<<<<<< HEAD # Determine which token parameter to use based on model type # o-series and gpt-5 models require max_completion_tokens instead of max_tokens vision_model_lower = vision_model.lower() @@ -3222,17 +3187,6 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): Ensure your entire response is valid JSON. Include all four keys even if some are empty strings or empty arrays.""" else: prompt_text = """Analyze this image and provide: -======= - response = gpt_client.chat.completions.create( - model=vision_model, - messages=[ - { - "role": "user", - "content": [ - { - "type": "text", - "text": """Analyze this image and provide: ->>>>>>> origin/main 1. A detailed description of what you see 2. List any objects, people, or notable elements 3. Extract any visible text (OCR) @@ -3245,7 +3199,6 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): "text": "...", "analysis": "..." }""" -<<<<<<< HEAD api_params = { "model": vision_model, @@ -3256,8 +3209,6 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): { "type": "text", "text": prompt_text -======= ->>>>>>> origin/main }, { "type": "image_url", @@ -3267,7 +3218,6 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): } ] } -<<<<<<< HEAD ] } @@ -3305,16 +3255,10 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): # Check finish reason if hasattr(response.choices[0], 'finish_reason'): debug_print(f" Finish reason: {response.choices[0].finish_reason}") -======= - ], - max_tokens=1000 - ) ->>>>>>> origin/main # Parse response content = response.choices[0].message.content -<<<<<<< HEAD # Handle None content if content is None: print(f"[VISION_ANALYSIS_V2] ⚠️ Response content is None!") @@ -3344,14 +3288,10 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): has_code_fence = '```' in content debug_print(f" Starts with JSON bracket: {is_json_like}") debug_print(f" Contains code fence: {has_code_fence}") -======= - debug_print(f"[VISION_ANALYSIS] Raw response for {document_id}: {content[:500]}...") ->>>>>>> origin/main # Try to parse as JSON, fallback to raw text try: # Clean up potential markdown code fences -<<<<<<< HEAD debug_print(f"[VISION_ANALYSIS] Attempting to clean JSON code fences...") content_cleaned = clean_json_codeFence(content) debug_print(f" Cleaned length: {len(content_cleaned)} characters") @@ -3376,23 +3316,10 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): 'parse_failed': True } debug_print(f"[VISION_ANALYSIS] Created fallback structure with raw response") -======= - content_cleaned = clean_json_codeFence(content) - vision_analysis = json.loads(content_cleaned) - debug_print(f"[VISION_ANALYSIS] Parsed JSON successfully for {document_id}") - except Exception as parse_error: - debug_print(f"[VISION_ANALYSIS] Vision response not valid JSON: {parse_error}") - print(f"Vision response not valid JSON, using raw text") - vision_analysis = { - 'description': content, - 'raw_response': content - } ->>>>>>> origin/main # Add model info to analysis vision_analysis['model'] = vision_model -<<<<<<< HEAD debug_print(f"[VISION_ANALYSIS] Final analysis structure for {document_id}:") debug_print(f" Model: {vision_model}") debug_print(f" Has 'description': {'description' in vision_analysis}") @@ -3414,13 +3341,6 @@ def analyze_image_with_vision_model(image_path, user_id, document_id, settings): txt = vision_analysis['text'] debug_print(f" Text length: {len(txt) if txt else 0} chars") debug_print(f" Text preview: {txt[:100] if txt else 'None'}...") -======= - debug_print(f"[VISION_ANALYSIS] Complete analysis for {document_id}:") - debug_print(f" Model: {vision_model}") - debug_print(f" Description: {vision_analysis.get('description', 'N/A')[:200]}...") - debug_print(f" Objects: {vision_analysis.get('objects', [])}") - debug_print(f" Text: {vision_analysis.get('text', 'N/A')[:100]}...") ->>>>>>> origin/main print(f"Vision analysis completed for document: {document_id}") return vision_analysis @@ -5195,79 +5115,10 @@ def process_di_document(document_id, user_id, temp_file_path, original_filename, # Don't fail the whole proc, total_embedding_tokens, embedding_model_nameess, just update status update_callback(status=f"Processing complete (metadata extraction warning)") -<<<<<<< HEAD # Note: Vision analysis now happens BEFORE save_chunks (moved earlier in the flow) # This ensures vision_analysis is available in metadata when chunks are being saved return total_final_chunks_processed, total_embedding_tokens, embedding_model_name -======= - # --- Multi-Modal Vision Analysis (for images only) --- - if is_image and enable_enhanced_citations: - enable_multimodal_vision = settings.get('enable_multimodal_vision', False) - if enable_multimodal_vision: - try: - update_callback(status="Performing AI vision analysis...") - - vision_analysis = analyze_image_with_vision_model( - temp_file_path, - user_id, - document_id, - settings - ) - - if vision_analysis: - print(f"Vision analysis completed for image: {original_filename}") - - # Update document with vision analysis results - update_fields = { - 'vision_analysis': vision_analysis, - 'vision_description': vision_analysis.get('description', ''), - 'vision_objects': vision_analysis.get('objects', []), - 'vision_extracted_text': vision_analysis.get('text', ''), - 'status': "AI vision analysis completed" - } - update_callback(**update_fields) - - # Save vision analysis as separate blob for citations - vision_json_path = temp_file_path + '_vision.json' - try: - with open(vision_json_path, 'w', encoding='utf-8') as f: - json.dump(vision_analysis, f, indent=2) - - vision_blob_filename = f"{os.path.splitext(original_filename)[0]}_vision_analysis.json" - - upload_blob_args = { - "temp_file_path": vision_json_path, - "user_id": user_id, - "document_id": document_id, - "blob_filename": vision_blob_filename, - "update_callback": update_callback - } - - if is_public_workspace: - upload_blob_args["public_workspace_id"] = public_workspace_id - elif is_group: - upload_blob_args["group_id"] = group_id - - upload_to_blob(**upload_blob_args) - print(f"Vision analysis saved to blob storage: {vision_blob_filename}") - - finally: - if os.path.exists(vision_json_path): - os.remove(vision_json_path) - else: - print(f"Vision analysis returned no results for: {original_filename}") - update_callback(status="Vision analysis completed (no results)") - - except Exception as e: - print(f"Warning: Error in vision analysis for {document_id}: {str(e)}") - import traceback - traceback.print_exc() - # Don't fail the whole process, just update status - update_callback(status=f"Processing complete (vision analysis warning)") - - return total_final_chunks_processed ->>>>>>> origin/main def _get_content_type(path: str) -> str: ext = os.path.splitext(path)[1].lower() @@ -5572,7 +5423,6 @@ def update_doc_callback(**kwargs): args["group_id"] = group_id if file_ext == '.txt': -<<<<<<< HEAD result = process_txt(**{k: v for k, v in args.items() if k != "file_ext"}) # Handle tuple return (chunks, tokens, model_name) if isinstance(result, tuple) and len(result) == 3: @@ -5603,17 +5453,6 @@ def update_doc_callback(**kwargs): total_chunks_saved, total_embedding_tokens, embedding_model_name = result else: total_chunks_saved = result -======= - total_chunks_saved = process_txt(**{k: v for k, v in args.items() if k != "file_ext"}) - elif file_ext == '.xml': - total_chunks_saved = process_xml(**{k: v for k, v in args.items() if k != "file_ext"}) - elif file_ext in ('.yaml', '.yml'): - total_chunks_saved = process_yaml(**{k: v for k, v in args.items() if k != "file_ext"}) - elif file_ext == '.log': - total_chunks_saved = process_log(**{k: v for k, v in args.items() if k != "file_ext"}) - elif file_ext in ('.doc', '.docm'): - total_chunks_saved = process_doc(**{k: v for k, v in args.items() if k != "file_ext"}) ->>>>>>> origin/main elif file_ext == '.html': result = process_html(**{k: v for k, v in args.items() if k != "file_ext"}) if isinstance(result, tuple) and len(result) == 3: diff --git a/deployers/bicep/README.md b/deployers/bicep/README.md index 492d569ec..c2c51bf08 100644 --- a/deployers/bicep/README.md +++ b/deployers/bicep/README.md @@ -89,6 +89,8 @@ Using the bash terminal in Visual Studio Code `cd ./deployers` +`azd config set cloud.name AzureCloud` - If you work with other Azure clouds, you may need to update your cloud like `azd config set cloud.name AzureUSGovernment` - more information here - [Use Azure Developer CLI in sovereign clouds | Microsoft Learn](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/sovereign-clouds) + `azd auth login` - this will open a browser window that the user with Owner level permissions to the target subscription will need to authenticate with. `azd env new ` - Use the same value for the \ that was used in the application registration. @@ -177,7 +179,7 @@ User shoud now be able to fully use Simple Chat application. "selected": [], "all": [] }, - ``` + ``` with @@ -205,7 +207,7 @@ User shoud now be able to fully use Simple Chat application. "selected": [], "all": [] }, - ``` + ``` with @@ -220,7 +222,7 @@ User shoud now be able to fully use Simple Chat application. "modelName": "text-embedding-3-small" ] }, - ``` + ``` - Update settings in the Cosmos UI and click Save. - Refresh web page and you shound now be able to Test the GPT and Embedding models.