Skip to content

add a document table to persist all the uploaded document names and sizes#9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alvinsjoy merged 5 commits into from
Apr 28, 2026
Merged

add a document table to persist all the uploaded document names and sizes #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
eba8ee7
add a document table to keep all the uploaded document names and sizes
alvinsjoy Apr 28, 2026
e1fefe2
use bigint for file size and useeffect for document state management
alvinsjoy Apr 28, 2026
261b1ee
update the loading skeletons similar to changed topbar of chat-area
alvinsjoy Apr 28, 2026
8892ef6
use uuids for document operations instead of filenames
alvinsjoy Apr 28, 2026
267abff
fix optimistic ui state bugs and add collection existence check befor…
alvinsjoy Apr 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 19 additions & 5 deletions app/api/upload/route.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -94,12 +94,26 @@ export async function POST(request: NextRequest) {
);
}

// 5. Increment document count in Prisma
await prisma.chat.update({
where: { id: chatId },
// 5. Create a Document record in Prisma
// Use storage_key as the Document ID for direct lookup during deletion
const filename = result?.filename || "unknown";
const storageKey = result?.storage_key;
const fileSize = result?.file_size || 0;

if (!storageKey) {
console.error("Backend did not return a storage_key");
return NextResponse.json(
{ error: "Backend returned an invalid response (missing storage_key)" },
{ status: 502 },
);
}

await prisma.document.create({
data: {
documentCount: { increment: 1 },
updatedAt: new Date(),
id: storageKey,
name: filename,
size: BigInt(fileSize),
chatId,
Comment on lines +97 to +116

CopilotAIApr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schema comment says Document.name is the original display name and Document.filename is the backend storage name, but this code sets both to result.filename (which the backend describes as the original filename). Either persist the original name separately from a stable storage key returned by the backend, or rename/adjust the schema fields/comments so they reflect what’s actually stored. Also, silently defaulting to "unknown" / 0 risks writing bad rows if the backend response changes—prefer validating required fields and returning an error instead.

Copilot uses AI. Check for mistakes.
Comment on lines +97 to +116

CopilotAIApr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This stores the same value into both name and filename. Since the backend currently returns the original upload filename, this makes filename redundant and contradicts the schema’s intent of separating display name vs storage key. Consider persisting the original filename to name and using a distinct storage identifier for filename (or drop one of the fields).

Copilot uses AI. Check for mistakes.
},
});
Comment on lines +111 to 118

CopilotAIApr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document upload no longer updates Chat.updatedAt. Since the chat list is ordered by updatedAt, uploads may not bump the chat in recency ordering (previously this was updated when incrementing documentCount). Consider updating the parent chat’s updatedAt (or using a dedicated “lastActivityAt”) when creating a Document.

Copilot uses AI. Check for mistakes.

Expand Down
9 changes: 4 additions & 5 deletions app/chat/loading.tsx
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -4,13 +4,12 @@ export default function Loading() {
return (
<div className="flex h-dvh min-h-0 w-full overflow-hidden bg-background">
<div className="flex min-h-0 flex-1 flex-col bg-background">
<div className="border-b border-border bg-card px-4 py-3 md:px-6">
<div className="border-b border-border bg-card px-4 py-2 md:px-6">
<div className="flex items-center justify-between gap-3">
<div className="min-w-0 space-y-1">
<Skeleton className="h-6 w-36 sm:w-56 rounded-md" />
<Skeleton className="h-4 w-24 sm:w-40 rounded-md" />
<div className="flex items-center gap-2">
<Skeleton className="h-7 w-36 sm:w-56 rounded-md" />
</div>
<Skeleton className="h-10 w-24 sm:w-40 rounded-md" />
<Skeleton className="h-9 w-20 sm:w-24 rounded-md" />
</div>
</div>

Expand Down
9 changes: 7 additions & 2 deletions backend/api/v1/ingestion_routes.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -3,6 +3,7 @@
Provides endpoint for file upload, parsing, chunking, and storage.
"""
import logging
import uuid

from fastapi import APIRouter, HTTPException, UploadFile, File, Query, Depends

Expand DownExpand Up@@ -98,13 +99,17 @@ async def ingest_document(
}
)

logger.info(f"Ingesting {filename} ({len(file_bytes)} bytes) for chat {chat_id}")
# Generate a unique storage key for this upload
storage_key = uuid.uuid4().hex

logger.info(f"Ingesting {filename} ({len(file_bytes)} bytes) for chat {chat_id} [key={storage_key}]")

# Process the file
result = service.ingest_file(
chat_id=chat_id,
file_bytes=file_bytes,
filename=filename
filename=filename,
storage_key=storage_key
)

logger.info(
Expand Down
12 changes: 6 additions & 6 deletions backend/api/v1/vector_routes.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -185,21 +185,21 @@ async def get_collection_stats(
@router.delete(
"/document/{chat_id}",
summary="Delete all chunks for a specific document",
description="Delete all chunks belonging to a specific file from a chat collection."
description="Delete all chunks belonging to a specific document upload from a chat collection.",
)
async def delete_document_by_filename(
async def delete_document_by_storage_key(
chat_id: str,
filename: str,
storage_key: str,
service: VectorService = Depends(get_service),
):
"""
Delete all vector chunks for a given filename from a chat collection.
Delete all vector chunks for a given storage key from a chat collection.

- **chat_id**: UUID of the chat session
- **filename**: The original filename as stored in chunk metadata
- **storage_key**: The unique storage key assigned during ingestion
"""
try:
result = service.delete_documents_by_filename(chat_id, filename)
result = service.delete_documents_by_storage_key(chat_id, storage_key)
if not result.get("success"):
raise HTTPException(status_code=500, detail=result.get("error", "Deletion failed"))
return result
Expand Down
2 changes: 2 additions & 0 deletions backend/schemas/ingestion.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,8 @@ class IngestResponse(BaseModel):
"""Successful response for document ingestion."""
chat_id: str = Field(..., description="Chat ID the documents were ingested into")
filename: str = Field(..., description="Original filename of the uploaded document")
storage_key: str = Field(..., description="Unique backend storage key for this document upload")
file_size: int = Field(..., description="File size in bytes")
chunks_created: int = Field(..., description="Number of text chunks created from the document")
document_ids: List[str] = Field(..., description="List of document IDs for the created chunks")
pages_processed: int = Field(default=1, description="Number of pages processed (for PDFs)")
Comment on lines 14 to 21

CopilotAIApr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IngestResponse now includes required file_size, but the json_schema_extra example below doesn’t include it. Consider adding file_size to the example so the generated OpenAPI docs stay accurate.

Copilot uses AI. Check for mistakes.
Expand Down
20 changes: 13 additions & 7 deletions backend/services/ingestion_service.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -214,14 +214,16 @@ def parse_text_file(self, file_bytes: bytes) -> Tuple[str, int]:
def chunk_text(
self,
text: str,
filename: str
filename: str,
storage_key: str
) -> List[Dict[str, Any]]:
"""
Split text into overlapping chunks with metadata.

Args:
text: Full text content to chunk.
filename: Source filename for metadata.
storage_key: Unique storage key for this upload.

Returns:
List of chunk dictionaries with text and metadata.
Expand All@@ -238,14 +240,14 @@ def chunk_text(
timestamp = datetime.utcnow().isoformat()

for idx, chunk_text in enumerate(chunks):
# Generate unique ID for each chunk
chunk_id = f"{filename.replace('.', '_')}_chunk_{idx}"
# Generate unique ID for each chunk using storage_key
chunk_id = f"{storage_key}_chunk_{idx}"

result.append({
"id": chunk_id,
"text": chunk_text,
"metadata": {
"source_id": filename,
"source_id": storage_key,
"file_name": filename,
"chunk_index": idx,
"total_chunks": len(chunks),
Expand All@@ -264,7 +266,8 @@ def ingest_file(
self,
chat_id: str,
file_bytes: bytes,
filename: str
filename: str,
storage_key: str
) -> IngestResponse:
"""
Full ingestion pipeline: parse → chunk → store.
Expand All@@ -273,6 +276,7 @@ def ingest_file(
chat_id: Target chat/collection ID.
file_bytes: Raw file bytes.
filename: Original filename (used for extension detection).
storage_key: Unique storage key for this upload.

Returns:
IngestResponse with ingestion statistics.
Expand All@@ -295,8 +299,8 @@ def ingest_file(
else:
text, page_count = self.parse_text_file(file_bytes)

# Chunk the text
chunks = self.chunk_text(text, filename)
# Chunk the text using storage_key for unique identification
chunks = self.chunk_text(text, filename, storage_key)

if not chunks:
raise ValueError("Document produced no text chunks")
Expand DownExpand Up@@ -328,6 +332,8 @@ def ingest_file(
return IngestResponse(
chat_id=chat_id,
filename=filename,
storage_key=storage_key,
file_size=len(file_bytes),
chunks_created=actual,
document_ids=add_response.document_ids,
pages_processed=page_count
Expand Down
20 changes: 12 additions & 8 deletions backend/services/vector_service.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -233,33 +233,37 @@ def delete_collection(self, chat_id: str) -> DeleteCollectionResponse:
message=f"Failed to delete collection: {str(e)}"
)

def delete_documents_by_filename(self, chat_id: str, filename: str) -> Dict[str, Any]:
def delete_documents_by_storage_key(self, chat_id: str, storage_key: str) -> Dict[str, Any]:
"""
Delete all chunks belonging to a specific file from a chat collection.
Delete all chunks belonging to a specific document upload from a chat collection.

Args:
chat_id: UUID of the chat session.
filename: The original file name used during ingestion (stored in metadata as file_name).
storage_key: The unique storage key assigned during ingestion (stored in metadata as source_id).

Returns:
Dictionary with success status and deleted count.
"""
if not self.collection_exists(chat_id):
logger.info(f"No collection exists for chat {chat_id}, nothing to delete")
return {"success": True, "deleted_chunks": 0}

collection = self.get_collection(chat_id)
try:
# ChromaDB supports metadata filtering via the 'where' clause
# First get the IDs so we can report the count
results = collection.get(
where={"file_name": {"$eq": filename}},
where={"source_id": {"$eq": storage_key}},
include=[]
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
chunk_ids = results.get("ids", [])
if chunk_ids:
collection.delete(ids=chunk_ids)
logger.info(f"Deleted {len(chunk_ids)} chunks for file '{filename}' from chat {chat_id}")
logger.info(f"Deleted {len(chunk_ids)} chunks for storage_key '{storage_key}' from chat {chat_id}")
else:
logger.info(f"No chunks found for file '{filename}' in chat {chat_id}")
logger.info(f"No chunks found for storage_key '{storage_key}' in chat {chat_id}")
return {"success": True, "deleted_chunks": len(chunk_ids)}
except Exception as e:
logger.error(f"Error deleting documents for file '{filename}' in chat {chat_id}: {e}")
logger.error(f"Error deleting documents for storage_key '{storage_key}' in chat {chat_id}: {e}")
return {"success": False, "deleted_chunks": 0, "error": str(e)}

def collection_exists(self, chat_id: str) -> bool:
Expand Down
78 changes: 50 additions & 28 deletions components/chat-area.tsx
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -36,10 +36,15 @@ interface UploadedDocument {
id: string;
name: string;
size: string;
filename: string;
uploading?: boolean;
}

function formatFileSize(bytes: number): string {
if (bytes < 1024) return `${bytes} B`;
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(2)} KB`;
return `${(bytes / (1024 * 1024)).toFixed(2)} MB`;
}

export function ChatArea({
currentChat,
messages,
Expand All@@ -48,14 +53,29 @@ export function ChatArea({
const [input, setInput] = useState("");
const [localMessages, setLocalMessages] = useState<Message[]>(messages);
const [isGenerating, setIsGenerating] = useState(false);
const [documents, setDocuments] = useState<UploadedDocument[]>([]);
const [documents, setDocuments] = useState<UploadedDocument[]>(
(currentChat?.documents ?? []).map((d) => ({
id: d.id,
name: d.name,
size: formatFileSize(d.size),
uploading: false,
})),
);
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Comment on lines +56 to +63

CopilotAIApr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

documents state is initialized from currentChat?.documents, but it won’t update when navigating to a different chat or after router.refresh() changes currentChat props. This can cause the document chips to show the previous chat’s documents or miss newly persisted documents. Add an effect keyed on currentChat?.id (and/or currentChat?.documents) to reset documents from the latest props when the active chat changes.

Copilot uses AI. Check for mistakes.
const fileInputRef = useRef<HTMLInputElement>(null);
const messagesContainerRef = useRef<HTMLDivElement>(null);
const router = useRouter();

// Reset internal state when navigating between chats
useEffect(() => {
setLocalMessages(messages);
setDocuments(
(currentChat?.documents ?? []).map((d) => ({
id: d.id,
name: d.name,
size: formatFileSize(d.size),
uploading: false,
})),
);
}, [messages, currentChat?.id]);

CopilotAIApr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documents state is only reset when currentChat?.id changes, so after router.refresh() the updated currentChat.documents won't repopulate local state (same chat id). This leaves the UI with stale/optimistic document IDs and breaks later delete calls. Update the effect dependencies (or derive documents directly from props) so local state syncs when the documents list changes.

Suggested change
},[messages,currentChat?.id]);
},[messages,currentChat?.id,currentChat?.documents]);

Copilot uses AI. Check for mistakes.

const userName = currentUser.name?.trim() || "User";
Expand DownExpand Up@@ -147,7 +167,6 @@ export function ChatArea({
id: `${Date.now()}-${index}-${file.name}`,
name: file.name,
size: (file.size / 1024).toFixed(2) + " KB",
filename: file.name,
uploading: true,
} satisfies UploadedDocument,
}));
Comment on lines 167 to 172

CopilotAIApr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optimistic document id is generated client-side, but deletes now use doc.id as the backend storage_key/Prisma Document.id. After upload succeeds you need to replace this optimistic id with the server-provided storage_key (or fully re-sync documents from refreshed currentChat.documents), otherwise delete will call the backend/DB with an ID that doesn’t exist.

Copilot uses AI. Check for mistakes.
Expand DownExpand Up@@ -179,9 +198,18 @@ export function ChatArea({
toast.promise(uploadPromise, {
loading: `Uploading "${file.name}"...`,
success: (data) => {
const serverKey = data?.result?.storage_key;
const serverSize = data?.result?.file_size;
setDocuments((prev) =>
prev.map((d) =>
d.id === optimisticDocument.id ? { ...d, uploading: false } : d,
d.id === optimisticDocument.id
? {
...d,
id: serverKey || d.id,
size: serverSize ? formatFileSize(serverSize) : d.size,
uploading: false,
}
: d,
),
);
router.refresh();
Expand DownExpand Up@@ -219,7 +247,7 @@ export function ChatArea({
// Delete from backend collection
if (doc && currentChat) {
try {
await deleteDocumentAction(currentChat.id, doc.filename);
await deleteDocumentAction(currentChat.id, doc.id);
router.refresh();
} catch (error) {
console.error("Failed to delete document from collection:", error);
Expand DownExpand Up@@ -250,29 +278,23 @@ export function ChatArea({

return (
<div className="flex min-h-0 flex-1 flex-col bg-background">
<div className="border-b border-border bg-card px-4 py-3 md:px-6">
<div className="border-b border-border bg-card px-4 py-2 md:px-6">
<div className="flex items-center justify-between gap-3">
<div className="flex items-center gap-2">
<SidebarTrigger className="md:hidden" />
<div>
<h1 className="text-xl font-semibold text-card-foreground">
{currentChat.title}
</h1>
<p className="mt-0.5 text-sm text-muted-foreground">
{currentChat.documentCount === 0
? "No documents uploaded"
: `${currentChat.documentCount} ${currentChat.documentCount === 1 ? "document" : "documents"} uploaded`}
</p>
</div>
<h1 className="text-xl font-semibold text-card-foreground">
{currentChat.title}
</h1>
</div>

<Button
onClick={() => fileInputRef.current?.click()}
variant="outline"
className="gap-2"
size="sm"
className="gap-1.5 text-xs"
>
<FileUp className="size-4" />
Upload Documents
<FileUp className="size-3.5" />
Upload
</Button>
<input
ref={fileInputRef}
Expand All@@ -286,32 +308,32 @@ export function ChatArea({
</div>

{documents.length > 0 && (
<div className="border-b border-border bg-card px-4 py-3 md:px-6">
<div className="flex flex-wrap gap-2">
<div className="border-b border-border bg-card px-4 py-1.5 md:px-6">
<div className="flex flex-wrap gap-1.5">
{documents.map((doc) => (
<div
key={doc.id}
className="group flex items-center gap-2 rounded-lg bg-accent px-3 py-1.5 text-sm"
className="group flex items-center gap-1.5 rounded-md bg-accent px-2 py-1 text-xs"
>
{doc.uploading ? (
<Loader2 className="size-4 text-muted-foreground animate-spin" />
<Loader2 className="size-3 text-muted-foreground animate-spin" />
) : (
<File className="size-4 text-muted-foreground" />
<File className="size-3 text-muted-foreground" />
)}
<span className="max-w-50 truncate text-accent-foreground">
<span className="max-w-32 truncate text-accent-foreground">
{doc.name}
</span>
<span className="text-xs text-muted-foreground">
<span className="text-[10px] text-muted-foreground">
({doc.size})
</span>
<Button
type="button"
variant="ghost"
size="icon-sm"
onClick={() => handleRemoveDocument(doc.id)}
className="ml-1 size-6 p-0 opacity-0 transition-opacity group-hover:opacity-100"
className="ml-0.5 size-5 p-0 opacity-0 transition-opacity group-hover:opacity-100"
>
<X className="size-3.5 text-muted-foreground hover:text-foreground" />
<X className="size-3 text-muted-foreground hover:text-foreground" />
</Button>
</div>
))}
Expand Down
Loading