Add scripts to download data for offline use - #120
Conversation
WalkthroughThe changes introduce a new task named Changes
Sequence Diagram(s)sequenceDiagram
participant U as User
participant T as Taskfile (get-offline-data)
participant O as ollama Script
participant K as kiwix Script
U->>T: Run get-offline-data task
T->>T: Load dotenv configs (docker/.env, localhost/.env)
T->>O: Execute get-offline-data-ollama.sh
T->>K: Execute get-offline-data-kiwix.sh with $STORAGE_KIWIX argument
sequenceDiagram
participant S as get-offline-data-kiwix.sh
participant W as Web (wget)
participant FS as Filesystem
S->>S: Validate target directory parameter
S->>W: Fetch HTML content from provided URL
W-->>S: Return HTML content
S->>S: Extract filenames and determine latest zim file
alt File exists locally
S->>S: Log "No download needed"
else File missing
S->>W: Download the latest zim file
W-->>S: Transfer file
S->>S: Log download success or failure
end
📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (6)
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
scripts/get-offline-data-ollama.sh (1)
22-25: Clarify LLM Section Comment
The addition of the# LLMcomment helps demarcate the section where language models are being pulled. Consider expanding the comment (e.g., "LLM - Pulling language models for offline processing") if further context may assist future maintainers.config-example/docker/myhost/apply.sh (1)
10-12: Verify Symlink Creation for 'localhost'
The newly added commands create a symbolic link namedlocalhostin the parent directory ofHOST_CONFIG_DIRthat points to$(hostname)/. Please verify that using a trailing slash in the target ($(hostname)/) meets your deployment needs and does not cause unexpected path resolution issues.scripts/get-offline-data-kiwix.sh (2)
28-42: HTML Fetch and File Extraction Logic
The function fetches HTML content withwgetand uses a regex to extract filenames matching${PATTERN}_[0-9]{4}-[0-9]{2}\.zim. This works well if file names include only a year and month. If a day component might be present in filenames, consider updating the regex accordingly.
59-66: Robustness in File Downloading
When downloading, the commandwget -q --show-progress "$URL$LATEST_FILE"concatenates the URL and filename directly. This assumes that$URLends with a trailing slash. To improve robustness, consider ensuring that the URL ends with a slash or appending one if missing.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
Taskfile.yaml(1 hunks)config-example/docker/myhost/.env(1 hunks)config-example/docker/myhost/apply.sh(1 hunks)docker/common.sh(1 hunks)scripts/get-offline-data-kiwix.sh(1 hunks)scripts/get-offline-data-ollama.sh(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: build
- GitHub Check: check
🔇 Additional comments (9)
docker/common.sh (1)
61-62: Improved Readability indocker_commandFunction
Inserting an empty line to separate output sections improves readability. This formatting tweak is a minor but useful enhancement.Taskfile.yaml (1)
125-133: New Offline Data Task Added
The newget-offline-datatask is correctly defined with the appropriate dotenv files and command invocations for the offline scripts. Ensure that theSTORAGE_KIWIXenvironment variable is set as expected so that thescripts/get-offline-data-kiwix.shscript functions properly.scripts/get-offline-data-kiwix.sh (6)
1-9: Initial Parameter and Environment Validation
The script sets strict error handling withset -euo pipefailand validates that a target directory is provided. Clear error messages guide the user if the required parameter is missing.
12-16: Directory Existence Check
The validation to confirm that the provided target directory exists is effective and prevents unintended operation in an invalid directory.
24-27: Definition ofkiwix_pullFunction
Thekiwix_pullfunction encapsulates the download logic neatly. Defining this functionality as a separate function aids maintainability and clarity.
44-49: Determining the Latest File
Sorting the list in reverse order and picking the first entry is an effective and simple method to select the latest file—provided file naming remains consistent.
53-57: Local File Check to Prevent Redundant Downloads
The script correctly checks if the latest file already exists locally before attempting a download, thereby avoiding unnecessary network usage.
69-71: Invocation of Kiwix Pull Operations
The calls tokiwix_pullwith different URL and pattern pairs are structured clearly. Confirm that these URLs and regex patterns remain valid and meet your download requirements in the long term.config-example/docker/myhost/.env (1)
7-11: Consolidated Storage Variable UpdateThe updated storage path variables (lines 8–11) now reference
${STORAGE_LOCAL_MEDIA}rather than hardcoding parts of the path. This centralization improves consistency and simplifies future updates to storage locations. Please ensure that any scripts or task configurations (e.g., inTaskfile.yaml) referencing these variables are also updated accordingly.
dfaa6d3 to
67512fd
Compare
Summary by CodeRabbit
New Features
Chores