Migrate storage from Hive to SQLite + add coach SQL query tool

docs/superpowers/specs/2026-08-11-health-data-sync-and-coach-sql-design.md

@@ -0,0 +1,135 @@
+# Health Data Sync (Sleep + HR) into SQLite — Design Spec
+
+**Date:** 2026-08-11
+**Status:** Approved
+**Feature area:** Storage layer (`lib/services/`) + AI Coach SQL tool (`lib/services/ai/`)
+
+---
+
+## 1. Problem
+
+The AI Coach's `run_sql_query` tool (added in `docs/superpowers/specs/2026-08-08-sqlite-migration-and-coach-sql-tool-design.md`) can query workouts, sets, targets, and PRs directly — but health data (sleep stages, heart rate, resting HR, HRV) is fetched live from Health Connect on every request via `HealthConnectService`/`HealthHistoryManager` and is never persisted. This means the coach cannot join health data against workout data in a single SQL query (e.g. "average sleep the night before a PR attempt" or "HR trend across the last 8 weeks of leg day sessions") — each half of the question requires a separate tool call and the model has to reconcile the join itself, unreliably.
+
+This spec adds three SQLite tables that mirror Health Connect data, plus a sync service that keeps them populated, so `run_sql_query` can join across workout and health data directly.
+
+---
+
+## 2. Goal
+
+1. Persist sleep sessions (with stage breakdown) and HR-related samples (raw heart rate, resting heart rate, HRV RMSSD) into the same SQLite database `SqliteStorageService` already owns.
+2. Keep this data reasonably fresh via sync-on-app-launch (throttled) plus a manual "Sync now" action — no background service.
+3. Extend `run_sql_query`'s schema description so the coach can query and join the new tables.
+4. Keep the existing live `get_health_metrics` coach tool as-is, for "right now" freshness the synced tables won't have until the next sync.
+
+Non-goals: no background/periodic sync (WorkManager or equivalent), no downsampling/compaction of old raw samples, no changes to `IStorageService`'s method signatures (this feature is additive on `SqliteStorageService` directly, matching how `SqlQueryService` already bypasses that interface), no UI beyond one manual sync button.
+
+---
+
+## 3. Schema
+
+Added to the same database `SqliteStorageService` manages, created in `onCreate` (and via a migration step for existing installs already past `onCreate` — see §6).
+
+```sql
+-- Raw heart rate, resting heart rate, and HRV RMSSD samples all share the
+-- same {time, value} shape from Health Connect; one EAV-style table avoids
+-- three near-identical tables and keeps the coach's query surface simple
+-- ("WHERE type = 'heart_rate'") instead of three tables to remember.
+CREATE TABLE health_samples (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ type TEXT NOT NULL, -- 'heart_rate' | 'resting_heart_rate' | 'hrv_rmssd'
+ timestamp TEXT NOT NULL, -- ISO8601
+ value REAL NOT NULL
+);
+CREATE UNIQUE INDEX idx_health_samples_unique ON health_samples(type, timestamp);
+CREATE INDEX idx_health_samples_type_ts ON health_samples(type, timestamp);
+
+CREATE TABLE sleep_sessions (
+ id TEXT PRIMARY KEY, -- synthetic: the start_ts ISO string
+ start_ts TEXT NOT NULL,
+ end_ts TEXT NOT NULL,
+ light_min INTEGER,
+ deep_min INTEGER,
+ rem_min INTEGER,
+ awake_min INTEGER
+);
+CREATE INDEX idx_sleep_sessions_start ON sleep_sessions(start_ts);
+
+CREATE TABLE sleep_stage_intervals (
+ sleep_session_id TEXT NOT NULL REFERENCES sleep_sessions(id),
+ start_ts TEXT NOT NULL,
+ end_ts TEXT NOT NULL,
+ stage TEXT NOT NULL -- 'deep' | 'rem' | 'light' | 'awake'
+);
+CREATE INDEX idx_sleep_stage_session ON sleep_stage_intervals(sleep_session_id);
+```
+
+Sync watermarks (one ISO8601 timestamp per data stream, e.g. key `health_sync.heart_rate`) are stored as ordinary rows in the existing `settings` table — no new table needed for that.
+
+`sleep_sessions.id` is derived from `start_ts` so re-syncing the same session (e.g. after a Health Connect correction) is a natural upsert target, not a duplicate.
+
+---
+
+## 4. `HealthSyncService`
+
+New file: `lib/services/health_sync_service.dart`.
+
+```dart
+class HealthSyncService {
+ HealthSyncService(this._hc, this._db);
+
+ final IHealthConnectService _hc;
+ final SqliteStorageService _db;
+
+ Future<void> sync({bool force = false}) async { ... }
+}
+```
+
+- **Throttle:** skip if the most recent sync (tracked via a `health_sync.last_run` watermark) was less than 30 minutes ago, unless `force: true`.
+- **Per-stream incremental fetch with look-back:** for each of `sleep`, `heart_rate`, `resting_heart_rate`, `hrv_rmssd`: read that stream's watermark from `settings` (default `now - 90 days` if absent — the agreed backfill window). Fetch from `watermark - 3 days` through `now` — the 3-day look-back re-pulls recent data even though it was already synced, to catch late corrections Health Connect or the watch itself makes to recent records (e.g. a sleep session Health Connect revises the next morning). Anything before the look-back window is assumed final and is never re-fetched.
+- **Upsert:**
+ - `health_samples`: `INSERT OR REPLACE` keyed by the `(type, timestamp)` unique index — naturally idempotent and self-correcting.
+ - `sleep_sessions` / `sleep_stage_intervals`: for each `SleepPeriod` in the fetch window, delete-then-reinsert `sleep_stage_intervals` for that session id and upsert the `sleep_sessions` row — same delete/reinsert-child-rows pattern the original migration spec already uses for `sets`/`exercise_logs`.
+ - After all four streams succeed, advance each stream's watermark to `now` and the `last_run` throttle marker to `now`.
+- **Failure handling:** any exception (permission not granted, Health Connect unavailable, one stream fails) is caught per-stream — a failed stream's watermark is left untouched so the next sync retries it, and does not block the other streams or crash the caller. Matches the existing best-effort caching posture in `HealthHistoryManager._readCachedHrDay`.
+
+### Wiring
+
+Only constructed when the active backend is `SqliteStorageService` — mirrors the existing guard in `main.dart:191-192` (`_storageService is SqliteStorageService ? SqlQueryService(...) : null`). Health data has no meaning under the pre-migration Hive fallback path.
+
+- `AppInitializer` calls `sync()` once after both `HealthConnectService` and `SqliteStorageService` are ready, fire-and-forget (does not block first frame).
+- A "Sync now" button is added to the existing health-permissions area of the Profile screen, calling `sync(force: true)`.
+
+---
+
+## 5. Coach SQL Tool Update
+
+`CoachToolService`'s embedded schema description (used by `run_sql_query`, §7 of the original migration spec) gets the three new tables appended in the same one-line-per-table/column format as the existing schema text, so the model can join them against `sessions`, `exercise_logs`, and `sets` without a separate discovery call.
+
+`get_health_metrics` (the existing live Health Connect tool) is unchanged — it remains the source for "right now" data that the synced tables won't have until the next app-open or manual sync.
+
+---
+
+## 6. Migration for Existing Installs
+
+Existing SQLite installs (already past `onCreate`) need the three new tables added without a fresh install. `SqliteStorageService.init()` bumps `_dbVersion` and adds an `onUpgrade` step that runs the `CREATE TABLE`/`CREATE INDEX` statements from §3 if the new tables don't already exist (`CREATE TABLE IF NOT EXISTS`, safe to run unconditionally on upgrade). No data migration needed — these are brand-new tables with no prior data to carry forward; the first post-upgrade sync populates them via the normal 90-day backfill path.
+
+---
+
+## 7. Testing
+
+- **`HealthSyncService`** (new test file, in-memory DB via `sqflite_common_ffi` + a fake `IHealthConnectService`):
+ - First sync with no prior watermark backfills the full 90-day window.
+ - Second sync only re-fetches from `watermark - 3 days` onward (verify the fake service receives the narrower range).
+ - Re-running sync is idempotent: no duplicate rows in `health_samples` or `sleep_sessions`, and changed values from a "corrected" fake response overwrite the prior row.
+ - A sync attempted less than 30 minutes after the last one is skipped unless `force: true`.
+ - An exception thrown by the fake health service for one stream doesn't propagate, doesn't advance that stream's watermark, and doesn't block the other streams from syncing.
+- **`SqliteStorageService`**: extend the existing test file to cover the new upsert methods and the `onUpgrade` path (open a v-1 schema DB, run `init()`, assert the new tables exist).
+- **`run_sql_query`**: extend `sql_query_service_test.dart` with a join query across `sessions`, `sets`, `sleep_sessions`, and `health_samples`, confirming the schema and join work end-to-end.
+
+---
+
+## 8. Rollout Notes
+
+- No new dependencies — reuses `sqflite`, `sqflite_common_ffi` (test), and the existing `IHealthConnectService`.
+- No changes to `IStorageService`, `MockStorageService`, or any manager/provider — additive on `SqliteStorageService` only, same boundary `SqlQueryService` already uses.
+- `CLAUDE.md`'s "6 boxes" / schema references would benefit from a follow-up doc note once this ships, but that's out of scope here (same deferral pattern as the original migration spec, §9).

workout-logger/lib/main.dart

@@ -3,16 +3,23 @@
 // Following Dependency Inversion Principle: we create concrete implementations
 // here at the composition root and inject them into high-level modules.
 
+import 'dart:async';
+
 import 'package:flutter/material.dart';
 import 'package:flutter/services.dart';
 import 'package:provider/provider.dart';
 
+import 'package:hive_flutter/hive_flutter.dart';
 import 'services/debug_log_buffer.dart';
 import 'services/storage_service.dart';
+import 'services/sqlite_storage_service.dart';
+import 'services/storage_backend_resolver.dart';
+import 'services/ai/sql_query_service.dart';
 import 'services/ml_service.dart';
 import 'services/ai/gemini_ai_service.dart';
 import 'services/ai/coach_tool_service.dart';
 import 'services/health_connect_service.dart';
+import 'services/health_data_sync_service.dart';
 import 'services/interfaces/storage_service_interface.dart';
 import 'services/interfaces/ml_service_interface.dart';
 import 'services/interfaces/health_connect_service_interface.dart';
@@ -32,6 +39,55 @@ import 'theme/a2ui_app_theme.dart';
 import 'screens/home_screen.dart';
 import 'screens/onboarding_screen.dart';
 
+/// Resolved once in main() before runApp(). Read lazily by
+/// WorkoutLoggerApp._storageService's static initializer, which only runs
+/// on first access (during build()) — by then this is already set.
+IStorageService? _resolvedStorageService;
+
+/// One-time, flag-gated, reversible Hive -> SQLite cutover. See
+/// docs/superpowers/specs/2026-08-08-sqlite-migration-and-coach-sql-tool-design.md §6.
+Future<void> _resolveStorageBackend() async {
+ // Hive stays initialized here even post-cutover: ApiService reads/writes
+ // an installation id directly against this settings box, independent of
+ // IStorageService. Do not remove this unconditional init.
+ await Hive.initFlutter();
+ final settingsBox = await Hive.openBox<String>('settings');
+ final alreadyMigrated = settingsBox.get(storageMigratedFlagKey) == 'true';
+
+ if (alreadyMigrated) {
+ final sqlite = SqliteStorageService();
+ try {
+ await sqlite.init();
+ } catch (e, st) {
+ debugPrint('SQLite init failed, staying on Hive: $e\n$st');
+ final hiveStorage = StorageService();
+ await hiveStorage.init();
+ _resolvedStorageService = hiveStorage;
+ return;
+ }
+ _resolvedStorageService = sqlite;
+ return;
+ }
+
+ final hiveStorage = StorageService();
+ await hiveStorage.init();
+ final sqliteStorage = SqliteStorageService();
+
+ try {
+ await sqliteStorage.init();
+ } catch (e, st) {
+ debugPrint('SQLite init failed, staying on Hive: $e\n$st');
+ _resolvedStorageService = hiveStorage;
+ return;
+ }
+
+ _resolvedStorageService = await resolveStorageBackend(
+ hiveStorage: hiveStorage,
+ sqliteStorage: sqliteStorage,
+ alreadyMigrated: false,
+ );
+}
+
 void main() async {
  DebugLogBuffer.attach();
  WidgetsFlutterBinding.ensureInitialized();
@@ -58,13 +114,15 @@ void main() async {
  ),
  );
 
+ await _resolveStorageBackend();
+
  runApp(const WorkoutLoggerApp());
 }
 
 class WorkoutLoggerApp extends StatelessWidget {
  // Singleton instances created once at app startup
  // This ensures the same instances are used throughout the app lifecycle
- static final IStorageService _storageService = StorageService();
+ static final IStorageService _storageService = _resolvedStorageService ?? StorageService();
  static final IMLService _mlService = MLService();
  static final IHealthConnectService _healthConnectService = HealthConnectService();
  static final ProgramManager _programManager = ProgramManager(_storageService);
@@ -83,6 +141,17 @@ class WorkoutLoggerApp extends StatelessWidget {
  // Serves arbitrary-range sleep/HR data to the detail screens.
  static final HealthHistoryManager _healthHistoryManager =
  HealthHistoryManager(_healthConnectService, _storageService);
+ // Populates the SQLite health tables the coach's run_sql_query tool joins
+ // against workout data. Null under the pre-migration Hive fallback path —
+ // there's no live SQLite database file to sync into. Mirrors the
+ // sqlQuery ? ... : null guard used for CoachToolService below.
+ static final HealthDataSyncService? _healthDataSyncService =
+ _storageService is SqliteStorageService
+ ? HealthDataSyncService(
+ healthConnectService: _healthConnectService,
+ storage: _storageService as SqliteStorageService,
+ )
+ : null;
  static final GeminiAiService _geminiService =
  GeminiAiService(storage: _storageService);
  static final ConversationManager _conversationManager =
@@ -115,6 +184,7 @@ class WorkoutLoggerApp extends StatelessWidget {
  ChangeNotifierProvider<PRManager>.value(value: _prManager),
  ChangeNotifierProvider<ReadinessManager>.value(value: _readinessManager),
  Provider<HealthHistoryManager>.value(value: _healthHistoryManager),
+ Provider<HealthDataSyncService?>.value(value: _healthDataSyncService),
  // GeminiAiService is the single AI backend instance. It's a ChangeNotifier
  // (settings UI watches isConfigured/model), so it's provided as such.
  // Consumers that should depend on the abstraction (the coach ViewModel,
@@ -134,11 +204,16 @@ class WorkoutLoggerApp extends StatelessWidget {
  ),
  ),
  // CoachToolService backs AI tool calls; reads from WorkoutProvider + PRManager.
+ // run_sql_query is only offered once the app has cut over to SQLite —
+ // it needs a live database file to open a read-only connection against.
  Provider<CoachToolService>(
  create: (ctx) => CoachToolService(
  workoutProvider: ctx.read<WorkoutProvider>(),
  prManager: ctx.read<PRManager>(),
  healthHistory: ctx.read<HealthHistoryManager>(),
+ sqlQuery: _storageService is SqliteStorageService
+ ? SqlQueryService((_storageService as SqliteStorageService).databasePath)
+ : null,
  ),
  ),
  ],
@@ -182,11 +257,16 @@ class _AppInitializerState extends State<AppInitializer> {
  final api = context.read<ApiService>();
  final gemini = context.read<GeminiAiService>();
  final readiness = context.read<ReadinessManager>();
+ final healthDataSync = context.read<HealthDataSyncService?>();
 
  try {
  await provider.init();
  await settings.init();
- gemini.init(settings.geminiApiKey, model: settings.geminiModel);
+ gemini.init(
+ settings.geminiApiKey,
+ model: settings.geminiModel,
+ maxToolRounds: settings.geminiMaxToolRounds,
+ );
  try {
  await gemini.loadUsage();
  } catch (e, st) {
@@ -206,6 +286,16 @@ class _AppInitializerState extends State<AppInitializer> {
  // so the opt-in flag is loaded; never blocks or fails app init.
  readiness.refresh();
 
+ // Fire-and-forget: populates the SQLite tables run_sql_query joins
+ // against. No-op under the pre-migration Hive fallback (null there).
+ // Errors are swallowed here since main.dart discards the returned
+ // Future — sync() has no caller to propagate a failure to.
+ unawaited(
+ healthDataSync?.sync().catchError(
+ (Object e, StackTrace st) => debugPrint('healthDataSync.sync failed: $e\n$st'),
+ ),
+ );
+
  // Fire-and-forget analytics in background.
  api.sendHeartbeat();
  api.trackEvent('app_open');

workout-logger/lib/models/models.dart

@@ -64,6 +64,19 @@ class MuscleActivation {
 
 // ==================== Exercise ====================
 
+// Exercise IDs treated as bodyweight-assisted (e.g. an assisted-dip/pull-up
+// machine). Computed once here so every consumer (the load panel, the input
+// row, set persistence) agrees on which exercises count as "assisted".
+const Set<String> _assistedBodyweightExerciseIds = {
+ 'pull_ups',
+ 'chin_ups',
+ 'dips',
+ 'push_ups',
+};
+
+bool isAssistedBodyweightExercise(String? exerciseId) =>
+ exerciseId != null && _assistedBodyweightExerciseIds.contains(exerciseId);
+
 class Exercise {
  final String id;
  final String name;
@@ -72,7 +85,7 @@ class Exercise {
  final bool isCustom; // User-created exercise
  final List<String>? availableHandles; // Attachment/handle options e.g. ['Rope', 'Bar']
 
- Exercise({
+const Exercise({
  required this.id,
  required this.name,
  required this.muscleActivations,
@@ -250,7 +263,7 @@ class ExerciseLog {
  final String? notes;
  final String? handle;
 
- ExerciseLog({
+const ExerciseLog({
  required this.exerciseId,
  required this.sets,
  this.notes,
@@ -936,12 +949,16 @@ class ChatMessage {
  final String role; // 'user' | 'model'
  final String text;
  final DateTime timestamp;
+ // Names of tools the model called (in order) while producing this reply.
+ // Null/empty for user messages and replies that used no tools.
+ final List<String>? toolCalls;
 
  ChatMessage({
  String? id,
  required this.role,
  required this.text,
  DateTime? timestamp,
+ this.toolCalls,
  }) : id = id ?? _uuid.v4(),
  timestamp = timestamp ?? DateTime.now();
 
@@ -950,24 +967,30 @@ class ChatMessage {
  'role': role,
  'text': text,
  'timestamp': timestamp.toIso8601String(),
+ if (toolCalls != null && toolCalls!.isNotEmpty) 'toolCalls': toolCalls,
  };
 
  factory ChatMessage.fromJson(Map<String, dynamic> json) => ChatMessage(
  id: json['id'] as String?,
  role: json['role'] as String,
  text: json['text'] as String,
  timestamp: DateTime.parse(json['timestamp'] as String),
+ toolCalls: (json['toolCalls'] as List?)?.cast<String>(),
  );
 
  ChatMessage copyWith({
  Object? role = _sentinel,
  Object? text = _sentinel,
  Object? timestamp = _sentinel,
+ Object? toolCalls = _sentinel,
  }) => ChatMessage(
  id: id,
  role: role == _sentinel ? this.role : role as String,
  text: text == _sentinel ? this.text : text as String,
  timestamp: timestamp == _sentinel ? this.timestamp : timestamp as DateTime,
+ toolCalls: toolCalls == _sentinel
+ ? this.toolCalls
+ : toolCalls as List<String>?,
  );
 }