AGENTS.md in BasedHardware/omi runs 1,300 words across 24 headings.
AI that sees your screen, listens to your conversations and tells you what to do
PythonCovers
11 of the 20 section tags
In the order a file is read inHeadings
24 headings, in the order the file writes them
01App (Flutter) — Operational Playbook
02Build Bootstrap
03Flavors
04Generated Files (never edit manually)
05Setup Sequence
06Firebase Config
07Native Bridge
08Pigeon Interface (bidirectional, iOS ↔ Dart)
09MethodChannel (Phone Calls)
10Pigeon (Phone Mic — conversation capture)
11Permission Matrix
12Test Strategy
13Test Structure
14Running Tests
15Test Patterns
16Localization (l10n)
17Auth & Security
18Token Lifecycle
19Auth Methods
20Request Headers
21API Base URLs
22Codegen Rules
23App Flows & E2E
24Verifying UI Changes (agent-flutter)
Commands
7 commands this file writes down
Extracted from the file, verbatimflutter test
flutter test test/unit/
flutter pub run build_runner build
flutter gen-l10n
dart run pigeon --input lib/phone_mic_interface.dart
flutter pub run build_runner build --delete-conflicting-outputs
npm install -g agent-flutter-cli
The file
app/AGENTS.md
158 lines1# App (Flutter) — Operational Playbook
2
3Inherits all rules from the root [`../AGENTS.md`](../AGENTS.md). This file adds app-specific operational guidance.
4
5## Build Bootstrap
6
7### Flavors
8- **dev**: Android `com.friend.ios.dev`, iOS `com.friend-app-with-wearable.ios12.development` — uses `.dev.env`, Firebase project `based-hardware-dev`
9- **prod**: Android `com.friend.ios`, iOS `com.friend-app-with-wearable.ios12` — uses `.prod.env`, Firebase project `based-hardware-prod`
10- **raybanDat**: camera-capable iOS target with the same iOS development identity; use `scripts/rayban_dat.sh`, which excludes mcumgr only for that transaction and restores the default graph.
11
12### Generated Files (never edit manually)
13| Generator | Source | Output | Command |
14|-----------|--------|--------|---------|
15| envied | `lib/env/dev_env.dart`, `lib/env/prod_env.dart` | `*.g.dart` (obfuscated secrets) | `flutter pub run build_runner build` |
16| json_serializable | `@JsonSerializable` models | `*.g.dart` (fromJson/toJson) | `flutter pub run build_runner build` |
17| pigeon | `lib/pigeon_interfaces.dart` | `lib/gen/pigeon_communicator.g.dart` + iOS/Android stubs | `flutter pub run build_runner build` |
18| flutter_gen | `pubspec.yaml` assets/fonts | `lib/gen/assets.gen.dart`, `lib/gen/fonts.gen.dart` | `flutter pub run build_runner build` |
19| flutter_localizations | `lib/l10n/*.arb` | `lib/gen_l10n/app_localizations*.dart` | `flutter gen-l10n` |
20
21### Setup Sequence
22```bash
23bash setup.sh ios # or: bash setup.sh android
24```
25This handles: pub get, build_runner, gen-l10n, and flavor configuration.
26
27### Firebase Config
28Never run `flutterfire configure` — it overwrites prod credentials. Config files:
29- Dev: `android/app/src/dev/`
30- Prod: `android/app/src/prod/`
31- Local emulator: `lib/firebase_options_local.dart`
32
33## Native Bridge
34
35### Pigeon Interface (bidirectional, iOS ↔ Dart)
36- Contract: `lib/pigeon_interfaces.dart` — paired host/Flutter APIs for the watch recorder, BLE, and Ray-Ban Meta
37- Dart side: `lib/gen/pigeon_communicator.g.dart`
38- iOS side: `ios/Runner/PigeonCommunicator.g.swift`
39- Android side: `android/app/src/main/kotlin/com/friend/ios/PigeonCommunicator.g.kt`
40- Implementation: `ios/Runner/RecorderHostApiImpl.swift`
41- After editing the contract, regenerate: `flutter pub run build_runner build`
42
43### MethodChannel (Phone Calls)
44- Channel: `com.omi/phone_calls` + EventChannel `com.omi/phone_calls/events`
45- Dart: `lib/services/phone_call_service.dart`
46- iOS: `ios/Runner/PhoneCalls/OmiPhoneCallsPlugin.swift`
47- Android: `android/app/src/main/kotlin/com/friend/ios/phonecalls/PhoneCallsPlugin.kt`
48- Methods: initialize, makeCall, endCall, toggleMute, toggleSpeaker
49
50### Pigeon (Phone Mic — conversation capture)
51- Contract: `lib/phone_mic_interface.dart` → `lib/gen/phone_mic_pigeon.g.dart` + `ios/Runner/PhoneMic/PhoneMicPigeon.g.swift` + `android/app/src/main/kotlin/com/friend/ios/phonemic/PhoneMicPigeon.g.kt`
52- Regenerate: `dart run pigeon --input lib/phone_mic_interface.dart`
53- iOS module: `ios/Runner/PhoneMic/` — self-healing AVAudioEngine capture (interruptions/route changes recover natively; Dart only mirrors state)
54- Android module: `android/app/src/main/kotlin/com/friend/ios/phonemic/` — AudioRecord capture with a self-healing rebuild loop + silencing detection (calls/assistant recover natively; Dart only mirrors state); `PhoneMicForegroundService` (microphone FGS) keeps background capture alive; batch opus encode via a JNI shim over the plugin-shipped libopus
55- Dart service: `lib/services/mic/native_mic_recorder_service.dart` behind `ServiceManager.phoneMic`; chat memos/speech profile stay on flutter_sound via `ServiceManager.mic`; `MicArbiter` prevents the two stacks contending
56- Events carry a Dart-minted session id (`start(mode, sessionId)`); Dart drops any event whose id is not the current session's, so a late/stale native event can't clobber a fresh session, and a `start()` onto a still-live native session adopts the new id and re-emits the current state so the caller converges. `stop()` always forwards to native (kills an orphaned session) and runs local teardown once
57- Two capture modes, fixed per session at `start(mode)`: `stream` (realtime frames → Dart → socket/WAL) and `batch` (Transcribe Later — native opus encode (OpusKit on iOS, libopus JNI shim on Android) → WAL-compatible `audio_omibatchphone[auto]_…bin`; no frames cross to Dart; liveness = 1Hz `onBatchProgress`). Mode selection lives in `CaptureController.streamRecording` (explicit `batchModeEnabled` or automatic offline fallback; iOS + Android); `omibatchphoneauto` recordings auto-upload on reconnect
58
59## Permission Matrix
60
61| Permission | Android | iOS | Feature |
62|-----------|---------|-----|---------|
63| Microphone | RECORD_AUDIO | NSMicrophoneUsageDescription | Recording, speech profile |
64| Bluetooth | BLUETOOTH_SCAN, BLUETOOTH_CONNECT | NSBluetoothAlwaysUsageDescription | Omi device connection |
65| Location | ACCESS_FINE_LOCATION | NSLocationUsageDescription | Background features |
66| Contacts | READ_CONTACTS | NSContactsUsageDescription | People recognition |
67| Calendar | READ/WRITE_CALENDAR | NSCalendarsUsageDescription | Calendar integration |
68| Camera | — | NSCameraUsageDescription | QR/photo features |
69| Notifications | POST_NOTIFICATIONS | (automatic) | Push notifications |
70| Background | FOREGROUND_SERVICE_* (4 types) | UIBackgroundModes (7 modes) | Continuous capture |
71
72Android has 26 total permissions in AndroidManifest.xml. iOS has 11 background modes + 10 consent strings.
73
74## Test Strategy
75
76### Test Structure
77- `test/unit/` — Auth, tokens, preferences, audio utils
78- `test/widgets/` — UI components (shimmer, waveform, transcript)
79- `test/providers/` — State management (capture_provider, device_provider)
80- `test/utils/` — Utility functions (localization helpers)
81
82### Running Tests
83```bash
84bash test.sh # runs all tests
85flutter test # same thing
86flutter test test/unit/ # specific directory
87```
88
89`bash test.sh` bootstraps missing local generated files with an empty `API_BASE_URL` so `test/` stays hermetic.
90
91PR CI runs `flutter test` and an analyzer ratchet (`app/scripts/analyze_ratchet.sh`) — analyzer errors always fail; new info/warning lint occurrences above `app/analysis_baseline.json` fail. Run the script locally before committing app Dart changes. Deliberate lint acceptances/improvements update the baseline via `--update-baseline` in the same PR.
92
93### Test Patterns
94- Mock singletons (SharedPreferencesUtil, AuthService, FirebaseAuth) since they aren't injectable
95- Test state machine logic via minimal abstractions mirroring production flow
96- Everything under `test/` must be hermetic — no network, live backends, or real devices — because `bash test.sh` (the CI suite) runs all of it.
97- A test that needs a live service, device, or real API goes under `integration_test/`, which `test.sh`/CI never runs. For integration tests against a local backend, set `OMI_APP_TEST_API_BASE_URL=http://127.0.0.1:<port>/`; use `OMI_APP_TEST_USE_PROD_API_DEFAULT=1` only when a test intentionally needs the prod API default. State in the PR how you ran it; it must not be the only evidence the change works.
98- Coverage rules (bug fix → regression test; feature → core + main error path): see root `AGENTS.md` → Testing.
99
100## Localization (l10n)
101
102- All user-facing strings must use `context.l10n.keyName`
103- 49 locales: English (template) + 48 translations in `lib/l10n/`. Don't trust this count from memory — enumerate with `ls lib/l10n/app_*.arb`.
104- Template: `lib/l10n/app_en.arb`
105- Add keys via `jq` (never read full ARB — they're large). Use skill `add-a-new-localization-key-l10n-arb`
106- Translate all locales — use skill `omi-add-missing-language-keys-l10n` for real translations
107- Regenerate after changes: `flutter gen-l10n`. Task is only complete when this command emits zero "untranslated message(s)" warnings. To get the exact missing-key list, temporarily add `untranslated-messages-file: /tmp/untranslated.json` to `l10n.yaml` and re-run.
108
109## Auth & Security
110
111### Token Lifecycle
1121. `getAuthHeader()` in `lib/backend/http/shared.dart` checks token expiry (5-minute buffer)
1132. If expired, calls `AuthService.instance.getIdToken()` for Firebase refresh
1143. Token stored via SharedPreferencesUtil in flutter_secure_storage (Keychain / EncryptedSharedPreferences); expiration timestamp stays in SharedPreferences. One-time migrateAuthTokenFromPrefs() runs at SharedPreferencesUtil.init() so existing sessions keep their token.
1154. 401 responses trigger automatic refresh + retry
116
117### Auth Methods
118- Google Sign In (`google_sign_in` package)
119- Apple Sign In (`sign_in_with_apple` package, includes PKCE via nonce+sha256)
120- Firebase Auth as the identity layer
121
122### Request Headers
123All API requests include: X-Request-Start-Time, X-App-Platform, X-Device-Id-Hash, X-App-Version, plus Bearer token.
124
125### API Base URLs
126- Dev: configured in `.dev.env` → `Env.apiBaseUrl`
127- Prod: configured in `.prod.env` → `Env.apiBaseUrl`
128
129## Codegen Rules
130
131- Run `flutter pub run build_runner build` after changing: env files, model annotations, pigeon contracts, or pubspec assets
132- Run `flutter gen-l10n` after changing ARB files
133- Never edit files ending in `.g.dart` or `.gen.dart`
134- If build_runner fails with conflicts: `flutter pub run build_runner build --delete-conflicting-outputs`
135
136## App Flows & E2E
137
138- See `e2e/SKILL.md` for navigation architecture, screen map, widget patterns, and 34 reference flows
139- See `e2e/flows/*.yaml` for individual flow definitions
140
141## Verifying UI Changes (agent-flutter)
142
143After any Flutter UI edit, verify programmatically with [agent-flutter](https://github.com/beastoin/agent-flutter) (Marionette is integrated in debug builds). Install once: `npm install -g agent-flutter-cli`.
144
145Edit → Verify → Evidence loop:
1461. Edit code, hot restart: `kill -SIGUSR2 $(pgrep -f "flutter run" | head -1)`
1472. Connect: `AGENT_FLUTTER_LOG=/tmp/flutter-run.log agent-flutter connect`
1483. Verify: `agent-flutter snapshot -i`
1494. Interact: `agent-flutter press @e3` / `press 540 1200` / `find type button press` / `fill @e5 "text"` / `dismiss`
1505. Evidence: `agent-flutter screenshot /tmp/evidence.png`
151
152Key rules:
153- Must reconnect after every hot restart (kills VM Service session).
154- Refs go stale frequently — always re-snapshot before every interaction. Use `press x y` as fallback.
155- `AGENT_FLUTTER_LOG` must point to flutter run stdout (not logcat).
156- Prefer `find type X` / `find key "name"` over hardcoded `@ref`. Add `Key('descriptive_name')` to new interactive widgets.
157- Full command reference: `agent-flutter schema`.
158
The rest of the repository
BasedHardware/omi ships 14 other instruction files
.cursor/rules/flutter-localization.mdc.cursor/rules/common-mistakes.mdc.cursor/rules/formatting.mdc.github/AGENTS.mdAGENTS.mdCLAUDE.mdbackend/AGENTS.mddesktop/macos/AGENTS.mddesktop/windows/AGENTS.mdomi/firmware/AGENTS.mdweb/app/AGENTS.md.cursor/rules/backend-imports.mdc.cursor/rules/backend-testing.mdcweb/admin/AGENTS.md
A row that is not a link is a file this repository ships that this app did not freeze a sheet for. It is listed because the corpus knows it exists, and it is not linked because there is nothing here to open.
This listing
Whoever runs BasedHardware/omi can claim it
This is yours? Claim this config and we will write to you when the measurement moves. The check is one token placed where only you can place it, and there is no account and no password.