04 · COOK CONSUMER UTILITY · LOCAL FIRST
Grocs, A Local First Recipe and Pantry Companion
A private kitchen companion that turns scattered recipes into meals users can cook with what they already have.
- Role
- Product, architecture and engineering, end to end
- Platform
- iOS + Android · Expo / React Native · SQLite
The problem
Recipes live in the messiest corners of the internet, links, captions, screenshots, videos, and 'just save it' apps quietly ship everything to a server.
My contribution
The whole product: a deterministic first extraction pipeline, a human review state machine, on device AI, SQLite full text search over a bundled index, and pantry aware meal matching, with no account and no backend.
Core stack
- Expo
- React Native
- TypeScript
- SQLite + FTS5
- ML Kit OCR
- whisper.rn
Proof it works
The shipped build captures from five source types, enforces review before save, ships a 2,500-record starter index (full 168k index installable), and matches recipes against the pantry, entirely on device.
What it is
Context
Grocs is a kitchen companion that answers two questions: where do my scattered recipes go, and what can I cook right now with what I already have?
It is local first as a hard rule, stated in the app itself: no account, no sync server, no ads, no telemetry. The network is touched in exactly three user initiated cases, fetching a page you pasted, downloading an optional model, opening a source link.
What made it hard
The challenge
Recipe sources are hostile input. Some pages carry clean schema.org data; most are a soup of markup, captions, login walls and video descriptions. A pipeline that pretends otherwise either fabricates recipes or fails silently.
The constraint that shaped everything: keep the honesty of the pipeline visible. Every uncertain extraction must say so, land in review, and never reach the library without a human pass, while the whole system runs offline on a phone.
The experience
A walk through the product
01
The promise is the privacy model: the first screen commits to no ads and user owned recipes before asking anything.
02
Local first is explained in plain words during onboarding, an architecture decision presented as a user promise.
03
AI is opt in and on device: models are optional downloads, removable any time, and arrive with an honest mistakes warning.
04
Five ways in, one rule out: every capture route feeds the same reviewed pipeline, including 'write it myself, no AI involved'.
05
Your library and the community index stay visibly separate: index records keep their source and never pretend to be full recipes.
06
The pantry is the other half of the product, what you have drives what Grocs suggests you cook.
07
The gap between a recipe and your pantry becomes a categorised shopping list in one tap.
What I built
The work, lane by lane
Extraction pipeline
- A best-evidence-first strategy chain: URL normalisation → JSON-LD Recipe → Microdata → YouTube description → public social captions → readability heuristics
- Honest failure modes with their own copy, login walls route to paste/screenshot; bare video links route to on device capture
- A capture-job state machine: queued → processing → needs_review → saved, with failed → queued on retry
On device AI
- ML Kit OCR for screenshots and video keyframes, feeding a rule based structurer
- whisper.rn transcription wired for audio files; models managed with size and free space checks, resumable installs and one tap removal
- A SmolVLM2 vision adapter with a validated output contract, deliberately shipped as adapter only, and described that way
Data & search
- SQLite with migrations and typed repositories; FTS5 full text search with a fallback path
- A bundled 2,500-record starter index; the optional full index (168,442 records) installs from Settings and removes in one tap
- Every index record keeps its source name, URL and licence, attribution is seeded into the database itself
Matching & lists
- Canonicalised ingredients matched pantry-to-recipe with a weighted score: match 0.45 · uses-expiring 0.25 · missing-count 0.15 · time 0.10 · preference 0.05
- Staples (salt, pepper, water, oil, sugar) never block 'can cook now'
- Missing ingredients become a categorised shopping list in one tap; backup is a single user owned JSON export
Design system & quality
- A three layer token architecture (primitives → semantic → component) with six switchable palettes, light and dark
- A test suite that enforces token contrast and pipeline behaviour, including that nothing reaches 'saved' without review
Engineering decisions
Decisions and their trade-offs
Real decisions cost something. Each of these names its price.
01Deterministic first, AI second
Structured data (JSON-LD, Microdata) is always preferred; OCR and heuristics only run when evidence is weaker, and their output is flagged.
Trade-offMore strategies to maintain than one model call, but clean sources stay perfect, and AI mistakes are contained to where AI actually ran.
02A human gate the code enforces
The needs_review state is not a suggestion: the transition to saved is only reachable through the review screen, and tests pin that behaviour.
Trade-offOne extra screen on every AI touched capture. Chosen deliberately, a wrong temperature in a recipe is a real world failure, not a UI bug.
03Ship a small index, make the big one optional
2,500 curated records bundle into the app; the 532 MB full index is a user initiated install with one tap removal that never touches personal recipes.
Trade-offPower users take one extra step, in exchange, the app stays a small download and the user decides what half a gigabyte of storage is worth.
04State the privacy model inside the product
Local first is written into onboarding and Settings in plain language, including the three exact cases where the network is touched.
Trade-offThe copy makes promises the architecture must keep forever, which is precisely the discipline a privacy product needs.
05Adapters over vapourware
The SmolVLM2 vision provider ships as an adapter with a validated output contract and written wiring docs, and the app never pretends the capability is live.
Trade-offA less impressive feature list today, in exchange for a codebase and a marketing story that agree with each other.
Under the hood
How the system holds together
Every input, link, text, image, video, share sheet, becomes a capture job in a four state machine, and only the review screen can move a job to saved. Around that core sit the deterministic parsers, the optional on device models, the SQLite/FTS5 store and the pantry matcher.
Deeper technical notes
- URL capture normalises first (youtu.be, Shorts and mobile links become canonical; tracking parameters are stripped) so downstream strategies see one shape of URL.
- YouTube extraction lifts the video description from the watch page's player response, the 'recipe in the description' norm, with oEmbed as a title/author fallback.
- The matcher canonicalises both pantry items and recipe ingredients before scoring, so 'scallions' and 'spring onions' meet in the middle.
- The icon system is generated: in app icons are produced from master sheets by repo tooling, and unknown icon names resolve to a safe fallback that logs once instead of crashing.
- Known limits are documented, not hidden: native OCR/ASR/share-sheet need a dev build rather than Expo Go; remote video streams are not downloaded or transcribed; walled gardens are not logged into.
Gallery
Every screen, up close
Real screenshots from the shipped build, open any of them for a closer look.
Quality, privacy, accessibility
Evidence of care
- Contrast is enforced by tests over the token layer, not by eyeballing, palettes cannot ship a failing pair.
- The review gate is pinned by tests: no capture path can write to the library without the human pass.
- Attribution is architectural: OpenRecipes records (CC BY 3.0) carry their source and licence in the database and in a Settings screen.
- AI honesty is product copy: 'AI capture can make mistakes. Review ingredients, quantities, temperatures, and cooking times before saving or cooking.'
- 48 dp touch targets and an 8-pt spacing grid are design system law in the app.
Where it stands
Outcome and current proof
- The shipped build demonstrates the full loop: five capture routes, review gated saving, index search, pantry, shopping list, themes, backup and attribution.
- The documented repository carries the state machine, the parser chain, the model manager and the token architecture exactly as described here.
- Store availability and usage are not claimed; SmolVLM2 is stated as adapter level, and remote video/audio extraction is stated as out of scope.
Technologies
Chosen for reasons
- Expo + React Native + TypeScript
- one typed codebase, with the native modules (OCR, whisper, share intent) behind a dev build
- SQLite + FTS5
- real relational storage and full text search that works on a phone with no server
- ML Kit OCR
- on device text recognition for screenshots and video keyframes
- whisper.rn
- on device speech to text for audio files, as an optional model
- Deterministic parsers
- JSON-LD, Microdata and heuristic extractors that keep AI out of the easy cases
- expo-share-intent
- capture from any app's share sheet straight into the pipeline
For your project
What this experience enables
- Local first and privacy conscious products where 'no server' is a feature, stated and kept
- Content extraction from messy real world sources with honest fallbacks
- On device AI done responsibly: optional models, visible warnings, human review gates
- Serious client side data work, SQLite schemas, migrations, FTS5 search, bundled datasets
- Design systems with enforced accessibility, not aspirational guidelines