> **Mellifera** > A shipped voice-first beekeeping record system, and the ML design study built on top of it: treatment timing, winter survival, and feeding models specified but not yet trained. > > Source: https://jasonstiltner.com/projects/mellifera/ --- *Part of my research on [coordination without collapse](https://jasonstiltner.com/corpus/): how does a beekeeper coordinate with a system they can't directly observe?* # Mellifera: Voice-First Beekeeping Records, and an ML Design Study A working MERN record system beekeepers can use with their gloves on — plus the decision-support models it was built to feed, specified in detail and not yet trained Updated Sep 8, 2026 ## What is built, and what is not ### Running in the repository - Bidirectional voice capture and TTS playback in the field - LLM extraction of structured observations from speech, via the OpenAI chat API - MongoDB domain model — apiaries, hives, boxes, inspections, treatments, feedings, queens - REST API with Swagger docs, JWT auth, Google OAuth, and a guest mode - React/Vite client, Docker and Kubernetes manifests ### Designed but not implemented - All three models below. No architecture is trained, and no weights exist. - The prediction endpoint is wired end to end, but `mlModel.js` is a stub whose `predict()` returns `Math.random()`. - TensorFlow.js is a declared dependency, not a used one. Nothing runs inference. - No labelled dataset has been assembled, so nothing has been measured. Voice-First UX · MERN + LLM Extraction · Feature Design · Untrained Models [View on GitHub](https://github.com/jstiltner/Mellifera-app) · [🐝 Jay's Bees](https://jaysbees.com) ## The ML Problem, and Three Designs Beekeeping has the shape of a hard decision-support problem: sparse, noisy observations, since inspections run one to two weeks apart; delayed feedback, since winter survival is not validated for three to six months; and high-stakes calls, since mistimed treatment kills a colony. Add class imbalance, because most colonies survive; temporal dependence, because treatment history matters; and geospatial variation, because climate moves the optimal timing. The structure mirrors clinical decision support, which is what drew me to it. Three models were specified against that structure. **None are trained**, no labelled dataset has been assembled, and no accuracy figures exist for any of them — see [Limitations](#limitations). - **Mite treatment timing.** Binary treat-now/wait over cyclical day-of-year encoding, sin/cos latitude-longitude plus elevation, and a mite-count trajectory weighted by sample size. The geospatial encoding exists to test one hypothesis: that a model can learn regional timing — which shifts two to four weeks between south Texas and north Minnesota — without explicit climate-zone labels. - **Winter survival risk.** Calibrated survival probability with bootstrap intervals, evaluated by temporal cross-validation rather than a random split, because a random split over seasonal data leaks. Uncertainty sits in the output shape deliberately: a fall prediction should not claim precision about a spring outcome. - **Feeding recommendation.** A multi-output head over a shared encoder — feed type as classification, amount as regression. This is the one worth attempting first: a colony's response is visible within days, which makes labels far cheaper than the other two. **Why the problem is worth the trouble:** honeybees pollinate roughly 35% of global food crops, and North American beekeepers lose 30-40% of colonies annually. Aiming intervention at the colonies most likely to fail is where a working model would pay for itself. ## Voice-First Data Collection (built) This is the part that exists. Nothing above can be trained until there is a labelled record of what beekeepers actually observed, and the reason that record does not exist for most operations is that inspections happen in gloves, in a veil, in direct sun. Mellifera's answer is a bidirectional voice interface: beekeepers speak observations while working, an LLM turns the transcript into structured fields, and the system speaks confirmations back. Getting this right is the prerequisite for the models, which is why it was built first. #### Traditional Data Entry - Remove gloves to use phone - Risk stings on exposed hands - Squint at screen in sunlight - **~8 minutes per hive** #### Voice with Mellifera - Keep gloves on, stay protected - Speak naturally while working - TTS speaks confirmations back - **Fully hands-free** Voice Command Processing LLM-powered NLU for structured data extraction 27 linesJavascriptExpress.js View code 27 lines · Javascript · Express.js Copy ```javascript // Voice Command Processing with LLM-powered NLU router.post('/voice-command', auth, async (req, res) => { const { audioTranscript, context } = req.body; const systemPrompt = `Extract structured observations from voice notes. Context: Apiary at (${context.lat}, ${context.lon}), Hive: ${context.hiveName} Extract if mentioned: - Mite count and sample size - Population estimate (frames of bees) - Brood pattern, honey stores, queen status - Treatments applied, feeding done Return JSON with extracted fields and confidence scores.`; const extraction = await llm.complete({ system: systemPrompt, user: audioTranscript, responseFormat: 'json' }); // The prediction call is wired, but the model behind it is a stub — // see server/mlModel.js, whose predict() returns Math.random(). const predictions = await runMLPredictions(context.hiveId); res.json({ extraction, predictions }); }); ``` ## System Architecture Full-stack MERN application with voice interface, LLM-powered NLU, and a prediction endpoint. Containerized with Docker and deployable to Kubernetes. The prediction service in the diagram is a real route with a placeholder behind it — the interface is settled, the model is not. [Diagram: Architecture diagram: voice-driven beekeeping application stack, layered from the beekeeper in the field through Web Speech API voice capture and AWS Polly text-to-speech, a React frontend, an Express.js API, a model-agnostic LLM and ML-ready schema, MongoDB persistence, and Kubernetes orchestration] ## Data Model & API MongoDB schema with full REST API documented via Swagger. The data model captures the natural hierarchy of beekeeping operations. #### Core Entities - **Users** - Auth, preferences, OAuth - **Apiaries** - Locations, metadata - **Hives** - Individual colony tracking - **Boxes** - Hive body components #### Activity Records - **Inspections** - Observations, scores - **Treatments** - Mite control, medications - **Feedings** - Supplemental nutrition - **Queens** - Lineage, performance **API Design:** Full REST API with Swagger documentation. JWT authentication with Google OAuth support. Specialized endpoints for voice commands, ML predictions, and aggregate reporting. ## Designed for Real Field Conditions Most software assumes a user sitting at a desk with keyboard and mouse. Beekeepers work in protective suits with thick leather gloves, mesh veils obscuring their vision, surrounded by thousands of stinging insects, often in direct sunlight that makes screens unreadable. #### Traditional Data Entry - Remove gloves to use phone - Risk stings on exposed hands - Squint at screen in sunlight - Navigate complex forms - **~8 minutes per hive** #### Bidirectional Voice with Mellifera - Keep gloves on, stay protected - Speak naturally while working - **TTS speaks responses back** - LLM extracts structured data - Fully hands-free operation **The UX insight:** Voice interfaces aren't just convenient—they're sometimes the *only* viable interface. This project taught me to start with user constraints, not technology capabilities. The same principle applies to clinical settings: surgeons can't touch keyboards mid-procedure, nurses have hands full with patients. ## Limitations The three designs above are a specification, not a report. What separates them from a working system is listed here rather than left for a reader to discover in the source. #### No trained model exists `server/mlModel.js` defines the predictor's interface — `createModel`, `trainModel`, `predict`, `saveModel`, `loadModel` — and every method is a stub that logs and returns. `predict()` returns `Math.random()`. There are no weights in the repository and no training script. #### No numbers, because nothing was measured An earlier version of this page reported accuracy, AUC and model-size figures. Those were targets written as if they were results, and they have been removed. Any metric that reappears here should come with the run that produced it. #### No labelled dataset The winter survival model needs multiple seasons of paired fall inspections and spring outcomes across varied latitudes. The voice capture layer is what makes collecting that plausible, and it has not been running long enough or widely enough to have produced it. #### TensorFlow.js is unused It is declared in `package.json` and referenced only in a test mock. Nothing loads a graph or runs inference, in the browser or on the server. The edge-inference story described here is a plan for how the models would ship, not something a reader can observe. **Why publish the design anyway:** the feature engineering is the part that took domain knowledge — cyclical date encoding so December sits next to January, sample-size weighting on mite counts, geospatial encoding as a proxy for climate zone, temporal cross-validation because the label arrives six months late. Those choices are legible and criticizable without a trained model. Presenting them as shipped classifiers was not. ## Explore the Project [View on GitHub Full source code, voice components, NLU](https://github.com/jstiltner/Mellifera-app) · [See the Healthcare Connection Document Understanding](https://jasonstiltner.com/projects/document-understanding/) Interested in voice-first interfaces or agricultural tech? [Email me](mailto:jason@jasonstiltner.com) or connect on [LinkedIn](https://linkedin.com/in/jasonlstiltner)