Machine learning · Tutorial
Give Watcher a face with the model zoo and inference engines
Make a SenseCAP Watcher notice a face, look toward it, and respond with happy eyes and a chirp. The downloadable himax.py example connects a native inference pipeline to SolarOS's emotion renderer. This tutorial follows its data path, from installing a model to reacting to detections in Python.
Watcher has two places to run models. Choosing a model in Zoo determines which engine will execute it:
| Engine | Runs on | Model lifecycle | Image sources |
|---|---|---|---|
| Himax / SSCMA | Watcher's HX6538 and its Ethos-U55 accelerator | Download a bundle, program a hardware slot, select the slot | Its attached camera only |
| ESP-DL | ESP32-S3 | Download a bundle and load it as a resident model | Compatible camera streams, image files, and JPEG RTSP sources |
The Himax face example uses the first path. The final section shows how the same pipeline service also drives ESP-DL. Model formats, slot numbers, and resident model handles belong to their respective engines and cannot be interchanged.
Prepare Watcher
Use the current SolarOS full image for seeed_sensecap_watcher. It includes the Himax processor provider, pipelines, Python, and the emotion renderer. Use the Watcher display shell with a paired BLE keyboard, or an optional UART shell for administration. Hold the wheel button to begin keyboard pairing. The bottom USB-C connector exposes separate ESP32 and Himax serial interfaces; SolarOS commands belong on the ESP32 interface at 115200 baud. For that optional console, create it with session create shell uart0 from the display shell. The Watcher board guide covers connector identification and the USB-to-UART bridge's reset-line settings.
In the SolarOS shell, connect Wi-Fi and inspect the devices:
wifi connect "YOUR_SSID" "YOUR_PASSWORD"
wifi status
expansion devices
stream list
pipeline info himax
pipeline info himax camera0 version
model slots himax camera0
camera0 should already be attached. Stop other camera consumers, such as Photo, rtspd, cam-webd, or an earlier pipeline, before querying slots, programming, or running this example. Camera acquisition is exclusive.
This tutorial uses mounted, writable internal flash storage. Zoo stores the downloaded files there, and the Himax programmer also needs internal flash for its per-chip slot inventory. SD storage is an alternative for bundles and the Python script; use the paths actually reported by Zoo.
Download a face model from Zoo
Run:
zoo storage flash
zoo refresh
zoo
In Zoo, expand hx6538 (himax) and inspect himax-face-detect-96, version 1.0.0. Its result is a face detector with SSCMA boxes. Inspect the declared backend, task, labels, license, and compatibility before installing it. Install the model and exit Zoo. The equivalent direct install is:
zoo install himax-face-detect-96 1.0.0
zoo installed
With the storage selection above, its manifest is:
/flash/dl/zoo/models/himax-face-detect-96/1.0.0/bundle.json
If your saved Zoo source still uses the older ESP-DL-only catalog, run zoo source reset, then zoo refresh to use the current mixed catalog. The 192-pixel face detector, himax-face-detect-192, is also recognized by the example script, but install the 96-pixel version for the steps below.
Installing a bundle downloads and checks files. It has not yet programmed the accelerator. Removing those files later will not erase a programmed Himax slot.
Program an inactive Himax slot
Inspect the live inventory again:
model slots himax camera0
Note the active slot and choose an inactive slot whose contents you intend to replace. The example below uses slot 4. If 4 is active, choose another inactive slot or first select another known-working programmed model. All four slots can be replaced while inactive; unknown means the inventory has no recorded identity, rather than proving the slot is empty.
Run the upload command on one line:
model upload himax camera0 /flash/dl/zoo/models/himax-face-detect-96/1.0.0/bundle.json 4 --replace
job status model-upload
--replace authorizes overwriting that slot, including unknown contents. Keep power connected and poll job status model-upload until it reports successful completion. Upload runs as a background job; returning to the prompt is not completion. Do not start the camera demo while it is running.
The programmer checks the bundle and the chip's supported slot layout, writes the model, selects it for one inference, then restores the previously active selection. A successful inventory entry has state programmed and records the model's ID. This recorded identity is what himax.py uses to find a face detector.
Inspect the result and try one inference:
model slots himax camera0
model select himax camera0 4
pipeline start himax camera0 1
The start command prints a pipeline ID. Substitute that value below, and wait for its status to report completion before reading the result:
pipeline status <PIPELINE_ID>
pipeline result <PIPELINE_ID>
pipeline destroy <PIPELINE_ID>
Look for boxes, classes, and perf in the processor's result. With a face in view, this detector should report face boxes; an empty list is a valid result when none is detected. Selection persists after destruction. If you want the demo to restore your original model on exit, select the original slot again before starting it.
See Himax model management for firmware/layout requirements, cancellation, and slot-inventory details.
Download and run himax.py
Download the complete himax.py. Save it in a directory on your desktop, then serve that directory on your local network:
python3 -m http.server 8000 --bind 0.0.0.0 --directory <DOWNLOAD_DIRECTORY>
Replace the directory placeholder with the directory containing himax.py. On Watcher, use your desktop's reachable LAN address:
curl -o /flash/himax.py http://<DESKTOP_IP>:8000/himax.py
python /flash/himax.py 60
Run the script from the display shell to put its eyes on Watcher's panel. The optional argument 60 stops it after about a minute. Omit it to keep running. Face the camera: after two qualifying detection results, the eyes become happy. They follow the face while inference is responsive, and return to bored after about 1.6 seconds without a detection.
Press the wheel button, q, or Esc to stop normally. The button must retain its default short-press Enter action. Stop the desktop HTTP server with Ctrl+C after transferring the file.
Follow the script's data path
The example imports native services rather than implementing image decoding, model execution, or rendering in Python:
from solaros import camera, emotion, pipeline, time, tui
First, pipeline.info("himax", "camera0", "slots") retrieves the slot inventory. face_slot() looks for a programmed record named himax-face-detect-96 or himax-face-detect-192. It saves the active slot, then starts this pipeline:
detector = pipeline.start(SOURCE, "himax", slot, 0, 100)
The positional arguments mean source, processor, model slot, result limit, and interval in milliseconds. Limit zero keeps a stream running. The 100 ms interval is a delay between requests, not a promised ten frames per second; capture, processing, and transport also take time.
Himax captures and processes its own camera image. The pipeline retains a matching JPEG and structured result. Python reads the newest record using a sequence cursor:
record = pipeline.result(detector, sequence)
if record is not None:
sequence = record["sequence"]
result = record["result"]
boxes = [b for b in result.get("boxes", ())
if len(b) >= 6 and b[4] >= SCORE and b[5] == 0]
For this face model, each box provides center x/y, width, height, score, and class ID. The script accepts class 0 with a score of at least 60, chooses the largest box, and requires two consecutive qualifying results before declaring presence. The result reader can skip frames: it receives the newest completed result, rather than draining a frame queue. Other models can have different labels and output structures; do not reuse this filter blindly.
emotion.open(sound=True, fps=20, motion=True, fullscreen=True) creates the animated face. A persistent face.set("happy", ...) keeps the happy expression while detections continue. face.react(..., sound="chirp") adds a temporary reaction, with at least eight seconds between sounds. A 1600 ms loss interval prevents a single missed frame from immediately switching back to bored.
The script smooths gaze changes and mirrors the horizontal direction with MIRROR_X. It suppresses tracking for slow results, rather than having the eyes chase an old position. SCORE, LOST_MS, and SOUND_GAP_MS near the top are useful first parameters to adjust for your application.
The finally block destroys the pipeline, closes the emotion renderer, and restores the previous model selection when it differs. Pipelines are OS-owned and can outlive a Python session, so explicitly releasing them matters. After exit, inspect pipeline list; this demo's detector should no longer be listed. If an interrupted older firmware leaves a record behind, inspect its ID and use pipeline destroy <PIPELINE_ID> before trying again.
Try the ESP-DL engine
Stop himax.py first so it releases the camera. ESP-DL runs on the ESP32-S3 and loads a model into native memory instead of writing a Himax slot. Inspect the engine:
model backend
Standard full firmware uses the downloadable ESP-DL library. If the engine is absent, install it from the signed package catalog, then inspect it again:
pkg install espdl
model backend
zoo install pedestrian-pico 1.0.0
zoo installed
model bundle /flash/dl/zoo/models/pedestrian-pico/1.0.0/bundle.json
Use the bundle path reported by your Zoo installation if it differs. The last command returns a resident model handle. Substitute it below:
pipeline start model camera0 <MODEL_HANDLE> 1
pipeline status <PIPELINE_ID>
pipeline result <PIPELINE_ID>
Use the new pipeline ID returned by start, and wait for completion before reading the result. Here SolarOS captures a camera image, decodes it, applies the bundle's image preprocessing, and runs the ESP-DL model on the ESP32-S3. The pedestrian bundle returns adapted detections with labels and scores; it does not return the face demo's SSCMA boxes structure.
Release the pipeline before unloading the model it references:
pipeline destroy <PIPELINE_ID>
model unload <MODEL_HANDLE>
Both engines use the same pipeline lifecycle, but have different execution locations, input contracts, and result adapters. Use the pipeline guide for Python detection consumers and the inference API for direct ESP-DL tensor access.
If setup fails, check the boundary involved: Zoo needs a reachable catalog and storage; upload needs a supported accelerator and an inactive slot; inference needs a free camera and working model; animation needs display and audio resources. mem and mem policy help distinguish internal-memory admission failures from free PSRAM. Model compatibility and successful execution do not establish detection accuracy for your lighting and scene.