As a former wearables designer I would go for something like radxa/orange (for exapmle rk3588 with built-in TPU for local inference) . Add off-the-shelf peripherals as needed, prototype and iterate on this open source hw/sw stack, than migrate to your own hw/pcb design utilizing same chips.