Local Edge AI on ESP32-S3 with ESPHome: YOLO11 Object and Face Detection Guide

Computer vision and object detection in smart home environments typically require dedicated local GPU servers (Frigate NVR, Coral TPU) or costly cloud AI subscriptions. However, the open-source s3_vision external component for ESPHome enables running YOLO11 (80 COCO classes), pedestrian, hand, and facial detection completely locally (on-device Edge AI) on a $5 ESP32-S3 board without any external server or cloud dependency.

If you are setting up your development toolchain, refer to our ESP32 Setup Guide, and explore our Home Assistant Architecture Guide for downstream smart home automation pipelines.

ESP32-S3 Vector Instructions and the ESP-DL Framework

The ESP32-S3 features dedicated Vector Instructions (SIMD) within its dual-core Tensilica Xtensa LX7 processor. These hardware accelerators compute neural network matrix multiplications in parallel, dramatically boosting dot-product operations per clock cycle.

Optimized using Espressif’s ESP-DL deep learning library, YOLO11 models are quantized to 8-bit integers (INT8/S8). The resulting coco_detect_yolo11n_320_s8_v3.espdl neural weights run directly in the microcontroller’s PSRAM, achieving real-time inference speeds.

Hardware Specifications and Verified Boards

Running complex neural networks on microcontrollers requires careful attention to memory bus bandwidth. Octal-SPI PSRAM modules deliver roughly twice the inference throughput of legacy Quad-SPI modules:

Verified Development Hardware: Freenove ESP32-S3 WROOM CAM, ESP32-S3-DevKitC-1 with OV2640 header, and Waveshare ESP32-S3-SIM7670G-4G boards.

Custom Partition Table Configuration

Because the ESP-DL runtime and quantized YOLO11 weights are bundled into the firmware binary, the application image reaches approximately 7 MB. To prevent flash allocation errors, place a custom partitions.csv file defining a 12 MB application partition in your project directory:

# Name,   Type, SubType, Offset,  Size,     Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0xC00000,
coredump, data, coredump,0xC10000,0x10000,

Step-by-Step ESPHome YAML Configuration

The YAML configuration below embeds the s3_vision component, binds the OV2640 sensor in RGB565 mode, and publishes detected objects and annotated bounding-box snapshots over MQTT to Home Assistant:

esphome:
  name: s3-ai-camera
  friendly_name: "ESP32-S3 Edge AI Camera"

esp32:
  board: esp32-s3-devkitc-1
  framework:
    type: esp-idf
  partitions: partitions.csv

external_components:
  - source:
      type: git
      url: https://github.com/youkorr/s3_vision
      ref: main
    components: [ vision ]

esp32_camera:
  name: "AI Camera Stream"
  id: my_camera
  resolution: 320x240
  pixel_format: RGB565
  frame_buffer_count: 1
  frame_buffer_location: PSRAM
  data_pins: [GPIO15, GPIO17, GPIO18, GPIO16, GPIO14, GPIO12, GPIO11, GPIO48]
  vsync_pin: GPIO38
  href_pin: GPIO47
  pixel_clock_pin: GPIO13
  external_clock:
    pin: GPIO10
    frequency: 20MHz
  i2c_pins:
    sda: GPIO40
    scl: GPIO39

vision:
  id: my_vision
  esp32_camera_id: my_camera
  model_family: coco_detect
  model_path: ./coco_detect_yolo11n_320_s8_v3.espdl
  score_threshold: 0.30
  detection_interval_ms: 200
  draw_boxes: true
  on_detection_image:
    - then:
        - mqtt.publish:
            topic: device/s3_ai_camera/camera/snapshot
            payload: !lambda 'return esphome::base64_encode(image.data, image.length);'
  on_object_detected:
    - then:
        - mqtt.publish:
            topic: device/s3_ai_camera/detection/state
            payload: !lambda |-
              auto dets = id(my_vision).get_detections();
              std::string out = "{\"count\":";
              out += std::to_string(dets.size());
              out += ",\"objects\":[";
              for (size_t i = 0; i < dets.size(); i++) {
                if (i) out += ",";
                out += "{\"class\":\"" + dets[i].category + "\",\"score\":" + std::to_string(dets[i].score) + "}";
              }
              out += "]}";
              return out;

Sources and References

• GitHub – youkorr/s3_vision: ESPHome Component for Edge AI Vision on ESP32-S3
• GitHub – espressif/esp-dl: Espressif Deep Learning Library
• Ultralytics – Official YOLO11 Architecture Documentation