Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,8 +172,8 @@ Versioning and Conventional Commits.
RTM-to-LCD indicator handling and device-service response parsing.
- Request CoreS3 DVP capture at most five times per second, holding capture buffers between
samples so idle periods stop DMA and copying. The sensor retains its 20 fps timing and JPEG
uplink remains capped at 1 fps. Increase the DVP DMA configuration to 16,384 bytes
(15 KiB allocated for QVGA YUYV), and reduce shared UI activity animation to 5 fps.
uplink remains capped at 1 fps. Use an 8,192-byte DVP DMA configuration
(7.5 KiB allocated for QVGA YUYV), and reduce shared UI activity animation to 5 fps.
- Count in-flight SPI callbacks across consecutive CoreS3 LVGL UI flushes so teardown waits
for all pending transfers.

Expand Down Expand Up @@ -224,6 +224,17 @@ Current build coverage, real-device results and unsupported features are tracked

### Fixed

- Move CoreS3's 7,168-byte LVGL task stack to PSRAM through the existing port's stack-capability
setting, preserving the internal display DMA stripe. Require external-stack support in the board
profile and log `ui_stack_bytes=7168 ui_stack=psram` during display initialization. This saves
7 KiB of internal stack storage; real-device audio/video/UI,
provisioning and volume-persistence NVS regressions remain to be verified.
- Reduce CoreS3's contiguous internal DMA buffer requirement from 15 KiB to 7.5 KiB with a
project Kconfig default override without changing vendored driver code. Log the configured DMA
bound and free/largest internal DMA memory before stream startup and when streaming does not
start. A device still failed with 13,187 bytes free but a largest block of 5,120 bytes; this
footprint reduction alone did not resolve the allocation failure. The smaller buffer increases
DMA events per active frame and requires concurrent audio/UI validation.
- Restrict batch-builder cleanup to dedicated, owned generated directories. Reject repository
ancestors, source directories, symlink components, traversal and overlapping roots before any
cleanup; preserve unmarked existing data and anchor child deletion to directory descriptors.
Expand Down
6 changes: 4 additions & 2 deletions VENDORED_SOURCES.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,10 @@ Firmware integration differences are limited to the active ESP32-S3 build:
- The video components include only the GC0308 DVP path, native I2C SCCB, and the ESP32-S3 JPEG
library. Local CMake/Kconfig replace registry metadata and pin component versions; sensor
detection uses the static table, the default format is QVGA YUYV, and SCCB timeout is 100 ms.
The local DVP DMA-size default is 16,384 bytes with video enabled; the driver's frame alignment
allocates 15 KiB of internal DMA memory for QVGA YUYV. The platform supplies capture buffers
With video enabled, `main/Kconfig.projbuild` overrides the component's 16,384-byte DVP DMA-size
default with 8,192 bytes; the vendored Kconfig and driver remain unchanged. The driver's frame
alignment allocates 7,680 bytes (7.5 KiB) of contiguous internal DMA memory for QVGA YUYV.
The platform supplies capture buffers
at most five times per second while retaining the sensor's 20 fps register configuration.
Unused transports, sensors, examples, and registry caches are omitted. Video sources and the
JPEG binary are linked only with `CONFIG_MYBOT_ENABLE_VIDEO=y`.
Expand Down
3 changes: 3 additions & 0 deletions components/mybot_platform/boards/m5stack-core-s3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@
DIN GPIO14、DOUT GPIO13;GPIO0 用于音频时钟,不作为配网按键。LCD SPI3 为 MOSI GPIO37、
SCLK GPIO36、CS GPIO3、DC GPIO35。

LVGL 对象与 7 KiB UI 任务栈使用 PSRAM,显示条带仍使用 10 KiB 内部 DMA 内存。
板级配置要求开启 `CONFIG_FREERTOS_TASK_CREATE_ALLOW_EXT_MEM`,为摄像头及音频保留内部内存。

无已保存的 Wi-Fi 信息时自动进入配网并显示热点名称、播放提示音。已有网络时联网后启动
SDK;长按触摸会先停止 SDK 再配网。设备音量保存在 NVS,重启后恢复。

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ set(MYBOT_BOARD_REQUIRED_CONFIGS
CONFIG_SPIRAM
CONFIG_SPIRAM_MODE_QUAD
CONFIG_SPIRAM_SPEED_80M
CONFIG_FREERTOS_TASK_CREATE_ALLOW_EXT_MEM
CONFIG_CODEC_ES7210_SUPPORT
CONFIG_CODEC_AW88298_SUPPORT
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ CONFIG_ESPTOOLPY_FLASHMODE_QIO=y
CONFIG_SPIRAM=y
CONFIG_SPIRAM_MODE_QUAD=y
CONFIG_SPIRAM_SPEED_80M=y
CONFIG_FREERTOS_TASK_CREATE_ALLOW_EXT_MEM=y

CONFIG_CODEC_ES7210_SUPPORT=y
CONFIG_CODEC_AW88298_SUPPORT=y
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

#include <mybot/platform/mybot_lcd.h>

#include "esp_heap_caps.h"
#include "esp_lvgl_port.h"
#include "esp_log.h"
#include "esp_timer.h"
Expand Down Expand Up @@ -240,6 +241,7 @@ int initialize(void **out_ctx) {
port_config.task_priority = 1;
port_config.task_affinity = 1;
port_config.task_stack = UI_STACK_BYTES;
port_config.task_stack_caps = MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT;
port_config.task_max_sleep_ms = 500;
port_config.timer_period_ms = 5;
lvgl_port_display_cfg_t config{};
Expand Down Expand Up @@ -302,9 +304,11 @@ int initialize(void **out_ctx) {
result = 0;
ESP_LOGI(TAG,
"event=lcd action=initialize backend=lvgl result=ok width=%d height=%d "
"dma_bytes=%u ui_core=1 ui_priority=1 objects=psram",
"dma_bytes=%u ui_core=1 ui_priority=1 ui_stack_bytes=%u ui_stack=psram "
"objects=psram",
MYBOT_DISPLAY_WIDTH, MYBOT_DISPLAY_HEIGHT,
(unsigned)(MYBOT_DISPLAY_WIDTH * ILI9342_LCD_TRANSFER_ROWS * sizeof(uint16_t)));
(unsigned)(MYBOT_DISPLAY_WIDTH * ILI9342_LCD_TRANSFER_ROWS * sizeof(uint16_t)),
(unsigned)UI_STACK_BYTES);
done:
if (result < 0) {
ESP_LOGE(TAG, "event=lcd action=initialize backend=lvgl result=error");
Expand Down
13 changes: 13 additions & 0 deletions components/mybot_platform/src/drivers/video/cores3_camera_video.c
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,17 @@ static bool wait_for_capture_slot(int64_t deadline_us) {
return false;
}

static void log_capture_memory(const char *stage) {
const uint32_t dma_caps = MALLOC_CAP_DMA | MALLOC_CAP_INTERNAL;
ESP_LOGI(TAG,
"event=video_memory stage=%s dma_buffer_max=%d dma_free=%zu dma_largest=%zu "
"internal_free=%zu psram_free=%zu",
stage, CONFIG_CAM_CTRL_DVP_DMA_BUFFER_SIZE, heap_caps_get_free_size(dma_caps),
heap_caps_get_largest_free_block(dma_caps),
heap_caps_get_free_size(MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT),
heap_caps_get_free_size(MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT));
}

static void video_worker(void *argument) {
(void)argument;
unsigned int captured = 0, encoded = 0, sent = 0, rejected = 0, dropped = 0, capture_errors = 0;
Expand All @@ -264,7 +275,9 @@ static void video_worker(void *argument) {
}
capture_pending = true;
const enum v4l2_buf_type type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
log_capture_memory("before_stream_on");
if (should_stop() || ioctl(s_video.fd, VIDIOC_STREAMON, &type) < 0) {
log_capture_memory("stream_on_not_started");
ESP_LOGW(TAG, "event=video action=stream_on result=not_started");
goto done;
}
Expand Down
17 changes: 13 additions & 4 deletions docs/CORES3_UI.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,10 +115,17 @@ saved configuration values. Build validation is recorded separately below.

## Rendering, resources and provenance

The existing LVGL task runs on Core 1 at priority 1. Objects and drawing scratch space use PSRAM;
SPI transfers retain a 320 × 16 RGB565 stripe, or 10,240 bytes of internal DMA memory. The UI task
stack and driver/synchronization allocations are additional memory. Partial redraws avoid a
full-screen page cache or decoded full-face buffer.
The existing LVGL task runs on Core 1 at priority 1. Its 7,168-byte stack now uses PSRAM through
the existing port's `task_stack_caps`, moving 7 KiB of stack storage out of internal RAM. The
CoreS3 profile requires `CONFIG_FREERTOS_TASK_CREATE_ALLOW_EXT_MEM=y`; board defaults enable it.
Objects and drawing scratch space also use PSRAM. SPI transfers retain a 320 × 16 RGB565 stripe,
or 10,240 bytes of internal DMA memory; driver/synchronization allocations are additional memory.
Partial redraws avoid a full-screen page cache or decoded full-face buffer.

The display initialization log includes `ui_stack_bytes=7168 ui_stack=psram`. Moving the stack
reduces internal memory use but does not guarantee the largest free DMA block required by the
camera. Check `event=video_memory` as described in [CoreS3 video](CORES3_VIDEO.md); total free
memory alone does not establish that streaming can start.

Fixed LVGL widgets and curves draw the face. Five immutable A8 masks contribute 13,631 alpha
bytes for hearts, stars, ellipses, cheeks and tears, excluding descriptor overhead. The terminal
Expand Down Expand Up @@ -152,6 +159,8 @@ input paths. They do not validate this new pixel renderer. Real-device testing r
- Test animation enabled/disabled, overflowing SSID/instructions, and web-console pairing.
- Regress short touch, three-second provisioning, Wi-Fi recovery and repeated SDK stop/start.
- Run full-duplex audio with video off and with the existing 1 fps camera uplink enabled.
- Validate the PSRAM UI stack with concurrent audio/video/UI and provisioning/volume-persistence
NVS writes; this real-device regression remains pending.
- Record CPU load, internal/PSRAM peaks, minimum free heap and largest blocks; repeat sessions
to check resource lifetime and sustained memory growth.

Expand Down
12 changes: 10 additions & 2 deletions docs/CORES3_UI.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,10 +100,17 @@ idf.py -B build/cores3-pixel -p <PORT> flash monitor

## 渲染、资源与来源

现有LVGL任务运行于Core 1、优先级1。对象与绘制临时空间使用PSRAM,SPI继续使用
320 × 16 RGB565条带,即10,240字节内部DMA内存;UI线程栈、驱动和同步资源另计。
现有LVGL任务运行于Core 1、优先级1。7,168字节任务栈现通过已有port的 `task_stack_caps`
放入PSRAM,减少7 KiB内部栈存储。CoreS3 profile要求
`CONFIG_FREERTOS_TASK_CREATE_ALLOW_EXT_MEM=y`,板级defaults已开启。
对象与绘制临时空间也使用PSRAM,SPI继续使用320 × 16 RGB565条带,即10,240字节
内部DMA内存;驱动和同步资源另计。
局部刷新不增加全屏页面缓存或整脸解码缓冲。

显示初始化日志包含 `ui_stack_bytes=7168 ui_stack=psram`。搬栈减少内部内存占用,不能保证
摄像头所需的最大DMA连续块;需结合[视频说明](CORES3_VIDEO.zh-CN.md)中的
`event=video_memory`检查,不能只凭空闲内存总量判断采集能够启动。

固定LVGL控件和曲线绘制表情,五个不可变A8蒙版保存心形、星形、椭圆、脸颊与眼泪,
alpha数据合计13,631字节,不含描述符。终端字库由DejaVu Sans Mono Bold派生并重命名,
保留Bitstream Vera许可;思源黑体UI子集仍为SIL OFL 1.1。蒙版与终端字库仅编入CoreS3
Expand Down Expand Up @@ -132,6 +139,7 @@ zh-CN耗时38秒、en-US耗时37秒,总计1分16秒。这是构建验证,不
- 开关动画,检查超宽SSID/提示滚动及网页控制台配对。
- 回归短触、三秒配网、断网恢复和反复SDK启停。
- 视频关闭及原1 fps摄像头上行开启时分别验证全双工音频。
- PSRAM UI栈改动后的音频、视频与UI并行,以及配网、音量持久化NVS写入回归仍待真机验证。
- 记录CPU、内部/PSRAM峰值、最小空闲堆及最大连续块,多轮会话观察资源释放和内存增长。

结果需包含板型、固件语言、视频/动画配置、音量及串口日志。
56 changes: 43 additions & 13 deletions docs/CORES3_VIDEO.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,10 @@ Keep the audio packet duration at the supported 60 ms. A fresh build uses Chines
`;ci/en-us.defaults` to the defaults list in a new build directory for English. All CoreS3 builds
use LVGL; no separate UI enable flag is needed.
When reusing a build directory, run `idf.py -B build/cores3-video reconfigure build` so changes
to the component's hidden DMA default are regenerated. Confirm
`CONFIG_CAM_CTRL_DVP_DMA_BUFFER_SIZE=16384` in that build's generated `sdkconfig`.
to the project's hidden DMA default are regenerated. Confirm
`CONFIG_CAM_CTRL_DVP_DMA_BUFFER_SIZE=8192` in that build's generated `sdkconfig`.
The CoreS3 profile also requires `CONFIG_FREERTOS_TASK_CREATE_ALLOW_EXT_MEM=y` for the PSRAM UI
stack; board defaults enable it. If an existing configuration saved `n`, enable it in menuconfig.

## Hardware and memory

Expand All @@ -48,16 +50,35 @@ to the component's hidden DMA default are regenerated. Confirm
| Reset | AW9523 P1_0; locked masked writes preserve LCD and audio reset lines |

Two 153,600-byte capture buffers and one 131,072-byte JPEG buffer use about 428 KiB of PSRAM.
The encoder also allocates workspace. With video enabled, the current
[`CAM_CTRL_DVP_DMA_BUFFER_SIZE`](../components/esp_cam_sensor/Kconfig) default is 16,384 bytes.
For 320×240 YUYV, the DVP driver rounds this to two 7,680-byte halves, allocating 15 KiB of
internal DMA memory, plus descriptors and a 3-KiB capture-task stack. These figures use the current
defaults; check the generated DMA configuration when comparing older firmware. The encoding worker
has a 6-KiB stack. Check internal free memory
and the largest free block during a call, not only total PSRAM. LVGL objects use PSRAM and the UI
has a separate 10 KiB internal DMA buffer; it does not allocate a full-screen page cache.
The DMA increase consumes 7.5 KiB more internal RAM than the previous configuration, excluding
descriptors. Both PSRAM frame buffers remain allocated and rotate through the single queued slot.
The encoder also allocates workspace. With video enabled,
[`main/Kconfig.projbuild`](../main/Kconfig.projbuild) sets the hidden
`CAM_CTRL_DVP_DMA_BUFFER_SIZE` default to 8,192 bytes before the vendored component's default is
loaded. The vendored Kconfig and driver remain unchanged. For 320×240 YUYV, the driver chooses
two 3,840-byte halves that divide the 153,600-byte frame, allocating 7,680 bytes (7.5 KiB) of
contiguous internal DMA memory, plus descriptors and a 3-KiB capture-task stack. The earlier
16,384-byte setting allocated 15,360 bytes (15 KiB). Check the generated DMA configuration when
comparing firmware. The encoding worker has a 6-KiB stack. Both PSRAM frame buffers remain
allocated and rotate through the single queued slot.

`no mem for CAM DVP DMA receive buffer` means the driver could not allocate its contiguous
internal DMA buffer when streaming starts, after RTC has connected. Free PSRAM cannot satisfy
this allocation; total free internal memory can also exceed the request while its largest DMA
block is too small. Descriptors, task stacks and other runtime allocations still need additional
memory. LVGL objects use PSRAM and the UI has a separate 10 KiB internal DMA buffer; it does not
allocate a full-screen page cache.

Before moving the UI stack, a device log still reported allocation failure with
`dma_free=13187` and `dma_largest=5120`: the largest block was below the 7,680-byte requirement.
The CoreS3 adapter now places its 7,168-byte LVGL task stack in PSRAM through the existing port's
`task_stack_caps`, moving 7 KiB of stack storage out of internal RAM. The display DMA stripe stays
internal. This change does not statically guarantee a sufficient contiguous block or successful
camera startup; check the new device logs.

The smaller buffer produces about 40 DMA data events per active frame instead of 20 and leaves
less time to service each half before reuse. The sensor's 20 fps timing during capture is unchanged
by the 5 fps request or 1 fps upload limits. Validate capture errors, timeouts and audio continuity
on a device with RTC audio and the UI active; this memory reduction alone does not establish that
the reported hardware failure is resolved.

The JPEG worker uses priority 3 without fixed core affinity and no additional encoder helper task.
The DVP copy task retains the upstream driver's higher priority for timely DMA service.
Expand Down Expand Up @@ -87,6 +108,14 @@ Current build and hardware status is tracked only in the
[validation matrix](VALIDATION.md). The following checks remain the device and server acceptance
procedure.

The display initialization log includes `ui_stack_bytes=7168 ui_stack=psram`.
`event=video_memory` records `stage=before_stream_on` and, if streaming does not start,
`stage=stream_on_not_started`. Its fields are `dma_buffer_max` (the configured upper bound,
8,192 bytes by default), `dma_free`, `dma_largest`, `internal_free`, and `psram_free`, all in bytes.
`dma_free` and `dma_largest` use the driver's internal DMA memory capabilities. Compare the
pre-start `dma_largest` with the actual 7,680-byte buffer requirement; sufficient space for this
buffer does not guarantee enough memory for the rest of stream startup.

`cores3_video` emits lifecycle logs and `event=video_stats` every 10 seconds during streaming:
`window_ms`, `captured`, `encoded`, `sent`, `rejected`, `dropped`, `capture_errors`, `last_bytes`, `quality`, `target_bps`,
and `encode_max_us`. Counters and `encode_max_us` reset each interval; `last_bytes`, `quality`, and
Expand All @@ -100,6 +129,7 @@ Validate GC0308 detection, correct colors/orientation, server reception/inferenc
JPEG quality, CPU/heap use, and audio continuity. Exercise repeated conversation start/stop,
Wi-Fi loss, long-touch provisioning, bitrate reduction, and missing camera. Confirm no frames after
stop and stable memory after repeated cycles. Hardware and server acceptance must be recorded
separately. Compare `captured/window_ms`,
separately. Validation of the PSRAM UI stack with concurrent video, full-duplex audio and UI,
including provisioning and volume-persistence NVS writes, remains pending. Compare `captured/window_ms`,
per-core CPU, internal free heap, and the largest free block while keeping the same scene and
audio activity. The expected reduction in DMA traffic is not a measured CPU reduction.
Loading
Loading