ESP32 PSRAM Memory Exception

[beckerzito]

Build environment: macOS
Target device: ESP32

Description
I have been working in an application with Main Thread + Worker (almost 50k bytes allocated for each machine) in the ESP32 SoC. With the increase of the application, we saw the need to start using an external PSRAM. To configure that, we have followed the espressif documentation on how to use the PSRAM.

Also, I have found in this repo, some issues raised when trying to use moddable in this setup, mainly related to the DEBUGGER SPEED, where I also had to decrease it to 115200 to be able to communicate with xsBug.

However, since we enabled the external RAM, the application got unstable. First of all, in debug mode, the WIFI is never able to connect (always getting event 5 - DISCONNECTED). In instrumented mode, I'm able to connect to the WIFI, however, Secure Socket connections get closed with -2 status.

Despite all that, what is really concerning me, is some memory exceptions being thrown randomly. Sometimes right after boot up, sometimes after hours. Those exceptions are (LoadProhibited or StoreProhibited) when trying to read or write to an invalid address. The weird part is that the crashes always happen in Moddable code (usually in the fxSeep function - see screenshot below).

Steps to Reproduce
Couldn't find a specific way to reproduce it. Sometimes the problem happens right after the bootup, sometimes it takes hours.

Expected behavior
What you expect to happen

Images

Other information

• Our current SoC revision is 1. We know that this revision used to have problems with Cache access when using PSRAM. But it seems that Espressif was able to work around all the reported issues through software. (I have tested lots of software that were used to reproduce the problems in the past, and all of them are working fine).
• I configured the sdk in such a way that both machines are now allocated in the external memory.

My question is: Have you seen similar behavior in this configuration? I'm not totally convinced that this is a moddable issue (maybe an integration problem between moddable and how Espressif handles PSRAM?). But what is bugging me is that all the crashes I have seen so far were always in a moddable code.

[phoddie]

Configuring PSRAM on the ESP32 has long been challenging to get right. When it is working correctly, it behaves like ordinary memory and the Moddable SDK is able to use it without reliability issues. A good example of this is the M5 Paper port.

[beckerzito]

Hi @phoddie

Thanks for the feedback.
Just have a final question. I see on the M5 Paper that it is requiring ESP32 revision 3 as the minimum revision. I know that in this revision 3 Espressif has fixed all the cache issues by hardware, but when you were porting the M5 Paper, was that a conclusion you got? Like, is it mandatory for the Moddable SDK to work properly, to have the Core Revision 3?

I was afraid that was the only solution, but just want to make sure from the Moddable SDK side if that's indeed the recommendation and the reason you have found issues in the past.

Really appreciate it if you can give a little bit more context around that!

Thanks!

[phoddie]

@beckerzito – Support for PSRAM on M5 Paper was done by @FWeinb, and he wrestled with the ESP-IDF configuration to get it working reliably.

Like, is it mandatory for the Moddable SDK to work properly, to have the Core Revision 3?

This is an incorrect characterization in two ways:

• The Moddable SDK expects RAM to be reliable. This is not an unusual requirement.
• The M5 Paper port is an example that happens to have Core Revision 3. That was not a statement that Core Revision 3 is required.

The problem to solve here is how to configure the ESP-IDF to work with the CPU and PSRAM components you have. That is outside the scope of the Moddable SDK.

[beckerzito]

Hi @phoddie thanks for the feedback.

I was reviewing the ESP32 documentation and here's what happens:

ESP32 Revision 3: Solves all the cache issue by hardware. No further action required. And that's the core available in the M5Paper.

ESP32 Revision 1: The memory issue is present but it was fixed by software (changes on the compiler rules to add a specific instruction in specific cases).
Originally the software solution was to add nop instructions. After some time it was detected it wasn't enough and a new workaround was proposed, usingmemw, which is described in this thread: espressif/esp-idf#2892

All of that is configured on the ESP-IDF by these two configurations:
CONFIG_SPIRAM_CACHE_WORKAROUND
CONFIG_SPIRAM_CACHE_WORKAROUND_STRATEGY

The documentation describes that when enabled, compilation flags are added to the compiler and linker.

Also, we can see that in the CMakeList in the ESP-IDF library:

What I have seen is that moddable build environment doesn't enable these flags during compilation time. For testing purposes, I have added them to the mcconfig/make.esp32.mk, and to my surprise, the Wifi modules worked right away (it wasn't working before).

I'm now running some long-term tests to verify if the memory access crash still exists, but only having better results from the Wifi module, I'm pretty confident that this might be the issue. Maybe that's the reason I was always seeing the crashes from moddable modules.

Is there any way we can reopen this defect and add these flags to the build env?

Thanks!

[phoddie]

To close the loop on this one... @beckerzito and I exchanged some emails to sort out the details. The problem here is pretty unique – to work around the PSRAM bug in ESP32 version 1 silicon, it is necessary to set some additional flags for the C compiler when building the Moddable SDK. There's now a C_FLAGS_SUBPLATFORM property in manifests for ESP32 subplatforms. It is documented here.