Backend Specific Configuration in Runtime

[cccclai]

🚀 The feature, motivation and pitch

Currently, ET users interact directly with the ET runtime, which in turn interacts with backends determined by the AoT process. Backend-specific configurations can only be changed during AoT or runtime build time, and users lack the ability to modify these configurations at runtime. This limitation prevents users from dynamically adjusting backend-specific settings during runtime.

Alternatives

No response

Additional context

No response

RFC

Option 1

Add an interface in the backend interface to get/set backend options, and the backend option instance is hosted inside each backend.

// users' code
module.load()
// Optional steps, return error code if fails
backend_options = module.get_options("MyBackend")
backend_options.set("power", 1)
module.execute()
// [infra code] // A new added class BackendOptions
// Supported types
enum class OptionType {
BOOL,
INT,
STRING
};
// Union to hold different data types
union OptionValue {
bool bool_value;
int int_value;
const char* string_value;
OptionValue() : int_value(0) {} };
// Key-value pair structure
struct BackendOption {
const char* key;
OptionType type;
OptionValue value;
};
// Options storage (array-based)
class BackendOptions {
public:
BackendOptions(size_t capacity) : size(0), capacity(capacity) {
options = new BackendOption[capacity];
}
void set_option(const char* key, bool value) {
set_option_internal(key, OptionType::BOOL, {.bool_value = value});
}
void set_option(const char* key, int value) {
set_option_internal(key, OptionType::INT, {.int_value = value});
}
void set_option(const char* key, const char* value) {
set_option_internal(key, OptionType::STRING, {.string_value = value});
}
Error<BackendOption*> get_option(const char* key) {
for (size_t i = 0; i < size; ++i) {
if (strcmp(options[i].key, key) == 0) {
return &options[i];
}
}
return nullptr; // Key not found
}
private:
BackendOption* options;
size_t size;
size_t capacity;
void set_option_internal(const char* key, OptionType type, OptionValue value) {
for (size_t i = 0; i < size; ++i) {
if (strcmp(options[i].key, key) == 0) {
options[i].type = type;
options[i].value = value;
return;
}
}
if (size < capacity) {
options[size++] = {key, type, value};
}
}
};
// backend interface code
class BackendInterface {
void is_available() {}
void init() {}
void execute() {}
virtual BackendOptions* get_options() {} // new API
}
// My backend interface code
class MyBackend {
void is_available() {}
void init() {}
BackendOptions* get_options() override {return &my_backend_options; }
private:
BackendOptions my_backend_options;
}

Pros:

• Can be called ad hoc, not tied with model loading or executions. Users can adjust the backend options after a few iterations of executions, or before model loading time.
• Unified data structure across backends for configuration

Cons

• Limited data type support (bool, int, string) for the backend options value.
• Introduce more APIs to configure the backend, though it’s optional and mainly for the last few percent optimization.
  Threading concerns

Option 2

Allow users to create a list like following users code and the list will be dispatched to the corresponding backend to allow them adjust based on the configuration.

// users' code
// Optional
BackendOptions options = {
{"CoreMLBackend", {{"power", "high"}}},
{"QNNBackend", {{"backend", "CPU_AND_ANE"}}}
};
module.load(file, /*optional*/options);
module.execute(inputs, /*optional*/options);
// [infra code] // Same BackendOptions class as option 1
// No change in Backend Interface class
// My backend interface code
class MyBackend {
void is_available() {}
void init(context, ...) {
BackendOptions my_backend_option = context.get_options() // only recieves options for my backend
}
void execute(context, ...) {
BackendOptions my_backend_option = context.get_options() // only recieves options for my backend
}
}
// In method.cpp, internal class BackendDelegate, add one line in init to pass backend specific option static Error Init(
const executorch_flatbuffer::BackendDelegate& delegate,
const Program* program,
BackendInitContext& backend_init_context,
const options,
BackendDelegate* out) {
const char* backend_id = delegate.id()->c_str();
// Get only the options relevant to this backend, new added line
+ BackendOptions backend_specific_options = global_options.get_options_for_backend(backend_id);
BackendInterface* backend = get_backend_class(backend_id);
// Initialize the delegate.
Result<DelegateHandle*> handle = backend->init(
backend_init_context,
&out->segment_,
ArrayRef<CompileSpec>(compile_specs, num_compile_specs));
return Error::Ok;
}
// In method.cpp, pass in backend specific option for load and execute
Result<Method> Method::load(
executorch_flatbuffer::ExecutionPlan* s_plan,
const Program* program,
MemoryManager* memory_manager,
EventTracer* event_tracer,
const NamedDataMap* named_data_map,
+ BackendOptions backend_options=None)
Error Method::execute(BackendOptions backend_options=None)

Pros:

• Simpler code to construct the backend specific options
• No extra external API, except introducing a new BackendOptions class
• Same users flow
• Thread safe

Cons:

• Modify the main load and execute API with backend options
• The backend specific change is tied with load/execute code, and can’t run ad hoc.

Option 3 [Preferred]

Adjusted from option 1, user will create a list of configs, and pass through via a separate API update. Instead of having a backend to host the backend option instance, it will be created by users and passed to the backend. The backend will be configured accordingly.

// users' code
BackendOptions backend_options = {
{"CoreMLBackend", {{"power", "high"}}},
{"QNNBackend", {{"backend", "CPU_AND_ANE"}}}
};
module.load(file);
// Optional
module.update(backend_options);
module.execute(inputs);
// [infra code] // Same BackendOptions class as option 1
// No change in Backend Interface class
// My backend interface code
class MyBackend {
void is_available() {}
void init(context, ...) {
...
}
// Keep the context to provide similar features as other API, like profiling etc.
void update(context, backend_options) {
for(size_t i=0; i<opts.size(); i++) {
const BackendOption& opt = opts.data()[i];
if(strcmp(opt.key, "power") == 0) {
power_level = opt.value;
}
else if(strcmp(opt.key, "profile") == 0) {
current_mode = opt.value;
}
else {
return false; // Unknown option
}
}
return true;
}
void execute(context, ...) {
...
}
private: std::string power_level;
bool profile;
}

Pros:

• Can run dynamically before/after load/execute
• Thread safety can be controlled inside the backend, which happens nowadays

Cons:

Summary table

	Option 1	Option 2	Option 3
Can run dynamically (before load, after load, between execute	Yes	No	Yes
Meet use cases requirement	Yes	Yes	Yes
Thread safety	High. Each backend maintains it’s own backend_option instance, and passes it to the users to update. It’s quite risky as it can be overwritten by multiple threads together.	Low, the backend option is constructed in users' code and passed together with load/execute. There might be some overhead for the cases when the backend option is not needed.	Low, the backend option is constructed in users' code. It will be passed to the backend by calling update explicitly. Each backend itself needs to ensure thread safe during update
User’s code complexity	Low	Low	Low
Backend’s code complexity	Low, might be slightly easier because of the backend update function can reuse the common one	Low	Low
API change	- New backend options class implementation- New user public API set_options- New backend public API set_options and get_options	- New backend options class implementation- Add optional args (backend options) to both load and execute- Add backend options to context	- New backend options class implementation- New user public API update- New backend public API update

No response

cc @larryliu0820@JacobSzwejbka@lucylq

[cccclai]

cc: @shewu-quic@chunit-quic@haowhsu-quic@winskuo-quic@DannyYuyang-quic

cc: @metascroy@cymbalrush@YifanShenSZ

cc: @neuropilot-captain

[cccclai]

Old discussion #1262

[digantdesai]

cc @gggekov FYI

[cccclai]

Starting the implementation in #11288

Start the new one here because it's easier to use ghstack #11389

[cccclai]

The stack is landed