Skip to content

feat: Centralized Request Processing middleware - #3847

Merged
mudler merged 8 commits into
mudler:masterfrom
dave-gray101:feat-request-middleware
Feb 10, 2025
Merged

feat: Centralized Request Processing middleware#3847
mudler merged 8 commits into
mudler:masterfrom
dave-gray101:feat-request-middleware

Conversation

@dave-gray101

@dave-gray101dave-gray101 commented Oct 16, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

This PR reworks how our endpoints parse and process user input, in order to centralize and commonize that code as much as possible. This will drastically cut down the endpoint-specific code to a minimum.

Additionally, the VAD subsystem was untested - making it hard to validate if this changeset was safe or not. Therefore, this PR also adds both manual testing tools (via bruno) and automated testing in the AIO image.

Note to @mudler - this PR looks worse than it is - here's some reviewing notes to help

  • core/http/middleware/request.go should be where you start - its the heart of the change and replaces core/http/ctx/fiber.go
  • configuration of defaults is now a responsibility of the route function - seems a bit weird at first, but allows endpoints to be easily reused with different defaults on different paths.
  • while all the endpoint files were updated, the changes all follow the same pattern:
    • A new standard header to the function was added which pulls the relevant input and config from the earlier middleware stage
    • A bunch of code is deleted as its now performed earlier by those middlewares
    • The functional code specific to the endpoint is unchanged except for variable references
  • Specific to the AIO CI test: I was seeing very unpredictable behavior on CI versus locally reproducing this test. It turns out that setting LOCALAI_SINGLE_ACTIVE_BACKEND=true helps with a memory usage issue!

@netlify

netlifyBot commented Oct 16, 2024•
edited
Loading

Copy link
Copy Markdown

✅ Deploy Preview for localai ready!

NameLink
🔨 Latest commitf7dab99
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67a9c57513819e00084f4390
😎 Deploy Previewhttps://deploy-preview-3847--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@dave-gray101
dave-gray101force-pushed the feat-request-middleware branch from ed7f03e to 7dfddc7CompareOctober 16, 2024 22:51
@dave-gray101dave-gray101 mentioned this pull request Oct 21, 2024
Merged
1 task
@dave-gray101
dave-gray101force-pushed the feat-request-middleware branch from 13a7411 to 85ed9aeCompareOctober 24, 2024 02:46
@dave-gray101
dave-gray101 enabled auto-merge (squash) October 24, 2024 03:50
@mudlermudler mentioned this pull request Nov 15, 2024
Merged
1 task
@dave-gray101

dave-gray101 commented Nov 21, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@mudler merged in VAD, but I don't know how to test it manually yet since it's new. Please double check that :)

@mudlermudler mentioned this pull request Dec 8, 2024
Merged
1 task
@dave-gray101
dave-gray101force-pushed the feat-request-middleware branch 2 times, most recently from df37c12 to d08311fCompareDecember 17, 2024 20:06
@dave-gray101

dave-gray101 commented Dec 18, 2024

Copy link
Copy Markdown
ContributorAuthor

@mudler Ready for review, now with working VAD test.

@dave-gray101
squash past, centralize request middleware PR
c1f30ba
Signed-off-by: Dave Lee <dave@gray101.com>
@dave-gray101
dave-gray101force-pushed the feat-request-middleware branch from 6050d80 to c1f30baCompareFebruary 5, 2025 19:14
@dave-gray101

dave-gray101 commented Feb 5, 2025

Copy link
Copy Markdown
ContributorAuthor

merged and squashed this again.

@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!

@mudler

mudler commented Feb 6, 2025

Copy link
Copy Markdown
Owner

merged and squashed this again.

@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!

Just had a quick look now and direction looks good, however since changes are big I need to take a closer look, I'll try to do this asap in these days as now would be a good merge window !

mudler
mudler reviewed Feb 6, 2025
View reviewed changes
Comment thread.bruno/LocalAI Test Requests/vad/vad test audio.bru Outdated
mudler
mudler reviewed Feb 6, 2025
View reviewed changes
Comment threadaio/cpu/vad.yaml
@@ -0,0 +1,8 @@
backend: silero-vad

@mudlermudlerFeb 6, 2025

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

mudler
mudler reviewed Feb 6, 2025
View reviewed changes
Comment threadcore/application/startup.go
config.LoadOptionF16(options.F16),
config.ModelPath(options.ModelPath),
)
cfg, err := application.BackendLoader().LoadBackendConfigFileByNameDefaultOptions(m, options)

@mudlermudlerFeb 6, 2025

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

mudler
mudler reviewed Feb 6, 2025
View reviewed changes
Comment threadcore/backend/tokenize.go Outdated
opts := ModelOptions(backendConfig, appConfig)

inferenceModel, err = loader.Load(opts...)
// TODO: looks weird, seems to be a correct merge?

@mudlermudlerFeb 6, 2025

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks going behind in history. this switch isn't needed anymore

@dave-gray101dave-gray101Feb 8, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have corrected this :)

mudler
mudler reviewed Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config_loader.go
return cfg, nil
}

func (bcl *BackendConfigLoader) LoadBackendConfigFileByNameDefaultOptions(modelName string, appConfig *ApplicationConfig) (*BackendConfig, error) {

@mudlermudlerFeb 6, 2025

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup really appreciated!

mudler
mudler reviewed Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config.go
}

if (u & FLAG_VAD) == FLAG_VAD {
if c.Backend != "silero-vad" {

@mudlermudlerFeb 6, 2025•
edited
Loading

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for later: at some point we should think about having these constants defined to be re-usable. There are some repetitions as well in the model initializer

@dave-gray101dave-gray101Feb 8, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - probably worth doing that as followup if you're OK with that

@mudlermudlerFeb 10, 2025

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely, was just a reminder/note

mudler
mudler reviewed Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/e2e_test.go Outdated
@mudler
Update tests/e2e-aio/e2e_test.go
b717f6f
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
mudler
mudler reviewed Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/sample_data_test.go

@mudlermudlerFeb 10, 2025

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems quite huge as a text golang file - can we maybe add the converted file to the repo in the fixture folder?

@mudler
mudler disabled auto-merge February 10, 2025 11:06
@mudler
mudler merged commit 3cddf24 into mudler:masterFeb 10, 2025
@BrewTestBotBrewTestBot mentioned this pull request Feb 15, 2025
Merged
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mudlermudlermudler approved these changes

Assignees

No one assigned

Labels

area/ai-modeldependencies

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dave-gray101@mudler
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
feat: Centralized Request Processing middleware by dave-gray101 · Pull Request #3847 · mudler/LocalAI · GitHub
Skip to content
feat: Centralized Request Processing middleware - #3847
Merged
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Feb 10, 2025
Merged
feat: Centralized Request Processing middleware#3847
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Conversation
@dave-gray101
@dave-gray101dave-gray101
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
This PR reworks how our endpoints parse and process user input, in order to centralize and commonize that code as much as possible. This will drastically cut down the endpoint-specific code to a minimum.
Additionally, the VAD subsystem was untested - making it hard to validate if this changeset was safe or not. Therefore, this PR also adds both manual testing tools (via bruno) and automated testing in the AIO image.
Note to @mudler - this PR looks worse than it is - here's some reviewing notes to help
  • core/http/middleware/request.go should be where you start - its the heart of the change and replaces core/http/ctx/fiber.go
  • configuration of defaults is now a responsibility of the route function - seems a bit weird at first, but allows endpoints to be easily reused with different defaults on different paths.
  • while all the endpoint files were updated, the changes all follow the same pattern:

    • A new standard header to the function was added which pulls the relevant input and config from the earlier middleware stage
    • A bunch of code is deleted as its now performed earlier by those middlewares
    • The functional code specific to the endpoint is unchanged except for variable references
  • Specific to the AIO CI test: I was seeing very unpredictable behavior on CI versus locally reproducing this test. It turns out that setting LOCALAI_SINGLE_ACTIVE_BACKEND=true helps with a memory usage issue!
@netlify
netlifyBot
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf7dab99
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67a9c57513819e00084f4390
😎 Deploy Previewhttps://deploy-preview-3847--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
ed7f03e to
7dfddc7CompareOctober 16, 2024 22:51
@dave-gray101dave-gray101
mentioned this pull request
Oct 21, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
13a7411 to
85ed9aeCompareOctober 24, 2024 02:46
@dave-gray101
dave-gray101
enabled auto-merge (squash)
October 24, 2024 03:50
@mudlermudler
mentioned this pull request
Nov 15, 2024
 Merged

1 task
@dave-gray101
dave-gray101
commented
Nov 21, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@mudler merged in VAD, but I don't know how to test it manually yet since it's new. Please double check that :)
@mudlermudler
mentioned this pull request
Dec 8, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
2 times, most recently
from
df37c12 to
d08311fCompareDecember 17, 2024 20:06
@dave-gray101
dave-gray101
commented
Dec 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@mudler Ready for review, now with working VAD test.
@dave-gray101
squash past, centralize request middleware PR
c1f30ba
Signed-off-by: Dave Lee <dave@gray101.com>
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
6050d80 to
c1f30baCompareFebruary 5, 2025 19:14
@dave-gray101
dave-gray101
commented
Feb 5, 2025

Copy link


Copy Markdown

ContributorAuthor
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
@mudler
mudler
commented
Feb 6, 2025

Copy link


Copy Markdown

Owner
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
Just had a quick look now and direction looks good, however since changes are big I need to take a closer look, I'll try to do this asap in these days as now would be a good merge window !
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment thread.bruno/LocalAI Test Requests/vad/vad test audio.bru
Outdated

mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadaio/cpu/vad.yaml
@@ -0,0 +1,8 @@
backend: silero-vad
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/application/startup.go
 config.LoadOptionF16(options.F16),
 config.ModelPath(options.ModelPath),
 )
 cfg, err := application.BackendLoader().LoadBackendConfigFileByNameDefaultOptions(m, options)
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/backend/tokenize.go
Outdated

	opts := ModelOptions(backendConfig, appConfig)

	inferenceModel, err = loader.Load(opts...)
	// TODO: looks weird, seems to be a correct merge?
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks going behind in history. this switch isn't needed anymore
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have corrected this :)
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config_loader.go
	return cfg, nil
}

func (bcl *BackendConfigLoader) LoadBackendConfigFileByNameDefaultOptions(modelName string, appConfig *ApplicationConfig) (*BackendConfig, error) {
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup really appreciated!
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config.go
	}

	if (u & FLAG_VAD) == FLAG_VAD {
 if c.Backend != "silero-vad" {
@mudlermudlerFeb 6, 2025•

edited

Loading

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for later: at some point we should think about having these constants defined to be re-usable. There are some repetitions as well in the model initializer
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - probably worth doing that as followup if you're OK with that
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely, was just a reminder/note
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/e2e_test.go
Outdated

@mudler
Update tests/e2e-aio/e2e_test.go
b717f6f
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/sample_data_test.go
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems quite huge as a text golang file - can we maybe add the converted file to the repo in the fixture folder?
@mudler
mudler
disabled auto-merge
February 10, 2025 11:06
@mudler
mudler
merged commit 3cddf24
into
mudler:masterFeb 10, 2025
@BrewTestBotBrewTestBot
mentioned this pull request
Feb 15, 2025
 Merged

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mudlermudlermudler approved these changes

Assignees


No one assigned


Labels

area/ai-modeldependencies

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@dave-gray101@mudler

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat: Centralized Request Processing middleware by dave-gray101 · Pull Request #3847 · mudler/LocalAI · GitHub
Skip to content
feat: Centralized Request Processing middleware - #3847
Merged
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Feb 10, 2025
Merged
feat: Centralized Request Processing middleware#3847
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Conversation
@dave-gray101
@dave-gray101dave-gray101
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
This PR reworks how our endpoints parse and process user input, in order to centralize and commonize that code as much as possible. This will drastically cut down the endpoint-specific code to a minimum.
Additionally, the VAD subsystem was untested - making it hard to validate if this changeset was safe or not. Therefore, this PR also adds both manual testing tools (via bruno) and automated testing in the AIO image.
Note to @mudler - this PR looks worse than it is - here's some reviewing notes to help
  • core/http/middleware/request.go should be where you start - its the heart of the change and replaces core/http/ctx/fiber.go
  • configuration of defaults is now a responsibility of the route function - seems a bit weird at first, but allows endpoints to be easily reused with different defaults on different paths.
  • while all the endpoint files were updated, the changes all follow the same pattern:

    • A new standard header to the function was added which pulls the relevant input and config from the earlier middleware stage
    • A bunch of code is deleted as its now performed earlier by those middlewares
    • The functional code specific to the endpoint is unchanged except for variable references
  • Specific to the AIO CI test: I was seeing very unpredictable behavior on CI versus locally reproducing this test. It turns out that setting LOCALAI_SINGLE_ACTIVE_BACKEND=true helps with a memory usage issue!
@netlify
netlifyBot
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf7dab99
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67a9c57513819e00084f4390
😎 Deploy Previewhttps://deploy-preview-3847--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
ed7f03e to
7dfddc7CompareOctober 16, 2024 22:51
@dave-gray101dave-gray101
mentioned this pull request
Oct 21, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
13a7411 to
85ed9aeCompareOctober 24, 2024 02:46
@dave-gray101
dave-gray101
enabled auto-merge (squash)
October 24, 2024 03:50
@mudlermudler
mentioned this pull request
Nov 15, 2024
 Merged

1 task
@dave-gray101
dave-gray101
commented
Nov 21, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@mudler merged in VAD, but I don't know how to test it manually yet since it's new. Please double check that :)
@mudlermudler
mentioned this pull request
Dec 8, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
2 times, most recently
from
df37c12 to
d08311fCompareDecember 17, 2024 20:06
@dave-gray101
dave-gray101
commented
Dec 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@mudler Ready for review, now with working VAD test.
@dave-gray101
squash past, centralize request middleware PR
c1f30ba
Signed-off-by: Dave Lee <dave@gray101.com>
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
6050d80 to
c1f30baCompareFebruary 5, 2025 19:14
@dave-gray101
dave-gray101
commented
Feb 5, 2025

Copy link


Copy Markdown

ContributorAuthor
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
@mudler
mudler
commented
Feb 6, 2025

Copy link


Copy Markdown

Owner
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
Just had a quick look now and direction looks good, however since changes are big I need to take a closer look, I'll try to do this asap in these days as now would be a good merge window !
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment thread.bruno/LocalAI Test Requests/vad/vad test audio.bru
Outdated

mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadaio/cpu/vad.yaml
@@ -0,0 +1,8 @@
backend: silero-vad
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/application/startup.go
 config.LoadOptionF16(options.F16),
 config.ModelPath(options.ModelPath),
 )
 cfg, err := application.BackendLoader().LoadBackendConfigFileByNameDefaultOptions(m, options)
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/backend/tokenize.go
Outdated

	opts := ModelOptions(backendConfig, appConfig)

	inferenceModel, err = loader.Load(opts...)
	// TODO: looks weird, seems to be a correct merge?
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks going behind in history. this switch isn't needed anymore
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have corrected this :)
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config_loader.go
	return cfg, nil
}

func (bcl *BackendConfigLoader) LoadBackendConfigFileByNameDefaultOptions(modelName string, appConfig *ApplicationConfig) (*BackendConfig, error) {
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup really appreciated!
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config.go
	}

	if (u & FLAG_VAD) == FLAG_VAD {
 if c.Backend != "silero-vad" {
@mudlermudlerFeb 6, 2025•

edited

Loading

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for later: at some point we should think about having these constants defined to be re-usable. There are some repetitions as well in the model initializer
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - probably worth doing that as followup if you're OK with that
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely, was just a reminder/note
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/e2e_test.go
Outdated

@mudler
Update tests/e2e-aio/e2e_test.go
b717f6f
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/sample_data_test.go
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems quite huge as a text golang file - can we maybe add the converted file to the repo in the fixture folder?
@mudler
mudler
disabled auto-merge
February 10, 2025 11:06
@mudler
mudler
merged commit 3cddf24
into
mudler:masterFeb 10, 2025
@BrewTestBotBrewTestBot
mentioned this pull request
Feb 15, 2025
 Merged

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mudlermudlermudler approved these changes

Assignees


No one assigned


Labels

area/ai-modeldependencies

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@dave-gray101@mudler

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat: Centralized Request Processing middleware by dave-gray101 · Pull Request #3847 · mudler/LocalAI · GitHub
Skip to content
feat: Centralized Request Processing middleware - #3847
Merged
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Feb 10, 2025
Merged
feat: Centralized Request Processing middleware#3847
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Conversation
@dave-gray101
@dave-gray101dave-gray101
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
This PR reworks how our endpoints parse and process user input, in order to centralize and commonize that code as much as possible. This will drastically cut down the endpoint-specific code to a minimum.
Additionally, the VAD subsystem was untested - making it hard to validate if this changeset was safe or not. Therefore, this PR also adds both manual testing tools (via bruno) and automated testing in the AIO image.
Note to @mudler - this PR looks worse than it is - here's some reviewing notes to help
  • core/http/middleware/request.go should be where you start - its the heart of the change and replaces core/http/ctx/fiber.go
  • configuration of defaults is now a responsibility of the route function - seems a bit weird at first, but allows endpoints to be easily reused with different defaults on different paths.
  • while all the endpoint files were updated, the changes all follow the same pattern:

    • A new standard header to the function was added which pulls the relevant input and config from the earlier middleware stage
    • A bunch of code is deleted as its now performed earlier by those middlewares
    • The functional code specific to the endpoint is unchanged except for variable references
  • Specific to the AIO CI test: I was seeing very unpredictable behavior on CI versus locally reproducing this test. It turns out that setting LOCALAI_SINGLE_ACTIVE_BACKEND=true helps with a memory usage issue!
@netlify
netlifyBot
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf7dab99
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67a9c57513819e00084f4390
😎 Deploy Previewhttps://deploy-preview-3847--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
ed7f03e to
7dfddc7CompareOctober 16, 2024 22:51
@dave-gray101dave-gray101
mentioned this pull request
Oct 21, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
13a7411 to
85ed9aeCompareOctober 24, 2024 02:46
@dave-gray101
dave-gray101
enabled auto-merge (squash)
October 24, 2024 03:50
@mudlermudler
mentioned this pull request
Nov 15, 2024
 Merged

1 task
@dave-gray101
dave-gray101
commented
Nov 21, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@mudler merged in VAD, but I don't know how to test it manually yet since it's new. Please double check that :)
@mudlermudler
mentioned this pull request
Dec 8, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
2 times, most recently
from
df37c12 to
d08311fCompareDecember 17, 2024 20:06
@dave-gray101
dave-gray101
commented
Dec 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@mudler Ready for review, now with working VAD test.
@dave-gray101
squash past, centralize request middleware PR
c1f30ba
Signed-off-by: Dave Lee <dave@gray101.com>
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
6050d80 to
c1f30baCompareFebruary 5, 2025 19:14
@dave-gray101
dave-gray101
commented
Feb 5, 2025

Copy link


Copy Markdown

ContributorAuthor
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
@mudler
mudler
commented
Feb 6, 2025

Copy link


Copy Markdown

Owner
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
Just had a quick look now and direction looks good, however since changes are big I need to take a closer look, I'll try to do this asap in these days as now would be a good merge window !
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment thread.bruno/LocalAI Test Requests/vad/vad test audio.bru
Outdated

mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadaio/cpu/vad.yaml
@@ -0,0 +1,8 @@
backend: silero-vad
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/application/startup.go
 config.LoadOptionF16(options.F16),
 config.ModelPath(options.ModelPath),
 )
 cfg, err := application.BackendLoader().LoadBackendConfigFileByNameDefaultOptions(m, options)
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/backend/tokenize.go
Outdated

	opts := ModelOptions(backendConfig, appConfig)

	inferenceModel, err = loader.Load(opts...)
	// TODO: looks weird, seems to be a correct merge?
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks going behind in history. this switch isn't needed anymore
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have corrected this :)
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config_loader.go
	return cfg, nil
}

func (bcl *BackendConfigLoader) LoadBackendConfigFileByNameDefaultOptions(modelName string, appConfig *ApplicationConfig) (*BackendConfig, error) {
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup really appreciated!
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config.go
	}

	if (u & FLAG_VAD) == FLAG_VAD {
 if c.Backend != "silero-vad" {
@mudlermudlerFeb 6, 2025•

edited

Loading

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for later: at some point we should think about having these constants defined to be re-usable. There are some repetitions as well in the model initializer
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - probably worth doing that as followup if you're OK with that
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely, was just a reminder/note
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/e2e_test.go
Outdated

@mudler
Update tests/e2e-aio/e2e_test.go
b717f6f
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/sample_data_test.go
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems quite huge as a text golang file - can we maybe add the converted file to the repo in the fixture folder?
@mudler
mudler
disabled auto-merge
February 10, 2025 11:06
@mudler
mudler
merged commit 3cddf24
into
mudler:masterFeb 10, 2025
@BrewTestBotBrewTestBot
mentioned this pull request
Feb 15, 2025
 Merged

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mudlermudlermudler approved these changes

Assignees


No one assigned


Labels

area/ai-modeldependencies

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@dave-gray101@mudler

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
feat: Centralized Request Processing middleware by dave-gray101 · Pull Request #3847 · mudler/LocalAI · GitHub
Skip to content
feat: Centralized Request Processing middleware - #3847
Merged
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Feb 10, 2025
Merged
feat: Centralized Request Processing middleware#3847
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Conversation
@dave-gray101
@dave-gray101dave-gray101
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
This PR reworks how our endpoints parse and process user input, in order to centralize and commonize that code as much as possible. This will drastically cut down the endpoint-specific code to a minimum.
Additionally, the VAD subsystem was untested - making it hard to validate if this changeset was safe or not. Therefore, this PR also adds both manual testing tools (via bruno) and automated testing in the AIO image.
Note to @mudler - this PR looks worse than it is - here's some reviewing notes to help
  • core/http/middleware/request.go should be where you start - its the heart of the change and replaces core/http/ctx/fiber.go
  • configuration of defaults is now a responsibility of the route function - seems a bit weird at first, but allows endpoints to be easily reused with different defaults on different paths.
  • while all the endpoint files were updated, the changes all follow the same pattern:

    • A new standard header to the function was added which pulls the relevant input and config from the earlier middleware stage
    • A bunch of code is deleted as its now performed earlier by those middlewares
    • The functional code specific to the endpoint is unchanged except for variable references
  • Specific to the AIO CI test: I was seeing very unpredictable behavior on CI versus locally reproducing this test. It turns out that setting LOCALAI_SINGLE_ACTIVE_BACKEND=true helps with a memory usage issue!
@netlify
netlifyBot
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf7dab99
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67a9c57513819e00084f4390
😎 Deploy Previewhttps://deploy-preview-3847--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
ed7f03e to
7dfddc7CompareOctober 16, 2024 22:51
@dave-gray101dave-gray101
mentioned this pull request
Oct 21, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
13a7411 to
85ed9aeCompareOctober 24, 2024 02:46
@dave-gray101
dave-gray101
enabled auto-merge (squash)
October 24, 2024 03:50
@mudlermudler
mentioned this pull request
Nov 15, 2024
 Merged

1 task
@dave-gray101
dave-gray101
commented
Nov 21, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@mudler merged in VAD, but I don't know how to test it manually yet since it's new. Please double check that :)
@mudlermudler
mentioned this pull request
Dec 8, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
2 times, most recently
from
df37c12 to
d08311fCompareDecember 17, 2024 20:06
@dave-gray101
dave-gray101
commented
Dec 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@mudler Ready for review, now with working VAD test.
@dave-gray101
squash past, centralize request middleware PR
c1f30ba
Signed-off-by: Dave Lee <dave@gray101.com>
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
6050d80 to
c1f30baCompareFebruary 5, 2025 19:14
@dave-gray101
dave-gray101
commented
Feb 5, 2025

Copy link


Copy Markdown

ContributorAuthor
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
@mudler
mudler
commented
Feb 6, 2025

Copy link


Copy Markdown

Owner
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
Just had a quick look now and direction looks good, however since changes are big I need to take a closer look, I'll try to do this asap in these days as now would be a good merge window !
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment thread.bruno/LocalAI Test Requests/vad/vad test audio.bru
Outdated

mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadaio/cpu/vad.yaml
@@ -0,0 +1,8 @@
backend: silero-vad
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/application/startup.go
 config.LoadOptionF16(options.F16),
 config.ModelPath(options.ModelPath),
 )
 cfg, err := application.BackendLoader().LoadBackendConfigFileByNameDefaultOptions(m, options)
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/backend/tokenize.go
Outdated

	opts := ModelOptions(backendConfig, appConfig)

	inferenceModel, err = loader.Load(opts...)
	// TODO: looks weird, seems to be a correct merge?
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks going behind in history. this switch isn't needed anymore
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have corrected this :)
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config_loader.go
	return cfg, nil
}

func (bcl *BackendConfigLoader) LoadBackendConfigFileByNameDefaultOptions(modelName string, appConfig *ApplicationConfig) (*BackendConfig, error) {
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup really appreciated!
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config.go
	}

	if (u & FLAG_VAD) == FLAG_VAD {
 if c.Backend != "silero-vad" {
@mudlermudlerFeb 6, 2025•

edited

Loading

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for later: at some point we should think about having these constants defined to be re-usable. There are some repetitions as well in the model initializer
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - probably worth doing that as followup if you're OK with that
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely, was just a reminder/note
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/e2e_test.go
Outdated

@mudler
Update tests/e2e-aio/e2e_test.go
b717f6f
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/sample_data_test.go
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems quite huge as a text golang file - can we maybe add the converted file to the repo in the fixture folder?
@mudler
mudler
disabled auto-merge
February 10, 2025 11:06
@mudler
mudler
merged commit 3cddf24
into
mudler:masterFeb 10, 2025
@BrewTestBotBrewTestBot
mentioned this pull request
Feb 15, 2025
 Merged

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mudlermudlermudler approved these changes

Assignees


No one assigned


Labels

area/ai-modeldependencies

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@dave-gray101@mudler

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat: Centralized Request Processing middleware by dave-gray101 · Pull Request #3847 · mudler/LocalAI · GitHub
Skip to content
feat: Centralized Request Processing middleware - #3847
Merged
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Feb 10, 2025
Merged
feat: Centralized Request Processing middleware#3847
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Conversation
@dave-gray101
@dave-gray101dave-gray101
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
This PR reworks how our endpoints parse and process user input, in order to centralize and commonize that code as much as possible. This will drastically cut down the endpoint-specific code to a minimum.
Additionally, the VAD subsystem was untested - making it hard to validate if this changeset was safe or not. Therefore, this PR also adds both manual testing tools (via bruno) and automated testing in the AIO image.
Note to @mudler - this PR looks worse than it is - here's some reviewing notes to help
  • core/http/middleware/request.go should be where you start - its the heart of the change and replaces core/http/ctx/fiber.go
  • configuration of defaults is now a responsibility of the route function - seems a bit weird at first, but allows endpoints to be easily reused with different defaults on different paths.
  • while all the endpoint files were updated, the changes all follow the same pattern:

    • A new standard header to the function was added which pulls the relevant input and config from the earlier middleware stage
    • A bunch of code is deleted as its now performed earlier by those middlewares
    • The functional code specific to the endpoint is unchanged except for variable references
  • Specific to the AIO CI test: I was seeing very unpredictable behavior on CI versus locally reproducing this test. It turns out that setting LOCALAI_SINGLE_ACTIVE_BACKEND=true helps with a memory usage issue!
@netlify
netlifyBot
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf7dab99
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67a9c57513819e00084f4390
😎 Deploy Previewhttps://deploy-preview-3847--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
ed7f03e to
7dfddc7CompareOctober 16, 2024 22:51
@dave-gray101dave-gray101
mentioned this pull request
Oct 21, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
13a7411 to
85ed9aeCompareOctober 24, 2024 02:46
@dave-gray101
dave-gray101
enabled auto-merge (squash)
October 24, 2024 03:50
@mudlermudler
mentioned this pull request
Nov 15, 2024
 Merged

1 task
@dave-gray101
dave-gray101
commented
Nov 21, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@mudler merged in VAD, but I don't know how to test it manually yet since it's new. Please double check that :)
@mudlermudler
mentioned this pull request
Dec 8, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
2 times, most recently
from
df37c12 to
d08311fCompareDecember 17, 2024 20:06
@dave-gray101
dave-gray101
commented
Dec 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@mudler Ready for review, now with working VAD test.
@dave-gray101
squash past, centralize request middleware PR
c1f30ba
Signed-off-by: Dave Lee <dave@gray101.com>
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
6050d80 to
c1f30baCompareFebruary 5, 2025 19:14
@dave-gray101
dave-gray101
commented
Feb 5, 2025

Copy link


Copy Markdown

ContributorAuthor
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
@mudler
mudler
commented
Feb 6, 2025

Copy link


Copy Markdown

Owner
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
Just had a quick look now and direction looks good, however since changes are big I need to take a closer look, I'll try to do this asap in these days as now would be a good merge window !
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment thread.bruno/LocalAI Test Requests/vad/vad test audio.bru
Outdated

mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadaio/cpu/vad.yaml
@@ -0,0 +1,8 @@
backend: silero-vad
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/application/startup.go
 config.LoadOptionF16(options.F16),
 config.ModelPath(options.ModelPath),
 )
 cfg, err := application.BackendLoader().LoadBackendConfigFileByNameDefaultOptions(m, options)
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/backend/tokenize.go
Outdated

	opts := ModelOptions(backendConfig, appConfig)

	inferenceModel, err = loader.Load(opts...)
	// TODO: looks weird, seems to be a correct merge?
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks going behind in history. this switch isn't needed anymore
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have corrected this :)
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config_loader.go
	return cfg, nil
}

func (bcl *BackendConfigLoader) LoadBackendConfigFileByNameDefaultOptions(modelName string, appConfig *ApplicationConfig) (*BackendConfig, error) {
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup really appreciated!
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config.go
	}

	if (u & FLAG_VAD) == FLAG_VAD {
 if c.Backend != "silero-vad" {
@mudlermudlerFeb 6, 2025•

edited

Loading

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for later: at some point we should think about having these constants defined to be re-usable. There are some repetitions as well in the model initializer
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - probably worth doing that as followup if you're OK with that
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely, was just a reminder/note
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/e2e_test.go
Outdated

@mudler
Update tests/e2e-aio/e2e_test.go
b717f6f
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/sample_data_test.go
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems quite huge as a text golang file - can we maybe add the converted file to the repo in the fixture folder?
@mudler
mudler
disabled auto-merge
February 10, 2025 11:06
@mudler
mudler
merged commit 3cddf24
into
mudler:masterFeb 10, 2025
@BrewTestBotBrewTestBot
mentioned this pull request
Feb 15, 2025
 Merged

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mudlermudlermudler approved these changes

Assignees


No one assigned


Labels

area/ai-modeldependencies

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@dave-gray101@mudler

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
feat: Centralized Request Processing middleware by dave-gray101 · Pull Request #3847 · mudler/LocalAI · GitHub
Skip to content
feat: Centralized Request Processing middleware - #3847
Merged
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Feb 10, 2025
Merged
feat: Centralized Request Processing middleware#3847
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Conversation
@dave-gray101
@dave-gray101dave-gray101
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
This PR reworks how our endpoints parse and process user input, in order to centralize and commonize that code as much as possible. This will drastically cut down the endpoint-specific code to a minimum.
Additionally, the VAD subsystem was untested - making it hard to validate if this changeset was safe or not. Therefore, this PR also adds both manual testing tools (via bruno) and automated testing in the AIO image.
Note to @mudler - this PR looks worse than it is - here's some reviewing notes to help
  • core/http/middleware/request.go should be where you start - its the heart of the change and replaces core/http/ctx/fiber.go
  • configuration of defaults is now a responsibility of the route function - seems a bit weird at first, but allows endpoints to be easily reused with different defaults on different paths.
  • while all the endpoint files were updated, the changes all follow the same pattern:

    • A new standard header to the function was added which pulls the relevant input and config from the earlier middleware stage
    • A bunch of code is deleted as its now performed earlier by those middlewares
    • The functional code specific to the endpoint is unchanged except for variable references
  • Specific to the AIO CI test: I was seeing very unpredictable behavior on CI versus locally reproducing this test. It turns out that setting LOCALAI_SINGLE_ACTIVE_BACKEND=true helps with a memory usage issue!
@netlify
netlifyBot
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf7dab99
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67a9c57513819e00084f4390
😎 Deploy Previewhttps://deploy-preview-3847--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
ed7f03e to
7dfddc7CompareOctober 16, 2024 22:51
@dave-gray101dave-gray101
mentioned this pull request
Oct 21, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
13a7411 to
85ed9aeCompareOctober 24, 2024 02:46
@dave-gray101
dave-gray101
enabled auto-merge (squash)
October 24, 2024 03:50
@mudlermudler
mentioned this pull request
Nov 15, 2024
 Merged

1 task
@dave-gray101
dave-gray101
commented
Nov 21, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@mudler merged in VAD, but I don't know how to test it manually yet since it's new. Please double check that :)
@mudlermudler
mentioned this pull request
Dec 8, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
2 times, most recently
from
df37c12 to
d08311fCompareDecember 17, 2024 20:06
@dave-gray101
dave-gray101
commented
Dec 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@mudler Ready for review, now with working VAD test.
@dave-gray101
squash past, centralize request middleware PR
c1f30ba
Signed-off-by: Dave Lee <dave@gray101.com>
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
6050d80 to
c1f30baCompareFebruary 5, 2025 19:14
@dave-gray101
dave-gray101
commented
Feb 5, 2025

Copy link


Copy Markdown

ContributorAuthor
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
@mudler
mudler
commented
Feb 6, 2025

Copy link


Copy Markdown

Owner
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
Just had a quick look now and direction looks good, however since changes are big I need to take a closer look, I'll try to do this asap in these days as now would be a good merge window !
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment thread.bruno/LocalAI Test Requests/vad/vad test audio.bru
Outdated

mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadaio/cpu/vad.yaml
@@ -0,0 +1,8 @@
backend: silero-vad
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/application/startup.go
 config.LoadOptionF16(options.F16),
 config.ModelPath(options.ModelPath),
 )
 cfg, err := application.BackendLoader().LoadBackendConfigFileByNameDefaultOptions(m, options)
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/backend/tokenize.go
Outdated

	opts := ModelOptions(backendConfig, appConfig)

	inferenceModel, err = loader.Load(opts...)
	// TODO: looks weird, seems to be a correct merge?
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks going behind in history. this switch isn't needed anymore
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have corrected this :)
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config_loader.go
	return cfg, nil
}

func (bcl *BackendConfigLoader) LoadBackendConfigFileByNameDefaultOptions(modelName string, appConfig *ApplicationConfig) (*BackendConfig, error) {
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup really appreciated!
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config.go
	}

	if (u & FLAG_VAD) == FLAG_VAD {
 if c.Backend != "silero-vad" {
@mudlermudlerFeb 6, 2025•

edited

Loading

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for later: at some point we should think about having these constants defined to be re-usable. There are some repetitions as well in the model initializer
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - probably worth doing that as followup if you're OK with that
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely, was just a reminder/note
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/e2e_test.go
Outdated

@mudler
Update tests/e2e-aio/e2e_test.go
b717f6f
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/sample_data_test.go
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems quite huge as a text golang file - can we maybe add the converted file to the repo in the fixture folder?
@mudler
mudler
disabled auto-merge
February 10, 2025 11:06
@mudler
mudler
merged commit 3cddf24
into
mudler:masterFeb 10, 2025
@BrewTestBotBrewTestBot
mentioned this pull request
Feb 15, 2025
 Merged

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mudlermudlermudler approved these changes

Assignees


No one assigned


Labels

area/ai-modeldependencies

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@dave-gray101@mudler

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
feat: Centralized Request Processing middleware by dave-gray101 · Pull Request #3847 · mudler/LocalAI · GitHub
Skip to content
feat: Centralized Request Processing middleware - #3847
Merged
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Feb 10, 2025
Merged
feat: Centralized Request Processing middleware#3847
mudler merged 8 commits into
mudler:masterfrom 
dave-gray101:feat-request-middleware
Conversation
@dave-gray101
@dave-gray101dave-gray101
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

Contributor
This PR reworks how our endpoints parse and process user input, in order to centralize and commonize that code as much as possible. This will drastically cut down the endpoint-specific code to a minimum.
Additionally, the VAD subsystem was untested - making it hard to validate if this changeset was safe or not. Therefore, this PR also adds both manual testing tools (via bruno) and automated testing in the AIO image.
Note to @mudler - this PR looks worse than it is - here's some reviewing notes to help
  • core/http/middleware/request.go should be where you start - its the heart of the change and replaces core/http/ctx/fiber.go
  • configuration of defaults is now a responsibility of the route function - seems a bit weird at first, but allows endpoints to be easily reused with different defaults on different paths.
  • while all the endpoint files were updated, the changes all follow the same pattern:

    • A new standard header to the function was added which pulls the relevant input and config from the earlier middleware stage
    • A bunch of code is deleted as its now performed earlier by those middlewares
    • The functional code specific to the endpoint is unchanged except for variable references
  • Specific to the AIO CI test: I was seeing very unpredictable behavior on CI versus locally reproducing this test. It turns out that setting LOCALAI_SINGLE_ACTIVE_BACKEND=true helps with a memory usage issue!
@netlify
netlifyBot
commented
Oct 16, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf7dab99
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67a9c57513819e00084f4390
😎 Deploy Previewhttps://deploy-preview-3847--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
ed7f03e to
7dfddc7CompareOctober 16, 2024 22:51
@dave-gray101dave-gray101
mentioned this pull request
Oct 21, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
13a7411 to
85ed9aeCompareOctober 24, 2024 02:46
@dave-gray101
dave-gray101
enabled auto-merge (squash)
October 24, 2024 03:50
@mudlermudler
mentioned this pull request
Nov 15, 2024
 Merged

1 task
@dave-gray101
dave-gray101
commented
Nov 21, 2024•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@mudler merged in VAD, but I don't know how to test it manually yet since it's new. Please double check that :)
@mudlermudler
mentioned this pull request
Dec 8, 2024
 Merged

1 task
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
2 times, most recently
from
df37c12 to
d08311fCompareDecember 17, 2024 20:06
@dave-gray101
dave-gray101
commented
Dec 18, 2024

Copy link


Copy Markdown

ContributorAuthor
@mudler Ready for review, now with working VAD test.
@dave-gray101
squash past, centralize request middleware PR
c1f30ba
Signed-off-by: Dave Lee <dave@gray101.com>
@dave-gray101
dave-gray101force-pushed
the
feat-request-middleware
branch
from
6050d80 to
c1f30baCompareFebruary 5, 2025 19:14
@dave-gray101
dave-gray101
commented
Feb 5, 2025

Copy link


Copy Markdown

ContributorAuthor
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
@mudler
mudler
commented
Feb 6, 2025

Copy link


Copy Markdown

Owner
merged and squashed this again.
@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!
Just had a quick look now and direction looks good, however since changes are big I need to take a closer look, I'll try to do this asap in these days as now would be a good merge window !
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment thread.bruno/LocalAI Test Requests/vad/vad test audio.bru
Outdated

mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadaio/cpu/vad.yaml
@@ -0,0 +1,8 @@
backend: silero-vad
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/application/startup.go
 config.LoadOptionF16(options.F16),
 config.ModelPath(options.ModelPath),
 )
 cfg, err := application.BackendLoader().LoadBackendConfigFileByNameDefaultOptions(m, options)
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/backend/tokenize.go
Outdated

	opts := ModelOptions(backendConfig, appConfig)

	inferenceModel, err = loader.Load(opts...)
	// TODO: looks weird, seems to be a correct merge?
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks going behind in history. this switch isn't needed anymore
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have corrected this :)
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config_loader.go
	return cfg, nil
}

func (bcl *BackendConfigLoader) LoadBackendConfigFileByNameDefaultOptions(modelName string, appConfig *ApplicationConfig) (*BackendConfig, error) {
@mudlermudlerFeb 6, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup really appreciated!
mudler
mudler
reviewed
Feb 6, 2025
View reviewed changes
Comment threadcore/config/backend_config.go
	}

	if (u & FLAG_VAD) == FLAG_VAD {
 if c.Backend != "silero-vad" {
@mudlermudlerFeb 6, 2025•

edited

Loading

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note for later: at some point we should think about having these constants defined to be re-usable. There are some repetitions as well in the model initializer
@dave-gray101dave-gray101Feb 8, 2025

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - probably worth doing that as followup if you're OK with that
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely, was just a reminder/note
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/e2e_test.go
Outdated

@mudler
Update tests/e2e-aio/e2e_test.go
b717f6f
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
mudler
mudler
reviewed
Feb 10, 2025
View reviewed changes
Comment threadtests/e2e-aio/sample_data_test.go
@mudlermudlerFeb 10, 2025

Copy link


Copy Markdown

Owner
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems quite huge as a text golang file - can we maybe add the converted file to the repo in the fixture folder?
@mudler
mudler
disabled auto-merge
February 10, 2025 11:06
@mudler
mudler
merged commit 3cddf24
into
mudler:masterFeb 10, 2025
@BrewTestBotBrewTestBot
mentioned this pull request
Feb 15, 2025
 Merged

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mudlermudlermudler approved these changes

Assignees


No one assigned


Labels

area/ai-modeldependencies

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@dave-gray101@mudler