feat: Realtime API support - #3722

Closed
mudler wants to merge 29 commits into
masterfrom
feat/realtime
Closed

feat: Realtime API support#3722
mudler wants to merge 29 commits into
masterfrom
feat/realtime

Conversation

@mudler

@mudlermudler commented Oct 3, 2024•
edited
Loading

Copy link
Copy Markdown
Owner

Description

This PR fixes#3714

And also covers #191

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

@netlify

netlifyBot commented Oct 3, 2024•
edited
Loading

Copy link
Copy Markdown

✅ Deploy Preview for localai ready!

NameLink
🔨 Latest commitf272605
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67868d4d9141c90008d963f5
😎 Deploy Previewhttps://deploy-preview-3722--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mudlermudler mentioned this pull request Oct 3, 2024
Closed
@mudler
mudlerforce-pushed the feat/realtime branch 3 times, most recently from 5435a07 to 2db3f3dCompareOctober 9, 2024 10:57
@github-actions

github-actionsBot commented Oct 9, 2024

Copy link
Copy Markdown
Contributor

yamllint Failed

Show Output
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery

@mudler

mudler commented Oct 14, 2024

Copy link
Copy Markdown
OwnerAuthor

Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59

github-advanced-security[bot]
github-advanced-securityAI found potential problems Oct 16, 2024
View reviewed changes
Comment threadcore/http/app.go Fixed
@mudler
mudlerforce-pushed the feat/realtime branch 2 times, most recently from 1c61aad to 22579bcCompareOctober 18, 2024 14:06
@mudler
mudlerforce-pushed the feat/realtime branch 2 times, most recently from 2886390 to deff060CompareOctober 30, 2024 15:10
@mudlermudler changed the title WIP: realtime API stubfeat: Realtime API supportOct 31, 2024
@mudler
mudlerforce-pushed the feat/realtime branch 4 times, most recently from 7a9d6e8 to ee4ae33CompareNovember 6, 2024 17:35
@mattkanwisher

mattkanwisher commented Nov 7, 2024

Copy link
Copy Markdown
Contributor

whats best option here if we want to contribute just make forks of the branch and PRS against this?

@mudler

mudler commented Nov 7, 2024

Copy link
Copy Markdown
OwnerAuthor

whats best option here if we want to contribute just make forks of the branch and PRS against this?

Yes, that would work just fine!

@mudler
mudlerforce-pushed the feat/realtime branch 2 times, most recently from 60579fb to 4034562CompareNovember 7, 2024 22:54
@mudlermudler mentioned this pull request Nov 8, 2024
Merged
1 task
@mudler
mudlerforce-pushed the feat/realtime branch 6 times, most recently from 818122a to a1da931CompareNovember 12, 2024 18:12
@mudler
Add model interface to sessions
f45d11c
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
setup ws upgrade
65f4c12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix route
05225c9
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: simplify passing options to ModelOptions
9e96503
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Load wrapper clients
b4fea58
Testing with:
```yaml
name: gpt-4o
pipeline:
tts: voice-it-riccardo_fasol-x-low
transcription: whisper-base-q5_1
llm: llama-3.2-1b-instruct:q4_k_m
```
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
refactor
60c99dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat: correctly detect when starting the vad server
4f69170
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: extract realtime models into two categories
1796a17
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat(vad): hook vad detection
a3fd8ca
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
9614422
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Attach context for VAD
0318434
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Move to debug calls
9273395
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix lock handling
5953156
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip(vad)
136fbd2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
gRPC client stubs
ebfe8dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip
3dd1b30
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
06e438d
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Small adaptations
c526f05
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Use template evaluator for preparing LLM prompt in wrapped mode
7592984
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - to drop
9020683
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
go tidy
5f2c837
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Tweak silero settings
01aace3
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Improve audio detection
30e3c47
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - improve start and end of speech detection
9a09820
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
more robust approach
f272605
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudlermudler mentioned this pull request May 17, 2025
Closed
@richiejprichiejp mentioned this pull request May 19, 2025
Merged
@mudler

mudler commented May 27, 2025

Copy link
Copy Markdown
OwnerAuthor

closing as we merged it in #5392

@mudlermudler closed this May 27, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

dependenciesenhancementNew feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add support for realtime API

3 participants

@mudler@mattkanwisher@github-advanced-security
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
feat: Realtime API support - #3722
Closed
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Closed
feat: Realtime API support#3722
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Conversation
@mudler
@mudlermudler
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

Owner
Description
This PR fixes#3714
And also covers #191
Notes for Reviewers
Signed commits
  •  Yes, I signed my commits.
@netlify
netlifyBot
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf272605
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67868d4d9141c90008d963f5
😎 Deploy Previewhttps://deploy-preview-3722--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@mudlermudler
mentioned this pull request
Oct 3, 2024

Closed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
3 times, most recently
from
5435a07 to
2db3f3dCompareOctober 9, 2024 10:57
@github-actions
github-actionsBot
commented
Oct 9, 2024

Copy link


Copy Markdown

Contributor
yamllint Failed
Show Output
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery
@mudler
mudler
commented
Oct 14, 2024

Copy link


Copy Markdown

OwnerAuthor
Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Oct 16, 2024
View reviewed changes
Comment threadcore/http/app.go
Fixed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
1c61aad to
22579bcCompareOctober 18, 2024 14:06
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
2886390 to
deff060CompareOctober 30, 2024 15:10
@mudlermudler
changed the title
WIP: realtime API stubfeat: Realtime API supportOct 31, 2024
@mudler
mudlerforce-pushed
the
feat/realtime
branch
4 times, most recently
from
7a9d6e8 to
ee4ae33CompareNovember 6, 2024 17:35
@mattkanwisher
mattkanwisher
commented
Nov 7, 2024

Copy link


Copy Markdown

Contributor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
@mudler
mudler
commented
Nov 7, 2024

Copy link


Copy Markdown

OwnerAuthor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
Yes, that would work just fine!
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
60579fb to
4034562CompareNovember 7, 2024 22:54
@mudlermudler
mentioned this pull request
Nov 8, 2024
 Merged

1 task
@mudler
mudlerforce-pushed
the
feat/realtime
branch
6 times, most recently
from
818122a to
a1da931CompareNovember 12, 2024 18:12
@mudler
Add model interface to sessions
f45d11c
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
setup ws upgrade
65f4c12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix route
05225c9
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: simplify passing options to ModelOptions
9e96503
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Load wrapper clients
b4fea58
Testing with:
```yaml
name: gpt-4o
pipeline:
tts: voice-it-riccardo_fasol-x-low
transcription: whisper-base-q5_1
llm: llama-3.2-1b-instruct:q4_k_m
```
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
refactor
60c99dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat: correctly detect when starting the vad server
4f69170
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: extract realtime models into two categories
1796a17
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat(vad): hook vad detection
a3fd8ca
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
9614422
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Attach context for VAD
0318434
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Move to debug calls
9273395
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix lock handling
5953156
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip(vad)
136fbd2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
gRPC client stubs
ebfe8dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip
3dd1b30
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
06e438d
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Small adaptations
c526f05
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Use template evaluator for preparing LLM prompt in wrapped mode
7592984
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - to drop
9020683
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
go tidy
5f2c837
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Tweak silero settings
01aace3
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Improve audio detection
30e3c47
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - improve start and end of speech detection
9a09820
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
more robust approach
f272605
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudlermudler
mentioned this pull request
May 17, 2025

Closed

@richiejprichiejp
mentioned this pull request
May 19, 2025
 Merged

@mudler
mudler
commented
May 27, 2025

Copy link


Copy Markdown

OwnerAuthor
closing as we merged it in #5392
@mudlermudler
closed this
May 27, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

dependenciesenhancementNew feature or request

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add support for realtime API


3 participants

@mudler@mattkanwisher@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat: Realtime API support - #3722
Closed
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Closed
feat: Realtime API support#3722
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Conversation
@mudler
@mudlermudler
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

Owner
Description
This PR fixes#3714
And also covers #191
Notes for Reviewers
Signed commits
  •  Yes, I signed my commits.
@netlify
netlifyBot
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf272605
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67868d4d9141c90008d963f5
😎 Deploy Previewhttps://deploy-preview-3722--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@mudlermudler
mentioned this pull request
Oct 3, 2024

Closed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
3 times, most recently
from
5435a07 to
2db3f3dCompareOctober 9, 2024 10:57
@github-actions
github-actionsBot
commented
Oct 9, 2024

Copy link


Copy Markdown

Contributor
yamllint Failed
Show Output
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery
@mudler
mudler
commented
Oct 14, 2024

Copy link


Copy Markdown

OwnerAuthor
Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Oct 16, 2024
View reviewed changes
Comment threadcore/http/app.go
Fixed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
1c61aad to
22579bcCompareOctober 18, 2024 14:06
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
2886390 to
deff060CompareOctober 30, 2024 15:10
@mudlermudler
changed the title
WIP: realtime API stubfeat: Realtime API supportOct 31, 2024
@mudler
mudlerforce-pushed
the
feat/realtime
branch
4 times, most recently
from
7a9d6e8 to
ee4ae33CompareNovember 6, 2024 17:35
@mattkanwisher
mattkanwisher
commented
Nov 7, 2024

Copy link


Copy Markdown

Contributor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
@mudler
mudler
commented
Nov 7, 2024

Copy link


Copy Markdown

OwnerAuthor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
Yes, that would work just fine!
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
60579fb to
4034562CompareNovember 7, 2024 22:54
@mudlermudler
mentioned this pull request
Nov 8, 2024
 Merged

1 task
@mudler
mudlerforce-pushed
the
feat/realtime
branch
6 times, most recently
from
818122a to
a1da931CompareNovember 12, 2024 18:12
@mudler
Add model interface to sessions
f45d11c
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
setup ws upgrade
65f4c12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix route
05225c9
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: simplify passing options to ModelOptions
9e96503
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Load wrapper clients
b4fea58
Testing with:
```yaml
name: gpt-4o
pipeline:
tts: voice-it-riccardo_fasol-x-low
transcription: whisper-base-q5_1
llm: llama-3.2-1b-instruct:q4_k_m
```
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
refactor
60c99dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat: correctly detect when starting the vad server
4f69170
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: extract realtime models into two categories
1796a17
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat(vad): hook vad detection
a3fd8ca
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
9614422
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Attach context for VAD
0318434
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Move to debug calls
9273395
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix lock handling
5953156
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip(vad)
136fbd2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
gRPC client stubs
ebfe8dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip
3dd1b30
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
06e438d
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Small adaptations
c526f05
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Use template evaluator for preparing LLM prompt in wrapped mode
7592984
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - to drop
9020683
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
go tidy
5f2c837
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Tweak silero settings
01aace3
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Improve audio detection
30e3c47
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - improve start and end of speech detection
9a09820
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
more robust approach
f272605
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudlermudler
mentioned this pull request
May 17, 2025

Closed

@richiejprichiejp
mentioned this pull request
May 19, 2025
 Merged

@mudler
mudler
commented
May 27, 2025

Copy link


Copy Markdown

OwnerAuthor
closing as we merged it in #5392
@mudlermudler
closed this
May 27, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

dependenciesenhancementNew feature or request

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add support for realtime API


3 participants

@mudler@mattkanwisher@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat: Realtime API support - #3722
Closed
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Closed
feat: Realtime API support#3722
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Conversation
@mudler
@mudlermudler
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

Owner
Description
This PR fixes#3714
And also covers #191
Notes for Reviewers
Signed commits
  •  Yes, I signed my commits.
@netlify
netlifyBot
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf272605
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67868d4d9141c90008d963f5
😎 Deploy Previewhttps://deploy-preview-3722--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@mudlermudler
mentioned this pull request
Oct 3, 2024

Closed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
3 times, most recently
from
5435a07 to
2db3f3dCompareOctober 9, 2024 10:57
@github-actions
github-actionsBot
commented
Oct 9, 2024

Copy link


Copy Markdown

Contributor
yamllint Failed
Show Output
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery
@mudler
mudler
commented
Oct 14, 2024

Copy link


Copy Markdown

OwnerAuthor
Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Oct 16, 2024
View reviewed changes
Comment threadcore/http/app.go
Fixed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
1c61aad to
22579bcCompareOctober 18, 2024 14:06
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
2886390 to
deff060CompareOctober 30, 2024 15:10
@mudlermudler
changed the title
WIP: realtime API stubfeat: Realtime API supportOct 31, 2024
@mudler
mudlerforce-pushed
the
feat/realtime
branch
4 times, most recently
from
7a9d6e8 to
ee4ae33CompareNovember 6, 2024 17:35
@mattkanwisher
mattkanwisher
commented
Nov 7, 2024

Copy link


Copy Markdown

Contributor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
@mudler
mudler
commented
Nov 7, 2024

Copy link


Copy Markdown

OwnerAuthor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
Yes, that would work just fine!
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
60579fb to
4034562CompareNovember 7, 2024 22:54
@mudlermudler
mentioned this pull request
Nov 8, 2024
 Merged

1 task
@mudler
mudlerforce-pushed
the
feat/realtime
branch
6 times, most recently
from
818122a to
a1da931CompareNovember 12, 2024 18:12
@mudler
Add model interface to sessions
f45d11c
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
setup ws upgrade
65f4c12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix route
05225c9
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: simplify passing options to ModelOptions
9e96503
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Load wrapper clients
b4fea58
Testing with:
```yaml
name: gpt-4o
pipeline:
tts: voice-it-riccardo_fasol-x-low
transcription: whisper-base-q5_1
llm: llama-3.2-1b-instruct:q4_k_m
```
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
refactor
60c99dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat: correctly detect when starting the vad server
4f69170
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: extract realtime models into two categories
1796a17
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat(vad): hook vad detection
a3fd8ca
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
9614422
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Attach context for VAD
0318434
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Move to debug calls
9273395
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix lock handling
5953156
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip(vad)
136fbd2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
gRPC client stubs
ebfe8dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip
3dd1b30
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
06e438d
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Small adaptations
c526f05
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Use template evaluator for preparing LLM prompt in wrapped mode
7592984
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - to drop
9020683
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
go tidy
5f2c837
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Tweak silero settings
01aace3
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Improve audio detection
30e3c47
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - improve start and end of speech detection
9a09820
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
more robust approach
f272605
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudlermudler
mentioned this pull request
May 17, 2025

Closed

@richiejprichiejp
mentioned this pull request
May 19, 2025
 Merged

@mudler
mudler
commented
May 27, 2025

Copy link


Copy Markdown

OwnerAuthor
closing as we merged it in #5392
@mudlermudler
closed this
May 27, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

dependenciesenhancementNew feature or request

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add support for realtime API


3 participants

@mudler@mattkanwisher@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
feat: Realtime API support - #3722
Closed
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Closed
feat: Realtime API support#3722
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Conversation
@mudler
@mudlermudler
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

Owner
Description
This PR fixes#3714
And also covers #191
Notes for Reviewers
Signed commits
  •  Yes, I signed my commits.
@netlify
netlifyBot
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf272605
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67868d4d9141c90008d963f5
😎 Deploy Previewhttps://deploy-preview-3722--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@mudlermudler
mentioned this pull request
Oct 3, 2024

Closed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
3 times, most recently
from
5435a07 to
2db3f3dCompareOctober 9, 2024 10:57
@github-actions
github-actionsBot
commented
Oct 9, 2024

Copy link


Copy Markdown

Contributor
yamllint Failed
Show Output
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery
@mudler
mudler
commented
Oct 14, 2024

Copy link


Copy Markdown

OwnerAuthor
Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Oct 16, 2024
View reviewed changes
Comment threadcore/http/app.go
Fixed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
1c61aad to
22579bcCompareOctober 18, 2024 14:06
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
2886390 to
deff060CompareOctober 30, 2024 15:10
@mudlermudler
changed the title
WIP: realtime API stubfeat: Realtime API supportOct 31, 2024
@mudler
mudlerforce-pushed
the
feat/realtime
branch
4 times, most recently
from
7a9d6e8 to
ee4ae33CompareNovember 6, 2024 17:35
@mattkanwisher
mattkanwisher
commented
Nov 7, 2024

Copy link


Copy Markdown

Contributor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
@mudler
mudler
commented
Nov 7, 2024

Copy link


Copy Markdown

OwnerAuthor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
Yes, that would work just fine!
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
60579fb to
4034562CompareNovember 7, 2024 22:54
@mudlermudler
mentioned this pull request
Nov 8, 2024
 Merged

1 task
@mudler
mudlerforce-pushed
the
feat/realtime
branch
6 times, most recently
from
818122a to
a1da931CompareNovember 12, 2024 18:12
@mudler
Add model interface to sessions
f45d11c
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
setup ws upgrade
65f4c12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix route
05225c9
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: simplify passing options to ModelOptions
9e96503
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Load wrapper clients
b4fea58
Testing with:
```yaml
name: gpt-4o
pipeline:
tts: voice-it-riccardo_fasol-x-low
transcription: whisper-base-q5_1
llm: llama-3.2-1b-instruct:q4_k_m
```
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
refactor
60c99dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat: correctly detect when starting the vad server
4f69170
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: extract realtime models into two categories
1796a17
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat(vad): hook vad detection
a3fd8ca
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
9614422
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Attach context for VAD
0318434
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Move to debug calls
9273395
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix lock handling
5953156
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip(vad)
136fbd2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
gRPC client stubs
ebfe8dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip
3dd1b30
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
06e438d
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Small adaptations
c526f05
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Use template evaluator for preparing LLM prompt in wrapped mode
7592984
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - to drop
9020683
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
go tidy
5f2c837
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Tweak silero settings
01aace3
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Improve audio detection
30e3c47
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - improve start and end of speech detection
9a09820
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
more robust approach
f272605
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudlermudler
mentioned this pull request
May 17, 2025

Closed

@richiejprichiejp
mentioned this pull request
May 19, 2025
 Merged

@mudler
mudler
commented
May 27, 2025

Copy link


Copy Markdown

OwnerAuthor
closing as we merged it in #5392
@mudlermudler
closed this
May 27, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

dependenciesenhancementNew feature or request

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add support for realtime API


3 participants

@mudler@mattkanwisher@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat: Realtime API support - #3722
Closed
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Closed
feat: Realtime API support#3722
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Conversation
@mudler
@mudlermudler
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

Owner
Description
This PR fixes#3714
And also covers #191
Notes for Reviewers
Signed commits
  •  Yes, I signed my commits.
@netlify
netlifyBot
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf272605
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67868d4d9141c90008d963f5
😎 Deploy Previewhttps://deploy-preview-3722--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@mudlermudler
mentioned this pull request
Oct 3, 2024

Closed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
3 times, most recently
from
5435a07 to
2db3f3dCompareOctober 9, 2024 10:57
@github-actions
github-actionsBot
commented
Oct 9, 2024

Copy link


Copy Markdown

Contributor
yamllint Failed
Show Output
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery
@mudler
mudler
commented
Oct 14, 2024

Copy link


Copy Markdown

OwnerAuthor
Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Oct 16, 2024
View reviewed changes
Comment threadcore/http/app.go
Fixed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
1c61aad to
22579bcCompareOctober 18, 2024 14:06
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
2886390 to
deff060CompareOctober 30, 2024 15:10
@mudlermudler
changed the title
WIP: realtime API stubfeat: Realtime API supportOct 31, 2024
@mudler
mudlerforce-pushed
the
feat/realtime
branch
4 times, most recently
from
7a9d6e8 to
ee4ae33CompareNovember 6, 2024 17:35
@mattkanwisher
mattkanwisher
commented
Nov 7, 2024

Copy link


Copy Markdown

Contributor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
@mudler
mudler
commented
Nov 7, 2024

Copy link


Copy Markdown

OwnerAuthor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
Yes, that would work just fine!
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
60579fb to
4034562CompareNovember 7, 2024 22:54
@mudlermudler
mentioned this pull request
Nov 8, 2024
 Merged

1 task
@mudler
mudlerforce-pushed
the
feat/realtime
branch
6 times, most recently
from
818122a to
a1da931CompareNovember 12, 2024 18:12
@mudler
Add model interface to sessions
f45d11c
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
setup ws upgrade
65f4c12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix route
05225c9
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: simplify passing options to ModelOptions
9e96503
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Load wrapper clients
b4fea58
Testing with:
```yaml
name: gpt-4o
pipeline:
tts: voice-it-riccardo_fasol-x-low
transcription: whisper-base-q5_1
llm: llama-3.2-1b-instruct:q4_k_m
```
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
refactor
60c99dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat: correctly detect when starting the vad server
4f69170
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: extract realtime models into two categories
1796a17
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat(vad): hook vad detection
a3fd8ca
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
9614422
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Attach context for VAD
0318434
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Move to debug calls
9273395
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix lock handling
5953156
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip(vad)
136fbd2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
gRPC client stubs
ebfe8dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip
3dd1b30
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
06e438d
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Small adaptations
c526f05
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Use template evaluator for preparing LLM prompt in wrapped mode
7592984
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - to drop
9020683
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
go tidy
5f2c837
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Tweak silero settings
01aace3
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Improve audio detection
30e3c47
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - improve start and end of speech detection
9a09820
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
more robust approach
f272605
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudlermudler
mentioned this pull request
May 17, 2025

Closed

@richiejprichiejp
mentioned this pull request
May 19, 2025
 Merged

@mudler
mudler
commented
May 27, 2025

Copy link


Copy Markdown

OwnerAuthor
closing as we merged it in #5392
@mudlermudler
closed this
May 27, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

dependenciesenhancementNew feature or request

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add support for realtime API


3 participants

@mudler@mattkanwisher@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
feat: Realtime API support - #3722
Closed
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Closed
feat: Realtime API support#3722
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Conversation
@mudler
@mudlermudler
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

Owner
Description
This PR fixes#3714
And also covers #191
Notes for Reviewers
Signed commits
  •  Yes, I signed my commits.
@netlify
netlifyBot
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf272605
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67868d4d9141c90008d963f5
😎 Deploy Previewhttps://deploy-preview-3722--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@mudlermudler
mentioned this pull request
Oct 3, 2024

Closed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
3 times, most recently
from
5435a07 to
2db3f3dCompareOctober 9, 2024 10:57
@github-actions
github-actionsBot
commented
Oct 9, 2024

Copy link


Copy Markdown

Contributor
yamllint Failed
Show Output
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery
@mudler
mudler
commented
Oct 14, 2024

Copy link


Copy Markdown

OwnerAuthor
Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Oct 16, 2024
View reviewed changes
Comment threadcore/http/app.go
Fixed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
1c61aad to
22579bcCompareOctober 18, 2024 14:06
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
2886390 to
deff060CompareOctober 30, 2024 15:10
@mudlermudler
changed the title
WIP: realtime API stubfeat: Realtime API supportOct 31, 2024
@mudler
mudlerforce-pushed
the
feat/realtime
branch
4 times, most recently
from
7a9d6e8 to
ee4ae33CompareNovember 6, 2024 17:35
@mattkanwisher
mattkanwisher
commented
Nov 7, 2024

Copy link


Copy Markdown

Contributor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
@mudler
mudler
commented
Nov 7, 2024

Copy link


Copy Markdown

OwnerAuthor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
Yes, that would work just fine!
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
60579fb to
4034562CompareNovember 7, 2024 22:54
@mudlermudler
mentioned this pull request
Nov 8, 2024
 Merged

1 task
@mudler
mudlerforce-pushed
the
feat/realtime
branch
6 times, most recently
from
818122a to
a1da931CompareNovember 12, 2024 18:12
@mudler
Add model interface to sessions
f45d11c
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
setup ws upgrade
65f4c12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix route
05225c9
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: simplify passing options to ModelOptions
9e96503
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Load wrapper clients
b4fea58
Testing with:
```yaml
name: gpt-4o
pipeline:
tts: voice-it-riccardo_fasol-x-low
transcription: whisper-base-q5_1
llm: llama-3.2-1b-instruct:q4_k_m
```
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
refactor
60c99dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat: correctly detect when starting the vad server
4f69170
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: extract realtime models into two categories
1796a17
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat(vad): hook vad detection
a3fd8ca
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
9614422
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Attach context for VAD
0318434
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Move to debug calls
9273395
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix lock handling
5953156
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip(vad)
136fbd2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
gRPC client stubs
ebfe8dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip
3dd1b30
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
06e438d
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Small adaptations
c526f05
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Use template evaluator for preparing LLM prompt in wrapped mode
7592984
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - to drop
9020683
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
go tidy
5f2c837
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Tweak silero settings
01aace3
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Improve audio detection
30e3c47
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - improve start and end of speech detection
9a09820
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
more robust approach
f272605
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudlermudler
mentioned this pull request
May 17, 2025

Closed

@richiejprichiejp
mentioned this pull request
May 19, 2025
 Merged

@mudler
mudler
commented
May 27, 2025

Copy link


Copy Markdown

OwnerAuthor
closing as we merged it in #5392
@mudlermudler
closed this
May 27, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

dependenciesenhancementNew feature or request

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add support for realtime API


3 participants

@mudler@mattkanwisher@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
feat: Realtime API support - #3722
Closed
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Closed
feat: Realtime API support#3722
mudler wants to merge 29 commits into
masterfrom 
feat/realtime
Conversation
@mudler
@mudlermudler
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

Owner
Description
This PR fixes#3714
And also covers #191
Notes for Reviewers
Signed commits
  •  Yes, I signed my commits.
@netlify
netlifyBot
commented
Oct 3, 2024•

edited

Loading

Copy link


Copy Markdown

✅ Deploy Preview for localai ready!
NameLink
🔨 Latest commitf272605
🔍 Latest deploy loghttps://app.netlify.com/sites/localai/deploys/67868d4d9141c90008d963f5
😎 Deploy Previewhttps://deploy-preview-3722--localai.netlify.app
📱 Preview on mobile
 Toggle QR Code... 

QR Code

Use your smartphone camera to open QR code link.
To edit notification comments on pull requests, go to your Netlify site configuration.
@mudlermudler
mentioned this pull request
Oct 3, 2024

Closed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
3 times, most recently
from
5435a07 to
2db3f3dCompareOctober 9, 2024 10:57
@github-actions
github-actionsBot
commented
Oct 9, 2024

Copy link


Copy Markdown

Contributor
yamllint Failed
Show Output
::group::gallery/arch-function.yaml
::error file=gallery/arch-function.yaml,line=66,col=22::66:22 [new-line-at-end-of-file] no new line character at the end of file
::endgroup::

Workflow: Yamllint GitHub Actions, Action: __karancode_yamllint-github-action, Lint: gallery
@mudler
mudler
commented
Oct 14, 2024

Copy link


Copy Markdown

OwnerAuthor
Just for reference, openai-realtime-console seems quite nice for testing things out especially at this stage, I've opened up a PR upstream to include a Dockerfile and instructions on how to use it with a local server: openai/openai-realtime-console#59
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Oct 16, 2024
View reviewed changes
Comment threadcore/http/app.go
Fixed

@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
1c61aad to
22579bcCompareOctober 18, 2024 14:06
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
2886390 to
deff060CompareOctober 30, 2024 15:10
@mudlermudler
changed the title
WIP: realtime API stubfeat: Realtime API supportOct 31, 2024
@mudler
mudlerforce-pushed
the
feat/realtime
branch
4 times, most recently
from
7a9d6e8 to
ee4ae33CompareNovember 6, 2024 17:35
@mattkanwisher
mattkanwisher
commented
Nov 7, 2024

Copy link


Copy Markdown

Contributor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
@mudler
mudler
commented
Nov 7, 2024

Copy link


Copy Markdown

OwnerAuthor
whats best option here if we want to contribute just make forks of the branch and PRS against this?
Yes, that would work just fine!
@mudler
mudlerforce-pushed
the
feat/realtime
branch
2 times, most recently
from
60579fb to
4034562CompareNovember 7, 2024 22:54
@mudlermudler
mentioned this pull request
Nov 8, 2024
 Merged

1 task
@mudler
mudlerforce-pushed
the
feat/realtime
branch
6 times, most recently
from
818122a to
a1da931CompareNovember 12, 2024 18:12
@mudler
Add model interface to sessions
f45d11c
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
setup ws upgrade
65f4c12
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix route
05225c9
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: simplify passing options to ModelOptions
9e96503
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Load wrapper clients
b4fea58
Testing with:
```yaml
name: gpt-4o
pipeline:
tts: voice-it-riccardo_fasol-x-low
transcription: whisper-base-q5_1
llm: llama-3.2-1b-instruct:q4_k_m
```
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
refactor
60c99dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat: correctly detect when starting the vad server
4f69170
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore: extract realtime models into two categories
1796a17
One is anyToAny models that requires a VAD model, and one is
wrappedModel that requires as well VAD models along others in the
pipeline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
feat(vad): hook vad detection
a3fd8ca
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
chore(vad): try to hook vad to received data from the API (WIP)
9614422
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Attach context for VAD
0318434
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Move to debug calls
9273395
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Fix lock handling
5953156
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip(vad)
136fbd2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
gRPC client stubs
ebfe8dd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
wip
3dd1b30
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP
06e438d
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Small adaptations
c526f05
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Use template evaluator for preparing LLM prompt in wrapped mode
7592984
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - to drop
9020683
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
go tidy
5f2c837
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Tweak silero settings
01aace3
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
Improve audio detection
30e3c47
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
WIP - improve start and end of speech detection
9a09820
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
more robust approach
f272605
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudlermudler
mentioned this pull request
May 17, 2025

Closed

@richiejprichiejp
mentioned this pull request
May 19, 2025
 Merged

@mudler
mudler
commented
May 27, 2025

Copy link


Copy Markdown

OwnerAuthor
closing as we merged it in #5392
@mudlermudler
closed this
May 27, 2025
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels

dependenciesenhancementNew feature or request

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add support for realtime API


3 participants

@mudler@mattkanwisher@github-advanced-security