feat(templates): use a single template for multimodals messages

[mudler]

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
video: "<|video_{{.ID}}|> {{.Text}}"image: "<|image_{{.ID}}|> {{.Text}}"audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
multimodal: "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

• Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

• Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

• Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	6f336da
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview	https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)

[mudler]

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
video: "<|video_{{.ID}}|> {{.Text}}"image: "<|image_{{.ID}}|> {{.Text}}"audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
multimodal: "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

• Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

• Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

• Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	6f336da
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview	https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)

[mudler]

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
video: "<|video_{{.ID}}|> {{.Text}}"image: "<|image_{{.ID}}|> {{.Text}}"audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
multimodal: "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

• Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

• Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

• Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	6f336da
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview	https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)

[mudler]

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
video: "<|video_{{.ID}}|> {{.Text}}"image: "<|image_{{.ID}}|> {{.Text}}"audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
multimodal: "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

• Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

• Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

• Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	6f336da
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview	https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)

[mudler]

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
video: "<|video_{{.ID}}|> {{.Text}}"image: "<|image_{{.ID}}|> {{.Text}}"audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
multimodal: "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

• Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

• Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

• Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	6f336da
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview	https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)

[mudler]

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
video: "<|video_{{.ID}}|> {{.Text}}"image: "<|image_{{.ID}}|> {{.Text}}"audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
multimodal: "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

• Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

• Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

• Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	6f336da
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview	https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)

[mudler]

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
video: "<|video_{{.ID}}|> {{.Text}}"image: "<|image_{{.ID}}|> {{.Text}}"audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
multimodal: "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

• Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

• Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

• Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	6f336da
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview	https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)

[mudler]

Description

This pull request includes significant changes to the way multimodal content (images, audio, and video) is handled and templated in the backend configuration and request processing. The most important changes include consolidating individual media fields into a single Multimodal field, updating the request configuration to handle multiple media types more efficiently, and enhancing the templating logic to support multimodal content.

As opposed to #3728 - this PR focuses on replacing the following YAML configuration:

template:
video: "<|video_{{.ID}}|> {{.Text}}"image: "<|image_{{.ID}}|> {{.Text}}"audio: "<|audio_{{.ID}}|> {{.Text}}"

With a simple:

template:
multimodal: "{{ range .Audio }}[audio-{{.ID}}]{{end}}{{ range .Images }}[img-{{.ID}}]{{end}}{{ range .Video }}[vid-{{.ID}}]{{end}}{{.Text}}"

that gives more granular control for each placeholder

Backend Configuration Updates:

• Consolidated Video, Image, and Audio fields into a single Multimodal field in the TemplateConfig struct in core/config/backend_config.go.

Templating Logic Improvements:

• Introduced MultiModalOptions and MultimodalContent types, along with a default multimodal template in pkg/templates/multimodal.go. The TemplateMultiModal function now uses these types to generate content placeholders dynamically.

Testing Enhancements:

• Added new test cases in pkg/templates/multimodal_test.go to ensure the TemplateMultiModal function correctly handles various scenarios, including messages with multiple types of media.

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	6f336da
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6716c5392a5cef0008dae872
😎 Deploy Preview	https://deploy-preview-3892--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dave-gray101]

Looks good to me. Will require a manual merge with #3847 but it's going to be easy.

(tagging PR number here so I remember to check for it)