Brandon/flux model loading - #6739

Merged

brandonrising merged 113 commits into

mainfrom

brandon/flux-model-loading

Aug 27, 2024

brandonrising commented Aug 12, 2024•
edited
Loading

Contributor

Summary

Add FLUX support to the model manager.

QA Instructions

Install the FLUX model via the "Starter Models" tab:

Test fresh install and inference of all FLUX variants
- FLUX schnell
- FLUX schnell quantized
- FLUX dev
- FLUX dev quantized
Test quantized on 16GB GPU
Maybe: Test quantized on 8GB GPU
Test that the app still runs on CPU and MPS systems (main concern is pip install, or that we try to import bitsandbytes)

Merge Plan

Can be merged with proper approvals

github-actionsBot added python invocations backend frontend labels

brandonrising commented

View reviewed changes

invokeai/backend/model_manager/probe.py Outdated

brandonrising commented

View reviewed changes

invokeai/backend/quantization/fast_quantized_diffusion_model.py Outdated

brandonrising commented

View reviewed changes

invokeai/backend/quantization/fast_quantized_transformers_model.py Outdated

sammcj mentioned this pull request

[enhancement]: Please support FLUX.1 #6712

Closed

1 task

brandonrising force-pushed the brandon/flux-model-loading branch from 2e978ad to 0e14fc6Compare

August 15, 2024 14:27

github-actionsBot added Root python-deps services labels

brandonrising force-pushed the brandon/flux-model-loading branch from 043df07 to 00b63afCompare

August 19, 2024 17:13

RyanJDick changed the base branch from ryan/flux to main

August 20, 2024 18:10

github-actionsBot added the python-tests label

RyanJDick reviewed

View reviewed changes

invokeai/app/invocations/fields.py Outdated

invokeai/app/invocations/fields.py Outdated

invokeai/app/invocations/flux_text_encoder.py

invokeai/app/invocations/flux_text_encoder.py Outdated

invokeai/app/invocations/flux_text_encoder.py Outdated

invokeai/backend/quantization/load_flux_model_bnb_llm_int8.py Outdated

invokeai/backend/requantize.py Outdated

invokeai/backend/requantize.py Outdated

invokeai/configs/flux/flux1-schnell.yaml Outdated

pyproject.toml Outdated

psychedelicious reviewed

View reviewed changes

invokeai/app/invocations/flux_text_encoder.py Outdated

psychedelicious reviewed

View reviewed changes

invokeai/backend/model_manager/config.py

psychedelicious reviewed

View reviewed changes

invokeai/backend/quantization/load_flux_model_bnb_llm_int8.py

psychedelicious reviewed

View reviewed changes

invokeai/frontend/web/src/features/nodes/types/v2/common.ts

psychedelicious reviewed

View reviewed changes

pyproject.toml Outdated

brandonrising force-pushed the brandon/flux-model-loading branch 2 times, most recently from 5606eea to 5edec7fCompare

August 21, 2024 13:11

RyanJDick force-pushed the brandon/flux-model-loading branch from ccb49c9 to e41025dCompare

August 21, 2024 18:21

RyanJDick reviewed

View reviewed changes

invokeai/backend/model_manager/load/model_loaders/flux.py

psychedelicious reviewed

View reviewed changes

invokeai/app/invocations/flux_text_encoder.py

StAlKeR7779 reviewed

View reviewed changes

invokeai/backend/flux/modules/autoencoder.py Outdated

StAlKeR7779 reviewed

View reviewed changes

invokeai/backend/flux/modules/autoencoder.py

RyanJDick reviewed

View reviewed changes

invokeai/app/invocations/flux_text_encoder.py

RyanJDick reviewed

View reviewed changes

pyproject.toml Outdated

RyanJDickand others added 25 commits

August 26, 2024 19:21

Update the T5 8-bit quantized starter model to use the BnB LLM.int8()…

54c48c3

… variant.

Remove all references to optimum-quanto and downgrade diffusers.

5af214b

Update docs for T5 quantization script.

f4612a9

Move quantization scripts to a scripts/ subdir.

18c0ec3

Downgrade revert torch version after removing optimum-qanto, and othe…

098db5c

…r minor version-related fixes.

Update t5 encoder formats to accurately reflect the quantization stra…

dbdd851

…tegy and data type

Switch the CLIP-L start model to use our hosted version - which is mu…

1d6c83b

…ch smaller.

Replace swish() with torch.nn.functional.silu(h). They are functional…

1413ff9

…ly equivalent, but in my test VAE deconding was ~8% faster after the change.

Setup scaffolding for in progress images and add ability to cancel th…

bd1b37d

…e flux node

Remove dependency on flux config files

d159fe6

ruff

877b88e

Remove flux repo dependency

ae94e48

Downgrade accelerate and huggingface-hub deps to original versions.

f046a38

ruff format

9f6f404

Remove outdated TODO.

9a530c7

Only install starter models if not already installed

bb80697

Remove in progress images until we're able to make the valuable

642a953

Remove no longer used code in the flux denoise function

a90d098

Fix type error in tsc

40a3fa5

Run ruff

b9238b6

Rename params for flux and flux vae, add comments explaining use of t…

5a5ca10

…he config_path in model config

update default workflow for flux

bf59ab3

remove prompt

bd2692b

Run ruff

5d42e67

default workflow: add steps to exposed fields, add more notes

c510234

brandonrising force-pushed the brandon/flux-model-loading branch from a30c2d9 to c510234Compare

August 26, 2024 23:21

hipsterusername approved these changes

View reviewed changes

Update starter model size estimates.

3b29bad

brandonrising merged commit 50085b4 into main

brandonrising deleted the brandon/flux-model-loading branch

August 27, 2024 00:17

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backendCI-CDfrontendinvocationspythonpython-depspython-testsRoot services