[Quantization] enable multi-backend bitsandbytes by hlky · Pull Request #10574 · huggingface/diffusers

hlky · 2025-01-14T11:03:19Z

What does this PR do?

Mainly copied from transformers PR.

May need to look at

diffusers/src/diffusers/pipelines/pipeline_utils.py

Lines 450 to 467 in fbff43a

_, is_loaded_in_4bit_bnb, is_loaded_in_8bit_bnb=_check_bnb_status(module) 
if (is_loaded_in_4bit_bnboris_loaded_in_8bit_bnb) anddtypeisnotNone: 
logger.warning( 
f"The module '{module.__class__.__name__}' has been loaded in `bitsandbytes` {'4bit'ifis_loaded_in_4bit_bnbelse'8bit'} and conversion to {dtype} is not supported. Module is still in {'4bit'ifis_loaded_in_4bit_bnbelse'8bit'} precision."
 ) 
ifis_loaded_in_8bit_bnbanddeviceisnotNone: 
logger.warning( 
f"The module '{module.__class__.__name__}' has been loaded in `bitsandbytes` 8bit and moving it to {device} via `.to()` is not supported. Module is still on {module.device}."
 ) 
# This can happen for `transformer` models. CPU placement was added in
# https://github.com/huggingface/transformers/pull/33122. So, we guard this accordingly.
ifis_loaded_in_4bit_bnbanddeviceisnotNoneandis_transformers_version(">", "4.44.0"): 
module.to(device=device) 
elifnotis_loaded_in_4bit_bnbandnotis_loaded_in_8bit_bnb: 
module.to(device, dtype) 

Test results are same as nightly https://github.com/huggingface/diffusers/actions/runs/12758480172/job/35560601164

RUN_SLOW=1 pytest -v -s tests/quantization/bnb/
===================================================================================== short test summary info =====================================================================================
FAILED tests/quantization/bnb/test_mixed_int8.py::SlowBnb8bitTests::test_generate_quality_dequantize - NotImplementedError: Only row-major format inputs are supported, but got format `col32`
FAILED tests/quantization/bnb/test_mixed_int8.py::SlowBnb8bitTests::test_quality - AssertionError: False is not true
===================================================================== 2 failed, 42 passed, 26 warnings in 1601.02s (0:26:41) ======================================================================

Fixes#10395

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sayakpaul

HuggingFaceDocBuilderDev · 2025-01-14T11:10:02Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sayakpaul · 2025-01-14T15:45:02Z

+ if set(device_map.values()) == {"cpu"} and bnb_multibackend_is_enabled:
+ pass


Because bnb is supported on intel CPUs?

sayakpaul · 2025-01-14T15:46:29Z

- if "cpu" in device_map_without_no_convert.values() or "disk" in device_map_without_no_convert.values():
+ if set(device_map.values()) == {"cpu"} and bnb_multibackend_is_enabled:
+ pass
+ elif "cpu" in device_map_without_no_convert.values() or "disk" in device_map_without_no_convert.values():


The common piece of code between the two utilities could be clubbed into a small function and reused?
Previously we didn't do because it was relatively small and was better off in-line.

sayakpaul · 2025-01-14T15:46:38Z

- if "cpu" in device_map_without_no_convert.values() or "disk" in device_map_without_no_convert.values():
+ if set(device_map.values()) == {"cpu"} and bnb_multibackend_is_enabled:
+ pass
+ elif "cpu" in device_map_without_no_convert.values() or "disk" in device_map_without_no_convert.values():


The common piece of code between the two utilities could be clubbed into a small function and reused?
Previously we didn't do because it was relatively small and was better off in-line.

sayakpaul · 2025-01-14T15:47:09Z

 state.CxB, state.SB = bnb.functional.transform(weight.data, to_order=state.formatB)
 out32, Sout32 = bnb.functional.igemmlt(im, state.CxB, Sim, state.SB)
- return bnb.functional.mm_dequant(out32, Sout32, SCim, state.SCB, bias=None).t()
+ return bnb.functional.mm_dequant(out32, Sout32, SCim, state.SCB, bias=None).t().to(dtype)


Note: #10401

Will rebase after that PR has merged.

sayakpaul · 2025-01-14T15:48:37Z

 return is_loaded_in_4bit_bnb or is_loaded_in_8bit_bnb, is_loaded_in_4bit_bnb, is_loaded_in_8bit_bnb
+
+
+def _validate_bnb_multi_backend_availability(raise_exception):


@matthewdouglas I wonder if it makes sense have these as utility functions in bitsandbytes so that they can be reused in transformers and diffusers (and any other libraries)?

sayakpaul · 2025-01-14T15:52:02Z

+ return True
+
+
+@lru_cache


We usually don't do lru_cache in import_utils.py. Any specific reasons?

Copied from transformers, not sure on the context.

sayakpaul

Left some comments but this is already very good!

May need to look at

Anything specific? Not seeing anything CUDA-specific.

sayakpaul · 2025-01-14T15:56:27Z

Test results are same as nightly

Were the tests run on the aws-g6e-xlarge-plus runner? If so, tests/quantization/bnb/test_mixed_int8.py::SlowBnb8bitTests::test_quality should have passed. Will take a look.

github-actions · 2025-02-13T15:02:58Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

vladmandic · 2025-02-13T15:09:42Z

ping to remove stale

github-actions · 2025-03-14T15:03:51Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Quantization] enable multi-backend bitsandbytes
2048646

sayakpaul reviewed Jan 14, 2025
View reviewed changes

sayakpaul requested a review from matthewdouglas January 14, 2025 15:53

Merge branch 'main' into bnb-multi-backend
0d39ea4

Merge branch 'main' into bnb-multi-backend
1079d18

github-actionsBot added the stale Issues that haven't received updates label Feb 13, 2025

github-actionsBot removed the stale Issues that haven't received updates label Feb 14, 2025

Merge branch 'main' into bnb-multi-backend
dce826d

github-actionsBot added the stale Issues that haven't received updates label Mar 14, 2025

hlky closed this Apr 15, 2025

hlky deleted the bnb-multi-backend branch April 15, 2025 12:29

	_, is_loaded_in_4bit_bnb, is_loaded_in_8bit_bnb=_check_bnb_status(module)

	if (is_loaded_in_4bit_bnboris_loaded_in_8bit_bnb) anddtypeisnotNone:
	logger.warning(
	f"The module '{module.__class__.__name__}' has been loaded in `bitsandbytes` {'4bit'ifis_loaded_in_4bit_bnbelse'8bit'} and conversion to {dtype} is not supported. Module is still in {'4bit'ifis_loaded_in_4bit_bnbelse'8bit'} precision."
	)

	ifis_loaded_in_8bit_bnbanddeviceisnotNone:
	logger.warning(
	f"The module '{module.__class__.__name__}' has been loaded in `bitsandbytes` 8bit and moving it to {device} via `.to()` is not supported. Module is still on {module.device}."
	)

	# This can happen for `transformer` models. CPU placement was added in
	# https://github.com/huggingface/transformers/pull/33122. So, we guard this accordingly.
	ifis_loaded_in_4bit_bnbanddeviceisnotNoneandis_transformers_version(">", "4.44.0"):
	module.to(device=device)
	elifnotis_loaded_in_4bit_bnbandnotis_loaded_in_8bit_bnb:
	module.to(device, dtype)

		if set(device_map.values()) == {"cpu"} and bnb_multibackend_is_enabled:
		pass

		return is_loaded_in_4bit_bnb or is_loaded_in_8bit_bnb, is_loaded_in_4bit_bnb, is_loaded_in_8bit_bnb


		def _validate_bnb_multi_backend_availability(raise_exception):

Uh oh!

Conversation

hlky commented Jan 14, 2025

What does this PR do?

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Jan 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Jan 14, 2025

Uh oh!

github-actionsBot commented Feb 13, 2025

Uh oh!

vladmandic commented Feb 13, 2025

Uh oh!

github-actionsBot commented Mar 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sayakpaul left a comment •
edited
Loading