remove unnecessary call to F.pad

[bm-synth]

remove one call to symmetric padding in F.pad when running with non-replicate pad mode, and instead let padding be done by Conv3d for a more efficient execution;

[hlky]

Hi @bm-synth. Thanks for your contribution. Can you share some figures on the memory and performance improvements?

[brunomaga]

Hi @hlky.

Running the following test_autoencoder.py

importtimeimporttorchimporttorch.nnasnnimporttorch.nn.functionalasFfromdiffusers.models.autoencoders.autoencoder_kl_cogvideoximportCogVideoXCausalConv3dtorch.manual_seed(42)
deftrain(model: nn.Module, video_input: torch.Tensor):
optimizer=torch.optim.Adam(model.parameters(), lr=0.001)
model.train()
start_train=time.time()
foriterationinrange(100): # Simulate 100 training iterationsoptimizer.zero_grad()
output=model(video_input)[0]
loss=F.mse_loss(output, output+iteration) # sum iteration to fake different grads per iterationloss.backward()
optimizer.step()
torch.cuda.synchronize()
train_time=time.time() -start_trainprint("train_time", train_time, "secs")
returnoutput.to("cpu")
defeval(model: nn.Module, video_input: torch.Tensor):
model.eval()
start_train=time.time()
withtorch.no_grad():
for_inrange(300): # Simulate 300 inference iterationsmodel(video_input)
torch.cuda.synchronize()
eval_time=time.time() -start_trainprint("eval_time", eval_time, "secs")

calling with that input shape [1, 128, 8, 544, 960], on the main branch, gives:

$ PYTHONPATH=./diffusers_main/src/ python test_autoencoder.py
input size: 0.498046875 GBs
eval_time 33.06385564804077 secs
train_time 34.33984375 secs
Max memory 22.18018913269043 GBs

calling this PR branch gives:

$ PYTHONPATH=./diffusers_PR/src/ python test_autoencoder.py
input size: 0.498046875 GBs
eval_time 31.588099241256714 secs
train_time 34.1251916885376 secs
Max memory 22.17398452758789 GBs

on the shape (1, 3, 300, 544, 960), main branch:

$ PYTHONPATH=./diffusers_main/src/ python test_autoencoder.py
input size: 0.43773651123046875 GBs
eval_time 17.759469032287598 secs
train_time 96.50320744514465 secs
Max memory 16.353439331054688 GBs

and this PR:

$ PYTHONPATH=./diffusers_PR/src/ python test_autoencoder.py
input size: 0.43773651123046875 GBs
eval_time 16.8880774974823 secs
train_time 96.04004764556885 secs
Max memory 16.34803009033203 GBs

I'll try to test more dimensions.

[hlky]

@bm-synth Great, thanks. Would it also be possible to verify numerical accuracy between the two versions? For a change like this we would expect between 0 to 1e-6 difference.

[brunomaga]

@hlky I updated the code above to fix a seed (torch.manual_seed(42)) and save the tensor with the model output after 100 training iterations. Then I ran this to compare both output_*.pt files:

if__name__=='__main__':
output_main: torch.Tensor=torch.load("output_main.pt")
output_PR: torch.Tensor=torch.load("output_PR.pt")
print("mean:", output_main.mean().item(), "vs", output_PR.mean().item())
print("std:", output_main.std().item(), "vs", output_PR.std().item())
print("max abs diff:", (output_PR-output_main).diff().abs().max().item())
asserttorch.allclose(output_main, output_PR)

output:

mean: -8.058547973632812e-05 vs -8.058547973632812e-05
std: 0.578125 vs 0.578125
max abs diff: 0.0

[bm-synth]

@hlky ping?

[hlky]

Hi @bm-synth. We need to verify the accuracy of CogVideoXCausalConv3d, not the output from your trained model.

Code

fromtypingimportOptional, Tuple, Unionimporttorchimporttorch.nnasnnimporttorch.nn.functionalasFfromdiffusers.models.autoencoders.autoencoder_kl_cogvideoximportCogVideoXCausalConv3dclassCogVideoXSafeConv3d_PR(nn.Conv3d):
r""" A 3D convolution layer that splits the input tensor into smaller parts to avoid OOM in CogVideoX Model. """defforward(self, input: torch.Tensor) ->torch.Tensor:
memory_count=torch.prod(torch.tensor(input.shape)) *2/1024**3# Set to 2GB, suitable for CuDNNifmemory_count>2:
kernel_size=self.kernel_size[0]
part_num=int(memory_count/2) +1input_chunks=torch.chunk(input, part_num, dim=2)
ifkernel_size>1:
input_chunks= [input_chunks[0]] + [
torch.cat((input_chunks[i-1][:, :, -kernel_size+1 :], input_chunks[i]), dim=2)
foriinrange(1, len(input_chunks))
]
output_chunks= []
forinput_chunkininput_chunks:
output_chunks.append(super().forward(input_chunk))
output=torch.cat(output_chunks, dim=2)
returnoutputelse:
returnsuper().forward(input)
classCogVideoXCausalConv3d_PR(nn.Module):
r"""A 3D causal convolution layer that pads the input tensor to ensure causality in CogVideoX Model. Args: in_channels (`int`): Number of channels in the input tensor. out_channels (`int`): Number of output channels produced by the convolution. kernel_size (`int` or `Tuple[int, int, int]`): Kernel size of the convolutional kernel. stride (`int`, defaults to `1`): Stride of the convolution. dilation (`int`, defaults to `1`): Dilation rate of the convolution. pad_mode (`str`, defaults to `"constant"`): Padding mode. """def__init__(
self,
in_channels: int,
out_channels: int,
kernel_size: Union[int, Tuple[int, int, int]],
stride: int=1,
dilation: int=1,
pad_mode: str="constant",
):
super().__init__()
ifisinstance(kernel_size, int):
kernel_size= (kernel_size,) *3time_kernel_size, height_kernel_size, width_kernel_size=kernel_size# TODO(aryan): configure calculation based on stride and dilation in the future.# Since CogVideoX does not use it, it is currently tailored to "just work" with Mochitime_pad=time_kernel_size-1height_pad= (height_kernel_size-1) //2width_pad= (width_kernel_size-1) //2self.pad_mode=pad_modeself.height_pad=height_padself.width_pad=width_padself.time_pad=time_padself.time_causal_padding= (width_pad, width_pad, height_pad, height_pad, time_pad, 0)
self.const_padding_conv3d= (0, self.width_pad, self.height_pad)
self.temporal_dim=2self.time_kernel_size=time_kernel_sizestride=strideifisinstance(stride, tuple) else (stride, 1, 1)
dilation= (dilation, 1, 1)
self.conv=CogVideoXSafeConv3d_PR(
in_channels=in_channels,
out_channels=out_channels,
kernel_size=kernel_size,
stride=stride,
dilation=dilation,
padding=0ifself.pad_mode=="replicate"elseself.const_padding_conv3d,
padding_mode="zeros",
)
deffake_context_parallel_forward(
self, inputs: torch.Tensor, conv_cache: Optional[torch.Tensor] =None
) ->torch.Tensor:
ifself.pad_mode=="replicate":
inputs=F.pad(inputs, self.time_causal_padding, mode="replicate")
else:
kernel_size=self.time_kernel_sizeifkernel_size>1:
cached_inputs= [conv_cache] ifconv_cacheisnotNoneelse [inputs[:, :, :1]] * (kernel_size-1)
inputs=torch.cat(cached_inputs+ [inputs], dim=2)
returninputsdefforward(self, inputs: torch.Tensor, conv_cache: Optional[torch.Tensor] =None) ->torch.Tensor:
inputs=self.fake_context_parallel_forward(inputs, conv_cache)
ifself.pad_mode=="replicate":
conv_cache=Noneelse:
conv_cache=inputs[:, :, -self.time_kernel_size+1 :].clone()
output=self.conv(inputs)
returnoutput, conv_cachemodel=CogVideoXCausalConv3d(in_channels=128, out_channels=512, kernel_size=3).eval()
withtorch.no_grad():
output=model(torch.randn([1, 128, 8, 544, 960], generator=torch.Generator().manual_seed(0)))[0]
withtorch.no_grad():
output_2=model(torch.randn([1, 128, 8, 544, 960], generator=torch.Generator().manual_seed(0)))[0]
torch.testing.assert_close(output, output_2)
print((output-output_2).abs().max())
model_pr=CogVideoXCausalConv3d_PR(in_channels=128, out_channels=512, kernel_size=3).eval()
withtorch.no_grad():
output_pr=model_pr(torch.randn([1, 128, 8, 544, 960], generator=torch.Generator().manual_seed(0)))[0]

torch.testing.assert_close(output, output_pr)
Mismatchedelements: 2139073042/2139095040 (100.0%)
Greatestabsolutedifference: 5.3313703536987305atindex (0, 421, 5, 286, 946) (upto1e-05allowed)
Greatestrelativedifference: 2893981952.0atindex (0, 348, 3, 142, 869) (upto1.3e-06allowed)

print((output-output_pr).abs().max())
tensor(5.3314)

The first check here torch.testing.assert_close(output, output_2) shows that CogVideoXCausalConv3d is deterministic so we would only accept up to around 1e-6 difference but preferably less or no change.

Also note the code will run on CPU in float32, this is to avoid other source of non-determinism when testing although it will use a large amount of memory. Generally we choose the smallest possible shape and model configuration for tests.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[bm-synth]

@hlky your code has an issue. the random state before initializing each CogVideoXCausalConv3D is different. Try to call torch.manual_seed(42) before each initialization, i.e.:

if__name__=='__main__':
print(f"diffusers version: {diffusers.__version__}")
torch.manual_seed(42) #### <---- added thismodel=CogVideoXCausalConv3d(in_channels=128, out_channels=512, kernel_size=3).eval()
withtorch.no_grad():
output=model(torch.randn([1, 128, 8, 544, 960], generator=torch.Generator().manual_seed(0)))[0]
withtorch.no_grad():
output_2=model(torch.randn([1, 128, 8, 544, 960], generator=torch.Generator().manual_seed(0)))[0]
torch.testing.assert_close(output, output_2)
print("max abs difference (output, output_2):", (output-output_2).abs().max().item())
print("number of different elements (output, output_2):", (output!=output_2).sum().item())
torch.manual_seed(42) ##### <---- added thismodel_pr=CogVideoXCausalConv3d_PR(in_channels=128, out_channels=512, kernel_size=3).eval()
withtorch.no_grad():
output_pr=model_pr(torch.randn([1, 128, 8, 544, 960], generator=torch.Generator().manual_seed(0)))[0]
torch.testing.assert_close(output, output_pr)
print("max abs difference (output, output_pr):", (output-output_pr).abs().max().item())
print("number of different elements (output, output_pr):", (output!=output_pr).sum().item())

output:

diffusers version: 0.32.2
abs difference (output, output_2): 0.0
number of different elements (output, output_2): 0
abs difference (output, output_pr): 0.0
number of different elements (output, output_pr): 0

[yiyixuxu]

thanks @bm-synth
the refactor looks really nice.

let's merge once @hlky confirm the results are identical

[hlky]

I'm still concerned by the reproducibility, especially as it's with float32 on CPU. Using torch.manual_seed isn't something we normally require users to do yet with this change they would need to use torch.manual_seed to get the same result from VAE decode in addition to passing a torch.Generator to the pipeline to get the same generated latent, and the issue would be compounded by the expected sources of non-determinism i.e. CUDA.

[bm-synth]

@hlky (cc @yiyixuxu ) your reproducibility concerns are not related to this PR. They are an issue in the production code. This PR gives the same behaviour as production. If you run the code below, where you only call your current CogVideoX module twice, without setting the random seed, it already fails. See below:

if__name__=='__main__':
print(f"diffusers version: {diffusers.__version__}")
torch.manual_seed(42)
model=CogVideoXCausalConv3d(in_channels=128, out_channels=512, kernel_size=3).eval()
withtorch.no_grad():
output=model(torch.randn([1, 128, 8, 544, 960], generator=torch.Generator().manual_seed(0)))[0]
withtorch.no_grad():
output_2=model(torch.randn([1, 128, 8, 544, 960], generator=torch.Generator().manual_seed(0)))[0]
torch.testing.assert_close(output, output_2)
print("max abs difference (output, output_2):", (output-output_2).abs().max().item())
print("number of different elements (output, output_2):", (output!=output_2).sum().item())
# torch.manual_seed(42) ##### <--- THE SECOND CHECK ONLY PASSES IF YOU UNCOMMENT THIS! model_main=CogVideoXCausalConv3d_PR(in_channels=128, out_channels=512, kernel_size=3).eval()
model_main=CogVideoXCausalConv3d_PR(in_channels=128, out_channels=512, kernel_size=3).eval()
withtorch.no_grad():
output_main=model_pr(torch.randn([1, 128, 8, 544, 960], generator=torch.Generator().manual_seed(0)))[0]
torch.testing.assert_close(output, output_main)
print("max abs difference (output, output_pr):", (output-output_main).abs().max().item())
print("number of different elements (output, output_pr):", (output!=output_main).sum().item())

[hlky]

We get the same output_2 as output with the same input (determined by torch.Generator().manual_seed(0)). We should expect the same output from PR version with the same input.

[bm-synth]

I'm still concerned by the reproducibility, especially as it's with float32 on CPU. Using torch.manual_seed isn't something we normally require users to do yet with this change they would need to use torch.manual_seed to get the same result from VAE decode in addition to passing a torch.Generator to the pipeline to get the same generated latent, and the issue would be compounded by the expected sources of non-determinism i.e. CUDA.

@hlky what i meant is:

• both this PR and the main give the same output==output_2 in this check here
• This PR's CogVideoX output matches the output of the CogVideoX in the current main ( see output==output_prhere), but you need to set the same random state when you initialise the module (ie set the seed) otherwise you have two modules with same architecture but different parameter state.
• This is exactly the same behaviour as the main branch: it gives the same output == output_mainhere but only if you set the seed.

What am I missing here?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hlky]

Can you run more benchmarks? From #10620 (comment) the benefit is not clear and it looks like the figures for memory only cover training. Follow our standard benchmarking methodology

diffusers/benchmarks/utils.py

Lines 52 to 58 in 1357931

	defbenchmark_fn(f, *args, **kwargs):
	t0=benchmark.Timer(
	stmt="f(*args, **kwargs)",
	globals={"args": args, "kwargs": kwargs, "f": f},
	num_threads=torch.get_num_threads(),
	)
	returnf"{(t0.blocked_autorange().mean):.3f}"

https://github.com/huggingface/diffusers/tree/main/benchmarks

#10620 (comment) also mentions improvement in torch.compile, can we demonstrate this? what specifically is the optimization it allows in torch.compile?

[bm-synth]

Can you run more benchmarks?

@hlky running this code that calls your benchmark_fn on 5 different shapes (feel free to change it):

defbenchmark_fn(f, *args, **kwargs): importtorch.utils.benchmarkasbenchmarkt0=benchmark.Timer( stmt="f(*args, **kwargs)", globals={"args": args, "kwargs": kwargs, "f": f}, num_threads=torch.get_num_threads(), ) returnf"{(t0.blocked_autorange().mean):.5f}"deftrain(model, inputs, targets, criterion, optimizer):
model.train()
optimizer.zero_grad()
outputs=model(inputs)
ifisinstance(outputs, tuple):
outputs=outputs[0] # only use the first tensor as outputloss=criterion(outputs, targets)
loss.backward()
optimizer.step()
returnloss.item()
definference(model, inputs):
model.eval()
withtorch.no_grad():
outputs=model(inputs)
returnoutputsif__name__=='__main__':
print(f"diffusers version: {diffusers.__version__}")
to_kw=dict(device="cuda", dtype=torch.bfloat16)
torch.manual_seed(42)
model_main=CogVideoXCausalConv3d(in_channels=128, out_channels=512, kernel_size=3).to(**to_kw)
model_main=torch.compile(model_main, mode="max-autotune-no-cudagraphs")
optimizer_main=torch.optim.Adam(model_main.parameters(), lr=1e-3)
torch.manual_seed(42)
model_pr=CogVideoXCausalConv3d(in_channels=128, out_channels=512, kernel_size=3).to(**to_kw)
model_pr=torch.compile(model_pr, mode="max-autotune-no-cudagraphs")
optimizer_pr=torch.optim.Adam(model_pr.parameters(), lr=1e-3)
criterion=nn.MSELoss()
# test 5 very different shape inputsforshapein [
(1, 128, 8, 544, 960),
(3, 128, 22, 12, 123),
(5, 128, 2, 4, 324),
(2, 128, 32, 128, 128),
(9, 128, 64, 24, 123),
]:
input=torch.randn(shape, generator=torch.Generator().manual_seed(0)).to(**to_kw)
output_shape= ( shape[0], 512, shape[2], shape[3], shape[4] )
target=torch.randn(output_shape, generator=torch.Generator().manual_seed(0)).to(**to_kw)
# few iterations to warm up the model and compilerfor_inrange(5):
benchmark_fn( train, model_main, input, target, criterion, optimizer_main)
benchmark_fn( train, model_pr, input, target, criterion, optimizer_pr)
# branchmark trainingruntime=benchmark_fn( train, model_main, input, target, criterion, optimizer_main)
print(f"shape {shape}, main runtime:", runtime) runtime=benchmark_fn( train, model_pr, input, target, criterion, optimizer_pr)
print(f"shape {shape}, PR runtime:", runtime)
print()

output:

diffusers version: 0.32.2
shape (1, 128, 8, 544, 960), main runtime: 0.12767
shape (1, 128, 8, 544, 960), PR runtime: 0.12665
shape (3, 128, 22, 12, 123), main runtime: 0.00345
shape (3, 128, 22, 12, 123), PR runtime: 0.00338
shape (5, 128, 2, 4, 324), main runtime: 0.00104
shape (5, 128, 2, 4, 324), PR runtime: 0.00103
shape (2, 128, 32, 128, 128), main runtime: 0.03298
shape (2, 128, 32, 128, 128), PR runtime: 0.03290
shape (9, 128, 64, 24, 123), main runtime: 0.05253
shape (9, 128, 64, 24, 123), PR runtime: 0.05283

so you're just avoiding the extra

inputs=F.pad(inputs, padding_2d, mode="constant", value=0)

and having the Conv3d cuda kernel handilng that instead. Plus the cleaner code.

[bm-synth]

#10620 (comment) also mentions improvement in torch.compile, can we demonstrate this? what specifically is the optimization it allows in torch.compile?

i'll cc here my colleague @ic-synth that first flagged that and let him answer with more accuracy. Either way, it's a cleaner way to write a product over all shape dimensions.

[hlky]

@bm-synth Can you benchmark inference? Training this autoencoder is not going to be a common use case. Timing is also worse in some cases and very minimal in others to the point it appears like random variation, the biggest difference is less than 1%. We can also include memory in benchmarking :)

[bm-synth]

@hlky cc @yiyixuxu here's the updated benchmark scritpt that includes inference:

defbenchmark_fn(f, *args, **kwargs): importtorch.utils.benchmarkasbenchmarkt0=benchmark.Timer( stmt="f(*args, **kwargs)", globals={"args": args, "kwargs": kwargs, "f": f}, num_threads=torch.get_num_threads(), ) returnf"{(t0.blocked_autorange().mean):.5f}"deftrain(model, inputs, targets, criterion, optimizer):
model.train()
optimizer.zero_grad()
outputs=model(inputs)
ifisinstance(outputs, tuple):
outputs=outputs[0] # only use the first tensor as outputloss=criterion(outputs, targets)
loss.backward()
optimizer.step()
returnloss.item()
definference(model, inputs):
model.eval()
withtorch.no_grad():
for_inrange(50):
outputs=model(inputs)
returnoutputsif__name__=='__main__':
print(f"diffusers version: {diffusers.__version__}")
to_kw=dict(device="cuda", dtype=torch.bfloat16)
torch.manual_seed(42)
model_main=CogVideoXCausalConv3d(in_channels=128, out_channels=512, kernel_size=3).to(**to_kw)
model_main=torch.compile(model_main, mode="max-autotune-no-cudagraphs")
optimizer_main=torch.optim.Adam(model_main.parameters(), lr=1e-3)
torch.manual_seed(42)
model_pr=CogVideoXCausalConv3d_PR(in_channels=128, out_channels=512, kernel_size=3).to(**to_kw)
model_pr=torch.compile(model_pr, mode="max-autotune-no-cudagraphs")
optimizer_pr=torch.optim.Adam(model_pr.parameters(), lr=1e-3)
criterion=nn.MSELoss()
# test 5 very different shape inputsforshapein (
(16, 128, 64, 64, 64),
(8, 128, 128, 32, 32),
(4, 128, 256, 128, 128),
(4, 128, 512, 256, 256),
(2, 128, 1024, 512, 512),
):
input=torch.randn(shape, generator=torch.Generator().manual_seed(0)).to(**to_kw)
output_shape= ( shape[0], 512, shape[2], shape[3], shape[4] )
target=torch.randn(output_shape, generator=torch.Generator().manual_seed(0)).to(**to_kw)
# few iterations of training to warm up the model and compilermodel_main.train()
model_pr.train()
for_inrange(5):
train(model_main, input, target, criterion, optimizer_main)
train(model_pr, input, target, criterion, optimizer_pr)
# benchmark trainingruntime_main=benchmark_fn( train, model_main, input, target, criterion, optimizer_main)
runtime_pr=benchmark_fn( train, model_pr, input, target, criterion, optimizer_pr)
print(f"shape {shape} train: main runtime {runtime_main} vs PR runtime {runtime_pr}")
# few iterations of inference to warm up the model and compilertorch.compiler.reset()
model_main.eval()
model_pr.eval()
withtorch.inference_mode():
for_inrange(5):
# few iterations of inference to warm up the model and compilerinference(model_main, input)
inference(model_pr, input)
runtime_main=benchmark_fn( inference, model_main, input)
runtime_pr=benchmark_fn( inference, model_pr, input)
print(f"shape {shape} inference: main runtime {runtime_main} vs PR runtime {runtime_pr}")
print()

to measure memory, i added to the forward of CogVideoXCausalConv3d and CogVideoXCausalConv3d_PR:

print(f"MAIN Memory BEFORE conv:", torch.cuda.memory_allocated())
output=self.conv(inputs)
print(f"MAIN Memory AFTER conv:", torch.cuda.memory_allocated())

to both PR and main branch. Memory usage for first shape:

main, train:

• MAIN Memory BEFORE conv: 6595022848
• MAIN Memory AFTER conv: 10889990144

PR, train:

• PR Memory BEFORE conv: 6523719680
• PR Memory AFTER conv: 10818686976

main, inference:

• MAIN Memory BEFORE conv: 10916464640
• MAIN Memory AFTER conv: 15211431936

PR, inference:

• PR Memory BEFORE conv: 10845161472
• PR Memory AFTER conv: 15140128768

Runtimes for first 2 shapes (NVIDIA H200):

• shape (16, 128, 64, 64, 64) train: main runtime 0.13034 vs PR runtime 0.07435
• shape (16, 128, 64, 64, 64) inference: main runtime 1.49722 vs PR runtime 1.46837
• shape (8, 128, 128, 32, 32) train: main runtime 0.03417 vs PR runtime 0.01869
• shape (8, 128, 128, 32, 32) inference: main runtime 0.38165 vs PR runtime 0.36572

Feel free to run it on your side and correct this benchmark script if you find any issue.

[ic-synth]

@bm-synth i think there's been some misunderstanding: doing torch.prod(torch.tensor(input.shape)) * 2 / 1024**3 is worse for compilation that writing down the product explicitly. i'd revert that change.

[bm-synth]

@bm-synth i think there's been some misunderstanding: doing torch.prod(torch.tensor(input.shape)) * 2 / 1024**3 is worse for compilation that writing down the product explicitly. i'd revert that change.

thank you @ic-synth for clarifying this. @hlky@yiyixuxu i reverted that change and updated the PR msg.

[hlky]

Thanks @bm-synth@ic-synth. Can we run the benchmarks again, torch.prod and creating a tensor likely affected the performance.

[bm-synth]

thank you @hlky here are the new results with the updated branch (NVIDIA H200):

Script that measures runtimes for 2 shapes:

shape (16, 128, 64, 64, 64) train: main runtime 0.12332 vs PR runtime 0.07097
shape (16, 128, 64, 64, 64) inference: main runtime 1.48820 vs PR runtime 1.46182
shape (8, 128, 128, 32, 32) train: main runtime 0.03254 vs PR runtime 0.01825
shape (8, 128, 128, 32, 32) inference: main runtime 0.37884 vs PR runtime 0.36265

the other test for the comparison of numerical difference:

max abs difference (output, output_2): 0.0
number of different elements (output, output_2): 0
max abs difference (output, output_pr): 0.0
number of different elements (output, output_pr): 0

loss for 10 iterations of train for shape (16, 128, 64, 64, 64):

iteration 0 loss: main 1.328125 vs PR 1.328125
iteration 1 loss: main 1.2734375 vs PR 1.2734375
iteration 2 loss: main 1.2265625 vs PR 1.2265625
iteration 3 loss: main 1.1875 vs PR 1.1875
iteration 4 loss: main 1.1484375 vs PR 1.1484375
iteration 5 loss: main 1.1171875 vs PR 1.1171875
iteration 6 loss: main 1.09375 vs PR 1.09375
iteration 7 loss: main 1.0703125 vs PR 1.0703125
iteration 8 loss: main 1.0546875 vs PR 1.0546875
iteration 9 loss: main 1.0390625 vs PR 1.0390625