[SOW MS3] Workaround MIOPEN output tensor memory format (NDHWC) issue

[micmelesse]

added a fix for these tests and enable them

python test_nn.py --verbose  TestNNDeviceTypeCUDA.test_conv_cudnn_ndhwc_cuda_float16
python test_nn.py --verbose  TestNNDeviceTypeCUDA.test_conv_cudnn_ndhwc_cuda_float32

[jithunnair-amd]

retest this please

[jithunnair-amd]

@micmelesse Tests failed in CI: http://rocmhead.amd.com:8080/job/pytorch/job/pytorch-test-1/235/consoleText

Aug 15 05:56:09   test_conv_cudnn_ndhwc_cuda_float16 (__main__.TestNNDeviceTypeCUDA) ... _convolution
Aug 15 05:56:09 select_conv_backend
Aug 15 05:56:09 _convolution
Aug 15 05:56:09 select_conv_backend
Aug 15 05:56:09 select_conv_backend
Aug 15 05:56:09 select_conv_backend
Aug 15 05:56:09 FAIL (0.010s)
Aug 15 05:56:09   test_conv_cudnn_ndhwc_cuda_float32 (__main__.TestNNDeviceTypeCUDA) ... _convolution
Aug 15 05:56:09 select_conv_backend
Aug 15 05:56:09 _convolution
Aug 15 05:56:09 select_conv_backend
Aug 15 05:56:09 select_conv_backend
Aug 15 05:56:09 select_conv_backend
Aug 15 05:56:09 FAIL (0.006s)
...
Aug 15 05:56:13   test_convert_conv2d_weight_memory_format_cuda (__main__.TestNNDeviceTypeCUDA) ... _convolution
Aug 15 05:56:13 select_conv_backend
Aug 15 05:56:13 miopen_convolution
Aug 15 05:56:13 FAIL (0.120s)

[jithunnair-amd]

@micmelesse Any progress on this PR?

[micmelesse]

@jithunnair-amd I am still working on it. Just disabling MIOPEN doesnot work. I am looking at another solution.

[micmelesse]

@jithunnair-amd Two of the tests pass. The failures donot seem related to my changes. The failures are in different files. The distributed test fails due to a docker error.

[micmelesse]

I will do a seperate pr for the third test

[jithunnair-amd]

Looks good to me, just want another review to confirm that this would be in line with how we can fix unit tests for sow ms3 branch.