Spectral norm improvements - #8590
Conversation
- Don't do iterations on weight in eval mode To facilitate this, register weight as buffer in order to be able to use module with spectral norm in eval mode after immediately after loading state dict (pytorch#8208) - Use weight instead of weight_orig as weight when removing spectral norm - Add dim parameter in case the normalization should occur w.r.t. a dimension other than 0 (pytorch#7865)
| delattr(module, self.name + '_u') | ||
| delattr(module, self.name + '_orig') | ||
| module.register_parameter(self.name, weight) | ||
| module.register_parameter(self.name, torch.nn.Parameter(weight)) |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
Code looks reasonable to me. Thanks! I think for the tests you proposed, 1 looks good. It might be hard to do a convergence test for 2. How about just test that |
|
Thanks! Here are the tests. |
|
@ezyang The failures are a timeout and a strange jit fault in LSTM (which I didn't get near to, I hope), also the caffee rocm seems to have completed, but it is not recognized as completed (the latter seems common today)... Do you have any advice for me? |
| self.assertTrue(hasattr(m, 'weight')) | ||
| self.assertTrue('weight' in m._parameters) | ||
|
|
||
| def test_spectral_norm_eval(self): |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
Thank you, Simon, for the suggestions.
|
"Aborted by user anonymous"? |
|
@pytorchbot retest this please |
|
@t-vi lol CI probably had some problem then. I restarted CI on this. Sorry! |
* Spectral norm improvements - Don't do iterations on weight in eval mode To facilitate this, register weight as buffer in order to be able to use module with spectral norm in eval mode after immediately after loading state dict (pytorch#8208) - Use weight instead of weight_orig as weight when removing spectral norm - Add dim parameter in case the normalization should occur w.r.t. a dimension other than 0 (pytorch#7865) * add and update spectral norm tests * More spectral norm tests Thank you, Simon, for the suggestions.
To facilitate this, register weight as buffer in order to be able
to use module with spectral norm in eval mode after immediately
after loading state dict ([discussion] SpectralNorm behavior in eval() and at remove() #8208)
spectral norm
a dimension other than 0 ([PyTorch] weight tensor dimension assumption #7865)
@ssnl is this about what you had in mind?
I have to add tests. What would be resonable here