add test for reloading lookahead optimizer

[BinyanHu]

Description

Brief Description of the PR: add the test for saving and reloading a Lookahead optimizer. The added test function reveals the bug that the states of a lookahead's inner optimizer are not properly saved and reloaded (#2094). So the test can not pass currently.

Type of change

• Bug fix
• New Tutorial
• Updated or additional documentation
• Additional Testing
• New Activation and the changes conform to the activation contribution guidelines
• New Callback and the changes conform to the callback contribution guidelines
• New Image addition and the changes conform to the image op contribution guidelines
• New Layer and the changes conform to the layer contribution guidelines
• New Loss and the changes conform to the loss contribution guidelines
• New Metric and the changes conform to the metric contribution guidelines
• New Optimizer and the changes conform to the optimizer contribution guidelines
• New RNN Cell and the changes conform to the rnn contribution guidelines
• New Seq2seq addition and the changes conform to the seq2seq contribution guidelines
• New Text addition and the changes conform to the text op contribution guidelines

Checklist:

• I've properly formatted my code according to the guidelines
  • By running Black + Flake8
  • By running pre-commit hooks
• This PR addresses an already submitted issue for TensorFlow Addons
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• This PR contains modifications to C++ custom-ops

How Has This Been Tested?

If you're adding a bugfix or new feature please describe the tests that you ran to verify your changes:

[bot-of-gabrieldemarmiesse]

@CyberZHG

You are owner of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

[bhack]

The original tests mentioned in the ticket was testing model serializzation?
What are you testing with save and load weights?

[BinyanHu]

The original tests mentioned in the ticket was testing model serializzation?
What are you testing with save and load weights?

The description from the ticket was inaccurate. I raised that issue and the problem is on save and reloading. Please check the the discussions in the issue(#2094) for more details.

[bhack]

I was talking about save and reloading original contributor test in https://github.com/CyberZHG/keras-lookahead/blob/master/tests/test_optimizers.py

[BinyanHu]

I was talking about save and reloading original contributor test in https://github.com/CyberZHG/keras-lookahead/blob/master/tests/test_optimizers.py

That is a different way to save a model. The contributor test uses model.save and keras.models.load_model to save and load the entire graph. In our case, we are using model.save_weights and model.load_weights to save and load only the weights of a model.

[BinyanHu]

I was talking about save and reloading original contributor test in https://github.com/CyberZHG/keras-lookahead/blob/master/tests/test_optimizers.py

Just updated the test file referring to the original contributor test. I added cases of whether the inner optimizer contains slots (adam) or not (sgd) and different saving and reloading methods (weights only and whole graph). All tests of "sgd" pass and those of "adam" fail, which I believe should be enough to demonstrate the problem.

[bhack]

Ok, have you double checked https://stackoverflow.com/questions/56805588/does-load-model-load-optimizer-state?

[bhack]

@WindQAQ Let me know if you still find useful to have a save_weight/load_weight test. In that case give a pass over the test.