Feature: Interfacing ML code with MiMA - #10

Open
mondus wants to merge 63 commits into
masterfrom
ML
Open

Feature: Interfacing ML code with MiMA#10
mondus wants to merge 63 commits into
masterfrom
ML

Conversation

@mondus

@mondusmondus commented May 15, 2023•
edited by SimonClifford
Loading

Copy link
Copy Markdown

Closes#7

@jatkinson1000 - covering #4 and #5

  • Adapt cg_drag.f90 to accommodate drag forcing from various different approaches
  • Create .f90 files with modules & subroutines for relevant drag calculations
  • Adapt PT NN to receive individual tensors of each variable and combine in a prepended layer
    This way each approach will call the NN twice, once with zonal, once with meridional, but tensors will only be written to shared memory once.

@SimonClifford - covering #3

  • Inspect PT model and construct matching TF model
  • Migrate weights
  • Add TF code into MiMA (requires @jatkinson1000 to write .f90 files above)

We then need to conduct benchmarking:

  • Instrument MiMA for ReBench
  • Plan a set of runs
  • Perform the set of runs

jatkinson1000and others added 30 commits January 4, 2023 15:46
@jatkinson1000
First cmake attempt.
3e18ca7
@jatkinson1000
Compiling CMake build
b757b3f
@jatkinson1000
Working CMake Build
9a7c075
@jatkinson1000
Update GettingStarted docfile to include CMake build process.
2c94d91
@SimonClifford
Correct some typos and infelicities.
88f3ebe
@jatkinson1000
Bug fix in CMakeLists appending NETCDF-C to NETCDF_INCLUDES
e95d118
@jatkinson1000
Merge branch 'cmake' of github.com:Cambridge-ICCS/MiMA into cmake
837aff9
@jatkinson1000
Correct typo in FindNetCDF.cmake filename.
f620e6e
@jatkinson1000
Added DaveNet pytorch code from Minah in src/shared/pytorch.
ed08814
@jatkinson1000
black and linting of python code.
ac4ac3c
@jatkinson1000
Fix arguments to extend requiring list.
7a39414
@jatkinson1000
Add script to save davenet as TorchScript.
6c6fc1b
@jatkinson1000
cg_drag updated with placeholders ready for ML changes.
913b4eb
@SimonClifford
Script to make TF davenet
c8bcda8
@SimonClifford
Add README.md
ae53fda
@SimonClifford
Lint and isort
8e88f8b
@jatkinson1000
Initial workings on pytorch coupling routine [WIP].
02a9e6c
@jatkinson1000
Updated net to take 3 inputs and test it behaves as expected.
cbeeb95
@jatkinson1000
tweak architecture and update pt2ts to run with 3 inputs.
63e5e99
@SimonClifford
Merge branch 'pytorch' into tensorflow
70d41b8
@SimonClifford
Update TF davenet construction and add test
e4b63b9
Update to reflect model now takes 3 inputs and concatenates
them, rather than a pre-concatted input.
Test is very basic but shows that TF davenet does the same
thing as PyTorch davenet.
@jatkinson1000
Coupled code for pytorch (untested compiling).
e56ac02
@jatkinson1000
Working pytorch bindings
1160352
@jatkinson1000
Updated pytorch bindings to handle output from model back to fortran
83fc53f
@jatkinson1000
Update pytorch build to match forpy inputs in nml
06d5e83
@jatkinson1000
WIP initial forpy bindings.
7575e9c
@SimonClifford
Implenent option for tensorflow coupling
62e5f66
@jatkinson1000
Forpy bindings uncompiled
d34f7fe
@jatkinson1000
Fix silly errors in forpy bindings.
2bb7462
@jatkinson1000
Update CMakeLists with forpy build option.
b9f021f
SimonCliffordand others added 22 commits April 6, 2023 15:26
@SimonClifford
Add null file for AD runs
2a98539
@jatkinson1000
add mpp timing for cg_drag
549b847
@SimonClifford
Fixes to get forpy working.
f44681a
@SimonClifford
Fixes to get direct TF working
0350424
@SimonClifford
Remove incorrect scaling for pressure
73df56d
It was there to test something.
@SimonClifford
Add a submit script for forpy+pytorch
cb5d9cb
@SimonClifford
Working forpy + Tensorflow
9b67fc1
@SimonClifford
Working Pytorch direct
d7cd746
@SimonClifford
Simplify (and optimise?) the pre-ML reshaping
0d8a4dd
@SimonClifford
Return cg_drag to normal, so job runs to completion
27339c9
@SimonClifford
Turn off verbose printing for Forpy+TF
04a54ce
@SimonClifford
Fix typo in submit_FP_PT script
029d167
@SimonClifford
Add scripts for AD99 runs, for completeness
1b97537
@jatkinson1000
Update README
1010f8c
@jatkinson1000
Merge branch 'ML' of github.com:DataWaveProject/MiMA into ML
22b8499
@jatkinson1000
Bug fix in surface pressure calculation
babba71
@jatkinson1000
Bug fix in surface pressure calculation
96f3dab
@SimonClifford
Apply fix for arg order and scaling pressure for models
d445996
The models' arguments were being sent in the wrong order.
It should be pressure, latitude, then wind.
Also the pressure should be in Pa, so no need for scale factor
of 100.
@SimonClifford
Add release submit scripts
5a27155
@SimonClifford
Make submit scripts use local disk
4183081
@SimonClifford
Remove debug printing
b319e62
@SimonClifford
Fix daft bug from previous commit
0a31ff3
@SimonClifford

SimonClifford commented May 23, 2023

Copy link
Copy Markdown

Closes #13 in the early @SimonClifford commits.

@SimonClifford

SimonClifford commented May 23, 2023

Copy link
Copy Markdown

@SimonClifford commits address TensorFlow coupling.
Closes #12
Closes #3

@SimonClifford

SimonClifford commented May 23, 2023

Copy link
Copy Markdown

@jatkinson1000 commits close #11 and close #12 and therefore close #4 .

@SimonClifford

SimonClifford commented May 23, 2023

Copy link
Copy Markdown

@jatkinson1000 closes #5, although we may revisit.

@jatkinson1000

jatkinson1000 commented May 24, 2023

Copy link
Copy Markdown
Member

Also closes #1 in earliest @jatkinson1000 commits (we could merge cmake into this branch to remove unused dependencies that caused failure on Sherlock).

CMake docs are included in docs/GettingStarted.md.

mondus
mondus commented May 24, 2023
View reviewed changes

@mondusmondus left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small suggestions.

  1. Move the CSD3 submissions scrips into a scripts\csd3 folder as they are hardware specific.
  2. Remove hardcoded home directories in the submissions scripts. Replace with a variable or argument.
  3. pytorch README.md could do with a bit more detail on what the scripts are. This could be opened as an issue for future work though.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@SimonCliffordSimonClifford

@jatkinson1000jatkinson1000

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Interfacing MiMA with ML codes

3 participants

@mondus@SimonClifford@jatkinson1000
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Feature: Interfacing ML code with MiMA - #10
Open
mondus wants to merge 63 commits into
masterfrom 
ML
Open
Feature: Interfacing ML code with MiMA#10
mondus wants to merge 63 commits into
masterfrom 
ML
Conversation
@mondus
@mondusmondus
commented
May 15, 2023•

edited by SimonClifford

Loading

Copy link


Copy Markdown

Closes#7
@jatkinson1000 - covering #4 and #5
  •  Adapt cg_drag.f90 to accommodate drag forcing from various different approaches
  •  Create .f90 files with modules & subroutines for relevant drag calculations
  •  Adapt PT NN to receive individual tensors of each variable and combine in a prepended layer
    
This way each approach will call the NN twice, once with zonal, once with meridional, but tensors will only be written to shared memory once.
@SimonClifford - covering #3
  •  Inspect PT model and construct matching TF model
  •  Migrate weights
  •  Add TF code into MiMA (requires @jatkinson1000 to write .f90 files above)
We then need to conduct benchmarking:
  •  Instrument MiMA for ReBench
  •  Plan a set of runs
  •  Perform the set of runs
jatkinson1000and others
added 30 commits
January 4, 2023 15:46
@jatkinson1000
First cmake attempt.
3e18ca7
@jatkinson1000
Compiling CMake build
b757b3f
@jatkinson1000
Working CMake Build
9a7c075
@jatkinson1000
Update GettingStarted docfile to include CMake build process.
2c94d91
@SimonClifford
Correct some typos and infelicities.
88f3ebe
@jatkinson1000
Bug fix in CMakeLists appending NETCDF-C to NETCDF_INCLUDES
e95d118
@jatkinson1000
Merge branch 'cmake' of github.com:Cambridge-ICCS/MiMA into cmake
837aff9
@jatkinson1000
Correct typo in FindNetCDF.cmake filename.
f620e6e
@jatkinson1000
Added DaveNet pytorch code from Minah in src/shared/pytorch.
ed08814
@jatkinson1000
black and linting of python code.
ac4ac3c
@jatkinson1000
Fix arguments to extend requiring list.
7a39414
@jatkinson1000
Add script to save davenet as TorchScript.
6c6fc1b
@jatkinson1000
cg_drag updated with placeholders ready for ML changes.
913b4eb
@SimonClifford
Script to make TF davenet
c8bcda8
@SimonClifford
Add README.md
ae53fda
@SimonClifford
Lint and isort
8e88f8b
@jatkinson1000
Initial workings on pytorch coupling routine [WIP].
02a9e6c
@jatkinson1000
Updated net to take 3 inputs and test it behaves as expected.
cbeeb95
@jatkinson1000
tweak architecture and update pt2ts to run with 3 inputs.
63e5e99
@SimonClifford
Merge branch 'pytorch' into tensorflow
70d41b8
@SimonClifford
Update TF davenet construction and add test
e4b63b9
Update to reflect model now takes 3 inputs and concatenates
them, rather than a pre-concatted input.
Test is very basic but shows that TF davenet does the same
thing as PyTorch davenet.
@jatkinson1000
Coupled code for pytorch (untested compiling).
e56ac02
@jatkinson1000
Working pytorch bindings
1160352
@jatkinson1000
Updated pytorch bindings to handle output from model back to fortran
83fc53f
@jatkinson1000
Update pytorch build to match forpy inputs in nml
06d5e83
@jatkinson1000
WIP initial forpy bindings.
7575e9c
@SimonClifford
Implenent option for tensorflow coupling
62e5f66
@jatkinson1000
Forpy bindings uncompiled
d34f7fe
@jatkinson1000
Fix silly errors in forpy bindings.
2bb7462
@jatkinson1000
Update CMakeLists with forpy build option.
b9f021f
SimonCliffordand others
added 22 commits
April 6, 2023 15:26
@SimonClifford
Add null file for AD runs
2a98539
@jatkinson1000
add mpp timing for cg_drag
549b847
@SimonClifford
Fixes to get forpy working.
f44681a
@SimonClifford
Fixes to get direct TF working
0350424
@SimonClifford
Remove incorrect scaling for pressure
73df56d
It was there to test something.
@SimonClifford
Add a submit script for forpy+pytorch
cb5d9cb
@SimonClifford
Working forpy + Tensorflow
9b67fc1
@SimonClifford
Working Pytorch direct
d7cd746
@SimonClifford
Simplify (and optimise?) the pre-ML reshaping
0d8a4dd
@SimonClifford
Return cg_drag to normal, so job runs to completion
27339c9
@SimonClifford
Turn off verbose printing for Forpy+TF
04a54ce
@SimonClifford
Fix typo in submit_FP_PT script
029d167
@SimonClifford
Add scripts for AD99 runs, for completeness
1b97537
@jatkinson1000
Update README
1010f8c
@jatkinson1000
Merge branch 'ML' of github.com:DataWaveProject/MiMA into ML
22b8499
@jatkinson1000
Bug fix in surface pressure calculation
babba71
@jatkinson1000
Bug fix in surface pressure calculation
96f3dab
@SimonClifford
Apply fix for arg order and scaling pressure for models
d445996
The models' arguments were being sent in the wrong order.
It should be pressure, latitude, then wind.
Also the pressure should be in Pa, so no need for scale factor
of 100.
@SimonClifford
Add release submit scripts
5a27155
@SimonClifford
Make submit scripts use local disk
4183081
@SimonClifford
Remove debug printing
b319e62
@SimonClifford
Fix daft bug from previous commit
0a31ff3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

Closes #13 in the early @SimonClifford commits.
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@SimonClifford commits address TensorFlow coupling.

Closes #12

Closes #3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 commits close #11 and close #12 and therefore close #4 .
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 closes #5, although we may revisit.
@jatkinson1000
jatkinson1000
commented
May 24, 2023

Copy link


Copy Markdown

Member
Also closes #1 in earliest @jatkinson1000 commits (we could merge cmake into this branch to remove unused dependencies that caused failure on Sherlock).
CMake docs are included in docs/GettingStarted.md.
mondus
mondus
commented
May 24, 2023
View reviewed changes
@mondusmondus
left a comment


Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small suggestions.
  1. Move the CSD3 submissions scrips into a scripts\csd3 folder as they are hardware specific.
  2. Remove hardcoded home directories in the submissions scripts. Replace with a variable or argument.
  3. pytorch README.md could do with a bit more detail on what the scripts are. This could be opened as an issue for future work though.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees

@SimonCliffordSimonClifford
@jatkinson1000jatkinson1000

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Interfacing MiMA with ML codes


3 participants

@mondus@SimonClifford@jatkinson1000

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Feature: Interfacing ML code with MiMA - #10
Open
mondus wants to merge 63 commits into
masterfrom 
ML
Open
Feature: Interfacing ML code with MiMA#10
mondus wants to merge 63 commits into
masterfrom 
ML
Conversation
@mondus
@mondusmondus
commented
May 15, 2023•

edited by SimonClifford

Loading

Copy link


Copy Markdown

Closes#7
@jatkinson1000 - covering #4 and #5
  •  Adapt cg_drag.f90 to accommodate drag forcing from various different approaches
  •  Create .f90 files with modules & subroutines for relevant drag calculations
  •  Adapt PT NN to receive individual tensors of each variable and combine in a prepended layer
    
This way each approach will call the NN twice, once with zonal, once with meridional, but tensors will only be written to shared memory once.
@SimonClifford - covering #3
  •  Inspect PT model and construct matching TF model
  •  Migrate weights
  •  Add TF code into MiMA (requires @jatkinson1000 to write .f90 files above)
We then need to conduct benchmarking:
  •  Instrument MiMA for ReBench
  •  Plan a set of runs
  •  Perform the set of runs
jatkinson1000and others
added 30 commits
January 4, 2023 15:46
@jatkinson1000
First cmake attempt.
3e18ca7
@jatkinson1000
Compiling CMake build
b757b3f
@jatkinson1000
Working CMake Build
9a7c075
@jatkinson1000
Update GettingStarted docfile to include CMake build process.
2c94d91
@SimonClifford
Correct some typos and infelicities.
88f3ebe
@jatkinson1000
Bug fix in CMakeLists appending NETCDF-C to NETCDF_INCLUDES
e95d118
@jatkinson1000
Merge branch 'cmake' of github.com:Cambridge-ICCS/MiMA into cmake
837aff9
@jatkinson1000
Correct typo in FindNetCDF.cmake filename.
f620e6e
@jatkinson1000
Added DaveNet pytorch code from Minah in src/shared/pytorch.
ed08814
@jatkinson1000
black and linting of python code.
ac4ac3c
@jatkinson1000
Fix arguments to extend requiring list.
7a39414
@jatkinson1000
Add script to save davenet as TorchScript.
6c6fc1b
@jatkinson1000
cg_drag updated with placeholders ready for ML changes.
913b4eb
@SimonClifford
Script to make TF davenet
c8bcda8
@SimonClifford
Add README.md
ae53fda
@SimonClifford
Lint and isort
8e88f8b
@jatkinson1000
Initial workings on pytorch coupling routine [WIP].
02a9e6c
@jatkinson1000
Updated net to take 3 inputs and test it behaves as expected.
cbeeb95
@jatkinson1000
tweak architecture and update pt2ts to run with 3 inputs.
63e5e99
@SimonClifford
Merge branch 'pytorch' into tensorflow
70d41b8
@SimonClifford
Update TF davenet construction and add test
e4b63b9
Update to reflect model now takes 3 inputs and concatenates
them, rather than a pre-concatted input.
Test is very basic but shows that TF davenet does the same
thing as PyTorch davenet.
@jatkinson1000
Coupled code for pytorch (untested compiling).
e56ac02
@jatkinson1000
Working pytorch bindings
1160352
@jatkinson1000
Updated pytorch bindings to handle output from model back to fortran
83fc53f
@jatkinson1000
Update pytorch build to match forpy inputs in nml
06d5e83
@jatkinson1000
WIP initial forpy bindings.
7575e9c
@SimonClifford
Implenent option for tensorflow coupling
62e5f66
@jatkinson1000
Forpy bindings uncompiled
d34f7fe
@jatkinson1000
Fix silly errors in forpy bindings.
2bb7462
@jatkinson1000
Update CMakeLists with forpy build option.
b9f021f
SimonCliffordand others
added 22 commits
April 6, 2023 15:26
@SimonClifford
Add null file for AD runs
2a98539
@jatkinson1000
add mpp timing for cg_drag
549b847
@SimonClifford
Fixes to get forpy working.
f44681a
@SimonClifford
Fixes to get direct TF working
0350424
@SimonClifford
Remove incorrect scaling for pressure
73df56d
It was there to test something.
@SimonClifford
Add a submit script for forpy+pytorch
cb5d9cb
@SimonClifford
Working forpy + Tensorflow
9b67fc1
@SimonClifford
Working Pytorch direct
d7cd746
@SimonClifford
Simplify (and optimise?) the pre-ML reshaping
0d8a4dd
@SimonClifford
Return cg_drag to normal, so job runs to completion
27339c9
@SimonClifford
Turn off verbose printing for Forpy+TF
04a54ce
@SimonClifford
Fix typo in submit_FP_PT script
029d167
@SimonClifford
Add scripts for AD99 runs, for completeness
1b97537
@jatkinson1000
Update README
1010f8c
@jatkinson1000
Merge branch 'ML' of github.com:DataWaveProject/MiMA into ML
22b8499
@jatkinson1000
Bug fix in surface pressure calculation
babba71
@jatkinson1000
Bug fix in surface pressure calculation
96f3dab
@SimonClifford
Apply fix for arg order and scaling pressure for models
d445996
The models' arguments were being sent in the wrong order.
It should be pressure, latitude, then wind.
Also the pressure should be in Pa, so no need for scale factor
of 100.
@SimonClifford
Add release submit scripts
5a27155
@SimonClifford
Make submit scripts use local disk
4183081
@SimonClifford
Remove debug printing
b319e62
@SimonClifford
Fix daft bug from previous commit
0a31ff3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

Closes #13 in the early @SimonClifford commits.
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@SimonClifford commits address TensorFlow coupling.

Closes #12

Closes #3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 commits close #11 and close #12 and therefore close #4 .
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 closes #5, although we may revisit.
@jatkinson1000
jatkinson1000
commented
May 24, 2023

Copy link


Copy Markdown

Member
Also closes #1 in earliest @jatkinson1000 commits (we could merge cmake into this branch to remove unused dependencies that caused failure on Sherlock).
CMake docs are included in docs/GettingStarted.md.
mondus
mondus
commented
May 24, 2023
View reviewed changes
@mondusmondus
left a comment


Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small suggestions.
  1. Move the CSD3 submissions scrips into a scripts\csd3 folder as they are hardware specific.
  2. Remove hardcoded home directories in the submissions scripts. Replace with a variable or argument.
  3. pytorch README.md could do with a bit more detail on what the scripts are. This could be opened as an issue for future work though.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees

@SimonCliffordSimonClifford
@jatkinson1000jatkinson1000

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Interfacing MiMA with ML codes


3 participants

@mondus@SimonClifford@jatkinson1000

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Feature: Interfacing ML code with MiMA - #10
Open
mondus wants to merge 63 commits into
masterfrom 
ML
Open
Feature: Interfacing ML code with MiMA#10
mondus wants to merge 63 commits into
masterfrom 
ML
Conversation
@mondus
@mondusmondus
commented
May 15, 2023•

edited by SimonClifford

Loading

Copy link


Copy Markdown

Closes#7
@jatkinson1000 - covering #4 and #5
  •  Adapt cg_drag.f90 to accommodate drag forcing from various different approaches
  •  Create .f90 files with modules & subroutines for relevant drag calculations
  •  Adapt PT NN to receive individual tensors of each variable and combine in a prepended layer
    
This way each approach will call the NN twice, once with zonal, once with meridional, but tensors will only be written to shared memory once.
@SimonClifford - covering #3
  •  Inspect PT model and construct matching TF model
  •  Migrate weights
  •  Add TF code into MiMA (requires @jatkinson1000 to write .f90 files above)
We then need to conduct benchmarking:
  •  Instrument MiMA for ReBench
  •  Plan a set of runs
  •  Perform the set of runs
jatkinson1000and others
added 30 commits
January 4, 2023 15:46
@jatkinson1000
First cmake attempt.
3e18ca7
@jatkinson1000
Compiling CMake build
b757b3f
@jatkinson1000
Working CMake Build
9a7c075
@jatkinson1000
Update GettingStarted docfile to include CMake build process.
2c94d91
@SimonClifford
Correct some typos and infelicities.
88f3ebe
@jatkinson1000
Bug fix in CMakeLists appending NETCDF-C to NETCDF_INCLUDES
e95d118
@jatkinson1000
Merge branch 'cmake' of github.com:Cambridge-ICCS/MiMA into cmake
837aff9
@jatkinson1000
Correct typo in FindNetCDF.cmake filename.
f620e6e
@jatkinson1000
Added DaveNet pytorch code from Minah in src/shared/pytorch.
ed08814
@jatkinson1000
black and linting of python code.
ac4ac3c
@jatkinson1000
Fix arguments to extend requiring list.
7a39414
@jatkinson1000
Add script to save davenet as TorchScript.
6c6fc1b
@jatkinson1000
cg_drag updated with placeholders ready for ML changes.
913b4eb
@SimonClifford
Script to make TF davenet
c8bcda8
@SimonClifford
Add README.md
ae53fda
@SimonClifford
Lint and isort
8e88f8b
@jatkinson1000
Initial workings on pytorch coupling routine [WIP].
02a9e6c
@jatkinson1000
Updated net to take 3 inputs and test it behaves as expected.
cbeeb95
@jatkinson1000
tweak architecture and update pt2ts to run with 3 inputs.
63e5e99
@SimonClifford
Merge branch 'pytorch' into tensorflow
70d41b8
@SimonClifford
Update TF davenet construction and add test
e4b63b9
Update to reflect model now takes 3 inputs and concatenates
them, rather than a pre-concatted input.
Test is very basic but shows that TF davenet does the same
thing as PyTorch davenet.
@jatkinson1000
Coupled code for pytorch (untested compiling).
e56ac02
@jatkinson1000
Working pytorch bindings
1160352
@jatkinson1000
Updated pytorch bindings to handle output from model back to fortran
83fc53f
@jatkinson1000
Update pytorch build to match forpy inputs in nml
06d5e83
@jatkinson1000
WIP initial forpy bindings.
7575e9c
@SimonClifford
Implenent option for tensorflow coupling
62e5f66
@jatkinson1000
Forpy bindings uncompiled
d34f7fe
@jatkinson1000
Fix silly errors in forpy bindings.
2bb7462
@jatkinson1000
Update CMakeLists with forpy build option.
b9f021f
SimonCliffordand others
added 22 commits
April 6, 2023 15:26
@SimonClifford
Add null file for AD runs
2a98539
@jatkinson1000
add mpp timing for cg_drag
549b847
@SimonClifford
Fixes to get forpy working.
f44681a
@SimonClifford
Fixes to get direct TF working
0350424
@SimonClifford
Remove incorrect scaling for pressure
73df56d
It was there to test something.
@SimonClifford
Add a submit script for forpy+pytorch
cb5d9cb
@SimonClifford
Working forpy + Tensorflow
9b67fc1
@SimonClifford
Working Pytorch direct
d7cd746
@SimonClifford
Simplify (and optimise?) the pre-ML reshaping
0d8a4dd
@SimonClifford
Return cg_drag to normal, so job runs to completion
27339c9
@SimonClifford
Turn off verbose printing for Forpy+TF
04a54ce
@SimonClifford
Fix typo in submit_FP_PT script
029d167
@SimonClifford
Add scripts for AD99 runs, for completeness
1b97537
@jatkinson1000
Update README
1010f8c
@jatkinson1000
Merge branch 'ML' of github.com:DataWaveProject/MiMA into ML
22b8499
@jatkinson1000
Bug fix in surface pressure calculation
babba71
@jatkinson1000
Bug fix in surface pressure calculation
96f3dab
@SimonClifford
Apply fix for arg order and scaling pressure for models
d445996
The models' arguments were being sent in the wrong order.
It should be pressure, latitude, then wind.
Also the pressure should be in Pa, so no need for scale factor
of 100.
@SimonClifford
Add release submit scripts
5a27155
@SimonClifford
Make submit scripts use local disk
4183081
@SimonClifford
Remove debug printing
b319e62
@SimonClifford
Fix daft bug from previous commit
0a31ff3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

Closes #13 in the early @SimonClifford commits.
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@SimonClifford commits address TensorFlow coupling.

Closes #12

Closes #3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 commits close #11 and close #12 and therefore close #4 .
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 closes #5, although we may revisit.
@jatkinson1000
jatkinson1000
commented
May 24, 2023

Copy link


Copy Markdown

Member
Also closes #1 in earliest @jatkinson1000 commits (we could merge cmake into this branch to remove unused dependencies that caused failure on Sherlock).
CMake docs are included in docs/GettingStarted.md.
mondus
mondus
commented
May 24, 2023
View reviewed changes
@mondusmondus
left a comment


Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small suggestions.
  1. Move the CSD3 submissions scrips into a scripts\csd3 folder as they are hardware specific.
  2. Remove hardcoded home directories in the submissions scripts. Replace with a variable or argument.
  3. pytorch README.md could do with a bit more detail on what the scripts are. This could be opened as an issue for future work though.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees

@SimonCliffordSimonClifford
@jatkinson1000jatkinson1000

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Interfacing MiMA with ML codes


3 participants

@mondus@SimonClifford@jatkinson1000

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Feature: Interfacing ML code with MiMA - #10
Open
mondus wants to merge 63 commits into
masterfrom 
ML
Open
Feature: Interfacing ML code with MiMA#10
mondus wants to merge 63 commits into
masterfrom 
ML
Conversation
@mondus
@mondusmondus
commented
May 15, 2023•

edited by SimonClifford

Loading

Copy link


Copy Markdown

Closes#7
@jatkinson1000 - covering #4 and #5
  •  Adapt cg_drag.f90 to accommodate drag forcing from various different approaches
  •  Create .f90 files with modules & subroutines for relevant drag calculations
  •  Adapt PT NN to receive individual tensors of each variable and combine in a prepended layer
    
This way each approach will call the NN twice, once with zonal, once with meridional, but tensors will only be written to shared memory once.
@SimonClifford - covering #3
  •  Inspect PT model and construct matching TF model
  •  Migrate weights
  •  Add TF code into MiMA (requires @jatkinson1000 to write .f90 files above)
We then need to conduct benchmarking:
  •  Instrument MiMA for ReBench
  •  Plan a set of runs
  •  Perform the set of runs
jatkinson1000and others
added 30 commits
January 4, 2023 15:46
@jatkinson1000
First cmake attempt.
3e18ca7
@jatkinson1000
Compiling CMake build
b757b3f
@jatkinson1000
Working CMake Build
9a7c075
@jatkinson1000
Update GettingStarted docfile to include CMake build process.
2c94d91
@SimonClifford
Correct some typos and infelicities.
88f3ebe
@jatkinson1000
Bug fix in CMakeLists appending NETCDF-C to NETCDF_INCLUDES
e95d118
@jatkinson1000
Merge branch 'cmake' of github.com:Cambridge-ICCS/MiMA into cmake
837aff9
@jatkinson1000
Correct typo in FindNetCDF.cmake filename.
f620e6e
@jatkinson1000
Added DaveNet pytorch code from Minah in src/shared/pytorch.
ed08814
@jatkinson1000
black and linting of python code.
ac4ac3c
@jatkinson1000
Fix arguments to extend requiring list.
7a39414
@jatkinson1000
Add script to save davenet as TorchScript.
6c6fc1b
@jatkinson1000
cg_drag updated with placeholders ready for ML changes.
913b4eb
@SimonClifford
Script to make TF davenet
c8bcda8
@SimonClifford
Add README.md
ae53fda
@SimonClifford
Lint and isort
8e88f8b
@jatkinson1000
Initial workings on pytorch coupling routine [WIP].
02a9e6c
@jatkinson1000
Updated net to take 3 inputs and test it behaves as expected.
cbeeb95
@jatkinson1000
tweak architecture and update pt2ts to run with 3 inputs.
63e5e99
@SimonClifford
Merge branch 'pytorch' into tensorflow
70d41b8
@SimonClifford
Update TF davenet construction and add test
e4b63b9
Update to reflect model now takes 3 inputs and concatenates
them, rather than a pre-concatted input.
Test is very basic but shows that TF davenet does the same
thing as PyTorch davenet.
@jatkinson1000
Coupled code for pytorch (untested compiling).
e56ac02
@jatkinson1000
Working pytorch bindings
1160352
@jatkinson1000
Updated pytorch bindings to handle output from model back to fortran
83fc53f
@jatkinson1000
Update pytorch build to match forpy inputs in nml
06d5e83
@jatkinson1000
WIP initial forpy bindings.
7575e9c
@SimonClifford
Implenent option for tensorflow coupling
62e5f66
@jatkinson1000
Forpy bindings uncompiled
d34f7fe
@jatkinson1000
Fix silly errors in forpy bindings.
2bb7462
@jatkinson1000
Update CMakeLists with forpy build option.
b9f021f
SimonCliffordand others
added 22 commits
April 6, 2023 15:26
@SimonClifford
Add null file for AD runs
2a98539
@jatkinson1000
add mpp timing for cg_drag
549b847
@SimonClifford
Fixes to get forpy working.
f44681a
@SimonClifford
Fixes to get direct TF working
0350424
@SimonClifford
Remove incorrect scaling for pressure
73df56d
It was there to test something.
@SimonClifford
Add a submit script for forpy+pytorch
cb5d9cb
@SimonClifford
Working forpy + Tensorflow
9b67fc1
@SimonClifford
Working Pytorch direct
d7cd746
@SimonClifford
Simplify (and optimise?) the pre-ML reshaping
0d8a4dd
@SimonClifford
Return cg_drag to normal, so job runs to completion
27339c9
@SimonClifford
Turn off verbose printing for Forpy+TF
04a54ce
@SimonClifford
Fix typo in submit_FP_PT script
029d167
@SimonClifford
Add scripts for AD99 runs, for completeness
1b97537
@jatkinson1000
Update README
1010f8c
@jatkinson1000
Merge branch 'ML' of github.com:DataWaveProject/MiMA into ML
22b8499
@jatkinson1000
Bug fix in surface pressure calculation
babba71
@jatkinson1000
Bug fix in surface pressure calculation
96f3dab
@SimonClifford
Apply fix for arg order and scaling pressure for models
d445996
The models' arguments were being sent in the wrong order.
It should be pressure, latitude, then wind.
Also the pressure should be in Pa, so no need for scale factor
of 100.
@SimonClifford
Add release submit scripts
5a27155
@SimonClifford
Make submit scripts use local disk
4183081
@SimonClifford
Remove debug printing
b319e62
@SimonClifford
Fix daft bug from previous commit
0a31ff3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

Closes #13 in the early @SimonClifford commits.
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@SimonClifford commits address TensorFlow coupling.

Closes #12

Closes #3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 commits close #11 and close #12 and therefore close #4 .
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 closes #5, although we may revisit.
@jatkinson1000
jatkinson1000
commented
May 24, 2023

Copy link


Copy Markdown

Member
Also closes #1 in earliest @jatkinson1000 commits (we could merge cmake into this branch to remove unused dependencies that caused failure on Sherlock).
CMake docs are included in docs/GettingStarted.md.
mondus
mondus
commented
May 24, 2023
View reviewed changes
@mondusmondus
left a comment


Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small suggestions.
  1. Move the CSD3 submissions scrips into a scripts\csd3 folder as they are hardware specific.
  2. Remove hardcoded home directories in the submissions scripts. Replace with a variable or argument.
  3. pytorch README.md could do with a bit more detail on what the scripts are. This could be opened as an issue for future work though.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees

@SimonCliffordSimonClifford
@jatkinson1000jatkinson1000

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Interfacing MiMA with ML codes


3 participants

@mondus@SimonClifford@jatkinson1000

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Feature: Interfacing ML code with MiMA - #10
Open
mondus wants to merge 63 commits into
masterfrom 
ML
Open
Feature: Interfacing ML code with MiMA#10
mondus wants to merge 63 commits into
masterfrom 
ML
Conversation
@mondus
@mondusmondus
commented
May 15, 2023•

edited by SimonClifford

Loading

Copy link


Copy Markdown

Closes#7
@jatkinson1000 - covering #4 and #5
  •  Adapt cg_drag.f90 to accommodate drag forcing from various different approaches
  •  Create .f90 files with modules & subroutines for relevant drag calculations
  •  Adapt PT NN to receive individual tensors of each variable and combine in a prepended layer
    
This way each approach will call the NN twice, once with zonal, once with meridional, but tensors will only be written to shared memory once.
@SimonClifford - covering #3
  •  Inspect PT model and construct matching TF model
  •  Migrate weights
  •  Add TF code into MiMA (requires @jatkinson1000 to write .f90 files above)
We then need to conduct benchmarking:
  •  Instrument MiMA for ReBench
  •  Plan a set of runs
  •  Perform the set of runs
jatkinson1000and others
added 30 commits
January 4, 2023 15:46
@jatkinson1000
First cmake attempt.
3e18ca7
@jatkinson1000
Compiling CMake build
b757b3f
@jatkinson1000
Working CMake Build
9a7c075
@jatkinson1000
Update GettingStarted docfile to include CMake build process.
2c94d91
@SimonClifford
Correct some typos and infelicities.
88f3ebe
@jatkinson1000
Bug fix in CMakeLists appending NETCDF-C to NETCDF_INCLUDES
e95d118
@jatkinson1000
Merge branch 'cmake' of github.com:Cambridge-ICCS/MiMA into cmake
837aff9
@jatkinson1000
Correct typo in FindNetCDF.cmake filename.
f620e6e
@jatkinson1000
Added DaveNet pytorch code from Minah in src/shared/pytorch.
ed08814
@jatkinson1000
black and linting of python code.
ac4ac3c
@jatkinson1000
Fix arguments to extend requiring list.
7a39414
@jatkinson1000
Add script to save davenet as TorchScript.
6c6fc1b
@jatkinson1000
cg_drag updated with placeholders ready for ML changes.
913b4eb
@SimonClifford
Script to make TF davenet
c8bcda8
@SimonClifford
Add README.md
ae53fda
@SimonClifford
Lint and isort
8e88f8b
@jatkinson1000
Initial workings on pytorch coupling routine [WIP].
02a9e6c
@jatkinson1000
Updated net to take 3 inputs and test it behaves as expected.
cbeeb95
@jatkinson1000
tweak architecture and update pt2ts to run with 3 inputs.
63e5e99
@SimonClifford
Merge branch 'pytorch' into tensorflow
70d41b8
@SimonClifford
Update TF davenet construction and add test
e4b63b9
Update to reflect model now takes 3 inputs and concatenates
them, rather than a pre-concatted input.
Test is very basic but shows that TF davenet does the same
thing as PyTorch davenet.
@jatkinson1000
Coupled code for pytorch (untested compiling).
e56ac02
@jatkinson1000
Working pytorch bindings
1160352
@jatkinson1000
Updated pytorch bindings to handle output from model back to fortran
83fc53f
@jatkinson1000
Update pytorch build to match forpy inputs in nml
06d5e83
@jatkinson1000
WIP initial forpy bindings.
7575e9c
@SimonClifford
Implenent option for tensorflow coupling
62e5f66
@jatkinson1000
Forpy bindings uncompiled
d34f7fe
@jatkinson1000
Fix silly errors in forpy bindings.
2bb7462
@jatkinson1000
Update CMakeLists with forpy build option.
b9f021f
SimonCliffordand others
added 22 commits
April 6, 2023 15:26
@SimonClifford
Add null file for AD runs
2a98539
@jatkinson1000
add mpp timing for cg_drag
549b847
@SimonClifford
Fixes to get forpy working.
f44681a
@SimonClifford
Fixes to get direct TF working
0350424
@SimonClifford
Remove incorrect scaling for pressure
73df56d
It was there to test something.
@SimonClifford
Add a submit script for forpy+pytorch
cb5d9cb
@SimonClifford
Working forpy + Tensorflow
9b67fc1
@SimonClifford
Working Pytorch direct
d7cd746
@SimonClifford
Simplify (and optimise?) the pre-ML reshaping
0d8a4dd
@SimonClifford
Return cg_drag to normal, so job runs to completion
27339c9
@SimonClifford
Turn off verbose printing for Forpy+TF
04a54ce
@SimonClifford
Fix typo in submit_FP_PT script
029d167
@SimonClifford
Add scripts for AD99 runs, for completeness
1b97537
@jatkinson1000
Update README
1010f8c
@jatkinson1000
Merge branch 'ML' of github.com:DataWaveProject/MiMA into ML
22b8499
@jatkinson1000
Bug fix in surface pressure calculation
babba71
@jatkinson1000
Bug fix in surface pressure calculation
96f3dab
@SimonClifford
Apply fix for arg order and scaling pressure for models
d445996
The models' arguments were being sent in the wrong order.
It should be pressure, latitude, then wind.
Also the pressure should be in Pa, so no need for scale factor
of 100.
@SimonClifford
Add release submit scripts
5a27155
@SimonClifford
Make submit scripts use local disk
4183081
@SimonClifford
Remove debug printing
b319e62
@SimonClifford
Fix daft bug from previous commit
0a31ff3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

Closes #13 in the early @SimonClifford commits.
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@SimonClifford commits address TensorFlow coupling.

Closes #12

Closes #3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 commits close #11 and close #12 and therefore close #4 .
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 closes #5, although we may revisit.
@jatkinson1000
jatkinson1000
commented
May 24, 2023

Copy link


Copy Markdown

Member
Also closes #1 in earliest @jatkinson1000 commits (we could merge cmake into this branch to remove unused dependencies that caused failure on Sherlock).
CMake docs are included in docs/GettingStarted.md.
mondus
mondus
commented
May 24, 2023
View reviewed changes
@mondusmondus
left a comment


Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small suggestions.
  1. Move the CSD3 submissions scrips into a scripts\csd3 folder as they are hardware specific.
  2. Remove hardcoded home directories in the submissions scripts. Replace with a variable or argument.
  3. pytorch README.md could do with a bit more detail on what the scripts are. This could be opened as an issue for future work though.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees

@SimonCliffordSimonClifford
@jatkinson1000jatkinson1000

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Interfacing MiMA with ML codes


3 participants

@mondus@SimonClifford@jatkinson1000

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Feature: Interfacing ML code with MiMA - #10
Open
mondus wants to merge 63 commits into
masterfrom 
ML
Open
Feature: Interfacing ML code with MiMA#10
mondus wants to merge 63 commits into
masterfrom 
ML
Conversation
@mondus
@mondusmondus
commented
May 15, 2023•

edited by SimonClifford

Loading

Copy link


Copy Markdown

Closes#7
@jatkinson1000 - covering #4 and #5
  •  Adapt cg_drag.f90 to accommodate drag forcing from various different approaches
  •  Create .f90 files with modules & subroutines for relevant drag calculations
  •  Adapt PT NN to receive individual tensors of each variable and combine in a prepended layer
    
This way each approach will call the NN twice, once with zonal, once with meridional, but tensors will only be written to shared memory once.
@SimonClifford - covering #3
  •  Inspect PT model and construct matching TF model
  •  Migrate weights
  •  Add TF code into MiMA (requires @jatkinson1000 to write .f90 files above)
We then need to conduct benchmarking:
  •  Instrument MiMA for ReBench
  •  Plan a set of runs
  •  Perform the set of runs
jatkinson1000and others
added 30 commits
January 4, 2023 15:46
@jatkinson1000
First cmake attempt.
3e18ca7
@jatkinson1000
Compiling CMake build
b757b3f
@jatkinson1000
Working CMake Build
9a7c075
@jatkinson1000
Update GettingStarted docfile to include CMake build process.
2c94d91
@SimonClifford
Correct some typos and infelicities.
88f3ebe
@jatkinson1000
Bug fix in CMakeLists appending NETCDF-C to NETCDF_INCLUDES
e95d118
@jatkinson1000
Merge branch 'cmake' of github.com:Cambridge-ICCS/MiMA into cmake
837aff9
@jatkinson1000
Correct typo in FindNetCDF.cmake filename.
f620e6e
@jatkinson1000
Added DaveNet pytorch code from Minah in src/shared/pytorch.
ed08814
@jatkinson1000
black and linting of python code.
ac4ac3c
@jatkinson1000
Fix arguments to extend requiring list.
7a39414
@jatkinson1000
Add script to save davenet as TorchScript.
6c6fc1b
@jatkinson1000
cg_drag updated with placeholders ready for ML changes.
913b4eb
@SimonClifford
Script to make TF davenet
c8bcda8
@SimonClifford
Add README.md
ae53fda
@SimonClifford
Lint and isort
8e88f8b
@jatkinson1000
Initial workings on pytorch coupling routine [WIP].
02a9e6c
@jatkinson1000
Updated net to take 3 inputs and test it behaves as expected.
cbeeb95
@jatkinson1000
tweak architecture and update pt2ts to run with 3 inputs.
63e5e99
@SimonClifford
Merge branch 'pytorch' into tensorflow
70d41b8
@SimonClifford
Update TF davenet construction and add test
e4b63b9
Update to reflect model now takes 3 inputs and concatenates
them, rather than a pre-concatted input.
Test is very basic but shows that TF davenet does the same
thing as PyTorch davenet.
@jatkinson1000
Coupled code for pytorch (untested compiling).
e56ac02
@jatkinson1000
Working pytorch bindings
1160352
@jatkinson1000
Updated pytorch bindings to handle output from model back to fortran
83fc53f
@jatkinson1000
Update pytorch build to match forpy inputs in nml
06d5e83
@jatkinson1000
WIP initial forpy bindings.
7575e9c
@SimonClifford
Implenent option for tensorflow coupling
62e5f66
@jatkinson1000
Forpy bindings uncompiled
d34f7fe
@jatkinson1000
Fix silly errors in forpy bindings.
2bb7462
@jatkinson1000
Update CMakeLists with forpy build option.
b9f021f
SimonCliffordand others
added 22 commits
April 6, 2023 15:26
@SimonClifford
Add null file for AD runs
2a98539
@jatkinson1000
add mpp timing for cg_drag
549b847
@SimonClifford
Fixes to get forpy working.
f44681a
@SimonClifford
Fixes to get direct TF working
0350424
@SimonClifford
Remove incorrect scaling for pressure
73df56d
It was there to test something.
@SimonClifford
Add a submit script for forpy+pytorch
cb5d9cb
@SimonClifford
Working forpy + Tensorflow
9b67fc1
@SimonClifford
Working Pytorch direct
d7cd746
@SimonClifford
Simplify (and optimise?) the pre-ML reshaping
0d8a4dd
@SimonClifford
Return cg_drag to normal, so job runs to completion
27339c9
@SimonClifford
Turn off verbose printing for Forpy+TF
04a54ce
@SimonClifford
Fix typo in submit_FP_PT script
029d167
@SimonClifford
Add scripts for AD99 runs, for completeness
1b97537
@jatkinson1000
Update README
1010f8c
@jatkinson1000
Merge branch 'ML' of github.com:DataWaveProject/MiMA into ML
22b8499
@jatkinson1000
Bug fix in surface pressure calculation
babba71
@jatkinson1000
Bug fix in surface pressure calculation
96f3dab
@SimonClifford
Apply fix for arg order and scaling pressure for models
d445996
The models' arguments were being sent in the wrong order.
It should be pressure, latitude, then wind.
Also the pressure should be in Pa, so no need for scale factor
of 100.
@SimonClifford
Add release submit scripts
5a27155
@SimonClifford
Make submit scripts use local disk
4183081
@SimonClifford
Remove debug printing
b319e62
@SimonClifford
Fix daft bug from previous commit
0a31ff3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

Closes #13 in the early @SimonClifford commits.
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@SimonClifford commits address TensorFlow coupling.

Closes #12

Closes #3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 commits close #11 and close #12 and therefore close #4 .
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 closes #5, although we may revisit.
@jatkinson1000
jatkinson1000
commented
May 24, 2023

Copy link


Copy Markdown

Member
Also closes #1 in earliest @jatkinson1000 commits (we could merge cmake into this branch to remove unused dependencies that caused failure on Sherlock).
CMake docs are included in docs/GettingStarted.md.
mondus
mondus
commented
May 24, 2023
View reviewed changes
@mondusmondus
left a comment


Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small suggestions.
  1. Move the CSD3 submissions scrips into a scripts\csd3 folder as they are hardware specific.
  2. Remove hardcoded home directories in the submissions scripts. Replace with a variable or argument.
  3. pytorch README.md could do with a bit more detail on what the scripts are. This could be opened as an issue for future work though.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees

@SimonCliffordSimonClifford
@jatkinson1000jatkinson1000

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Interfacing MiMA with ML codes


3 participants

@mondus@SimonClifford@jatkinson1000

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Feature: Interfacing ML code with MiMA - #10
Open
mondus wants to merge 63 commits into
masterfrom 
ML
Open
Feature: Interfacing ML code with MiMA#10
mondus wants to merge 63 commits into
masterfrom 
ML
Conversation
@mondus
@mondusmondus
commented
May 15, 2023•

edited by SimonClifford

Loading

Copy link


Copy Markdown

Closes#7
@jatkinson1000 - covering #4 and #5
  •  Adapt cg_drag.f90 to accommodate drag forcing from various different approaches
  •  Create .f90 files with modules & subroutines for relevant drag calculations
  •  Adapt PT NN to receive individual tensors of each variable and combine in a prepended layer
    
This way each approach will call the NN twice, once with zonal, once with meridional, but tensors will only be written to shared memory once.
@SimonClifford - covering #3
  •  Inspect PT model and construct matching TF model
  •  Migrate weights
  •  Add TF code into MiMA (requires @jatkinson1000 to write .f90 files above)
We then need to conduct benchmarking:
  •  Instrument MiMA for ReBench
  •  Plan a set of runs
  •  Perform the set of runs
jatkinson1000and others
added 30 commits
January 4, 2023 15:46
@jatkinson1000
First cmake attempt.
3e18ca7
@jatkinson1000
Compiling CMake build
b757b3f
@jatkinson1000
Working CMake Build
9a7c075
@jatkinson1000
Update GettingStarted docfile to include CMake build process.
2c94d91
@SimonClifford
Correct some typos and infelicities.
88f3ebe
@jatkinson1000
Bug fix in CMakeLists appending NETCDF-C to NETCDF_INCLUDES
e95d118
@jatkinson1000
Merge branch 'cmake' of github.com:Cambridge-ICCS/MiMA into cmake
837aff9
@jatkinson1000
Correct typo in FindNetCDF.cmake filename.
f620e6e
@jatkinson1000
Added DaveNet pytorch code from Minah in src/shared/pytorch.
ed08814
@jatkinson1000
black and linting of python code.
ac4ac3c
@jatkinson1000
Fix arguments to extend requiring list.
7a39414
@jatkinson1000
Add script to save davenet as TorchScript.
6c6fc1b
@jatkinson1000
cg_drag updated with placeholders ready for ML changes.
913b4eb
@SimonClifford
Script to make TF davenet
c8bcda8
@SimonClifford
Add README.md
ae53fda
@SimonClifford
Lint and isort
8e88f8b
@jatkinson1000
Initial workings on pytorch coupling routine [WIP].
02a9e6c
@jatkinson1000
Updated net to take 3 inputs and test it behaves as expected.
cbeeb95
@jatkinson1000
tweak architecture and update pt2ts to run with 3 inputs.
63e5e99
@SimonClifford
Merge branch 'pytorch' into tensorflow
70d41b8
@SimonClifford
Update TF davenet construction and add test
e4b63b9
Update to reflect model now takes 3 inputs and concatenates
them, rather than a pre-concatted input.
Test is very basic but shows that TF davenet does the same
thing as PyTorch davenet.
@jatkinson1000
Coupled code for pytorch (untested compiling).
e56ac02
@jatkinson1000
Working pytorch bindings
1160352
@jatkinson1000
Updated pytorch bindings to handle output from model back to fortran
83fc53f
@jatkinson1000
Update pytorch build to match forpy inputs in nml
06d5e83
@jatkinson1000
WIP initial forpy bindings.
7575e9c
@SimonClifford
Implenent option for tensorflow coupling
62e5f66
@jatkinson1000
Forpy bindings uncompiled
d34f7fe
@jatkinson1000
Fix silly errors in forpy bindings.
2bb7462
@jatkinson1000
Update CMakeLists with forpy build option.
b9f021f
SimonCliffordand others
added 22 commits
April 6, 2023 15:26
@SimonClifford
Add null file for AD runs
2a98539
@jatkinson1000
add mpp timing for cg_drag
549b847
@SimonClifford
Fixes to get forpy working.
f44681a
@SimonClifford
Fixes to get direct TF working
0350424
@SimonClifford
Remove incorrect scaling for pressure
73df56d
It was there to test something.
@SimonClifford
Add a submit script for forpy+pytorch
cb5d9cb
@SimonClifford
Working forpy + Tensorflow
9b67fc1
@SimonClifford
Working Pytorch direct
d7cd746
@SimonClifford
Simplify (and optimise?) the pre-ML reshaping
0d8a4dd
@SimonClifford
Return cg_drag to normal, so job runs to completion
27339c9
@SimonClifford
Turn off verbose printing for Forpy+TF
04a54ce
@SimonClifford
Fix typo in submit_FP_PT script
029d167
@SimonClifford
Add scripts for AD99 runs, for completeness
1b97537
@jatkinson1000
Update README
1010f8c
@jatkinson1000
Merge branch 'ML' of github.com:DataWaveProject/MiMA into ML
22b8499
@jatkinson1000
Bug fix in surface pressure calculation
babba71
@jatkinson1000
Bug fix in surface pressure calculation
96f3dab
@SimonClifford
Apply fix for arg order and scaling pressure for models
d445996
The models' arguments were being sent in the wrong order.
It should be pressure, latitude, then wind.
Also the pressure should be in Pa, so no need for scale factor
of 100.
@SimonClifford
Add release submit scripts
5a27155
@SimonClifford
Make submit scripts use local disk
4183081
@SimonClifford
Remove debug printing
b319e62
@SimonClifford
Fix daft bug from previous commit
0a31ff3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

Closes #13 in the early @SimonClifford commits.
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@SimonClifford commits address TensorFlow coupling.

Closes #12

Closes #3
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 commits close #11 and close #12 and therefore close #4 .
@SimonClifford
SimonClifford
commented
May 23, 2023

Copy link


Copy Markdown

@jatkinson1000 closes #5, although we may revisit.
@jatkinson1000
jatkinson1000
commented
May 24, 2023

Copy link


Copy Markdown

Member
Also closes #1 in earliest @jatkinson1000 commits (we could merge cmake into this branch to remove unused dependencies that caused failure on Sherlock).
CMake docs are included in docs/GettingStarted.md.
mondus
mondus
commented
May 24, 2023
View reviewed changes
@mondusmondus
left a comment


Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small suggestions.
  1. Move the CSD3 submissions scrips into a scripts\csd3 folder as they are hardware specific.
  2. Remove hardcoded home directories in the submissions scripts. Replace with a variable or argument.
  3. pytorch README.md could do with a bit more detail on what the scripts are. This could be opened as an issue for future work though.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees

@SimonCliffordSimonClifford
@jatkinson1000jatkinson1000

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Interfacing MiMA with ML codes


3 participants

@mondus@SimonClifford@jatkinson1000