Ocn 027 rw0 - #292

Open
clairehemmerly wants to merge 23 commits into
masterfrom
ocn_027_rw0
Open

Ocn 027 rw0#292
clairehemmerly wants to merge 23 commits into
masterfrom
ocn_027_rw0

Conversation

@clairehemmerly

@clairehemmerlyclairehemmerly commented Jun 7, 2022•
edited by chrowe
Loading

Copy link
Copy Markdown

Checklist for Reviewing a Pre-Processing Script

  • Does the python script contain the following 4 sections: Download data and save to your data directory, Process data, Upload processed data to Carto/Upload processed data to Google Earth Engine, Upload original data and processed data to Amazon S3 storage?
  • Does the script have the standardized variable names for: dataset_name, raw_data_file, processed_data_file?
  • Does the script create and use a 'data' directory?
  • Does the script use a python module to automatically download the data? If this is not possible, are there explicit instructions for how to download the data (step by step with instructions about every input parameter and button to click to find the exact data) and does it use shutil to move the data from 'Downloads' into the data directory?
  • Is the script automated as much as possible to minimize rewriting code the next time the dataset updates (ex 1: can you automatically pull out column names instead of typing them yourself? ex 2: if you are dropping columns with no data, did you use pandas to find the nodata columns instead of dropping the column by name?)
  • Are there comments on almost every line of code to explicitly state what is being done and why?
  • Are you uploading to the correct AWS location?
  • For GEE scripts, did you explicitly define the band manifest with a pyramiding policy so that we can easily change it later if we need to?
  • Does the README contain all relevant links?
  • Does the README state the original file type?
  • Does the README list all of the processing steps that were taken in the script?
  • For netcdfs, does the README state which variables were pulled from the netcdf?
  • Did you use the util functions whenever possible?
  • Have you checked the processed data file on your computer to make sure it matches the data you uploaded to Carto? (spaces or symbol names in column titles often get changed in the upload process-please change these before uploading so that the backed up processed data matches the data on Carto)
  • Does the folder name and python script file name match the dataset_name variable in the script? Are they all lowercase? Does the processing script end in '_processing.py'? (this part is often forgotten)

@clairehemmerlyclairehemmerly self-assigned this Jun 7, 2022
chrowe
chrowe requested changes Jul 8, 2022
View reviewed changes

@chrowechrowe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor change requests and a couple questions but overall looks good.
Only thing I didn't check was if the column names in Carto match those in the files sent to s3. Did you do that?

Comment threadocn_027a_rw0_nitrogen_plumes/README.md Outdated

You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).

###### Note: This dataset processing was done by Claire Hemmerly, and QC'd by [Chris Rowe](https://www.wri.org/profile/chris-rowe).

@chrowechroweJun 8, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add any link here if you want.

Comment threadocn_027a_rw0_nitrogen_plumes/README.md Outdated
Comment threadocn_027a_rw0_nitrogen_plumes/README.md Outdated

You can view the processed Wastewater Plumes in Coastal Areas dataset [on Resource Watch](https://resourcewatch.org/data/explore/11804f04-d9c7-47b9-8d27-27ce6ed6c042).

You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).

@chrowechroweJul 8, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this if a raster file we don't link to our s3 version. You can just link to the source.
Side note, do you know why the zip file on our s3 is so large? My browser says it is 11GB but the source file looks to be 250MB.

@clairehemmerlyclairehemmerlyJul 12, 2022•
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compressed the zipfile and uploaded to the correct folder. I deleted the one that was 11G

Comment threadocn_027a_rw0_nitrogen_plumes/README.md Outdated
Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py

# convert the data type of columns to integer
for col in gdf.columns[1:9]:
gdf[col] = gdf[col].fillna(0).astype('int')

@chrowechroweJul 8, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure we want na values to be zero? Are there any existing zero values?

@clairehemmerlyclairehemmerlyJul 13, 2022•
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NA values are in the columns that show % nitrogen input, which we're not using, and the same rows have 0 for the nitrogen input in g/yr, so I think updating the NAs to 0 should be fine. I had trouble creating the processed shapefile when the values were floats not integers, I got an error basically saying the numbers were too big.

Comment threadocn_027c_rw0_wastewater_watersheds/README.md Outdated
Comment threadocn_027c_rw0_wastewater_watersheds/README.md Outdated
Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py

# convert the data type of columns to integer
for col in gdf.columns[1:9]:
gdf[col] = gdf[col].fillna(0).astype('int')

@chrowechroweJul 8, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, just want to make sure we are accurately translating the data here.

Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py Outdated
'''

# load in the polygon shapefile
shapes = os.path.join(raw_data_file_unzipped, 'effluent_N_pourpoints_all.shp')

@chrowechroweJul 8, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use processed_data_file here instead of shapes for consistency with our other scripts?

@chrowechroweJul 8, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think the main thing is that the final thing you upload is called processed_data_file.

@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ive updated this section, let me know if it looks ok

Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py
set_default_credentials(username=CARTO_USER, base_url="https://{user}.carto.com/".format(user=CARTO_USER),api_key=CARTO_KEY)

# upload data frame to Carto
to_carto(gdf, dataset_name + '_edit', if_exists='replace')

@chrowechroweJul 8, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency can you use processed_data_file instead of gdf?

chrowe
chrowe reviewed Jul 12, 2022
View reviewed changes
Comment threadocn_027a_rw0_nitrogen_plumes/ocn_027a_rw0_nitrogen_plumes_processing.py Outdated
Upload original data and processed data to Amazon S3 storage
'''
# initialize AWS variables
aws_bucket = 'wri-public-data'

@chrowechroweJul 12, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should actually be aws_bucket = 'wri-projects' for raster datasets.

@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@clairehemmerly

clairehemmerly commented Jul 13, 2022

Copy link
Copy Markdown
Author

I added the processed data file steps and checked the col names, should match now

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chrowechrowechrowe requested changes

Requested changes must be addressed to merge this pull request.

Assignees

@clairehemmerlyclairehemmerly

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@clairehemmerly@chrowe
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Ocn 027 rw0 - #292
Open
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Open
Ocn 027 rw0#292
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Conversation
@clairehemmerly
@clairehemmerlyclairehemmerly
commented
Jun 7, 2022•

edited by chrowe

Loading

Copy link


Copy Markdown

Checklist for Reviewing a Pre-Processing Script
  •  Does the python script contain the following 4 sections: Download data and save to your data directory, Process data, Upload processed data to Carto/Upload processed data to Google Earth Engine, Upload original data and processed data to Amazon S3 storage?
  •  Does the script have the standardized variable names for: dataset_name, raw_data_file, processed_data_file?
  •  Does the script create and use a 'data' directory?
  •  Does the script use a python module to automatically download the data? If this is not possible, are there explicit instructions for how to download the data (step by step with instructions about every input parameter and button to click to find the exact data) and does it use shutil to move the data from 'Downloads' into the data directory?
  •  Is the script automated as much as possible to minimize rewriting code the next time the dataset updates (ex 1: can you automatically pull out column names instead of typing them yourself? ex 2: if you are dropping columns with no data, did you use pandas to find the nodata columns instead of dropping the column by name?)
  •  Are there comments on almost every line of code to explicitly state what is being done and why?
  •  Are you uploading to the correct AWS location?
  •  For GEE scripts, did you explicitly define the band manifest with a pyramiding policy so that we can easily change it later if we need to?
  •  Does the README contain all relevant links?
  •  Does the README state the original file type?
  •  Does the README list all of the processing steps that were taken in the script?
  •  For netcdfs, does the README state which variables were pulled from the netcdf?
  •  Did you use the util functions whenever possible?
  •  Have you checked the processed data file on your computer to make sure it matches the data you uploaded to Carto? (spaces or symbol names in column titles often get changed in the upload process-please change these before uploading so that the backed up processed data matches the data on Carto)
  •  Does the folder name and python script file name match the dataset_name variable in the script? Are they all lowercase? Does the processing script end in '_processing.py'? (this part is often forgotten)
@clairehemmerlyclairehemmerly
self-assigned this
Jun 7, 2022
chrowe
chrowe
requested changes
Jul 8, 2022
View reviewed changes
@chrowechrowe
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor change requests and a couple questions but overall looks good.

Only thing I didn't check was if the column names in Carto match those in the files sent to s3. Did you do that?
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).

###### Note: This dataset processing was done by Claire Hemmerly, and QC'd by [Chris Rowe](https://www.wri.org/profile/chris-rowe).
@chrowechroweJun 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add any link here if you want.
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can view the processed Wastewater Plumes in Coastal Areas dataset [on Resource Watch](https://resourcewatch.org/data/explore/11804f04-d9c7-47b9-8d27-27ce6ed6c042).

You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this if a raster file we don't link to our s3 version. You can just link to the source.

Side note, do you know why the zip file on our s3 is so large? My browser says it is 11GB but the source file looks to be 250MB.
@clairehemmerlyclairehemmerlyJul 12, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compressed the zipfile and uploaded to the correct folder. I deleted the one that was 11G
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure we want na values to be zero? Are there any existing zero values?
@clairehemmerlyclairehemmerlyJul 13, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NA values are in the columns that show % nitrogen input, which we're not using, and the same rows have 0 for the nitrogen input in g/yr, so I think updating the NAs to 0 should be fine. I had trouble creating the processed shapefile when the values were floats not integers, I got an error basically saying the numbers were too big.
Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, just want to make sure we are accurately translating the data here.
Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py
Outdated

'''

# load in the polygon shapefile
shapes = os.path.join(raw_data_file_unzipped, 'effluent_N_pourpoints_all.shp')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use processed_data_file here instead of shapes for consistency with our other scripts?
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think the main thing is that the final thing you upload is called processed_data_file.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ive updated this section, let me know if it looks ok
Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py
set_default_credentials(username=CARTO_USER, base_url="https://{user}.carto.com/".format(user=CARTO_USER),api_key=CARTO_KEY)

# upload data frame to Carto
to_carto(gdf, dataset_name + '_edit', if_exists='replace')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency can you use processed_data_file instead of gdf?
chrowe
chrowe
reviewed
Jul 12, 2022
View reviewed changes
Comment threadocn_027a_rw0_nitrogen_plumes/ocn_027a_rw0_nitrogen_plumes_processing.py
Outdated

Upload original data and processed data to Amazon S3 storage
'''
# initialize AWS variables
aws_bucket = 'wri-public-data'
@chrowechroweJul 12, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should actually be aws_bucket = 'wri-projects' for raster datasets.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed
@clairehemmerly
clairehemmerly
commented
Jul 13, 2022

Copy link


Copy Markdown

Author
I added the processed data file steps and checked the col names, should match now
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@chrowechrowechrowe requested changes

Requested changes must be addressed to merge this pull request.


Assignees

@clairehemmerlyclairehemmerly

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@clairehemmerly@chrowe

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Ocn 027 rw0 - #292
Open
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Open
Ocn 027 rw0#292
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Conversation
@clairehemmerly
@clairehemmerlyclairehemmerly
commented
Jun 7, 2022•

edited by chrowe

Loading

Copy link


Copy Markdown

Checklist for Reviewing a Pre-Processing Script
  •  Does the python script contain the following 4 sections: Download data and save to your data directory, Process data, Upload processed data to Carto/Upload processed data to Google Earth Engine, Upload original data and processed data to Amazon S3 storage?
  •  Does the script have the standardized variable names for: dataset_name, raw_data_file, processed_data_file?
  •  Does the script create and use a 'data' directory?
  •  Does the script use a python module to automatically download the data? If this is not possible, are there explicit instructions for how to download the data (step by step with instructions about every input parameter and button to click to find the exact data) and does it use shutil to move the data from 'Downloads' into the data directory?
  •  Is the script automated as much as possible to minimize rewriting code the next time the dataset updates (ex 1: can you automatically pull out column names instead of typing them yourself? ex 2: if you are dropping columns with no data, did you use pandas to find the nodata columns instead of dropping the column by name?)
  •  Are there comments on almost every line of code to explicitly state what is being done and why?
  •  Are you uploading to the correct AWS location?
  •  For GEE scripts, did you explicitly define the band manifest with a pyramiding policy so that we can easily change it later if we need to?
  •  Does the README contain all relevant links?
  •  Does the README state the original file type?
  •  Does the README list all of the processing steps that were taken in the script?
  •  For netcdfs, does the README state which variables were pulled from the netcdf?
  •  Did you use the util functions whenever possible?
  •  Have you checked the processed data file on your computer to make sure it matches the data you uploaded to Carto? (spaces or symbol names in column titles often get changed in the upload process-please change these before uploading so that the backed up processed data matches the data on Carto)
  •  Does the folder name and python script file name match the dataset_name variable in the script? Are they all lowercase? Does the processing script end in '_processing.py'? (this part is often forgotten)
@clairehemmerlyclairehemmerly
self-assigned this
Jun 7, 2022
chrowe
chrowe
requested changes
Jul 8, 2022
View reviewed changes
@chrowechrowe
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor change requests and a couple questions but overall looks good.

Only thing I didn't check was if the column names in Carto match those in the files sent to s3. Did you do that?
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).

###### Note: This dataset processing was done by Claire Hemmerly, and QC'd by [Chris Rowe](https://www.wri.org/profile/chris-rowe).
@chrowechroweJun 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add any link here if you want.
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can view the processed Wastewater Plumes in Coastal Areas dataset [on Resource Watch](https://resourcewatch.org/data/explore/11804f04-d9c7-47b9-8d27-27ce6ed6c042).

You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this if a raster file we don't link to our s3 version. You can just link to the source.

Side note, do you know why the zip file on our s3 is so large? My browser says it is 11GB but the source file looks to be 250MB.
@clairehemmerlyclairehemmerlyJul 12, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compressed the zipfile and uploaded to the correct folder. I deleted the one that was 11G
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure we want na values to be zero? Are there any existing zero values?
@clairehemmerlyclairehemmerlyJul 13, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NA values are in the columns that show % nitrogen input, which we're not using, and the same rows have 0 for the nitrogen input in g/yr, so I think updating the NAs to 0 should be fine. I had trouble creating the processed shapefile when the values were floats not integers, I got an error basically saying the numbers were too big.
Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, just want to make sure we are accurately translating the data here.
Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py
Outdated

'''

# load in the polygon shapefile
shapes = os.path.join(raw_data_file_unzipped, 'effluent_N_pourpoints_all.shp')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use processed_data_file here instead of shapes for consistency with our other scripts?
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think the main thing is that the final thing you upload is called processed_data_file.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ive updated this section, let me know if it looks ok
Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py
set_default_credentials(username=CARTO_USER, base_url="https://{user}.carto.com/".format(user=CARTO_USER),api_key=CARTO_KEY)

# upload data frame to Carto
to_carto(gdf, dataset_name + '_edit', if_exists='replace')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency can you use processed_data_file instead of gdf?
chrowe
chrowe
reviewed
Jul 12, 2022
View reviewed changes
Comment threadocn_027a_rw0_nitrogen_plumes/ocn_027a_rw0_nitrogen_plumes_processing.py
Outdated

Upload original data and processed data to Amazon S3 storage
'''
# initialize AWS variables
aws_bucket = 'wri-public-data'
@chrowechroweJul 12, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should actually be aws_bucket = 'wri-projects' for raster datasets.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed
@clairehemmerly
clairehemmerly
commented
Jul 13, 2022

Copy link


Copy Markdown

Author
I added the processed data file steps and checked the col names, should match now
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@chrowechrowechrowe requested changes

Requested changes must be addressed to merge this pull request.


Assignees

@clairehemmerlyclairehemmerly

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@clairehemmerly@chrowe

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Ocn 027 rw0 - #292
Open
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Open
Ocn 027 rw0#292
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Conversation
@clairehemmerly
@clairehemmerlyclairehemmerly
commented
Jun 7, 2022•

edited by chrowe

Loading

Copy link


Copy Markdown

Checklist for Reviewing a Pre-Processing Script
  •  Does the python script contain the following 4 sections: Download data and save to your data directory, Process data, Upload processed data to Carto/Upload processed data to Google Earth Engine, Upload original data and processed data to Amazon S3 storage?
  •  Does the script have the standardized variable names for: dataset_name, raw_data_file, processed_data_file?
  •  Does the script create and use a 'data' directory?
  •  Does the script use a python module to automatically download the data? If this is not possible, are there explicit instructions for how to download the data (step by step with instructions about every input parameter and button to click to find the exact data) and does it use shutil to move the data from 'Downloads' into the data directory?
  •  Is the script automated as much as possible to minimize rewriting code the next time the dataset updates (ex 1: can you automatically pull out column names instead of typing them yourself? ex 2: if you are dropping columns with no data, did you use pandas to find the nodata columns instead of dropping the column by name?)
  •  Are there comments on almost every line of code to explicitly state what is being done and why?
  •  Are you uploading to the correct AWS location?
  •  For GEE scripts, did you explicitly define the band manifest with a pyramiding policy so that we can easily change it later if we need to?
  •  Does the README contain all relevant links?
  •  Does the README state the original file type?
  •  Does the README list all of the processing steps that were taken in the script?
  •  For netcdfs, does the README state which variables were pulled from the netcdf?
  •  Did you use the util functions whenever possible?
  •  Have you checked the processed data file on your computer to make sure it matches the data you uploaded to Carto? (spaces or symbol names in column titles often get changed in the upload process-please change these before uploading so that the backed up processed data matches the data on Carto)
  •  Does the folder name and python script file name match the dataset_name variable in the script? Are they all lowercase? Does the processing script end in '_processing.py'? (this part is often forgotten)
@clairehemmerlyclairehemmerly
self-assigned this
Jun 7, 2022
chrowe
chrowe
requested changes
Jul 8, 2022
View reviewed changes
@chrowechrowe
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor change requests and a couple questions but overall looks good.

Only thing I didn't check was if the column names in Carto match those in the files sent to s3. Did you do that?
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).

###### Note: This dataset processing was done by Claire Hemmerly, and QC'd by [Chris Rowe](https://www.wri.org/profile/chris-rowe).
@chrowechroweJun 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add any link here if you want.
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can view the processed Wastewater Plumes in Coastal Areas dataset [on Resource Watch](https://resourcewatch.org/data/explore/11804f04-d9c7-47b9-8d27-27ce6ed6c042).

You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this if a raster file we don't link to our s3 version. You can just link to the source.

Side note, do you know why the zip file on our s3 is so large? My browser says it is 11GB but the source file looks to be 250MB.
@clairehemmerlyclairehemmerlyJul 12, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compressed the zipfile and uploaded to the correct folder. I deleted the one that was 11G
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure we want na values to be zero? Are there any existing zero values?
@clairehemmerlyclairehemmerlyJul 13, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NA values are in the columns that show % nitrogen input, which we're not using, and the same rows have 0 for the nitrogen input in g/yr, so I think updating the NAs to 0 should be fine. I had trouble creating the processed shapefile when the values were floats not integers, I got an error basically saying the numbers were too big.
Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, just want to make sure we are accurately translating the data here.
Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py
Outdated

'''

# load in the polygon shapefile
shapes = os.path.join(raw_data_file_unzipped, 'effluent_N_pourpoints_all.shp')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use processed_data_file here instead of shapes for consistency with our other scripts?
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think the main thing is that the final thing you upload is called processed_data_file.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ive updated this section, let me know if it looks ok
Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py
set_default_credentials(username=CARTO_USER, base_url="https://{user}.carto.com/".format(user=CARTO_USER),api_key=CARTO_KEY)

# upload data frame to Carto
to_carto(gdf, dataset_name + '_edit', if_exists='replace')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency can you use processed_data_file instead of gdf?
chrowe
chrowe
reviewed
Jul 12, 2022
View reviewed changes
Comment threadocn_027a_rw0_nitrogen_plumes/ocn_027a_rw0_nitrogen_plumes_processing.py
Outdated

Upload original data and processed data to Amazon S3 storage
'''
# initialize AWS variables
aws_bucket = 'wri-public-data'
@chrowechroweJul 12, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should actually be aws_bucket = 'wri-projects' for raster datasets.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed
@clairehemmerly
clairehemmerly
commented
Jul 13, 2022

Copy link


Copy Markdown

Author
I added the processed data file steps and checked the col names, should match now
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@chrowechrowechrowe requested changes

Requested changes must be addressed to merge this pull request.


Assignees

@clairehemmerlyclairehemmerly

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@clairehemmerly@chrowe

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Ocn 027 rw0 - #292
Open
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Open
Ocn 027 rw0#292
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Conversation
@clairehemmerly
@clairehemmerlyclairehemmerly
commented
Jun 7, 2022•

edited by chrowe

Loading

Copy link


Copy Markdown

Checklist for Reviewing a Pre-Processing Script
  •  Does the python script contain the following 4 sections: Download data and save to your data directory, Process data, Upload processed data to Carto/Upload processed data to Google Earth Engine, Upload original data and processed data to Amazon S3 storage?
  •  Does the script have the standardized variable names for: dataset_name, raw_data_file, processed_data_file?
  •  Does the script create and use a 'data' directory?
  •  Does the script use a python module to automatically download the data? If this is not possible, are there explicit instructions for how to download the data (step by step with instructions about every input parameter and button to click to find the exact data) and does it use shutil to move the data from 'Downloads' into the data directory?
  •  Is the script automated as much as possible to minimize rewriting code the next time the dataset updates (ex 1: can you automatically pull out column names instead of typing them yourself? ex 2: if you are dropping columns with no data, did you use pandas to find the nodata columns instead of dropping the column by name?)
  •  Are there comments on almost every line of code to explicitly state what is being done and why?
  •  Are you uploading to the correct AWS location?
  •  For GEE scripts, did you explicitly define the band manifest with a pyramiding policy so that we can easily change it later if we need to?
  •  Does the README contain all relevant links?
  •  Does the README state the original file type?
  •  Does the README list all of the processing steps that were taken in the script?
  •  For netcdfs, does the README state which variables were pulled from the netcdf?
  •  Did you use the util functions whenever possible?
  •  Have you checked the processed data file on your computer to make sure it matches the data you uploaded to Carto? (spaces or symbol names in column titles often get changed in the upload process-please change these before uploading so that the backed up processed data matches the data on Carto)
  •  Does the folder name and python script file name match the dataset_name variable in the script? Are they all lowercase? Does the processing script end in '_processing.py'? (this part is often forgotten)
@clairehemmerlyclairehemmerly
self-assigned this
Jun 7, 2022
chrowe
chrowe
requested changes
Jul 8, 2022
View reviewed changes
@chrowechrowe
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor change requests and a couple questions but overall looks good.

Only thing I didn't check was if the column names in Carto match those in the files sent to s3. Did you do that?
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).

###### Note: This dataset processing was done by Claire Hemmerly, and QC'd by [Chris Rowe](https://www.wri.org/profile/chris-rowe).
@chrowechroweJun 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add any link here if you want.
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can view the processed Wastewater Plumes in Coastal Areas dataset [on Resource Watch](https://resourcewatch.org/data/explore/11804f04-d9c7-47b9-8d27-27ce6ed6c042).

You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this if a raster file we don't link to our s3 version. You can just link to the source.

Side note, do you know why the zip file on our s3 is so large? My browser says it is 11GB but the source file looks to be 250MB.
@clairehemmerlyclairehemmerlyJul 12, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compressed the zipfile and uploaded to the correct folder. I deleted the one that was 11G
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure we want na values to be zero? Are there any existing zero values?
@clairehemmerlyclairehemmerlyJul 13, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NA values are in the columns that show % nitrogen input, which we're not using, and the same rows have 0 for the nitrogen input in g/yr, so I think updating the NAs to 0 should be fine. I had trouble creating the processed shapefile when the values were floats not integers, I got an error basically saying the numbers were too big.
Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, just want to make sure we are accurately translating the data here.
Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py
Outdated

'''

# load in the polygon shapefile
shapes = os.path.join(raw_data_file_unzipped, 'effluent_N_pourpoints_all.shp')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use processed_data_file here instead of shapes for consistency with our other scripts?
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think the main thing is that the final thing you upload is called processed_data_file.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ive updated this section, let me know if it looks ok
Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py
set_default_credentials(username=CARTO_USER, base_url="https://{user}.carto.com/".format(user=CARTO_USER),api_key=CARTO_KEY)

# upload data frame to Carto
to_carto(gdf, dataset_name + '_edit', if_exists='replace')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency can you use processed_data_file instead of gdf?
chrowe
chrowe
reviewed
Jul 12, 2022
View reviewed changes
Comment threadocn_027a_rw0_nitrogen_plumes/ocn_027a_rw0_nitrogen_plumes_processing.py
Outdated

Upload original data and processed data to Amazon S3 storage
'''
# initialize AWS variables
aws_bucket = 'wri-public-data'
@chrowechroweJul 12, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should actually be aws_bucket = 'wri-projects' for raster datasets.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed
@clairehemmerly
clairehemmerly
commented
Jul 13, 2022

Copy link


Copy Markdown

Author
I added the processed data file steps and checked the col names, should match now
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@chrowechrowechrowe requested changes

Requested changes must be addressed to merge this pull request.


Assignees

@clairehemmerlyclairehemmerly

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@clairehemmerly@chrowe

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Ocn 027 rw0 - #292
Open
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Open
Ocn 027 rw0#292
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Conversation
@clairehemmerly
@clairehemmerlyclairehemmerly
commented
Jun 7, 2022•

edited by chrowe

Loading

Copy link


Copy Markdown

Checklist for Reviewing a Pre-Processing Script
  •  Does the python script contain the following 4 sections: Download data and save to your data directory, Process data, Upload processed data to Carto/Upload processed data to Google Earth Engine, Upload original data and processed data to Amazon S3 storage?
  •  Does the script have the standardized variable names for: dataset_name, raw_data_file, processed_data_file?
  •  Does the script create and use a 'data' directory?
  •  Does the script use a python module to automatically download the data? If this is not possible, are there explicit instructions for how to download the data (step by step with instructions about every input parameter and button to click to find the exact data) and does it use shutil to move the data from 'Downloads' into the data directory?
  •  Is the script automated as much as possible to minimize rewriting code the next time the dataset updates (ex 1: can you automatically pull out column names instead of typing them yourself? ex 2: if you are dropping columns with no data, did you use pandas to find the nodata columns instead of dropping the column by name?)
  •  Are there comments on almost every line of code to explicitly state what is being done and why?
  •  Are you uploading to the correct AWS location?
  •  For GEE scripts, did you explicitly define the band manifest with a pyramiding policy so that we can easily change it later if we need to?
  •  Does the README contain all relevant links?
  •  Does the README state the original file type?
  •  Does the README list all of the processing steps that were taken in the script?
  •  For netcdfs, does the README state which variables were pulled from the netcdf?
  •  Did you use the util functions whenever possible?
  •  Have you checked the processed data file on your computer to make sure it matches the data you uploaded to Carto? (spaces or symbol names in column titles often get changed in the upload process-please change these before uploading so that the backed up processed data matches the data on Carto)
  •  Does the folder name and python script file name match the dataset_name variable in the script? Are they all lowercase? Does the processing script end in '_processing.py'? (this part is often forgotten)
@clairehemmerlyclairehemmerly
self-assigned this
Jun 7, 2022
chrowe
chrowe
requested changes
Jul 8, 2022
View reviewed changes
@chrowechrowe
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor change requests and a couple questions but overall looks good.

Only thing I didn't check was if the column names in Carto match those in the files sent to s3. Did you do that?
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).

###### Note: This dataset processing was done by Claire Hemmerly, and QC'd by [Chris Rowe](https://www.wri.org/profile/chris-rowe).
@chrowechroweJun 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add any link here if you want.
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can view the processed Wastewater Plumes in Coastal Areas dataset [on Resource Watch](https://resourcewatch.org/data/explore/11804f04-d9c7-47b9-8d27-27ce6ed6c042).

You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this if a raster file we don't link to our s3 version. You can just link to the source.

Side note, do you know why the zip file on our s3 is so large? My browser says it is 11GB but the source file looks to be 250MB.
@clairehemmerlyclairehemmerlyJul 12, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compressed the zipfile and uploaded to the correct folder. I deleted the one that was 11G
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure we want na values to be zero? Are there any existing zero values?
@clairehemmerlyclairehemmerlyJul 13, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NA values are in the columns that show % nitrogen input, which we're not using, and the same rows have 0 for the nitrogen input in g/yr, so I think updating the NAs to 0 should be fine. I had trouble creating the processed shapefile when the values were floats not integers, I got an error basically saying the numbers were too big.
Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, just want to make sure we are accurately translating the data here.
Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py
Outdated

'''

# load in the polygon shapefile
shapes = os.path.join(raw_data_file_unzipped, 'effluent_N_pourpoints_all.shp')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use processed_data_file here instead of shapes for consistency with our other scripts?
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think the main thing is that the final thing you upload is called processed_data_file.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ive updated this section, let me know if it looks ok
Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py
set_default_credentials(username=CARTO_USER, base_url="https://{user}.carto.com/".format(user=CARTO_USER),api_key=CARTO_KEY)

# upload data frame to Carto
to_carto(gdf, dataset_name + '_edit', if_exists='replace')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency can you use processed_data_file instead of gdf?
chrowe
chrowe
reviewed
Jul 12, 2022
View reviewed changes
Comment threadocn_027a_rw0_nitrogen_plumes/ocn_027a_rw0_nitrogen_plumes_processing.py
Outdated

Upload original data and processed data to Amazon S3 storage
'''
# initialize AWS variables
aws_bucket = 'wri-public-data'
@chrowechroweJul 12, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should actually be aws_bucket = 'wri-projects' for raster datasets.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed
@clairehemmerly
clairehemmerly
commented
Jul 13, 2022

Copy link


Copy Markdown

Author
I added the processed data file steps and checked the col names, should match now
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@chrowechrowechrowe requested changes

Requested changes must be addressed to merge this pull request.


Assignees

@clairehemmerlyclairehemmerly

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@clairehemmerly@chrowe

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Ocn 027 rw0 - #292
Open
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Open
Ocn 027 rw0#292
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Conversation
@clairehemmerly
@clairehemmerlyclairehemmerly
commented
Jun 7, 2022•

edited by chrowe

Loading

Copy link


Copy Markdown

Checklist for Reviewing a Pre-Processing Script
  •  Does the python script contain the following 4 sections: Download data and save to your data directory, Process data, Upload processed data to Carto/Upload processed data to Google Earth Engine, Upload original data and processed data to Amazon S3 storage?
  •  Does the script have the standardized variable names for: dataset_name, raw_data_file, processed_data_file?
  •  Does the script create and use a 'data' directory?
  •  Does the script use a python module to automatically download the data? If this is not possible, are there explicit instructions for how to download the data (step by step with instructions about every input parameter and button to click to find the exact data) and does it use shutil to move the data from 'Downloads' into the data directory?
  •  Is the script automated as much as possible to minimize rewriting code the next time the dataset updates (ex 1: can you automatically pull out column names instead of typing them yourself? ex 2: if you are dropping columns with no data, did you use pandas to find the nodata columns instead of dropping the column by name?)
  •  Are there comments on almost every line of code to explicitly state what is being done and why?
  •  Are you uploading to the correct AWS location?
  •  For GEE scripts, did you explicitly define the band manifest with a pyramiding policy so that we can easily change it later if we need to?
  •  Does the README contain all relevant links?
  •  Does the README state the original file type?
  •  Does the README list all of the processing steps that were taken in the script?
  •  For netcdfs, does the README state which variables were pulled from the netcdf?
  •  Did you use the util functions whenever possible?
  •  Have you checked the processed data file on your computer to make sure it matches the data you uploaded to Carto? (spaces or symbol names in column titles often get changed in the upload process-please change these before uploading so that the backed up processed data matches the data on Carto)
  •  Does the folder name and python script file name match the dataset_name variable in the script? Are they all lowercase? Does the processing script end in '_processing.py'? (this part is often forgotten)
@clairehemmerlyclairehemmerly
self-assigned this
Jun 7, 2022
chrowe
chrowe
requested changes
Jul 8, 2022
View reviewed changes
@chrowechrowe
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor change requests and a couple questions but overall looks good.

Only thing I didn't check was if the column names in Carto match those in the files sent to s3. Did you do that?
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).

###### Note: This dataset processing was done by Claire Hemmerly, and QC'd by [Chris Rowe](https://www.wri.org/profile/chris-rowe).
@chrowechroweJun 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add any link here if you want.
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can view the processed Wastewater Plumes in Coastal Areas dataset [on Resource Watch](https://resourcewatch.org/data/explore/11804f04-d9c7-47b9-8d27-27ce6ed6c042).

You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this if a raster file we don't link to our s3 version. You can just link to the source.

Side note, do you know why the zip file on our s3 is so large? My browser says it is 11GB but the source file looks to be 250MB.
@clairehemmerlyclairehemmerlyJul 12, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compressed the zipfile and uploaded to the correct folder. I deleted the one that was 11G
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure we want na values to be zero? Are there any existing zero values?
@clairehemmerlyclairehemmerlyJul 13, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NA values are in the columns that show % nitrogen input, which we're not using, and the same rows have 0 for the nitrogen input in g/yr, so I think updating the NAs to 0 should be fine. I had trouble creating the processed shapefile when the values were floats not integers, I got an error basically saying the numbers were too big.
Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, just want to make sure we are accurately translating the data here.
Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py
Outdated

'''

# load in the polygon shapefile
shapes = os.path.join(raw_data_file_unzipped, 'effluent_N_pourpoints_all.shp')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use processed_data_file here instead of shapes for consistency with our other scripts?
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think the main thing is that the final thing you upload is called processed_data_file.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ive updated this section, let me know if it looks ok
Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py
set_default_credentials(username=CARTO_USER, base_url="https://{user}.carto.com/".format(user=CARTO_USER),api_key=CARTO_KEY)

# upload data frame to Carto
to_carto(gdf, dataset_name + '_edit', if_exists='replace')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency can you use processed_data_file instead of gdf?
chrowe
chrowe
reviewed
Jul 12, 2022
View reviewed changes
Comment threadocn_027a_rw0_nitrogen_plumes/ocn_027a_rw0_nitrogen_plumes_processing.py
Outdated

Upload original data and processed data to Amazon S3 storage
'''
# initialize AWS variables
aws_bucket = 'wri-public-data'
@chrowechroweJul 12, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should actually be aws_bucket = 'wri-projects' for raster datasets.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed
@clairehemmerly
clairehemmerly
commented
Jul 13, 2022

Copy link


Copy Markdown

Author
I added the processed data file steps and checked the col names, should match now
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@chrowechrowechrowe requested changes

Requested changes must be addressed to merge this pull request.


Assignees

@clairehemmerlyclairehemmerly

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@clairehemmerly@chrowe

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Ocn 027 rw0 - #292
Open
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Open
Ocn 027 rw0#292
clairehemmerly wants to merge 23 commits into
masterfrom 
ocn_027_rw0
Conversation
@clairehemmerly
@clairehemmerlyclairehemmerly
commented
Jun 7, 2022•

edited by chrowe

Loading

Copy link


Copy Markdown

Checklist for Reviewing a Pre-Processing Script
  •  Does the python script contain the following 4 sections: Download data and save to your data directory, Process data, Upload processed data to Carto/Upload processed data to Google Earth Engine, Upload original data and processed data to Amazon S3 storage?
  •  Does the script have the standardized variable names for: dataset_name, raw_data_file, processed_data_file?
  •  Does the script create and use a 'data' directory?
  •  Does the script use a python module to automatically download the data? If this is not possible, are there explicit instructions for how to download the data (step by step with instructions about every input parameter and button to click to find the exact data) and does it use shutil to move the data from 'Downloads' into the data directory?
  •  Is the script automated as much as possible to minimize rewriting code the next time the dataset updates (ex 1: can you automatically pull out column names instead of typing them yourself? ex 2: if you are dropping columns with no data, did you use pandas to find the nodata columns instead of dropping the column by name?)
  •  Are there comments on almost every line of code to explicitly state what is being done and why?
  •  Are you uploading to the correct AWS location?
  •  For GEE scripts, did you explicitly define the band manifest with a pyramiding policy so that we can easily change it later if we need to?
  •  Does the README contain all relevant links?
  •  Does the README state the original file type?
  •  Does the README list all of the processing steps that were taken in the script?
  •  For netcdfs, does the README state which variables were pulled from the netcdf?
  •  Did you use the util functions whenever possible?
  •  Have you checked the processed data file on your computer to make sure it matches the data you uploaded to Carto? (spaces or symbol names in column titles often get changed in the upload process-please change these before uploading so that the backed up processed data matches the data on Carto)
  •  Does the folder name and python script file name match the dataset_name variable in the script? Are they all lowercase? Does the processing script end in '_processing.py'? (this part is often forgotten)
@clairehemmerlyclairehemmerly
self-assigned this
Jun 7, 2022
chrowe
chrowe
requested changes
Jul 8, 2022
View reviewed changes
@chrowechrowe
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor change requests and a couple questions but overall looks good.

Only thing I didn't check was if the column names in Carto match those in the files sent to s3. Did you do that?
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).

###### Note: This dataset processing was done by Claire Hemmerly, and QC'd by [Chris Rowe](https://www.wri.org/profile/chris-rowe).
@chrowechroweJun 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add any link here if you want.
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated


You can view the processed Wastewater Plumes in Coastal Areas dataset [on Resource Watch](https://resourcewatch.org/data/explore/11804f04-d9c7-47b9-8d27-27ce6ed6c042).

You can also download the original dataset [directly through Resource Watch](http://wri-public-data.s3.amazonaws.com/resourcewatch/raster/ocn_027_rw0_nitrogen_plumes.zip), or [from the source website](https://knb.ecoinformatics.org/view/urn%3Auuid%3Ac7bdc77e-6c7d-46b6-8bfc-a66491119d07).
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this if a raster file we don't link to our s3 version. You can just link to the source.

Side note, do you know why the zip file on our s3 is so large? My browser says it is 11GB but the source file looks to be 250MB.
@clairehemmerlyclairehemmerlyJul 12, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compressed the zipfile and uploaded to the correct folder. I deleted the one that was 11G
Comment threadocn_027a_rw0_nitrogen_plumes/README.md
Outdated

Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure we want na values to be zero? Are there any existing zero values?
@clairehemmerlyclairehemmerlyJul 13, 2022•

edited

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NA values are in the columns that show % nitrogen input, which we're not using, and the same rows have 0 for the nitrogen input in g/yr, so I think updating the NAs to 0 should be fine. I had trouble creating the processed shapefile when the values were floats not integers, I got an error basically saying the numbers were too big.
Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/README.md
Outdated

Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py

# convert the data type of columns to integer 
for col in gdf.columns[1:9]:
 gdf[col] = gdf[col].fillna(0).astype('int')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, just want to make sure we are accurately translating the data here.
Comment threadocn_027b_rw0_wastewater_pourpoints/ocn_027b_rw0_wastewater_pourpoints_processing.py
Outdated

'''

# load in the polygon shapefile
shapes = os.path.join(raw_data_file_unzipped, 'effluent_N_pourpoints_all.shp')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use processed_data_file here instead of shapes for consistency with our other scripts?
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think the main thing is that the final thing you upload is called processed_data_file.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ive updated this section, let me know if it looks ok
Comment threadocn_027c_rw0_wastewater_watersheds/ocn_027c_rw0_wastewater_watersheds_processing.py
set_default_credentials(username=CARTO_USER, base_url="https://{user}.carto.com/".format(user=CARTO_USER),api_key=CARTO_KEY)

# upload data frame to Carto
to_carto(gdf, dataset_name + '_edit', if_exists='replace')
@chrowechroweJul 8, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency can you use processed_data_file instead of gdf?
chrowe
chrowe
reviewed
Jul 12, 2022
View reviewed changes
Comment threadocn_027a_rw0_nitrogen_plumes/ocn_027a_rw0_nitrogen_plumes_processing.py
Outdated

Upload original data and processed data to Amazon S3 storage
'''
# initialize AWS variables
aws_bucket = 'wri-public-data'
@chrowechroweJul 12, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should actually be aws_bucket = 'wri-projects' for raster datasets.
@clairehemmerlyclairehemmerlyJul 13, 2022

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed
@clairehemmerly
clairehemmerly
commented
Jul 13, 2022

Copy link


Copy Markdown

Author
I added the processed data file steps and checked the col names, should match now
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@chrowechrowechrowe requested changes

Requested changes must be addressed to merge this pull request.


Assignees

@clairehemmerlyclairehemmerly

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@clairehemmerly@chrowe