Nrel atb usa costs

[finozzifa]

Goals

This pull request contains the changes performed by @danielelerede-oet and myself (@finozzifa) as agreed with @martacki and @euronion.

Proposed final goal: this pull request is the first (intermediate) step of a set of changes that aim at granting the possibility to the model users to use country-specific cost assumptions.

Goal of this pull request: This work in particular proposes an intermediate step and creates a sub-folder outputs/US, where the outputs/costs_yyyy.csv files are copied and updated with NREL/ATB data.

Input and output schemas

NREL/ATB input values and schema

The NREL/ATB electricity data source is available here.

We require the cost assumptions for the years 2020, 2025, 2030, 2035, 2040, 2045 and 2050. The cost assumptions for 2020 are obtained from atb_e_2022 dataset, whereas those for the other years from the atb_e_2024 dataset. The schema of these files is unfortunately slightly different. Namely:

Schema of atb_e_2022

 # Column Non-Null Count Dtype --- ------ -------------- ----- 0 atb_year 286998 non-null int64 1 core_metric_key 286812 non-null object 2 core_metric_parameter 286998 non-null object 3 core_metric_case 286998 non-null object 4 crpyears 286998 non-null object 5 technology 286998 non-null object 6 technology_alias 285510 non-null object 7 techdetail 286998 non-null object 8 display_name 285510 non-null object 9 default 285510 non-null float64
10 scenario 286998 non-null object 11 core_metric_variable 286998 non-null object 12 units 286998 non-null object 13 value 286998 non-null float64

Schema of atb_e_2024

 # Column Non-Null Count Dtype --- ------ -------------- ----- 0 atb_year 572232 non-null int64 1 core_metric_key 572232 non-null object 2 core_metric_parameter 572232 non-null object 3 core_metric_case 572232 non-null object 4 tax_credit_case 212388 non-null object 5 crpyears 572232 non-null object 6 technology 572232 non-null object 7 technology_alias 572232 non-null object 8 techdetail 572232 non-null object 9 techdetail2 506772 non-null object 10 resourcedetail 488328 non-null object 11 display_name 572232 non-null object 12 default 572232 non-null int64 13 scale 506772 non-null object 14 maturity 506772 non-null object 15 scenario 572232 non-null object 16 core_metric_variable 572232 non-null int64 17 units 572232 non-null object 18 value 572232 non-null float64

We consider a subset of such columns. This can be configured with the configuration from the config.yaml from config["nrel_atb"]["nrel_atb_columns_to_keep"]. For this pull request, the columns taken are

["atb_year", "core_metric_parameter", "core_metric_case", "core_metric_variable", "technology", "technology_alias", "display_name", "scenario", "units", "value"]

where in particular

• atb_year equals the year in the file name. For example atb_year = 2022 for atb_e_2022 or atb_year = 2024 for atb_e_2024
• core_metric_variable equals the year for which the cost assumption is made
• core_metric_parameter has various values. We consider

Column value	Unit
CAPEX	$/kW
CF
Fixed O&M	$/KW-yr
Variable O&M	$/MWh
Fuel	$/MWh
Additional OCC	$/KW
WACC Real

• scenario equals Moderate, Conservative, Advanced

outputs/US/costs_yyyy.csv values and schema

NREL/ATB column	outputs/US/costs_yyyy.csv column	Notes
display_name	technology
core_metric_parameter	parameter
units	unit
	source	taken from config["nrel_atb"]["nrel_atb_source_link"]
	further description	left blank
atb_year	currency_year
scenario	scenario
core_metric_case	financial_case

Changes

Changes in the workflow

The workflow has been updated as follows:

1. Step 1: rule compile_cost_assumptions generates outputs/costs_yyyy.csv files
2. Step 2: rule compile_cost_assumptions_nrel takes the outputs/costs_yyyy.csv files, reads-in the nrel/atb inputs, processes them and outputs a dedicated set of costs for the US in outputs/US/costs_yyyy.csv

The "high level" description of what compile_cost_assumptions_nrel.py does

The "high level" description of what the script does is:

• it loops through the years 2020, 2025, 2030, 2035, 2040, 2045, 2050
• for each year, it reads the corresponding outputs/cost_yyyy.csv file
• for each year, it reads the corresponding atb input file, filters the rows corresponding to the given year and extracts the necessary columns
• for each year, it normalizes the Fixed O&M by Additional OCC (for retrofits technologies) or CAPEX (for any other technology) and changes its unit from $/KW-yr to %-yr
• for each year, it performs the following technology renaming. NREL/ATB contains many more technologies than those listed in the table below. Those technologies are included anyway in the final outputs/US/costs_year.csv files but not renamed.

display_name	source NREL/ATB file	PyPSA technology name
Coal-new -> 2nd Gen Tech	atb_e_2022.parquet	coal
Coal-new	atb_e_2022.parquet, atb_e_2024.parquet	coal
NG F-Frame CT	atb_e_2022.parquet	CCGT
NG Combustion Turbine (F-Frame)	atb_e_2024.parquet	CCGT
Hydropower - NPD 1	atb_e_2022.parquet, atb_e_2024.parquet	hydro
Hydropower - NSD 1	atb_e_2022.parquet, atb_e_2024.parquet	ror
Pumped Storage Hydropower - National Class 1	atb_e_2022.parquet, atb_e_2024.parquet	PHS
Nuclear - Large	atb_e_2024.parquet	nuclear
Nuclear - AP1000	atb_e_2022.parquet	nuclear
Geothermal - Hydro / Flash	atb_e_2022.parquet, atb_e_2024.parquet	geothermal
Land-Based Wind - Class 1	atb_e_2022.parquet	onwind
Land-Based Wind - Class 1 - Technology 1	atb_e_2024.parquet	onwind
Offshore Wind - Class 1	atb_e_2022.parquet, atb_e_2024.parquet	offwind
Utility PV - Class 1	atb_e_2022.parquet, atb_e_2024.parquet	solar-utility
Commercial PV - Class 1	atb_e_2022.parquet, atb_e_2024.parquet	solar-rooftop
Utility-Scale Battery Storage - 6Hr	atb_e_2022.parquet, atb_e_2024.parquet	battery storage
Biopower	atb_e_2022.parquet	biomass
Biopower - Dedicated	atb_e_2022.parquet, atb_e_2024.parquet	biomass
CSP - Class 2	atb_e_2022.parquet, atb_e_2024.parquet	csp-tower

• for each year, it updates the cost values in the corresponding outputs/US/costs_year.csv file so that, the script
  • appends NREL/ATB technology entries not present in the original file outputs/costs_year.csv
  • leaves untouched technology entries not present in the NREL/ATB dataset
  • updates the technology entries present in both the NREL/ATB dataset and outputs/costs_year.csv, with NREL/ATB values
  • appends the values of the discount_rate from the input file discount_rates_usa.csv. This step is necessary because the NREL/ATB dataset provides "grouped names" instead of single technology names. For example, for the parameters CAPEX, CF, Fixed O&M we have the technology Land-Based Wind - Class 1, Land-Based Wind - Class 2 etc. Instead for WACC Real we have just Land-Based Wind
  • appends the values of the fuel from the input file fuel_costs_usa.csv. This step is necessary because the NREL/ATB dataset only presents fuel costs for nuclear and biomass. Oil and gas US-specific fuel costs are fetched from the World Bank's annual prices with projections up to 2030 (based on World Bank's 2026 estimations). Coal fuel cost is fetched from the EIA Annual Coal Report 2023.

Other noteworthy changes

Aligning the technology names of atb_e_2022 to the names of atb_e_2024

The technologies listed below have different name between atb_e_2022.parquet and atb_e_2024.parquet. Therefore the technologies on the left-hand side are renamed to the names on the right-hand side.

atb_e_2022 display_name	atb_e_2024 display_name
Land-Based Wind - Class 2	Land-Based Wind - Class 2 - Technology 1
Land-Based Wind - Class 3	Land-Based Wind - Class 3 - Technology 1
Land-Based Wind - Class 4	Land-Based Wind - Class 4 - Technology 1
Land-Based Wind - Class 5	Land-Based Wind - Class 5 - Technology 1
Land-Based Wind - Class 6	Land-Based Wind - Class 6 - Technology 1
Land-Based Wind - Class 7	Land-Based Wind - Class 7 - Technology 1
Land-Based Wind - Class 8	Land-Based Wind - Class 8 - Technology 2
Land-Based Wind - Class 9	Land-Based Wind - Class 9 - Technology 3
Land-Based Wind - Class 10	Land-Based Wind - Class 10 - Technology 4
NG F-Frame CC	NG 2-on-1 Combined Cycle (F-Frame)
NG H-Frame CC	NG 2-on-1 Combined Cycle (H-Frame)
NG combined cycle 95% CCS (F-frame basis -> 2nd Gen Tech)	NG 2-on-1 Combined Cycle (F-Frame) 95% CCS
NG combined cycle 95% CCS (H-frame basis -> 2nd Gen Tech)	NG 2-on-1 Combined Cycle (H-Frame) 95% CCS
NG combined cycle Max CCS (F-frame basis -> 2nd Gen Tech)	NG 2-on-1 Combined Cycle (F-Frame) 97% CCS
NG combined cycle Max CCS (H-frame basis -> 2nd Gen Tech)	NG 2-on-1 Combined Cycle (H-Frame) 97% CCS
Coal-CCS-95% -> 2nd Gen Tech	Coal-95%-CCS
Coal-Max-CCS -> 2nd Gen Tech	Coal-99%-CCS
Coal-IGCC	Coal - IGCC
CSP - Class 7	CSP - Class 8
Nuclear - Small Modular Reactor	Nuclear - Small

Discarded technologies

The following technologies are present in the input file atb_e_2022.parquet. They are however not present in atb_e_2024.parquet. They are therefore discarded from the final cost output files. They are:

• Coal-CCS-95% -> Transformational Tech
• Coal-Max-CCS -> Transformational Tech
• Coal-new -> Transformational Tech
• NG combined cycle 95% CCS (F-frame basis -> Transformational Tech)
• NG combined cycle 95% CCS (H-frame basis -> Transformational Tech)
• NG combined cycle Max CCS (F-frame basis -> Transformational Tech)
• NG combined cycle Max CCS (H-frame basis -> Transformational Tech)

environment.yaml

We choose to take the the input datasets atb_e_2022 and atb_e_2024 in parquet format. This is because the corresponding csv files have a size which is significantly larger. This choice brings about the following addition to the environment.yaml file

 - pyarrow

unit tests

We added a test folder to include unit tests for the functions included in scripts/compile_cost_assumptions_nrel.py

inputs/manual_input.csv

New technologies have been added to manual_inputs.csv.

Checklist

• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Data source for new technologies is clearly stated.
• Newly introduced dependencies are added to environment.yaml (if applicable).
• A note for the release notes doc/release_notes.rst of the upcoming release is included.
• I consent to the release of this PR's code under the GPLv3 license.

[euronion]

Thanks @finozzifa for the PR - I haven't had a full look yet. Just a few codestyle comments from a first pass. I'm trying to simplify it a bit to make it easier for me to make sense of it.

I noticed you're making heavy use of .casefold() - is this really necessary, maybe even creating more issues downstream? I'm thinking of occassions where the case does not match, .casefold() catches it and allows the code to run, but then the (wrongly) cased words get added to the file later, which could mess up the capitalisation? I'd rather catch wrongly cased input right at the start and ask the user to fix. ("In the face of ambiguity, refuse the temptation to guess.")

I'll try to run it in the next days.

[euronion]

Ok, seems to be running smoothly!

@lkstrp : Is there any code conventions you'd like to see for newly added code? The repo has been wildwest in the past, so I as long it is working, I consider it good enough.

@finozzifa If you could just address the following things:

• Code simplifications mentioned
• Source indications
• Documentation update (mention NREL ATB option and output for US, how to run, and added keys to config file)

That would be great!

[finozzifa]

Hi @euronion,
thanks a lot for your review :) we really appreciate it.
I will propose a few answers to each of your comments and set the comments as resolved.
Of course, feel free to re-open them if you deem our answers as unsatisfactory.

Let us know when you feel happy to approve the changes and merge the PR :)

[finozzifa]

Thanks @finozzifa for the PR - I haven't had a full look yet. Just a few codestyle comments from a first pass. I'm trying to simplify it a bit to make it easier for me to make sense of it.

I noticed you're making heavy use of .casefold() - is this really necessary, maybe even creating more issues downstream? I'm thinking of occassions where the case does not match, .casefold() catches it and allows the code to run, but then the (wrongly) cased words get added to the file later, which could mess up the capitalisation? I'd rather catch wrongly cased input right at the start and ask the user to fix. ("In the face of ambiguity, refuse the temptation to guess.")

I'll try to run it in the next days.

Hi @euronion, I think it is way safer and a standard practice to use .casefold() when performing string comparisons. I would therefore keep it if this is fine for you.

[finozzifa]

Hi @pz-max, thank you for giving it a look. Actually, the idea was to use atb_e_2022 for 2020 only and atb_e_2024 for future years. Indeed, atb_e_2024 does not contain any data for 2020 and that would obviously cause issues in the model, while we would prefer to only have the most recent data for future years (so the ones coming from atb_e_2024). Does it sound good to you?

I prefer your solution. Historic values stay historic, future values are based on the most recent numbers. If there is a use case and need for other, we can raise an issue an adapt in the future, I'd prefer to keep it for another iteration though :)

Regarding dependencies (curiosity): why pyarrow and fastparquet? One parquet backend for pandas should be enough?

Solved as well. Thanks for noticing it. fastparquet was a left over from a test :)

[lkstrp]

As discussed @finozzifa

[fneum]

Ehm, there are some technologies where the costs outputs/costs....csv get overwritten with US cost assumptions, e.g., Fischer-Tropsch, direct air capture, and hydrogen storage. That should not happen—or at least it should have been discussed before merging.

Also, several electrolysis technologies (PEM, Alkaline, SOEC) have been added without removing or adjusting "electrolysis," which should be avoided. We have not been very clear about this in the past, but looking forward, we should not add the same technologies under different names.

@finozzifa, can I ask you to revise?