DIA template generation how-to#49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
drphilmarshall wants to merge 14 commits into
base:master
Choose a base branch
from
Open

DIA template generation how-to #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c5bc1ff
DIA project set-up
drphilmarshall Jul 11, 2018
41b5a89
Merge branch 'master' into project/DIA/drphilmarshall
drphilmarshall Jul 13, 2018
fca437b
Started template generation how-to
drphilmarshall Jul 13, 2018
713e188
Trying to show image...
drphilmarshall Jul 13, 2018
72d3f1d
Fixed figure, guessed section headings
drphilmarshall Jul 13, 2018
97e7e88
Merge branch 'master' into project/DIA/drphilmarshall
drphilmarshall Aug 8, 2018
90e8772
Merge branch 'master' into project/DIA/drphilmarshall
drphilmarshall Aug 24, 2018
5d1dc32
Started looking at Twinkles data
drphilmarshall Aug 24, 2018
dd9cad0
Merge branch 'project/DIA/drphilmarshall' of github.com:LSSTScienceCo…
drphilmarshall Aug 24, 2018
6b009b3
Upgrade to enable command line tasks to be found
drphilmarshall Aug 24, 2018
4f19e8c
Finding task docs
drphilmarshall Aug 24, 2018
eac65bd
Guessed MakeDiscreteSkyMap task set-up
drphilmarshall Aug 31, 2018
27cf16e
Merge branch 'master' into project/DIA/drphilmarshall
drphilmarshall Jan 25, 2019
a6c06ad
Replaced markdown with rst
drphilmarshall Jan 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
300 changes: 300 additions & 0 deletions DIA/DIA_How_To_Generate_a_Template_Image.ipynb
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# DIA: How To Generate a Template Image\n",
"<br>Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
"<br>Last Verified to Run: **2018-08-24**\n",
"<br>Verified Stack Release: **16.0**\n",
"<br>Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n",
"\n",
"In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n",
"\n",
"### Learning Objectives:\n",
"\n",
"After working through this tutorial you should be able to: \n",
"1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n",
"2. Combine these images into a decorrelated template image.\n",
"\n",
"### Logistics\n",
"This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
"\n",
"## Set-up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# What version of the Stack am I using?\n",
"! echo $HOSTNAME\n",
"! eups list -s | grep lsst_distrib"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! cd .. && python setup.py -q develop --user && cd -"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from stackclub import where_is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Background\n",
"\n",
"The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n",
"Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n",
"\n",
"> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n",
">\n",
"> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n",
"\n",
"These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n",
"\n",
"> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n",
"\n",
"* **Standard decorrelated coadds**\n",
"\n",
"* **Constant-PSF partially-decorrelated coadds**\n",
"\n",
"* **PSF-matched coadds**\n",
"\n",
"* **\"The result of inference on resampled exposures with no PSF-matching\"**\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The Twinkles_subset Dataset\n",
"\n",
"We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's what this data repository contains:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!ls -lh TWINKLES_DATA/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!cat TWINKLES_DATA/_mapper"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All of the relavent images and calibrations have already been ingested into the Butler for this data set."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Making a SkyMap\n",
"\n",
"We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n",
"\n",
"# cf the following line from Dominique Fouchez's minicookbook:\n",
"# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above command fails with the following error:\n",
"```\n",
"makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n",
"makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n",
"```\n",
"\n",
"We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"help(MakeDiscreteSkyMapTask)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"skyMapConfig = MakeDiscreteSkyMapConfig()\n",
"skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n",
"skyMapTask.run()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Making the Coadd Images"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
"#find the tract/patches overlapping visit 182014\n",
"./reportPatchesWithImages.py output --visits 182014 --filt r\n",
"sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n",
"\n",
"#make coadd on these patches to create template\n",
"makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n",
"assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n",
"\"\"\";"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
29 changes: 29 additions & 0 deletions DIA/README.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
Difference Image Analysis
-------------------------

This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.
To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)

.. list-table::
:widths: 10 20 10 10
:header-rows: 1

* - Notebook
- Short description
- Links
- Owner


* - **How_To_Generate_a_Template_Image**
- Options for template coadd generation, demonstration on Twinkles images.
- `ipynb <https://github.com/LSSTScienceCollaborations/StackClub/blob/master/DIA/DIA_How_To_Generate_a_Template_Image.ipynb>`__,
`rendered <https://nbviewer.jupyter.org/github/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/DIA_How_To_Generate_a_Template_Image.nbconvert.ipynb>`__

.. raw:: html

<a href="https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.log">
<img width="72" height="16" src="https://raw.githubusercontent.com/LSSTScienceCollaborations/StackClub/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.png">
</img>
</a>

- `Phil Marshall <https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall>`__
34 changes: 21 additions & 13 deletions GettingStarted/templates/template_Notebook.ipynb
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -42,24 +42,16 @@
"```\n",
"pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
"```\n",
"If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
"If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"python: can't open file 'setup.py': [Errno 2] No such file or directory\n"
]
}
],
"outputs": [],
"source": [
"! cd .. && python setup.py -q develop --user && cd -"
"! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -"
]
},
{
Expand All@@ -71,14 +63,30 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"During development, it's helpful to have the `where_is` documentation finder loaded in:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from stackclub import where_is"
]
},
{
"cell_type": "markdown",
"metadata": {
Expand Down
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -16,6 +16,7 @@ the LSST Science Collaborations.
| Basics | Guided tours of various key Stack classes and functions, data structures, etc. | [StackClub/Basics](Basics) |
| Visualization | Displaying images and catalogs. | [StackClub/Visualization](Visualization) |
| Image Processing | From raw images to `calexp`s and `coadd`s. | [StackClub/ImageProcessing](ImageProcessing) |
| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [StackClub/DIA](DIA) |
| SourceDetection | Detection of sources in images - including low surface brightness galaxies. | [StackClub/SourceDetection](SourceDetection) |
| Deblending | Deblending the objects | [StackClub/Deblending](Deblending) |
| Measurement | Measuring the objects | [StackClub/Measurement](Measurement) |
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
DIA template generation how-to#49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

drphilmarshall
wants to merge
14
commits into

base:master
Choose a base branch

from

 Open


DIA template generation how-to
#49
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
c5bc1ff

DIA project set-up


drphilmarshall Jul 11, 2018
41b5a89

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jul 13, 2018
fca437b

Started template generation how-to


drphilmarshall Jul 13, 2018
713e188

Trying to show image...


drphilmarshall Jul 13, 2018
72d3f1d

Fixed figure, guessed section headings


drphilmarshall Jul 13, 2018
97e7e88

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 8, 2018
90e8772

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 24, 2018
5d1dc32

Started looking at Twinkles data


drphilmarshall Aug 24, 2018
dd9cad0

Merge branch 'project/DIA/drphilmarshall' of github.com:LSSTScienceCo…


drphilmarshall Aug 24, 2018
6b009b3

Upgrade to enable command line tasks to be found


drphilmarshall Aug 24, 2018
4f19e8c

Finding task docs


drphilmarshall Aug 24, 2018
eac65bd

Guessed MakeDiscreteSkyMap task set-up


drphilmarshall Aug 31, 2018
27cf16e

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jan 25, 2019
a6c06ad

Replaced markdown with rst


drphilmarshall Jan 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


300 changes: 300 additions & 0 deletions
DIA/DIA_How_To_Generate_a_Template_Image.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
{
 "cells": [
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "# DIA: How To Generate a Template Image\n",
 "<br>Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
 "<br>Last Verified to Run: **2018-08-24**\n",
 "<br>Verified Stack Release: **16.0**\n",
 "<br>Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n",
 "\n",
 "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n",
 "\n",
 "### Learning Objectives:\n",
 "\n",
 "After working through this tutorial you should be able to: \n",
 "1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n",
 "2. Combine these images into a decorrelated template image.\n",
 "\n",
 "### Logistics\n",
 "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
 "\n",
 "## Set-up"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "# What version of the Stack am I using?\n",
 "! echo $HOSTNAME\n",
 "! eups list -s | grep lsst_distrib"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Background\n",
 "\n",
 "The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n",
 "Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n",
 "\n",
 "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n",
 ">\n",
 "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n",
 "\n",
 "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n",
 "\n",
 "> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\""
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n",
 "\n",
 "* **Standard decorrelated coadds**\n",
 "\n",
 "* **Constant-PSF partially-decorrelated coadds**\n",
 "\n",
 "* **PSF-matched coadds**\n",
 "\n",
 "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## The Twinkles_subset Dataset\n",
 "\n",
 "We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%%bash\n",
 "if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Here's what this data repository contains:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!ls -lh TWINKLES_DATA/"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!cat TWINKLES_DATA/_mapper"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "All of the relavent images and calibrations have already been ingested into the Butler for this data set."
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making a SkyMap\n",
 "\n",
 "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n",
 "\n",
 "# cf the following line from Dominique Fouchez's minicookbook:\n",
 "# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The above command fails with the following error:\n",
 "```\n",
 "makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n",
 "makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n",
 "```\n",
 "\n",
 "We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "help(MakeDiscreteSkyMapTask)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "skyMapConfig = MakeDiscreteSkyMapConfig()\n",
 "skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n",
 "skyMapTask.run()"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making the Coadd Images"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "\"\"\"\n",
 "#find the tract/patches overlapping visit 182014\n",
 "./reportPatchesWithImages.py output --visits 182014 --filt r\n",
 "sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n",
 "\n",
 "#make coadd on these patches to create template\n",
 "makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n",
 "assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n",
 "\"\"\";"
 ]
 }
 ],
 "metadata": {
 "kernelspec": {
 "display_name": "Python 2",
 "language": "python",
 "name": "python2"
 },
 "language_info": {
 "codemirror_mode": {
 "name": "ipython",
 "version": 2
 },
 "file_extension": ".py",
 "mimetype": "text/x-python",
 "name": "python",
 "nbconvert_exporter": "python",
 "pygments_lexer": "ipython2",
 "version": "2.7.12"
 }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

29 changes: 29 additions & 0 deletions
DIA/README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
Difference Image Analysis
-------------------------

This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.
To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)

.. list-table::
 :widths: 10 20 10 10
 :header-rows: 1

 * - Notebook
 - Short description
 - Links
 - Owner


 * - **How_To_Generate_a_Template_Image**
 - Options for template coadd generation, demonstration on Twinkles images.
 - `ipynb <https://github.com/LSSTScienceCollaborations/StackClub/blob/master/DIA/DIA_How_To_Generate_a_Template_Image.ipynb>`__,
 `rendered <https://nbviewer.jupyter.org/github/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/DIA_How_To_Generate_a_Template_Image.nbconvert.ipynb>`__

 .. raw:: html

 <a href="https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.log">
 <img width="72" height="16" src="https://raw.githubusercontent.com/LSSTScienceCollaborations/StackClub/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.png">
 </img>
 </a>

 - `Phil Marshall <https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall>`__

34 changes: 21 additions & 13 deletions
GettingStarted/templates/template_Notebook.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,24 +42,16 @@
 "```\n",
 "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
 "```\n",
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "execution_count": null,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "python: can't open file 'setup.py': [Errno 2] No such file or directory\n"
 ]
 }
 ],
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 "! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -"
 ]
 },
 {
Expand All@@ -71,14 +63,30 @@
 },
 {
 "cell_type": "code",
 "execution_count": 2,
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%load_ext autoreload\n",
 "%autoreload 2"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `where_is` documentation finder loaded in:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {
Expand Down

1 change: 1 addition & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,6 +16,7 @@ the LSST Science Collaborations.
| Basics | Guided tours of various key Stack classes and functions, data structures, etc. | [StackClub/Basics](Basics) |
| Visualization | Displaying images and catalogs. | [StackClub/Visualization](Visualization) |
| Image Processing | From raw images to `calexp`s and `coadd`s. | [StackClub/ImageProcessing](ImageProcessing) |
| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [StackClub/DIA](DIA) |
| SourceDetection | Detection of sources in images - including low surface brightness galaxies. | [StackClub/SourceDetection](SourceDetection) |
| Deblending | Deblending the objects | [StackClub/Deblending](Deblending) |
| Measurement | Measuring the objects | [StackClub/Measurement](Measurement) |
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
DIA template generation how-to#49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

drphilmarshall
wants to merge
14
commits into

base:master
Choose a base branch

from

 Open


DIA template generation how-to
#49
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
c5bc1ff

DIA project set-up


drphilmarshall Jul 11, 2018
41b5a89

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jul 13, 2018
fca437b

Started template generation how-to


drphilmarshall Jul 13, 2018
713e188

Trying to show image...


drphilmarshall Jul 13, 2018
72d3f1d

Fixed figure, guessed section headings


drphilmarshall Jul 13, 2018
97e7e88

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 8, 2018
90e8772

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 24, 2018
5d1dc32

Started looking at Twinkles data


drphilmarshall Aug 24, 2018
dd9cad0

Merge branch 'project/DIA/drphilmarshall' of github.com:LSSTScienceCo…


drphilmarshall Aug 24, 2018
6b009b3

Upgrade to enable command line tasks to be found


drphilmarshall Aug 24, 2018
4f19e8c

Finding task docs


drphilmarshall Aug 24, 2018
eac65bd

Guessed MakeDiscreteSkyMap task set-up


drphilmarshall Aug 31, 2018
27cf16e

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jan 25, 2019
a6c06ad

Replaced markdown with rst


drphilmarshall Jan 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


300 changes: 300 additions & 0 deletions
DIA/DIA_How_To_Generate_a_Template_Image.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
{
 "cells": [
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "# DIA: How To Generate a Template Image\n",
 "<br>Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
 "<br>Last Verified to Run: **2018-08-24**\n",
 "<br>Verified Stack Release: **16.0**\n",
 "<br>Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n",
 "\n",
 "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n",
 "\n",
 "### Learning Objectives:\n",
 "\n",
 "After working through this tutorial you should be able to: \n",
 "1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n",
 "2. Combine these images into a decorrelated template image.\n",
 "\n",
 "### Logistics\n",
 "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
 "\n",
 "## Set-up"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "# What version of the Stack am I using?\n",
 "! echo $HOSTNAME\n",
 "! eups list -s | grep lsst_distrib"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Background\n",
 "\n",
 "The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n",
 "Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n",
 "\n",
 "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n",
 ">\n",
 "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n",
 "\n",
 "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n",
 "\n",
 "> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\""
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n",
 "\n",
 "* **Standard decorrelated coadds**\n",
 "\n",
 "* **Constant-PSF partially-decorrelated coadds**\n",
 "\n",
 "* **PSF-matched coadds**\n",
 "\n",
 "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## The Twinkles_subset Dataset\n",
 "\n",
 "We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%%bash\n",
 "if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Here's what this data repository contains:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!ls -lh TWINKLES_DATA/"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!cat TWINKLES_DATA/_mapper"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "All of the relavent images and calibrations have already been ingested into the Butler for this data set."
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making a SkyMap\n",
 "\n",
 "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n",
 "\n",
 "# cf the following line from Dominique Fouchez's minicookbook:\n",
 "# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The above command fails with the following error:\n",
 "```\n",
 "makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n",
 "makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n",
 "```\n",
 "\n",
 "We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "help(MakeDiscreteSkyMapTask)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "skyMapConfig = MakeDiscreteSkyMapConfig()\n",
 "skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n",
 "skyMapTask.run()"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making the Coadd Images"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "\"\"\"\n",
 "#find the tract/patches overlapping visit 182014\n",
 "./reportPatchesWithImages.py output --visits 182014 --filt r\n",
 "sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n",
 "\n",
 "#make coadd on these patches to create template\n",
 "makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n",
 "assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n",
 "\"\"\";"
 ]
 }
 ],
 "metadata": {
 "kernelspec": {
 "display_name": "Python 2",
 "language": "python",
 "name": "python2"
 },
 "language_info": {
 "codemirror_mode": {
 "name": "ipython",
 "version": 2
 },
 "file_extension": ".py",
 "mimetype": "text/x-python",
 "name": "python",
 "nbconvert_exporter": "python",
 "pygments_lexer": "ipython2",
 "version": "2.7.12"
 }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

29 changes: 29 additions & 0 deletions
DIA/README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
Difference Image Analysis
-------------------------

This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.
To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)

.. list-table::
 :widths: 10 20 10 10
 :header-rows: 1

 * - Notebook
 - Short description
 - Links
 - Owner


 * - **How_To_Generate_a_Template_Image**
 - Options for template coadd generation, demonstration on Twinkles images.
 - `ipynb <https://github.com/LSSTScienceCollaborations/StackClub/blob/master/DIA/DIA_How_To_Generate_a_Template_Image.ipynb>`__,
 `rendered <https://nbviewer.jupyter.org/github/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/DIA_How_To_Generate_a_Template_Image.nbconvert.ipynb>`__

 .. raw:: html

 <a href="https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.log">
 <img width="72" height="16" src="https://raw.githubusercontent.com/LSSTScienceCollaborations/StackClub/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.png">
 </img>
 </a>

 - `Phil Marshall <https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall>`__

34 changes: 21 additions & 13 deletions
GettingStarted/templates/template_Notebook.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,24 +42,16 @@
 "```\n",
 "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
 "```\n",
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "execution_count": null,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "python: can't open file 'setup.py': [Errno 2] No such file or directory\n"
 ]
 }
 ],
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 "! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -"
 ]
 },
 {
Expand All@@ -71,14 +63,30 @@
 },
 {
 "cell_type": "code",
 "execution_count": 2,
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%load_ext autoreload\n",
 "%autoreload 2"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `where_is` documentation finder loaded in:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {
Expand Down

1 change: 1 addition & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,6 +16,7 @@ the LSST Science Collaborations.
| Basics | Guided tours of various key Stack classes and functions, data structures, etc. | [StackClub/Basics](Basics) |
| Visualization | Displaying images and catalogs. | [StackClub/Visualization](Visualization) |
| Image Processing | From raw images to `calexp`s and `coadd`s. | [StackClub/ImageProcessing](ImageProcessing) |
| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [StackClub/DIA](DIA) |
| SourceDetection | Detection of sources in images - including low surface brightness galaxies. | [StackClub/SourceDetection](SourceDetection) |
| Deblending | Deblending the objects | [StackClub/Deblending](Deblending) |
| Measurement | Measuring the objects | [StackClub/Measurement](Measurement) |
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
DIA template generation how-to#49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

drphilmarshall
wants to merge
14
commits into

base:master
Choose a base branch

from

 Open


DIA template generation how-to
#49
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
c5bc1ff

DIA project set-up


drphilmarshall Jul 11, 2018
41b5a89

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jul 13, 2018
fca437b

Started template generation how-to


drphilmarshall Jul 13, 2018
713e188

Trying to show image...


drphilmarshall Jul 13, 2018
72d3f1d

Fixed figure, guessed section headings


drphilmarshall Jul 13, 2018
97e7e88

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 8, 2018
90e8772

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 24, 2018
5d1dc32

Started looking at Twinkles data


drphilmarshall Aug 24, 2018
dd9cad0

Merge branch 'project/DIA/drphilmarshall' of github.com:LSSTScienceCo…


drphilmarshall Aug 24, 2018
6b009b3

Upgrade to enable command line tasks to be found


drphilmarshall Aug 24, 2018
4f19e8c

Finding task docs


drphilmarshall Aug 24, 2018
eac65bd

Guessed MakeDiscreteSkyMap task set-up


drphilmarshall Aug 31, 2018
27cf16e

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jan 25, 2019
a6c06ad

Replaced markdown with rst


drphilmarshall Jan 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


300 changes: 300 additions & 0 deletions
DIA/DIA_How_To_Generate_a_Template_Image.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
{
 "cells": [
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "# DIA: How To Generate a Template Image\n",
 "<br>Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
 "<br>Last Verified to Run: **2018-08-24**\n",
 "<br>Verified Stack Release: **16.0**\n",
 "<br>Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n",
 "\n",
 "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n",
 "\n",
 "### Learning Objectives:\n",
 "\n",
 "After working through this tutorial you should be able to: \n",
 "1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n",
 "2. Combine these images into a decorrelated template image.\n",
 "\n",
 "### Logistics\n",
 "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
 "\n",
 "## Set-up"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "# What version of the Stack am I using?\n",
 "! echo $HOSTNAME\n",
 "! eups list -s | grep lsst_distrib"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Background\n",
 "\n",
 "The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n",
 "Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n",
 "\n",
 "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n",
 ">\n",
 "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n",
 "\n",
 "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n",
 "\n",
 "> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\""
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n",
 "\n",
 "* **Standard decorrelated coadds**\n",
 "\n",
 "* **Constant-PSF partially-decorrelated coadds**\n",
 "\n",
 "* **PSF-matched coadds**\n",
 "\n",
 "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## The Twinkles_subset Dataset\n",
 "\n",
 "We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%%bash\n",
 "if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Here's what this data repository contains:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!ls -lh TWINKLES_DATA/"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!cat TWINKLES_DATA/_mapper"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "All of the relavent images and calibrations have already been ingested into the Butler for this data set."
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making a SkyMap\n",
 "\n",
 "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n",
 "\n",
 "# cf the following line from Dominique Fouchez's minicookbook:\n",
 "# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The above command fails with the following error:\n",
 "```\n",
 "makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n",
 "makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n",
 "```\n",
 "\n",
 "We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "help(MakeDiscreteSkyMapTask)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "skyMapConfig = MakeDiscreteSkyMapConfig()\n",
 "skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n",
 "skyMapTask.run()"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making the Coadd Images"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "\"\"\"\n",
 "#find the tract/patches overlapping visit 182014\n",
 "./reportPatchesWithImages.py output --visits 182014 --filt r\n",
 "sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n",
 "\n",
 "#make coadd on these patches to create template\n",
 "makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n",
 "assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n",
 "\"\"\";"
 ]
 }
 ],
 "metadata": {
 "kernelspec": {
 "display_name": "Python 2",
 "language": "python",
 "name": "python2"
 },
 "language_info": {
 "codemirror_mode": {
 "name": "ipython",
 "version": 2
 },
 "file_extension": ".py",
 "mimetype": "text/x-python",
 "name": "python",
 "nbconvert_exporter": "python",
 "pygments_lexer": "ipython2",
 "version": "2.7.12"
 }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

29 changes: 29 additions & 0 deletions
DIA/README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
Difference Image Analysis
-------------------------

This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.
To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)

.. list-table::
 :widths: 10 20 10 10
 :header-rows: 1

 * - Notebook
 - Short description
 - Links
 - Owner


 * - **How_To_Generate_a_Template_Image**
 - Options for template coadd generation, demonstration on Twinkles images.
 - `ipynb <https://github.com/LSSTScienceCollaborations/StackClub/blob/master/DIA/DIA_How_To_Generate_a_Template_Image.ipynb>`__,
 `rendered <https://nbviewer.jupyter.org/github/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/DIA_How_To_Generate_a_Template_Image.nbconvert.ipynb>`__

 .. raw:: html

 <a href="https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.log">
 <img width="72" height="16" src="https://raw.githubusercontent.com/LSSTScienceCollaborations/StackClub/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.png">
 </img>
 </a>

 - `Phil Marshall <https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall>`__

34 changes: 21 additions & 13 deletions
GettingStarted/templates/template_Notebook.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,24 +42,16 @@
 "```\n",
 "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
 "```\n",
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "execution_count": null,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "python: can't open file 'setup.py': [Errno 2] No such file or directory\n"
 ]
 }
 ],
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 "! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -"
 ]
 },
 {
Expand All@@ -71,14 +63,30 @@
 },
 {
 "cell_type": "code",
 "execution_count": 2,
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%load_ext autoreload\n",
 "%autoreload 2"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `where_is` documentation finder loaded in:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {
Expand Down

1 change: 1 addition & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,6 +16,7 @@ the LSST Science Collaborations.
| Basics | Guided tours of various key Stack classes and functions, data structures, etc. | [StackClub/Basics](Basics) |
| Visualization | Displaying images and catalogs. | [StackClub/Visualization](Visualization) |
| Image Processing | From raw images to `calexp`s and `coadd`s. | [StackClub/ImageProcessing](ImageProcessing) |
| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [StackClub/DIA](DIA) |
| SourceDetection | Detection of sources in images - including low surface brightness galaxies. | [StackClub/SourceDetection](SourceDetection) |
| Deblending | Deblending the objects | [StackClub/Deblending](Deblending) |
| Measurement | Measuring the objects | [StackClub/Measurement](Measurement) |
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
DIA template generation how-to#49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

drphilmarshall
wants to merge
14
commits into

base:master
Choose a base branch

from

 Open


DIA template generation how-to
#49
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
c5bc1ff

DIA project set-up


drphilmarshall Jul 11, 2018
41b5a89

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jul 13, 2018
fca437b

Started template generation how-to


drphilmarshall Jul 13, 2018
713e188

Trying to show image...


drphilmarshall Jul 13, 2018
72d3f1d

Fixed figure, guessed section headings


drphilmarshall Jul 13, 2018
97e7e88

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 8, 2018
90e8772

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 24, 2018
5d1dc32

Started looking at Twinkles data


drphilmarshall Aug 24, 2018
dd9cad0

Merge branch 'project/DIA/drphilmarshall' of github.com:LSSTScienceCo…


drphilmarshall Aug 24, 2018
6b009b3

Upgrade to enable command line tasks to be found


drphilmarshall Aug 24, 2018
4f19e8c

Finding task docs


drphilmarshall Aug 24, 2018
eac65bd

Guessed MakeDiscreteSkyMap task set-up


drphilmarshall Aug 31, 2018
27cf16e

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jan 25, 2019
a6c06ad

Replaced markdown with rst


drphilmarshall Jan 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


300 changes: 300 additions & 0 deletions
DIA/DIA_How_To_Generate_a_Template_Image.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
{
 "cells": [
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "# DIA: How To Generate a Template Image\n",
 "<br>Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
 "<br>Last Verified to Run: **2018-08-24**\n",
 "<br>Verified Stack Release: **16.0**\n",
 "<br>Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n",
 "\n",
 "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n",
 "\n",
 "### Learning Objectives:\n",
 "\n",
 "After working through this tutorial you should be able to: \n",
 "1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n",
 "2. Combine these images into a decorrelated template image.\n",
 "\n",
 "### Logistics\n",
 "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
 "\n",
 "## Set-up"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "# What version of the Stack am I using?\n",
 "! echo $HOSTNAME\n",
 "! eups list -s | grep lsst_distrib"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Background\n",
 "\n",
 "The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n",
 "Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n",
 "\n",
 "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n",
 ">\n",
 "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n",
 "\n",
 "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n",
 "\n",
 "> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\""
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n",
 "\n",
 "* **Standard decorrelated coadds**\n",
 "\n",
 "* **Constant-PSF partially-decorrelated coadds**\n",
 "\n",
 "* **PSF-matched coadds**\n",
 "\n",
 "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## The Twinkles_subset Dataset\n",
 "\n",
 "We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%%bash\n",
 "if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Here's what this data repository contains:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!ls -lh TWINKLES_DATA/"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!cat TWINKLES_DATA/_mapper"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "All of the relavent images and calibrations have already been ingested into the Butler for this data set."
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making a SkyMap\n",
 "\n",
 "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n",
 "\n",
 "# cf the following line from Dominique Fouchez's minicookbook:\n",
 "# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The above command fails with the following error:\n",
 "```\n",
 "makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n",
 "makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n",
 "```\n",
 "\n",
 "We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "help(MakeDiscreteSkyMapTask)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "skyMapConfig = MakeDiscreteSkyMapConfig()\n",
 "skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n",
 "skyMapTask.run()"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making the Coadd Images"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "\"\"\"\n",
 "#find the tract/patches overlapping visit 182014\n",
 "./reportPatchesWithImages.py output --visits 182014 --filt r\n",
 "sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n",
 "\n",
 "#make coadd on these patches to create template\n",
 "makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n",
 "assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n",
 "\"\"\";"
 ]
 }
 ],
 "metadata": {
 "kernelspec": {
 "display_name": "Python 2",
 "language": "python",
 "name": "python2"
 },
 "language_info": {
 "codemirror_mode": {
 "name": "ipython",
 "version": 2
 },
 "file_extension": ".py",
 "mimetype": "text/x-python",
 "name": "python",
 "nbconvert_exporter": "python",
 "pygments_lexer": "ipython2",
 "version": "2.7.12"
 }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

29 changes: 29 additions & 0 deletions
DIA/README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
Difference Image Analysis
-------------------------

This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.
To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)

.. list-table::
 :widths: 10 20 10 10
 :header-rows: 1

 * - Notebook
 - Short description
 - Links
 - Owner


 * - **How_To_Generate_a_Template_Image**
 - Options for template coadd generation, demonstration on Twinkles images.
 - `ipynb <https://github.com/LSSTScienceCollaborations/StackClub/blob/master/DIA/DIA_How_To_Generate_a_Template_Image.ipynb>`__,
 `rendered <https://nbviewer.jupyter.org/github/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/DIA_How_To_Generate_a_Template_Image.nbconvert.ipynb>`__

 .. raw:: html

 <a href="https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.log">
 <img width="72" height="16" src="https://raw.githubusercontent.com/LSSTScienceCollaborations/StackClub/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.png">
 </img>
 </a>

 - `Phil Marshall <https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall>`__

34 changes: 21 additions & 13 deletions
GettingStarted/templates/template_Notebook.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,24 +42,16 @@
 "```\n",
 "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
 "```\n",
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "execution_count": null,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "python: can't open file 'setup.py': [Errno 2] No such file or directory\n"
 ]
 }
 ],
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 "! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -"
 ]
 },
 {
Expand All@@ -71,14 +63,30 @@
 },
 {
 "cell_type": "code",
 "execution_count": 2,
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%load_ext autoreload\n",
 "%autoreload 2"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `where_is` documentation finder loaded in:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {
Expand Down

1 change: 1 addition & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,6 +16,7 @@ the LSST Science Collaborations.
| Basics | Guided tours of various key Stack classes and functions, data structures, etc. | [StackClub/Basics](Basics) |
| Visualization | Displaying images and catalogs. | [StackClub/Visualization](Visualization) |
| Image Processing | From raw images to `calexp`s and `coadd`s. | [StackClub/ImageProcessing](ImageProcessing) |
| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [StackClub/DIA](DIA) |
| SourceDetection | Detection of sources in images - including low surface brightness galaxies. | [StackClub/SourceDetection](SourceDetection) |
| Deblending | Deblending the objects | [StackClub/Deblending](Deblending) |
| Measurement | Measuring the objects | [StackClub/Measurement](Measurement) |
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
DIA template generation how-to#49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

drphilmarshall
wants to merge
14
commits into

base:master
Choose a base branch

from

 Open


DIA template generation how-to
#49
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
c5bc1ff

DIA project set-up


drphilmarshall Jul 11, 2018
41b5a89

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jul 13, 2018
fca437b

Started template generation how-to


drphilmarshall Jul 13, 2018
713e188

Trying to show image...


drphilmarshall Jul 13, 2018
72d3f1d

Fixed figure, guessed section headings


drphilmarshall Jul 13, 2018
97e7e88

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 8, 2018
90e8772

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 24, 2018
5d1dc32

Started looking at Twinkles data


drphilmarshall Aug 24, 2018
dd9cad0

Merge branch 'project/DIA/drphilmarshall' of github.com:LSSTScienceCo…


drphilmarshall Aug 24, 2018
6b009b3

Upgrade to enable command line tasks to be found


drphilmarshall Aug 24, 2018
4f19e8c

Finding task docs


drphilmarshall Aug 24, 2018
eac65bd

Guessed MakeDiscreteSkyMap task set-up


drphilmarshall Aug 31, 2018
27cf16e

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jan 25, 2019
a6c06ad

Replaced markdown with rst


drphilmarshall Jan 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


300 changes: 300 additions & 0 deletions
DIA/DIA_How_To_Generate_a_Template_Image.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
{
 "cells": [
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "# DIA: How To Generate a Template Image\n",
 "<br>Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
 "<br>Last Verified to Run: **2018-08-24**\n",
 "<br>Verified Stack Release: **16.0**\n",
 "<br>Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n",
 "\n",
 "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n",
 "\n",
 "### Learning Objectives:\n",
 "\n",
 "After working through this tutorial you should be able to: \n",
 "1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n",
 "2. Combine these images into a decorrelated template image.\n",
 "\n",
 "### Logistics\n",
 "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
 "\n",
 "## Set-up"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "# What version of the Stack am I using?\n",
 "! echo $HOSTNAME\n",
 "! eups list -s | grep lsst_distrib"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Background\n",
 "\n",
 "The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n",
 "Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n",
 "\n",
 "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n",
 ">\n",
 "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n",
 "\n",
 "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n",
 "\n",
 "> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\""
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n",
 "\n",
 "* **Standard decorrelated coadds**\n",
 "\n",
 "* **Constant-PSF partially-decorrelated coadds**\n",
 "\n",
 "* **PSF-matched coadds**\n",
 "\n",
 "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## The Twinkles_subset Dataset\n",
 "\n",
 "We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%%bash\n",
 "if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Here's what this data repository contains:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!ls -lh TWINKLES_DATA/"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!cat TWINKLES_DATA/_mapper"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "All of the relavent images and calibrations have already been ingested into the Butler for this data set."
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making a SkyMap\n",
 "\n",
 "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n",
 "\n",
 "# cf the following line from Dominique Fouchez's minicookbook:\n",
 "# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The above command fails with the following error:\n",
 "```\n",
 "makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n",
 "makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n",
 "```\n",
 "\n",
 "We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "help(MakeDiscreteSkyMapTask)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "skyMapConfig = MakeDiscreteSkyMapConfig()\n",
 "skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n",
 "skyMapTask.run()"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making the Coadd Images"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "\"\"\"\n",
 "#find the tract/patches overlapping visit 182014\n",
 "./reportPatchesWithImages.py output --visits 182014 --filt r\n",
 "sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n",
 "\n",
 "#make coadd on these patches to create template\n",
 "makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n",
 "assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n",
 "\"\"\";"
 ]
 }
 ],
 "metadata": {
 "kernelspec": {
 "display_name": "Python 2",
 "language": "python",
 "name": "python2"
 },
 "language_info": {
 "codemirror_mode": {
 "name": "ipython",
 "version": 2
 },
 "file_extension": ".py",
 "mimetype": "text/x-python",
 "name": "python",
 "nbconvert_exporter": "python",
 "pygments_lexer": "ipython2",
 "version": "2.7.12"
 }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

29 changes: 29 additions & 0 deletions
DIA/README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
Difference Image Analysis
-------------------------

This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.
To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)

.. list-table::
 :widths: 10 20 10 10
 :header-rows: 1

 * - Notebook
 - Short description
 - Links
 - Owner


 * - **How_To_Generate_a_Template_Image**
 - Options for template coadd generation, demonstration on Twinkles images.
 - `ipynb <https://github.com/LSSTScienceCollaborations/StackClub/blob/master/DIA/DIA_How_To_Generate_a_Template_Image.ipynb>`__,
 `rendered <https://nbviewer.jupyter.org/github/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/DIA_How_To_Generate_a_Template_Image.nbconvert.ipynb>`__

 .. raw:: html

 <a href="https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.log">
 <img width="72" height="16" src="https://raw.githubusercontent.com/LSSTScienceCollaborations/StackClub/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.png">
 </img>
 </a>

 - `Phil Marshall <https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall>`__

34 changes: 21 additions & 13 deletions
GettingStarted/templates/template_Notebook.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,24 +42,16 @@
 "```\n",
 "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
 "```\n",
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "execution_count": null,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "python: can't open file 'setup.py': [Errno 2] No such file or directory\n"
 ]
 }
 ],
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 "! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -"
 ]
 },
 {
Expand All@@ -71,14 +63,30 @@
 },
 {
 "cell_type": "code",
 "execution_count": 2,
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%load_ext autoreload\n",
 "%autoreload 2"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `where_is` documentation finder loaded in:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {
Expand Down

1 change: 1 addition & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,6 +16,7 @@ the LSST Science Collaborations.
| Basics | Guided tours of various key Stack classes and functions, data structures, etc. | [StackClub/Basics](Basics) |
| Visualization | Displaying images and catalogs. | [StackClub/Visualization](Visualization) |
| Image Processing | From raw images to `calexp`s and `coadd`s. | [StackClub/ImageProcessing](ImageProcessing) |
| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [StackClub/DIA](DIA) |
| SourceDetection | Detection of sources in images - including low surface brightness galaxies. | [StackClub/SourceDetection](SourceDetection) |
| Deblending | Deblending the objects | [StackClub/Deblending](Deblending) |
| Measurement | Measuring the objects | [StackClub/Measurement](Measurement) |
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
DIA template generation how-to#49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

drphilmarshall
wants to merge
14
commits into

base:master
Choose a base branch

from

 Open


DIA template generation how-to
#49
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
c5bc1ff

DIA project set-up


drphilmarshall Jul 11, 2018
41b5a89

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jul 13, 2018
fca437b

Started template generation how-to


drphilmarshall Jul 13, 2018
713e188

Trying to show image...


drphilmarshall Jul 13, 2018
72d3f1d

Fixed figure, guessed section headings


drphilmarshall Jul 13, 2018
97e7e88

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 8, 2018
90e8772

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 24, 2018
5d1dc32

Started looking at Twinkles data


drphilmarshall Aug 24, 2018
dd9cad0

Merge branch 'project/DIA/drphilmarshall' of github.com:LSSTScienceCo…


drphilmarshall Aug 24, 2018
6b009b3

Upgrade to enable command line tasks to be found


drphilmarshall Aug 24, 2018
4f19e8c

Finding task docs


drphilmarshall Aug 24, 2018
eac65bd

Guessed MakeDiscreteSkyMap task set-up


drphilmarshall Aug 31, 2018
27cf16e

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jan 25, 2019
a6c06ad

Replaced markdown with rst


drphilmarshall Jan 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


300 changes: 300 additions & 0 deletions
DIA/DIA_How_To_Generate_a_Template_Image.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
{
 "cells": [
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "# DIA: How To Generate a Template Image\n",
 "<br>Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
 "<br>Last Verified to Run: **2018-08-24**\n",
 "<br>Verified Stack Release: **16.0**\n",
 "<br>Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n",
 "\n",
 "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n",
 "\n",
 "### Learning Objectives:\n",
 "\n",
 "After working through this tutorial you should be able to: \n",
 "1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n",
 "2. Combine these images into a decorrelated template image.\n",
 "\n",
 "### Logistics\n",
 "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
 "\n",
 "## Set-up"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "# What version of the Stack am I using?\n",
 "! echo $HOSTNAME\n",
 "! eups list -s | grep lsst_distrib"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Background\n",
 "\n",
 "The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n",
 "Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n",
 "\n",
 "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n",
 ">\n",
 "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n",
 "\n",
 "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n",
 "\n",
 "> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\""
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n",
 "\n",
 "* **Standard decorrelated coadds**\n",
 "\n",
 "* **Constant-PSF partially-decorrelated coadds**\n",
 "\n",
 "* **PSF-matched coadds**\n",
 "\n",
 "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## The Twinkles_subset Dataset\n",
 "\n",
 "We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%%bash\n",
 "if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Here's what this data repository contains:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!ls -lh TWINKLES_DATA/"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!cat TWINKLES_DATA/_mapper"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "All of the relavent images and calibrations have already been ingested into the Butler for this data set."
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making a SkyMap\n",
 "\n",
 "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n",
 "\n",
 "# cf the following line from Dominique Fouchez's minicookbook:\n",
 "# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The above command fails with the following error:\n",
 "```\n",
 "makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n",
 "makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n",
 "```\n",
 "\n",
 "We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "help(MakeDiscreteSkyMapTask)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "skyMapConfig = MakeDiscreteSkyMapConfig()\n",
 "skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n",
 "skyMapTask.run()"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making the Coadd Images"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "\"\"\"\n",
 "#find the tract/patches overlapping visit 182014\n",
 "./reportPatchesWithImages.py output --visits 182014 --filt r\n",
 "sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n",
 "\n",
 "#make coadd on these patches to create template\n",
 "makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n",
 "assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n",
 "\"\"\";"
 ]
 }
 ],
 "metadata": {
 "kernelspec": {
 "display_name": "Python 2",
 "language": "python",
 "name": "python2"
 },
 "language_info": {
 "codemirror_mode": {
 "name": "ipython",
 "version": 2
 },
 "file_extension": ".py",
 "mimetype": "text/x-python",
 "name": "python",
 "nbconvert_exporter": "python",
 "pygments_lexer": "ipython2",
 "version": "2.7.12"
 }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

29 changes: 29 additions & 0 deletions
DIA/README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
Difference Image Analysis
-------------------------

This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.
To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)

.. list-table::
 :widths: 10 20 10 10
 :header-rows: 1

 * - Notebook
 - Short description
 - Links
 - Owner


 * - **How_To_Generate_a_Template_Image**
 - Options for template coadd generation, demonstration on Twinkles images.
 - `ipynb <https://github.com/LSSTScienceCollaborations/StackClub/blob/master/DIA/DIA_How_To_Generate_a_Template_Image.ipynb>`__,
 `rendered <https://nbviewer.jupyter.org/github/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/DIA_How_To_Generate_a_Template_Image.nbconvert.ipynb>`__

 .. raw:: html

 <a href="https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.log">
 <img width="72" height="16" src="https://raw.githubusercontent.com/LSSTScienceCollaborations/StackClub/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.png">
 </img>
 </a>

 - `Phil Marshall <https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall>`__

34 changes: 21 additions & 13 deletions
GettingStarted/templates/template_Notebook.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,24 +42,16 @@
 "```\n",
 "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
 "```\n",
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "execution_count": null,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "python: can't open file 'setup.py': [Errno 2] No such file or directory\n"
 ]
 }
 ],
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 "! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -"
 ]
 },
 {
Expand All@@ -71,14 +63,30 @@
 },
 {
 "cell_type": "code",
 "execution_count": 2,
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%load_ext autoreload\n",
 "%autoreload 2"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `where_is` documentation finder loaded in:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {
Expand Down

1 change: 1 addition & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,6 +16,7 @@ the LSST Science Collaborations.
| Basics | Guided tours of various key Stack classes and functions, data structures, etc. | [StackClub/Basics](Basics) |
| Visualization | Displaying images and catalogs. | [StackClub/Visualization](Visualization) |
| Image Processing | From raw images to `calexp`s and `coadd`s. | [StackClub/ImageProcessing](ImageProcessing) |
| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [StackClub/DIA](DIA) |
| SourceDetection | Detection of sources in images - including low surface brightness galaxies. | [StackClub/SourceDetection](SourceDetection) |
| Deblending | Deblending the objects | [StackClub/Deblending](Deblending) |
| Measurement | Measuring the objects | [StackClub/Measurement](Measurement) |
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
DIA template generation how-to#49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Open

drphilmarshall
wants to merge
14
commits into

base:master
Choose a base branch

from

 Open


DIA template generation how-to
#49
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


14 commits


Select commit
Hold shift + click to select a range
c5bc1ff

DIA project set-up


drphilmarshall Jul 11, 2018
41b5a89

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jul 13, 2018
fca437b

Started template generation how-to


drphilmarshall Jul 13, 2018
713e188

Trying to show image...


drphilmarshall Jul 13, 2018
72d3f1d

Fixed figure, guessed section headings


drphilmarshall Jul 13, 2018
97e7e88

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 8, 2018
90e8772

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Aug 24, 2018
5d1dc32

Started looking at Twinkles data


drphilmarshall Aug 24, 2018
dd9cad0

Merge branch 'project/DIA/drphilmarshall' of github.com:LSSTScienceCo…


drphilmarshall Aug 24, 2018
6b009b3

Upgrade to enable command line tasks to be found


drphilmarshall Aug 24, 2018
4f19e8c

Finding task docs


drphilmarshall Aug 24, 2018
eac65bd

Guessed MakeDiscreteSkyMap task set-up


drphilmarshall Aug 31, 2018
27cf16e

Merge branch 'master' into project/DIA/drphilmarshall


drphilmarshall Jan 25, 2019
a6c06ad

Replaced markdown with rst


drphilmarshall Jan 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


300 changes: 300 additions & 0 deletions
DIA/DIA_How_To_Generate_a_Template_Image.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
{
 "cells": [
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "# DIA: How To Generate a Template Image\n",
 "<br>Owner(s): **Phil Marshall** ([@drphilmarshall](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall))\n",
 "<br>Last Verified to Run: **2018-08-24**\n",
 "<br>Verified Stack Release: **16.0**\n",
 "<br>Discussion: **[issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)**\n",
 "\n",
 "In this tutorial we will understand the possible algorithm choices for DIA template image generation, see how these are currently implemented in the science pipelines, and demonstrate them on the Twinkles images.\n",
 "\n",
 "### Learning Objectives:\n",
 "\n",
 "After working through this tutorial you should be able to: \n",
 "1. Use a SkyMap to identify all the visit images in a given filter you want to coadd ;\n",
 "2. Combine these images into a decorrelated template image.\n",
 "\n",
 "### Logistics\n",
 "This notebook is intended to be runnable on `lsst-lspdev.ncsa.illinois.edu` from a local git clone of https://github.com/LSSTScienceCollaborations/StackClub.\n",
 "\n",
 "## Set-up"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "# What version of the Stack am I using?\n",
 "! echo $HOSTNAME\n",
 "! eups list -s | grep lsst_distrib"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `stackclub` library available - and the `where_is` documentation finder in particular."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Background\n",
 "\n",
 "The DRP pipelines are summarized in Figure 9 of the _Data Management Science Pipelines\n",
 "Design Document_, [LDM-151](http://ls.st/ldm-151), reproduced here:"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "![](https://github.com/lsst/LDM-151/raw/master/figures/drp_coaddition_and_diffim.png)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Difference imaging is shown in block 4, and includes creation of TemplateCoadd images, image differencing, and DIASource creation. The relevant pipeline design text on template generation, for the data release processing, is in Section 5.2:\n",
 "\n",
 "> \"4. We run the WarpTemplates, CoaddTemplates, and DiffIm pipelines to generate the DIASource and DiffExp datasets. We may then be able to generate better CalExp Masks than we can obtain from BackgroundMatchAndReject by comparing the DiffExp masks across visits in the UpdateMasks pipeline.\n",
 ">\n",
 "> \"5. After all CalExp components have been finalized, we run the WarpRemaining and CoaddRemaining to build additional coadd data products.\"\n",
 "\n",
 "These steps assume that the final visit CalExp images have already been made. The main coadd data product that is needed for DIA is the TemplateCoadd:\n",
 "\n",
 "> \"**TemplateCoadd**: A coadd data product used for difference imaging in both DRP and AP. In order to produce templates appropriate for the level of DCR in a given science image, these coadds may require a third dimension in addition to the usual two image dimen- sions (likely either wavelength or a quantity that is a function of airmass).\""
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Exactly how this TemplateCoadd is made is still under development: LDM-151 lists several possible scenarios, in which different algorithms for the TemplateCoadd generation are employed. The four choices for TemplateCoadd generation seem to be as follows:\n",
 "\n",
 "* **Standard decorrelated coadds**\n",
 "\n",
 "* **Constant-PSF partially-decorrelated coadds**\n",
 "\n",
 "* **PSF-matched coadds**\n",
 "\n",
 "* **\"The result of inference on resampled exposures with no PSF-matching\"**\n"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## The Twinkles_subset Dataset\n",
 "\n",
 "We'll use the `Twinkles_subset` data, in a local data \"repository\" copied from the shared project space:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%%bash\n",
 "if [ ! -d TWINKLES_DATA ]; then cp -r /project/shared/data/Twinkles_subset/input_data_v2 TWINKLES_DATA; fi"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Here's what this data repository contains:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!ls -lh TWINKLES_DATA/"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The Butler uses a mapper to find and organize data in a format specific to each camera. Here we're using `lsst.obs.lsstSim.LsstSimMapper` mapper for the Twinkles simulated data:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "!cat TWINKLES_DATA/_mapper"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "All of the relavent images and calibrations have already been ingested into the Butler for this data set."
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making a SkyMap\n",
 "\n",
 "We need to make a SkyMap, to define the _tracts_ and _patches_ of sky that will be covered by our coadd image. For this, we could use the [`lsst.pipe.tasks.makeDiscreteSkyMap.py`](https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeDiscreteSkyMap.py) command line task, as described in the [pipelines.lsst.io Getting started part 4](https://pipelines.lsst.io/getting-started/coaddition.html) tutorial."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "! makeDiscreteSkyMap.py TWINKLES_DATA --id --rerun coadd --config skyMap.projection=\"TAN\"\n",
 "\n",
 "# cf the following line from Dominique Fouchez's minicookbook:\n",
 "# makeSkyMap.py input --output output --configfile makeSkyMapConfig.py"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "The above command fails with the following error:\n",
 "```\n",
 "makeDiscreteSkyMap INFO: Extracting bounding boxes of 0 images\n",
 "makeDiscreteSkyMap FATAL: Failed: No data found from which to compute convex hull\n",
 "```\n",
 "\n",
 "We need to somehow specify which images in the Twinkles_subset to use - and these are e-images, rather than outputs from `processCcd`."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "where_is(\"makeDiscreteSkyMap.py\", in_the=\"source\", assuming_its_a=\"cmdlinetask\")"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Let's unpack the `makeDiscreteSkyMap` command line task, and configure and run it from python. First we need to import the task and its configuration object."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from lsst.pipe.tasks.makeDiscreteSkyMap import MakeDiscreteSkyMapTask, MakeDiscreteSkyMapConfig"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "help(MakeDiscreteSkyMapTask)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "Setting up the task with the default configuration, let's run it in this vanilla mode before reconfiguring."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "skyMapConfig = MakeDiscreteSkyMapConfig()\n",
 "skyMapTask = MakeDiscreteSkyMapTask(config=skyMapConfig)\n",
 "skyMapTask.run()"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## Making the Coadd Images"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "\"\"\"\n",
 "#find the tract/patches overlapping visit 182014\n",
 "./reportPatchesWithImages.py output --visits 182014 --filt r\n",
 "sed -e 's/^/--id filter=r /' r_182014_patches.list > patches_r.txt\n",
 "\n",
 "#make coadd on these patches to create template\n",
 "makeCoaddTempExp.py input --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32\n",
 "assembleCoadd.py output --output output @patches_r.txt --selectId filter=r visit=182014^452599 -j 32 \n",
 "\"\"\";"
 ]
 }
 ],
 "metadata": {
 "kernelspec": {
 "display_name": "Python 2",
 "language": "python",
 "name": "python2"
 },
 "language_info": {
 "codemirror_mode": {
 "name": "ipython",
 "version": 2
 },
 "file_extension": ".py",
 "mimetype": "text/x-python",
 "name": "python",
 "nbconvert_exporter": "python",
 "pygments_lexer": "ipython2",
 "version": "2.7.12"
 }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

29 changes: 29 additions & 0 deletions
DIA/README.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
Difference Image Analysis
-------------------------

This folder contains a set of tutorial notebooks exploring the Difference Image Analysis (DIA) parts of the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.
To join the discussion of this project, please see [issue #48](https://github.com/LSSTScienceCollaborations/DMStackClub/issues/48)

.. list-table::
 :widths: 10 20 10 10
 :header-rows: 1

 * - Notebook
 - Short description
 - Links
 - Owner


 * - **How_To_Generate_a_Template_Image**
 - Options for template coadd generation, demonstration on Twinkles images.
 - `ipynb <https://github.com/LSSTScienceCollaborations/StackClub/blob/master/DIA/DIA_How_To_Generate_a_Template_Image.ipynb>`__,
 `rendered <https://nbviewer.jupyter.org/github/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/DIA_How_To_Generate_a_Template_Image.nbconvert.ipynb>`__

 .. raw:: html

 <a href="https://github.com/LSSTScienceCollaborations/StackClub/blob/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.log">
 <img width="72" height="16" src="https://raw.githubusercontent.com/LSSTScienceCollaborations/StackClub/rendered/DIA/log/DIA_How_To_Generate_a_Template_Image.png">
 </img>
 </a>

 - `Phil Marshall <https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@drphilmarshall>`__

34 changes: 21 additions & 13 deletions
GettingStarted/templates/template_Notebook.ipynb

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,24 +42,16 @@
 "```\n",
 "pip install git+git://github.com/LSSTScienceCollaborations/StackClub.git#egg=stackclub\n",
 "```\n",
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation. In the top level folder of the `StackClub` repo, do:"
 "If you are developing the `stackclub` package (eg by adding modules to it to support the Stack Club tutorial that you are writing, you'll need to make a local, editable installation, like this:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "execution_count": null,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "python: can't open file 'setup.py': [Errno 2] No such file or directory\n"
 ]
 }
 ],
 "outputs": [],
 "source": [
 "! cd .. && python setup.py -q develop --user && cd -"
 "! cd $HOME/notebooks/StackClub && python setup.py -q develop --user && cd -"
 ]
 },
 {
Expand All@@ -71,14 +63,30 @@
 },
 {
 "cell_type": "code",
 "execution_count": 2,
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "%load_ext autoreload\n",
 "%autoreload 2"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "During development, it's helpful to have the `where_is` documentation finder loaded in:"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": [
 "from stackclub import where_is"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {
Expand Down

1 change: 1 addition & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -16,6 +16,7 @@ the LSST Science Collaborations.
| Basics | Guided tours of various key Stack classes and functions, data structures, etc. | [StackClub/Basics](Basics) |
| Visualization | Displaying images and catalogs. | [StackClub/Visualization](Visualization) |
| Image Processing | From raw images to `calexp`s and `coadd`s. | [StackClub/ImageProcessing](ImageProcessing) |
| Difference Image Analysis | Walkthroughs of the DIA parts of the science pipeline | [StackClub/DIA](DIA) |
| SourceDetection | Detection of sources in images - including low surface brightness galaxies. | [StackClub/SourceDetection](SourceDetection) |
| Deblending | Deblending the objects | [StackClub/Deblending](Deblending) |
| Measurement | Measuring the objects | [StackClub/Measurement](Measurement) |
Expand Down
Loading