Raw retrieval mode

[mikelangmayr]

Archon RAW-Mode Retrieval

What RAW mode is

The Archon continuously samples each CCD output at 16-bit / 100 MHz. Normal image
data is the result of on-chip CDS (correlated double sampling) and is useless
for many electrical measurements. RAW mode instead captures the raw ADC samples
from a single AD channel, before CDS, an oscilloscope view of the CCD output
waveform. Use it to inspect settling time, reset-level stability, clock
feedthrough, and to choose optimal CDS sample windows (SHP/SHD), or to optimise
read noise and linearity.

RAW is captured alongside the normal image during the same exposure, from the
same frame buffer. Because the frame buffer is finite, only a portion of the frame
can be captured this way.

The raw command

Command	Effect
raw config	report the current RAW keyword values
raw set <KEY> <VAL> [<KEY> <VAL> ...]	set RAW keyword(s), then apply (APPLYCDS)
raw read	retrieve RAW from the newest buffer and deliver it in-band
raw ?	help

The six configuration keys

Key	Meaning
RAWENABLE	1 to enable RAW capture, 0 to disable
RAWSEL	AD channel to capture: 0 = ch1 of the ADC in slot 5 … 15 = ch4 of the ADC in slot 8
RAWSTARTLINE	first line to capture (0–65535)
RAWENDLINE	last line to capture (0–65535) — inclusive
RAWSTARTPIXEL	pixel within each line where sampling begins
RAWSAMPLES	16-bit samples to store per line (rounded up to whole 1024-byte blocks)

RAWSTARTLINE=0, RAWENDLINE=1 captures two lines. RAWSTARTPIXEL only sets
where in the line sampling starts; it does not change the number of samples.

The ACF must already define these six keys — raw set writes existing keys, it
cannot create them.

Retrieving RAW data

raw config # inspect current settings
raw set RAWENABLE 1 RAWSEL 0 RAWSTARTLINE 0 RAWENDLINE 1 RAWSTARTPIXEL 0 RAWSAMPLES 2048
expose # RAW captured alongside the image
raw read # retrieve it

raw read returns a summary line, e.g.:

samples=2048 lines=2 bytes=8192 DONE

and delivers the data in-band to whatever frame-output sink is configured
(a FITS file, or a shared-memory object with SHM_ENABLED=yes) — no separate
file to go and load.

The retrieved data

The delivered frame is a lines × RAWSAMPLES array of uint16 (always 16-bit,
even when the pixel mode is HDR/32-bit):

• each row is one CCD line's raw waveform;
• the x axis is sample number (10 ns per sample at 100 MHz);
• the y axis is ADC counts (DN).

Plot a row and you have the oscilloscope trace of that line's readout.

fromastropy.ioimportfitsimportmatplotlib.pyplotaspltraw=fits.getdata("raw_00000001_1.fits") # shape (lines, RAWSAMPLES), uint16plt.plot(raw[0]) # first line's raw waveform

Watch the size

Bytes ≈ (RAWENDLINE − RAWSTARTLINE + 1) × RAWSAMPLES × 2, padded up per line to
whole 1024-byte blocks. RAW fills the frame buffer fast, per the Archon manual,
1000 lines × 1000 samples at a 100 kHz pixel clock is ~2 GB. Start small.

How it works (reference)

Mirrors the pixel-data path, per the Archon ICD:

1. FRAME? reports BUFnRAWOFFSET, BUFnRAWBLOCKS (blocks per line) and
   BUFnRAWLINES for each buffer.
2. camerad locks the buffer (LOCKn) and issues FETCH from
   BUFnBASE + BUFnRAWOFFSET (not BUFnBASE), sized from the reported raw
   blocks/lines (falling back to the config keys if the controller does not
   report them).
3. The fetched buffer is stripped of its per-line block padding, reshaped to
   lines × RAWSAMPLES, and dispatched in-band as 16-bit samples.

Status / caveats

• Validated end-to-end against the Archon emulator: a known ramp fed through
  round-trips byte-for-byte at the correct lines × RAWSAMPLES shape.
• The emulator serves RAW at offset 0, so the nonzero-BUFnRAWOFFSET path should
  be confirmed on real hardware during first use.
• Retrieving RAW-only (raw read) and pixel-only (expose) from the same buffer
  works today; a single combined command is a possible future addition.

[weatherhead99]

LGTM, the only real comment is more of a meta-comment.

I understand (I think) the reasoning for the ACF file to already have to define these keywords, and it's no problem for that to be the case here. HOWEVER, in the more general archon case I think a problem arises, specifically with the definition of taplines.

For example, imagine I have 3 taplines defined as such
TAPLINE0="AM37L,1,100"
TAPLINE1="AM38R,1,100"
TAPLINE2="AM39L,1,100"
TAPLINES=3

Now, for example, I want to switch modes and add another tapline
TAPLINE0="AM37L,1,100"
TAPLINE1="AM38R,1,100"
TAPLINE2="AM39L,1,100"
TAPLINE3="AM40R,1,100"
TAPLINES=4

This involves writing another key to the archon config. This occurs pretty frequently, example I can cite is in DEIMOS switching from FCS to SCIENCE mode (where we go from 2 taplines to 16). Now, we currently handle that by pre-defining all keys and then simply changing from e.g. TAPLINES=16 to TAPLINES=2, which causes the last 14 taplines to be ignored. We then change back to TAPLINES=16 to "un-ignore" those taplines. At least I think that's how it works, @astronomerdave feel free to correct me. This is a slightly ugly but workable solution. If we intend to support that level of configurability in camerad2, I think I am satisfied with it. OTOH it's definitely not as fflexible as being able to programatically build up the number of taplines "from scratch".

The same issue arises, for example, if I want to alter the timing script, and in doing so change the number of lines in a timing script. Same true also for parameters and constants. If we think doing that is outside even the eventual functionality of camerad2, I can definitely live with it.

[weatherhead99]

nothing important, and probably not possible due to API consistency

[weatherhead99]

out of curiosity what's the reason for std::strncmp here rather than std::string::starts_with?

[mikelangmayr]

The parser works on raw const char* slices, not std::string, so starts_with isn't available without adding one per key

[weatherhead99]

understood. std::string:;starts_with doesn't work. But std::string_view::starts_with might? (Although possibly unsafe in the raw const char* situation?)

UPDATE just looked it up. Looks like you can directly to string_view::starts_with on a const char* (https://en.cppreference.com/cpp/string/basic_string_view/starts_with)

[mikelangmayr]

Switched to std::string_view comparisons, that drops the manual length and first-char pre-checks while still just doing a length check plus memcmp under the hood.

[weatherhead99]

why not just return a std::optionalstd::string here? RTVO is guaranteed to elide that copy since c++17 AFAICS if performance is an issue

[mikelangmayr]

It matches the signature, and the long return encodes three states (NO_ERROR/ERROR/HELP), the optional can't do that

[weatherhead99]

fair enough.... we do have std::expected for that case ;-).
Broader question then why is this a long and not an enum class?
Or if there are only 3 states why is it a long and not a short?#

Anyway who cares, not to worry

[mikelangmayr]

long matches the whole codebase here, NO_ERROR/ERROR/HELP are constexpr long constants in common/common.h reused by every method in this class and others. An enum class would be nicer but touching just this one return type would break that convention without fixing anything, so leaving it for now. Agree std::expected would be the right shape if we ever do a broader pass on the error codes.

[prkrtg]

Might be good to handle the case of get_frame_status returns with an error before computing the raw geometry

[mikelangmayr]

Good catch, read_raw now checks the return value and bails out before computing raw geometry.

[prkrtg]

A question about the use of the changed bool -
So let's say one RAW setting changed, but another one doesn't. Regardless do we apply CDS if one of the RAW values change?

If so would the following case fail: first key RAWSEL changed would be true, then the second key RAWSAMPLES changed would be false so APPLYCDS would be skipped?

[weatherhead99]

every change of any RAW related keyword or TAPLINE related keyword or CDS related keyword requires an APPLYCDS command before it takes effect. There is no way to know strictly what the current status is of these keywords because one can update the config but not do an APPLYCDS. This is the reason (though it's annoying) why in foxtrot you can't individually set any of those, you have to provide a struct containing all CDS related keywords, so there is never this inconsistency.

Therefore, in my opinion, it's very hard to decide on any situation when you should NOT do APPLYCDS. Because it might be that the values in your config are simply not reflected in the real world anyway, there's no way to tell from just querying the archon.

[mikelangmayr]

Added a comment: changed is never reset to false, so it accumulates across all keys and APPLYCDS fires if any of them changed.

[mikelangmayr]

Simplified this: set_raw_config now always sends APPLYCDS after writing the RAW keys, no more changed flag. Per the manual APPLYCDS just parses and applies the CDS/deinterlacing settings from config memory, there's no documented cost to calling it when nothing changed, and as you pointed out there's no reliable way to know from the archon side whether a pending write still needs applying anyway.

[mikelangmayr]

@weatherhead99

• Mode switching already only rewrites values for keys already in the ACF; write_config_key errors on any key not already in the configmap.
• TAPLINE already works this way: hispecatc.acf predeclares TAPLINE0..TAPLINE4 plus TAPLINES=5, and each mode just rewrites TAPLINES and refills the preexisting slots.
• Same for parameters, constants, and timing script line count: changing the key set or line count requires a full load_acf/load_timing reparse, not an incremental add.