Images read from camera in incorrect order

[Cody-G]

When acquiring images at nearly maximal frame rates with PCO cameras, sometimes images are written to disk out-of-order. This can be confirmed by examining the frame numbers encoded in the upper left corner of the images. Sometimes images are read out quite late. For instance, the frame order may go something like this: 1,2,4,5,6,3,7 (frame #3 is delayed).

This may be a bug in the PCO camera / SDK rather than in Imagine, I'm not sure yet. But in any case we may be able to fix this by changing the buffer mode or the number of buffers used by the camera (currently we just use 2 buffers, reading one out to the PC as the camera writes to the other).

[timholy]

To get started, it should also be straightforward to correct this after acquisition. It could be done inplace (without writing a new file), since the frames are of the same size.

While it could be a PCO bug, I wouldn't be surprised if it's an Imagine bug, since Imagine is multithreaded. One way to get a little more insight might be to allocate a vector at the beginning and write the frame number as each frame is grabbed to the vector; at the end of acquisition, dump the vector to a file and then compare the temporal order of the vector to the temporal order of the frames. There are of course ways that Imagine could get both of those wrong, but there are also certain types of bugs that might catch.

[Cody-G]

Sounds like a plan. I do have code in Julia to correct this in-place after the acquisition. I will post it somewhere whenever it's needed. For now I think I'm the only one to have encountered the issue.

[Cody-G]

I think I have a better solution for this, waiting for PCO to fix an (unrelated) problem before I merge

[Cody-G]

The problem was that the windows API function WaitForMultipleObjects prioritizes events based on the order in which they appear in the event list. When recording at high speeds, the two buffer events trigger almost simultaneously. It seems that sometimes windows was handling the high-priority event twice in a row before servicing the low-priority event. To address this in #30 we swap the order of the events handed to WaitForMultipleObjects every time we handle a frame, so that the most recent image is always matched with the highest-priority event.

[timholy]

You're becoming an expert on scheduling!