Gain settings for Flats vs Lights [Deep Sky] Processing techniques · Willem Jan Drijfhout · ... · 45 · 1552 · 2

Rick Veregin:
I never got good consistent flats until I started using the same gain as the lights, and using temperature/time match darks and dark flats.

Tim Hawkes:
I am in the no camp  (basically because I have never detected a particular problem with flats as I make them now).  It is worth reading Robin Glover on the sharpcap forum on this topic.  His analysis appears entirely logical to me anyhow


My understanding is 

1) For flat frames you only need to measure the relative amount of light falling on each part of the sensor  so no need for flats to match the gain of lights

2) Thus it is important to record flats at a level where the response is linear (so not close to LH axis and neither above say 80% saturation?)

3) For artificial light sources the exposure time also needs to exceed that of any flicker

So generally the specific gain used for flats doesn't matter


But there are also some important camera-specific aspects to this question.  There is - for example - apparently a specific issue for example with 294 sensors and for example in ASI 294 MM and MC cameras wherein at certain gain levels some small number of the pixels fail to saturate whatever the exposure time.   That means for example that it is recommended not to make flats within a certain range of gain settings 120-200  (~ 8.7 dB I think) if you use these cameras.

Tim

Tim Hawkes:
It's a relative response because you subtract the dark and divide by the flat in calibration?

The issue with the 294 does appear to be  real Andrea  - that is hidden somewhere on the ZWO website and relates to the Sony chip  - I have only just become aware of it myself.  It may of course be that the actual effect of the problem on flats is subtle or not easy to notice - I don't know.   

It is very easy to test anyway.  Hook up the camera set it to gain 130 or whatever (into HCG mode) and increase exposure while observing the saturation response.  Apparently with our chip -- I have the same -  even  at exposures well 'above' saturation' a few percent of the pixels will still be showing on the graph at around 70-80%.

Willem Jan Drijfhout:
Thank you for your extensive response, Rick. And by the way you have an impressive portfolio of long exposure images!
Your first argument points in a similar direction as Andrea's, that the PRNU fixed pattern noise changes with gain. PRNU itself is a pixel-level difference in response to gain, which causes the fixed pattern of non-uniformity as a kind of fingerprint of the sensor. The question is then if the pattern/fingerprint is different at different gain levels?

The other point you bring up is about offset (or bias as you call it). So perhaps the idea of keeping gain constant is about keeping offset constant? Offset is meant to avoid any 0-pixels and essentially avoid dark clipping. This is important for bias/dark frames and should therefore be the same in any frame you calibrate with them. But once properly calibrated it is hard to see what further effect it can have. Especially on flat frames that typically have 25-50% illumination. On another note, with modern cameras it is not too difficult to keep offset the same within the normal range of gains used. I've tested this for both the ASI533MM and the QHY268c, and both have such clean signal that a fairly low similar offset for both typical gain settings avoids any clipping.

Going the safe route makes a lot of sense, but there will still be that itch of wanting to know why we do it. And for me it has practical consequences. To get to let's say 30% illumination, there are three variables to work with: light intensity, gain and exposure time. So far I used a flat cap that allowed automated fine adjustment of brightness per filter in software. As I'm preparing for a bigger OTA, the options for flat panels become less. And light intensity can only be manually adjusted in software or by adding additional diffusers/pieces paper etc. So that leaves only gain and exposure time to play with. And if gain is not an option, it's only exposure time which will likely lead to some very long exposures. Anyway, rattling here a bit. Nothing that can be overcome, it just got me into this rather fundamental question.

Rick Veregin:

Willem Jan Drijfhout:
Thank you for your extensive response, Rick. And by the way you have an impressive portfolio of long exposure images!
Your first argument points in a similar direction as Andrea's, that the PRNU fixed pattern noise changes with gain. PRNU itself is a pixel-level difference in response to gain, which causes the fixed pattern of non-uniformity as a kind of fingerprint of the sensor. The question is then if the pattern/fingerprint is different at different gain levels?

The other point you bring up is about offset (or bias as you call it). So perhaps the idea of keeping gain constant is about keeping offset constant? Offset is meant to avoid any 0-pixels and essentially avoid dark clipping. This is important for bias/dark frames and should therefore be the same in any frame you calibrate with them. But once properly calibrated it is hard to see what further effect it can have. Especially on flat frames that typically have 25-50% illumination. On another note, with modern cameras it is not too difficult to keep offset the same within the normal range of gains used. I've tested this for both the ASI533MM and the QHY268c, and both have such clean signal that a fairly low similar offset for both typical gain settings avoids any clipping.

Going the safe route makes a lot of sense, but there will still be that itch of wanting to know why we do it. And for me it has practical consequences. To get to let's say 30% illumination, there are three variables to work with: light intensity, gain and exposure time. So far I used a flat cap that allowed automated fine adjustment of brightness per filter in software. As I'm preparing for a bigger OTA, the options for flat panels become less. And light intensity can only be manually adjusted in software or by adding additional diffusers/pieces paper etc. So that leaves only gain and exposure time to play with. And if gain is not an option, it's only exposure time which will likely lead to some very long exposures. Anyway, rattling here a bit. Nothing that can be overcome, it just got me into this rather fundamental question.

Yes, I am saying the same thing as Andrea. The answer is it might be affected by gain, or better, there is no reason to expect the fixed noise must be identical.  Though possibly it might be good enough for your camera sensor and your images. 

Yes, it is a bias or offset signal added to all pixels, and results in a bias frame when you use a short exposure. Yes, if you set your offset high enough you can run with that for all gains, only downside is you lose a bit of dynamic range potentially, but generally the offset is small enough that it doesn't matter. This is in fact what I do now, I am using a higher offset at higher gain, so if I drop gain I leave the offset where it is. I assume other cameras are similar. This would yes remove my reservation about the different offset, but not the potential effect of pattern noise. Not sure what your question about calibration is. Correct calibration is:

CalibratedLight = (Light - Dark) / (Flat - FlatDark)

The offset/thermal dark noise in either case is not a light signal. Flats are used to map the non-uniform light response, so if you don't subtract the respective darks from the lights and flats you will not get the correct light signal correction. Some people get away with this if there is no vignetting, but I have an SCT which cuts into my APS-C field, if I don't do the darks absolutely right (light and dark match settings, while flat and flat dark match each other) then my field is not properly corrected when I stretch. As I mentioned I find gain matching light and flats works better too. My camera is very linear response over the range I use (SharpCap has a nice routine to semi-automatically work you through testing out your camera--you can determine linearity, read noise, etc), so exposure differences for flats do not affect the linearity compared to lights. As for noise, remember each flat has the same read noise as a light. The good thing as you point out is that the flats are divided (or multiplied) depending how your SW handles them, which means the S/N is what is important. And as you say the signal is strong. However, if you do hundreds of images and only 10 flats, the noise in the flats can contribute. Note as well fixed pattern noise is not averaged in flats, so you do not want the fixed pattern noise to be different in flats and lights, it will show up (dithering your lights helps this, as it moves the target relative to the flats, averaging out the pattern noise--but better not to have it to begin with).

I use an artists A3 flat panel from Amazon with my 9.25" SCT, probably this would be good up to 10 or 11" scope, and was only about $50 if I recall. It also has 3 light settings, so is adjustable. I set it at the lowest value for high gain then adjust exposure. Uniformity from it has never been an issue. My own flats are usually a fraction of a second at high gain. At low gain I turn up the light. Even running one hundred flats is done is less than a minute. The RASC robotic 16" scope uses just sky flats, works well, though each exposure may need to be slightly different unless you can take them all very quickly one after the other.

My advice always is integrate advice, be wary of absolute NO/YES statements with no reasoning, the theory is around if you look for it, and put it all together as best you can, and then test it yourself. Note measuring noise accurately is not that easy. So for example, take a good set of lights so you noise will be low. Then use just a few flats a few darks and change things up. Stretch all images the same exactly, then see what noise and flatness look like. Come to your own conclusions of what affects what for you. Taking extra flats and darks with varying conditions is not a lot of work to understand your system--and there are many cloudy nights to experiment. It is worth it, the unfortunate fact, that is not often talked about, is the longer your total exposure and the better the image data, the better your flats and darks need to be, and the more of them you need if you really want to get all you can out of that data.
Rick

Tim Hawkes:

Willem Jan Drijfhout:
to be seen how big of an impact this has on actual images.

Interesting.  It is probably possible to work through some numbers to calculate the potential for any impact?  Trouble is questions always beget more questions and rarely answers? :-) 

Firstly am just wondering about

1)  What is the amplitude of the difference signal  relative to the  expected error  level  ( ~  200 ADU ?)   

2)  Why is there a numerically overall net positive difference signal (rather than a signal about equally populated with black zero pixels as with positive pixels) at all ?  It  seems most likely that one image actually ended up just a bit more saturated than the other (say 22.5K and 22K  thus giving a mean signal of 500 etc) -  raising the question of whether the same or some different signal would arise from any other arbitrary close to starting balance  of the two signals?

Because of the problem of  trying to measure what is likely to be a small number from the difference between two large numbers  I am thinking that this is difficult.   But also to add that --if the difference signal is down at something like 0.5% amplitude of the 22k signal -  then not only is it an awful lot of work to measure but it would also have very little potential for impact  surely  --  i.e.   a +/- 1 %  variation in some adjacent pixels being beneath notice in most images  -  and anyway lost in the effect of dithering -  where dithering is essential in any case to lose the bigger impact of pattern effects  in the dark subtraction?



Tim

andrea tasselli:
which makes even the tiniest difference will crop up in the final integrated light