Don't bother with dark frames when using a DSLR [Deep Sky] Processing techniques · Stuart Taylor · ... · 64 · 2613 · 6

I think it is very camera specific, I can't comment too much on DSLR imaging and I only did that very briefly, with both of the cameras I used at that time it was not practicable to shoot darks seperately and I was limited to 30 seconds, so I turned on the "Long exposure noise reduction" function of both, which simply shoots a dark frame after each light, and subtracts is in camera. Without that the images were notably noisy.

Regards bias not being useful for CMOS sensors as a rule, I think that is too generalised. I moved from the DSLRs to an ASI294MC Pro, and bias frames were not useful due to the issues mentioned above, I used a range of fixed length exposure times and fixed length flats with matching dark flats and never shot any bias frames.

I have since replaced that camera with an ASI2600MC Pro and it is a different beast entirely, I shot 100 bias frames at 0 gain and 100 gain (The only two I use) and while I still shot a range of dark frames to make masters with no bias frame extraction, I shoot varying length flats and calibrate them with the master bias only. I did process one set of data with bias frames and no darks as I did not have a master dark for 100 gain at that time, and when I shot darks for that gain and reprocessed it later I was hard pressed to see a difference in the integration.

TL;DR:

Bias frames do work with at least some CMOS sensors and in almost every image I have ever produced, darks improved the image, that may or may not carry over to a DSLR and it might come down to how difficult it is to shoot darks to match your lights as to if it is worthwhile.

Calibration always improves an image, *if it is done correctly*. The reasons are obvious if you understand the concepts involved and process behind it (which is surprisingly simple). There's a reason why 'professional' astronomical observations are thoroughly calibrated, and why 99% of great images by experienced imagers (including much more than myself) are calibrated. The theory is the same regardless of sensor type (DSLR/CCD/CMOS, mono/color, etc). There are some specific questions that may differ.

The temperature question with uncooled cameras (i.e. DSLRs) is important. Ideally it should match perfectly, but +- 2C is fine.

The question that dark frames "waste" time is pointless - you should use twilight time or cloudy nights (with similar ambient/sensor temperature) or fridge, or whataver way you can to try to match the temperatures as well as possible. No clear sky time wasted.

The idea that DSLRs sensor temperatures are "random" is wrong. IF the ambient temperature is held constant, once the camera enters "in permanent regime" - after a few long squential exposure shots (such as nighttime), the temperature of the sensor should stay the same! Besides, some software (I know BYEOS and MagicLantern) report the internal temperature of the camera, which should make matching temperatures easier (although not reporting the sensor itself, it is close enough).

Now, people try to cut corners to simplify the workflow. That is a decision everyone should make. The difference might be minimal so that one may consider the "hassle" of dealing with calibration worthless. If you are happy with your images, that is fine.

Some points I want to keep noted:

• The best results always come from properly calibrated data. There are very good reasons to do dark subtraction and flat fielding.
• The need for perfect calibration increases as you try do dig in for the deepest details. That is, of course, best done using sufficient integration from dark skies.
• If calibration doesn't apparently make your images better, you might choose skip a few steps. But do so knowing what you are actually doing, and the consequences.
• If calibration degrades the quality of the image, you (either image acquisition or software) are doing something wrong. Track down the issue, and fix it.
• The quesion of how much you want to "fiddle" and "fuss" about things (including the small amount of work calibration means) is a personal one. But know that the best results comes from calibration, and beginners should learn it as a skill useful especially down the road.
• CMOS sensors with ampglow should never have dark scaling enabled, and usually don't work well with biases. In that case, use flat darks. When in doubt, use flat darks and skip biases/scaling.
• My experience tells that DSLRs do work well with biases. Flat darks work just as well.
• Dark temperature should match +- 2C (but possibly that deviation could be a bit more - experiment)
• Dark subtraction (and flat fielding) is NOT about removing random noise in time (that noise is actually increased by calibration, although a proper master dark/flat (with sufficient number of subframes) really minimize this contribution!). Calibration aims to remove *FIXED PATTERN noise* (DFPN and PRNU/vignetting).
• Fixed pattern can be averaged out by dithering + stacking. But calibration addresses the issue from the beginning, and the residuals can then be averaged out by dithering.

If you want to know more about it, I strongly recommend searching for Jon Rista's posts on Cloudy Nights, over several different threads (this is one of those questions that get asked again and again).

Personally, I don't like the approach of just stacking light frames (and some even advocate doing that with stretched files from Camera Raw/TIFF for instance) for serious DSO work (especially for faint stuff). This approach is used by some astrophotographers (Tony Hallas, Roger Clark). Easier, yes. Best possible results, no.

Bottom line. I calibrate my DSLR images. I use darks taken on the fridge. I have neved wasted clear sky time taking darks. I have had cases where calibration didn't work, usually from human error during acquision (especially stray lights) or incorrect software setup. If you want to skip calibration (for darks or for all calibration) know your results will be suboptimal, but might be good enough. And always experiment with your gear and learn about the intricacies behind the processes.

my 2cents

Best regards to all
Gabriel

Gabriel R. Santos (grsotnas):
my 2cents

Best regards to all
Gabriel

More like an entire dollar! Thank you :-)

It's a dumb argument, if I don't use darks hot pixels often destroy the image, even with dithering.

Nothing simple about manually aligning the black points of all your subs one by one for example, which is an extremely crucial step

If you (or the video poster or anyone else) are doing that manually, then you are using improper techniques on improper software.

Any serious preprocessing software (APP, PI, probably DSS, MaximDL, ImagesPlus.... you name it) has a form of NORMALIZATION, which does exacly that.

And more than just matching additive effects (which is what you would do with manual black point correction), these algorithms can be much more advanced in terms of statistics, local normalization, etc - that is to match frames from different nights or places, and make them statistically comparable so integration works best.

Or figuring out a light fall-off pattern that matches your gear

If you are doing that in CameraRaw or Photoshop and doing it on NON-LINEAR data (which is what Hallas does), then Calibration will never work as intended, integration will be suboptimal, and there is no point in discussing it further - mathematically, dark frame and flat fielding will not be performed correclty. It must be applied to raw LINEAR images.

Why would you figure out (model) a light falloff pattern when a FLAT FRAME DOES EXACLY THAT measuring your actual data, PERFECTLY, and accounting for dust motes and PRNU?

The subs you end up stacking are not "just the light frames", at all. They are color corrected, flattened, sharpened, denoised, normalized and have problematic pixels removed, chromatic aberration cancelled and highlights reconstructed.

You seem to be using Hallas/Clark workflow. It inverts the order of things, makes proper calibration mathematically impossible, and doesn't do preprocessing in a "standard way" (i.e. the standard workflow and Calibration frames amateur and professionals use). It can lead to good results. But not best ones. Depends ultimately on your goals, but also on your processing skill, and your targets (the fainter the harder). I stand by my arguments, and agree with what @Arun H. has added.

Again, I do not disencourage anyone from trying new techniques, and see what works well enough for you. But all the points of my previous post are valid. "Learn about the intricacies behind the processes" (includes calibration, registration, normalization, integration - pre processing, post processing, and more), and the best possible results come from correctly calibrated (and exposed, and framed... and carefully processed) data. And all that is a decision every imager should make, based on your goals and your enjoyment of the hobby. =)

Best regards, 
Gabriel

If you are doing that in CameraRaw or Photoshop and doing it on NON-LINEAR data (which is what Hallas does), then Calibration will never work as intended, integration will be suboptimal, and there is no point in discussing it further - mathematically, dark frame and flat fielding will not be performed correclty. It must be applied to raw LINEAR images.

Why would you figure out (model) a light falloff pattern when a FLAT FRAME DOES EXACLY THAT measuring your actual data, PERFECTLY, and accounting for dust motes and PRNU?

It is this point that really needs to be emphasized. Especially for complex lenses as opposed to optically simpler telescopes, the light fall off pattern will depend on focus location as the location of the optical components internal to the lens change. I've had enormous trouble with a 70-200 f/2.8L II lens trying to just use the lens profiles in Lightroom to correct for light fall off. Flats do it very, very well. And if you happen to have dust on your sensor and are trying to image a frame that has weak signal (empty space or IFN), it is an absolute nightmare to correct without flats, but simplicity itself if you have a well matched set of flats. It is now part of my routine to take flats after every session. Yes, a bit of work, but saves a ton of headache during post processing.

Finally, the point made by @Gabriel R. Santos (grsotnas) about linear data is important. Flat and dark correction convert, in a mathematical way, your raw data to purely linear form - your data are directly proportional to photons recorded. Without calibration with darks and flats, this is impossible.

Ok, you all got me thinking so I have today stacked the same batch of 366 lights of M57;

1. alone
2. with darks
3. with darks and flats
4. with darks, flats and biases

I have tried to then postprocess them in photoshop all exactly the same way. See if you can see any differences (I'll rely on your expert eyes)

Die Launische Diva:
All images have a very low black point

Sorry, I am very new to all this. Can you explain what you mean by low black point? Did I move the left hand arrow too far to the right in the levels adjust?

I do adjust the 'offset' in the Exposure panel too as it makes the sky darker. Maybe that is my error?

Gabriel R. Santos (grsotnas):

Nothing simple about manually aligning the black points of all your subs one by one for example, which is an extremely crucial step

If you (or the video poster or anyone else) are doing that manually, then you are using improper techniques on improper software.

Any serious preprocessing software (APP, PI, probably DSS, MaximDL, ImagesPlus.... you name it) has a form of NORMALIZATION, which does exacly that.

And more than just matching additive effects (which is what you would do with manual black point correction), these algorithms can be much more advanced in terms of statistics, local normalization, etc - that is to match frames from different nights or places, and make them statistically comparable so integration works best.

Or figuring out a light fall-off pattern that matches your gear

If you are doing that in CameraRaw or Photoshop and doing it on NON-LINEAR data (which is what Hallas does), then Calibration will never work as intended, integration will be suboptimal, and there is no point in discussing it further - mathematically, dark frame and flat fielding will not be performed correclty. It must be applied to raw LINEAR images.

Why would you figure out (model) a light falloff pattern when a FLAT FRAME DOES EXACLY THAT measuring your actual data, PERFECTLY, and accounting for dust motes and PRNU?

The subs you end up stacking are not "just the light frames", at all. They are color corrected, flattened, sharpened, denoised, normalized and have problematic pixels removed, chromatic aberration cancelled and highlights reconstructed.

You seem to be using Hallas/Clark workflow. It inverts the order of things, makes proper calibration mathematically impossible, and doesn't do preprocessing in a "standard way" (i.e. the standard workflow and Calibration frames amateur and professionals use). It can lead to good results. But not best ones. Depends ultimately on your goals, but also on your processing skill, and your targets (the fainter the harder). I stand by my arguments, and agree with what @Arun H. has added.

Again, I do not disencourage anyone from trying new techniques, and see what works well enough for you. But all the points of my previous post are valid. "Learn about the intricacies behind the processes" (includes calibration, registration, normalization, integration - pre processing, post processing, and more), and the best possible results come from correctly calibrated (and exposed, and framed... and carefully processed) data. And all that is a decision every imager should make, based on your goals and your enjoyment of the hobby. =)

Best regards, 
Gabriel

Like I said, it's a heated subject

To answer your three questions
1) the normalisation routines offered by the various integrators align the peaks of the histograms, not the black points. Thereby, they reduce chroma which is something I care about a lot. Also, when I set the black point I know what is supposed to look black, thereby starting the integration with a great deal less gradients. But that is immaterial. My argument was that the subs you are supposed to "just stack" are normalized, and much better than they'd been done by an automatic routine. They are not "plain lights".
2) I use the Roger Clark method when imaging with a DSLR but I don't mind throwing in in a dark or a flat if deemed appropriate. The core of the method for me is letting the raw converter do all the heavy lifting, it's not so much about whether darks are used or not. On the other hand, I use traditional calibration when imaging with the astrocam, simply because Rawtherapee does not support my astrocam. I very much wish it did.
3) I would correct the light fall off algorithmically if I didn't have dust motes (or if I could get rid of them with massive dithering) because a clean mathematical transformation is much simpler than dividing every set with a master flat that has its own noise and its own patterns if it hasn't been taken correctly, which in my experience is not the most unthinkable of "ifs" with a CMOS sensor and a setup that must be disassembled at the end of the imaging session which is usually much earlier than dusk. You should see me taking flats for the astrocam, capping the Newt with a 4kg laptop that I hold with my right hand while I am trying to start shooting by pointing the mouse of a second laptop to the "play" button using my left hand. I am right handed. It's a very miserable situation. 

So I am not religious about it, I have objectively (well, objectively in my subjective setup ) found that Rawtherapee with the DCP profile for my Nikon, a better debayering algorithm than VNG, carefully aligned black points, some noise reduction, some sharpening, some highlight reconstruction and all the other stuff simply yields superior results. By far. I cannot use this method with the astrocam so I don't. I am conscious that my results with the astrocam are suboptimal, despite taking dozens of darks, flats and dark flats at every session.

The way I see it, you are going to do extremely non linear stuff to your integrated image anyway. Why does it matter (mathematically) if you do them to the individual subs before integration, provided the individual subs are not stretched (i.e. do not have tone curve applied)? Is it such a crime if you increase SNR by 5% by applying a gentle noise reduction to the raw data? Why use a dark frame or dithering to correct hot pixels (in particular) when they are the one class of imperfection that can be detected algorithmically with very good precision? Etc. And what you are doing is actually more advanced, not less. It is by no means "just stacking the lights". I would not recommend the workflow to a beginner.

It is true that the full power of a raw converter probably falls apart when imaging a very faint object and it is completely useless with an unsupported sensor, but when imaging something visible with a stock DSLR there simply is no way you can do better color rendition, noise reduction and all the other stuff that has been optimised by professionals for your camera. Pixinsight in particular does not have AmaZE and the other extremely capable demosaicing algorithms  purely for licensing reasons, neither does it support DCP profiles. Those two things alone can very well make a tremendous difference, definitely larger than a dark frame, if you are imaging a reasonably visible target and care about colors, dynamic range and CA around stars  more than you do about hot pixels and whether noise reduction happens before stretching.  

BUT, for one more time, I feel I must repeat my core point which is: whether you believe that method to be superior or not (and it is I think established that the answer varies), it is not "stacking just the light frames".  The way I perceive it, what you are stacking are subs as close as possible to fully preprocessed while still remaining linear (linear, not raw). 

Cheers,
Dimitris

The best description I have seen of a good background in a dark area is that the image background should have a texture like an old school chalkboard, not just black.

I usually when processing crop the image so that there is no black space around the edges from dithering (assuming you dither) then examine the histogram in photoshop to see where the real data starts, I set the black point to a couple of levels below the first one that has at least one pixel after black space has been removed. Then you can darken the background somewhat using curves if you want to, which might suppress some faint detail but setting the black point higher eliminates it completely.

To calibrate the background after stacking, I set the black point to all 35's on the R, G and B. Not out back, but chalkboard like was stated.

Olaf Fritsche:
I might be crazy, but I choose the background the way I like it best. At least for me AP is fun. :-)

And of course you are entirely free to do so ! But, many novice astrophotographers set the background way too dark as they try to minimize the effects of light pollution, wrong or no flat-fielding and probably of some calibration mistakes. It is perfectly fine to do so if you wish to quickly produce an image for you and your friends to enjoy, but this strategy often leads to harsh transitions between the bright parts of the image and the sky, which is unnatural. Obtaining good flats and applying them is the first step in producing an image with smoother light pollution gradients which then are easier to remove with the proper software tools. This in turn results in less or no strange color casts and a smoother background to work with.

Stuart Taylor:
Interesting that you say it's important to get good flats. I am not sure mine are looking quite right (here is an example). I place white paper over the aperture and point the scope straight up at the daytime sky adjusting the exposure to get the histogram peak about half way along the x-axis.

Getting good flats and using them well is the alpha and omega of AP. The way you do it I don't think I would in a 1000 years. You need to get proper flats, either with a flat box, a flat-field panel or shooting dusk flats (or even better sky flats when is cloudy and you have evenly distributed LP). Bigger telescopes would go the latter way 100% of the times.  And even with the best of flats and the darkest skies on earth I still have to deal with gradients in the end. Moonlight is one of the major bugbears when dealing with flats. Well, unless  you only shoot with no moon in the sky.