Every now and then there is a dramatic TV show on where the good guys are ‘digitally enhancing’ a few pixels to create an ultra-sharp image of the face of a villain. The bad guy is recognized, he is caught, and the world is again safe. Of course we all know this is impossible. There were only a few pixels to work with, it can’t be done.
Right?
Well, it is indeed impossible if you only had those few pixels to work with, but if you have more than one image of the same target, either from slightly different viewpoints or taken over time with a slight offset in the images, it turns out it actually is possible. It is only possible though if the images are both under-sampled and you can determine the offset of the target with sub-pixel accuracy.
If the images are not under-sampled – like it is the case for most high resolution astrophotography images – the first frame would contain the same information as any of the other frames. Apart from a little bit of noise of course. Stacking the images would certainly increase the signal to noise ratio, and you can even reject frames that are too blurry or compensate for the movement a bit, but you will never be able to go beyond the diffraction limit of the optics. You will need a larger aperture to get more detail.
If you can’t determine the offset of the target in the image, then even if it did contain extra information, we wouldn’t know where to place it! Luckily, accurately determining offset in images is no problem at all for AutoStakkert!2. If you do have under-sampled images to play around with, the fun can begin. I found an interesting data-set of a tiny video containing several frames of a moving car. One of these images is shown above. It is enlarged to 300% to show the individual pixels it was made up from.
Now let’s use AutoStakkert!2 to stack twenty of these images from a small sequence where the car is moving through the field, and sharpen the results a little bit.
All of a suddon we can read the license plate, and even the brand at the back of the car! We actually cleaned up the image and enhanced it. Super Resolution does indeed give you super resolution.
AutoStakkert!2 uses an advanced technique called drizzling – officially known as Variable Pixel Linear Reconstruction – which was originally developed for the hubble telescope to achieve sharper results for undersampled images. Drizzling was applied to several tiny sections in the images to compensate for any image distortions. Combine this with an accurate estimation of the location of the features throughout the image, and you can end up with a lot more resolution than you started with even when the field of view is changing.
Apart from making Hubble images sharper, Super Resolution is also applied to telescopes actually peaking down to earth. Some might find it interesting to see an enemy tank or structure when it was hardly visible in a single image. More down to earth implications are to actually do what we did here: read license plates of speeding cars, or indeed to recognize the bad guys in a video of a robbery. Unless the bad guys wore masks of course.
To sum things up: Super Resolution is real. If you have just one image containing a few pixels there is little you can do. But if you have a lot of slightly different and under-sampled versions of those pixels, then you can significantly increase the resolution of your images! For planetary astrophotography this is hardly ever the case however. Sometimes drizzling can give sharper results for low focal length recordings: when imaging the Sun in good seeing conditions at low magnifications for example. For short exposures of deepsky targets at lower focal lengths there is a much bigger chance it will actually increase the effective resolution. For most planetary recordings there simply is little to gain by drizzling.
AutoStakkert!2 does not use the MAP-uHMT method shown in the image above. The MAP-uHMT technique was developed by Dr. Feng Li at the University of New South Wales. AutoStakkert!2 only produces raw stacks, and to correct for residual image blurring these stacks have been manually sharpened in Photoshop using the smart-sharpening tool. Better results can likely be obtained when using more advanced deconvolution methods to get rid of residual image blurring.
Magnify that death sphere!
http://www.youtube.com/watch?v=vHjuV7kRpFQ
Hi Emil,
Excellent in-depth article. I really enjoyed reading.
Thanks, Torsten
Hi Emil,
Quite interesting, to put it mildly. I think I can forget about Focus-magic now.
AS!2 does the job and FAST.
Hi Wim,
You would still need something to sharpen your images, as this is not included in AutoStakkert!2. When stacking many images that are slightly blurry to begin with – as is pretty much always the case in astrophotography – the results will inevitably be blurry themselves. And even if all the images were perfectly sharp, there would be something to gain by applying deconvolution methods to correct for the distortions introduced by the optical system itself. So please don’t get rid of any sharpening techniques just yet 😉
P.s. The reason image sharpening is not in AS!2, is that this itself is a very tricky subject to get just right. There is plenty of other software that gives you options for sharpening your images. In the future I might add some real sharpening routines in AS!2, but only if I’m satisfied with what it can do for planetary imaging.
0. no tutorial, help guide is troll nonsense
1. this software never open avi files, if yes
2. never get the same results in examples
3. i wasted 3 hours of try all possible variants
4. i think the programmer not published the 100% working program.
WHY?
This post was written to explain a little bit about super resolution, for people who already know quite a lot about astrophotography and typical image stacking and processing routines. In astrophotography it is quite common that the image stacks are further processed (wavelets, deconvolving, tools like that) in order to get the most detail out of them (because the stacks are inherently a little bit blurry still). I did this as well for the AS!2 images here, because it has no build in sharpening or deconvolution techniques.
The techniques I used for this image sharpening are nothing special though, in fact I could have used more advanced deconvolution techniques that would likely have given better results, but that really was outside the scope of this article. Just like how I did not tell how the input images of the car needed to be made such that AS!2 could open them properly (AS!2 can only handle a fairly specific set of video and image formats, those most typically used in planetary astrophotography).
The point of the article was simply to show that super resolution is real, and that drizzling – the technique invented for the hubble space telescope images, and implemented in the AS!2 software as well – do actually work. A secundary goal was to brag a bit about the image alignment quality in AS!2, especially considering this is just a (big) hobby project of mine.
You start counting with 0, I’m guessing you have been programming too often ;). I’ll start with 1 instead…
1: see above
2: see above
3: That’s too bad. I’ll give you a refund…
4: I think I don’t know any more funny things to reply with
Why? Because!
i have a video and i need to find the number of plates of the car who crash my car out from my home and left. if anybody can do i can pay for this
Sounds like a nice challenge. Send me an email and I’ll have a look for you.
Hello Emil,
Do you remember the parameters you used in AS!2 to achieve this fantastic result of the license plate?
Not really, but if I had to do it again it would be something like this:
You have to be very careful with the initial image stabilization phase, you basically need to stabilize the images on the car. If that is not done correctly, you will not get any good results. I also used ‘more accurate’ during this stage, and disabled all the frames where the car was basically too small to do accurate aligning, so I only kept the first 20 frames or so (it might be easier to do this before running it through AS!2 actually, maybe using VirtualDub. Manually disabling frames can be done in AS!2 using the space bar, just make sure you do this on the unordered frames: after analysing left click on the ‘Frames’ label left to the frame slider).
Then you need to experiment with the size and placement of the alignment points. Better start with just one small ap right on top of the license plate, and make some test stacks to see what works best. Stacking goes fast anyway.
After stacking you can do some quick post-processing in your favourite software to sharpen things up a bit.
You’ll see that it can be quite tricky to get the results, AS!2 is not specifically designed for stacking a handful of license plate images, but the drizzling technique still works if applied carefully. If you need help, let me know (send me an email).
Luckily for you it’s a kind of AstroPhotography even though it’s a car: Subaru is the Japanese name for the Pleiades constellation!
i need a video enhanced so i can get a license plate number please contact me and il email u the video please thank you
sincerely mike…..
I’m not doing that anymore. It takes a lot of time and effort, and the result is often not worth it.
Also, if you really want something like this done, you should:
– put in a bit more effort in writing proper sentences. They don’t have to be perfect, just a bit less like a 10 year old wrote them.
– and contact me (or someone else) directly instead of posting on a blog.
Great program! I really want to try myself and see its performance. Can you prepare some tutorials and share them with us? Especially the ones which you have given as example. Maybe recording your desktop while you are working is an easy way for you also. Please add some tutorials. Thank you again.