Smarter Sharpen

This article will introduce a new approach for sharpening with the Smart Sharpen filter, which does not use masks or selections and minimises excessive noise amplification and distracting halos. The method outlined below makes use of a 25+ year old image processing technique that may be unknown to many Photoshop users, in order to facilitate the application and improve the results of the Smart Sharpen filter. Although the sharpening properties of this approach are best suited for acquisition/capture and monitor viewing sharpening, it is also possible to exaggerate the effect to produce wider, less subtle halos which are suitable for output sharpening for halftone or stochastic based print.

Introduction

Sharpening is one of the most popular and subjective topics in modern digital image processing, with many people developing a wide range of competing sharpening workflows; actions and other tools. Sharpening results and the various options that can be employed in a sharpening workflow are often discussed and debated with great enthusiasm in online forums and email lists. In Photoshop CS2, Adobe introduced the "Smart Sharpen" filter, which was a major addition to the existing sharpening options which had not seen any updates with new version releases over the years. Smart Sharpen uses an undisclosed "Deconvolution" algorithm to remove blur in a mathematical attempt to restore focus to an image, rather than simply adding edge contrast to blurred pixels to simulate focus.

Smart Sharpen was initially met with interest, however many Photoshop users reverted to using their older, trusted Unsharp Mask, Band Pass or Difference of Gaussians based sharpening techniques. The lack of a "Threshold" or other noise avoidance option in Smart Sharpen is often viewed from two extreme opposite viewpoints from users familiar with traditional sharpening methods. One camp considers Smart Sharpen to be smart as it does not have a Threshold type command, while the other camp views the lack of a Threshold type command to be a serious design flaw - as noise and sharp, fine detail are often overly amplified. Even with further modifications such as masks or various blending options, Smart Sharpen may not be employed to it's full potential when used with common editing approaches.

Smart Sharpen - Basics

Unlike the Unsharp Mask filter, Smart Sharpen requires viewing at 100% (1:1 pixel) magnification. Giving the Smart Sharpen filter a cursory view, there are familiar Amount and Radius sliders, without a Threshold option. There is the added choice of three different blur types, with optional tonal blending options similar to those found in the Shadow/Highlight command. With simple application, one may conclude that results are the same as Unsharp Mask. This is true when Smart Sharpen is set to Basic mode, Gaussian Blur with the "More Accurate" option disabled. Results are identical to USM of the same values with zero Threshold.

When the More Accurate checkbox is enabled, the Smart Sharpen filter becomes smart, using a processor intensive iterative deconvolution process to remove blur to improve acutance. Of note, when run in More Accurate mode, visible artifacts may be introduced with some images when the filter is applied to 8 bpc data. It is strongly recommended that filtering be performed in 16 bpc mode, even if the image was in 8 bpc mode prior to running Smart Sharpen (even more so if the image is grayscale or has had JPEG compression applied).

Deconvolution and the Point Spread Function (PSF)

The Point Spread Function describes the convolution (blur) in the image and is critical to successful deconvolution. The closer that the PSF describes the image blur, the more accurate the final result. In astronomy, a blurred star or other point light source provides an ideal measure of the image blur, however in regular photography it may be much harder to assess the image blur. Smart Sharpen offers three simple blur types with fixed parameters: Gaussian Blur, Lens Blur and directional Motion Blur. 

Gaussian Blur is the "Swiss Army Knife" of the Smart Sharpen filter. Simple blur in natural images will likely be different to the Gaussian weighting, however the Gaussian function can often be successfully used as a generic substitute with the knowledge that better results could be obtained with a more accurate PSF description. 

Lens Blur differs from Gaussian Blur and only provides accurate results when the image blur matches the Lens Blur PSF description. Once the appropriate Amount and Radius settings are found, Lens Blur results should be compared against the Gaussian Blur results. One may find that the Gaussian PSF will provide more pleasing results, even if the blur is truly lens related. 

Motion Blur removal is controlled via an interface similar to the Blur/Motion Blur filter, with a choice of directional blur angle. Selecting the correct blur angle is critical, otherwise noticeable artifacts may be generated with little appreciable restoration of focus.

Smart Sharpen - Review

• View the image at 100% magnification
• Process in 16 bpc mode where possible
• Process with the More Accurate option enabled
• Accurate blur description is critical to the deconvolution process
• Radius - start slightly smaller than the estimated blur radius and adjust higher in small steps (0.1, 0.2)
• Amount - a larger radius blur requires minor amount values, while a smaller radius blur can accommodate high/maximum values

Smarter Sharpen Technique - Introduction

As previously noted, without a Threshold type command, Smart Sharpen using the More Accurate option can be very unforgiving when it comes to image noise, lossy compression damage and high spatial frequency detail. When the More Accurate option is disabled, results are the same as USM, which defeats the purpose of using Smart Sharpen over traditional methods. Traditional approaches to using the Smart Sharpen filter often process both low and high frequencies using the same settings. I believe that successful use of the Smart Sharpen filter requires the targeting of various ranges of spatial frequency, with different Amount and Radius values appropriate to each frequency range.

In the spirit of the Adobe naming convention, I have named this technique Smarter Sharpen. More descriptive terms may include "Bi-Frequency Decomposition & Deconvolution" or "Iterative Multi-Scale Focus Restoration". To my knowledge, this method has not been publicly proposed for dedicated use with the Smart Sharpen filter prior to the publication of this article in December 2008. 

The Smarter Sharpen technique targets the various image spatial frequencies by "decomposing" or splitting the original image data into a layer stack of two images, with the low frequency (Gaussian) data in the lower layer and the high frequency (Laplacian) data in the upper layer. This bi-frequency decomposition forms part of a classic image processing technique known as "Gaussian & Laplacian Pyramid Processing" (however the Smarter Sharpen technique does not use a decreasing resolution pyramid structure). Once the image has been decomposed into separate high and low frequency layers, Smart Sharpening can be applied with greater success than when processing regular image data. 

Smarter Sharpen, Basic Technique - Gaussian & Laplacian Decomposition

Stage 1: Bi-Frequency Decomposition

Creating the Gaussian and Laplacian bi-frequency data layers is a fairly simple process. When performed in 16 bpc mode, the decomposition is lossless, resulting in no change to the original image when it is recombined. This is critical, as even the "negligible" 0.5 standard deviation error in 8 bpc mode processing is enough to generate artifacts when iterative decomposition is performed. 

Presuming that one is working on a flattened duplicate image or history snapshot in 16 bpc mode:

1. Duplicate the background image into a second layer

2. Target the original lower background image and Gaussian blur (example, 2 pixel radius)

3. Target the upper layer containing the duplicated original data

4. Apply Image command: source background layer, subtract blending mode, 100% opacity, offset 128, scale 2

5. Change the upper layer to Linear Light blending mode retaining the default 100% opacity

The two layer set of high and low band passed frequencies can be flattened and compared in difference blending mode to the original image at this point to verify that this technique is lossless. The (refreshed) Histogram statistics or Equalize command can be used to assess any potential errors in the frequency splitting process. 

Note: Step 4 does not create errors or artifacts, unlike High Pass filtering (2 pixel radius, blended at 50% opacity, Linear Light blending mode). Conclusion - Although similar, the Photoshop High Pass filter is not the true inverse of Gaussian Blur. High Pass filtering should be avoided due to the high standard deviation errors affecting high frequency detail.

Stage 2: Low Frequency Deconvolution & Supplemental Sharpening

In a basic single decomposition Smarter Sharpen workflow, the low frequency data layer is deconvolved with the Smart Sharpen filter using the Gaussian + More Accurate option, with appropriate Amount and Radius settings for the image. If the low frequency layer was created with a 2 pixel radius blur, then an appropriate initial radius for the Smart Sharpen Gaussian removal filter could be around 1.7 pixels. Next, gradually increase the radius value by steps of 0.1 or 0.2 pixels until the low frequency layer "pops" into better focus, without amplifying noise or adding objectionable halos. For a 2 pixel Gaussian Blur, the radius "sweet spot" setting may be around 1.8 to 2.2 pixels (dependent on the sharpness and noise levels of the original image). This is a general guide only, depending on image variables one may need to select lower or higher values. When in doubt, a lower Radius and Amount setting may be appropriate.

The following options further enhance the image sharpness over the initial application of the Smart Sharpen filter to the low frequency data.

Option 1: After Smart Sharpening, with the low frequency (LF) layer as the target layer, the Apply Image command is used to blend the high frequency data into the low frequency layer. Appropriate blending modes include: Soft Light, Overlay, Hard Light, Vivid Light and Linear Light using an appropriate reduced opacity level to further control intensity.

Option 2: Finally, the high frequency (HF) data layer is targeted and sharpened. As the HF data has very different characteristics, the sharpening radius will require a smaller value with an appropriate intensity for the image content. Simple Unsharp Mask filtering may be used in order to take advantage of the Threshold command, however one may also choose to use Smart Sharpen or minor noise reduction and High Pass filtering, using the fade command to a suitable blending mode and opacity. 

Stage 3: Flattening and Blending

The final stage is to "collapse" or Flatten the sharpened low and high frequency layers into a regular composite image. The sharpened layer is then blended over the original colour image in Luminosity layer blending mode. Opacity may be globally reduced, or the Layer Option Blend If sliders and or layer masks may be incorporated to further control the blended results of the Smarter Sharpen technique.

Smarter Sharpen, Advanced Technique - Iterative Multi-scale Deconvolution

The basic Smarter Sharpen technique makes use of a single decomposition to the user defined Gaussian low frequency level. Taking the technique to the next obvious step, one may perform multiple serial decompositions from coarser to finer Gaussian Blur levels (or vice versa). Appropriate subtle deconvolution sharpening of the LF data is performed at each Gaussian level, with HF data at each level mostly being ignored. 

For example, one may choose to use a five pass approach, using Gaussian blur steps of 3, 2, 1, 0.6 and 0.3 pixel radius. Low frequency sharpening results from each manual deconvolution iteration are compounded to some extent, therefore a higher blur requires lesser amount settings, while a smaller blur can use higher intensities. Appropriate Radius selection at each LF level is critical. Sharpening of the HF data should commonly be performed in the later finer scale decompositions, in order to reduce the influence of HF data on the next finer scale level low frequency decomposition deconvolution.

Smarter Sharpen - Processing Variables

Colour Modes & Processing Speed

Smarter Sharpen can be used to process colour data in RGB, Lab and CMYK colour modes, or single channel mode data such as Grayscale mode. Using channel blends in lieu of layer blends, Multichannel mode images can also be directly processed using this technique. As this technique relies on the accuracy of 16 bpc processing, performance may become an issue. Additionally, the slower More Accurate Smart Sharpen option compounds the need for high bit data processing which also adds to the processing time. 

One may speed Smarter Sharpening of RGB images by only processing the single RGB Luminosity "channel" data instead of all three channels. Unlike the lossy conversion to Lab mode for L* channel processing, the extraction of the RGB Luminosity data is a lossless move. To create this Luminosity data, simply fill a duped RGB image with a flat neutral white/black/gray tone in Saturation or Colour blending mode. Next, target a single channel and use the mode/grayscale command (do not use the Convert to Profile command in this case). Finally one manually assigns a grayscale ICC profile of the same gamma as the RGB original. 

Once this Grayscale mode faux RGB Luminosity "channel" has been sharpened using the Smarter Sharpen technique, it can be blended into the original RGB image in Luminosity blending mode. When this technique is correctly performed, the original RGB image will not change in overall luminosity (even in 8 bpc mode, Luminosity extraction and blending is a lossless process). The net result should be very similar to RGB mode sharpening, using only one third of the resources as regular RGB processing.

Linear Deconvolution

Linear gamma sharpening may be preferred for some image content. The simplest way to access linear RGB processing in Photoshop is to convert to 32 bpc mode. As linear gamma involves major gamma transforms from/to standard 1.8 or 2.2 gamma RGB working spaces, this move is not recommended for underexposed or noisy originals, which may suffer visual degradation from the conversion in and out of linear 32 bpc mode. The results of linear sharpening are similar to standard gamma sharpening, however linear processing results in less obvious light halos, with more obvious dark halos. 

Note: when converting from 32 bpc to a lower bit depth, the image will be converted to the RGB working space instead of the original ICC profile of the document (this is by design, as Photoshop can't keep track of the document profile). One should either change their RGB working space in Colour Settings to match the image, or convert the image to working RGB.

 

Smarter Sharpen Technique - Review

Deconvolution requires an accurate Point Spread Function description to provide pleasing results. If the PSF is incorrect, results may be poor or unwanted artifacts may be introduced. The Smart Sharpen filter provides three simple PSF choices, with lens blur and motion blur added to the traditional USM (Gaussian blur) based edge sharpening. A PSF can be idealised or based on an actual real world image sample. As there is no way to sample and use the unique Lens PSF description in the simple Smart Sharpen deconvolution filter, results may be less than satisfactory.

The Smarter Sharpen technique uses an idealised Gaussian Blur PSF, which is not a part of a natural image. By decomposing an image into separate low and high frequency data, one can then deconvolve the known Gaussian low frequency data with the matching idealised Gaussian PSF. This idealised PSF will often result in more accurate deconvolution than when using the unknown Lens Blur PSF, which likely differs from the blur found in the image. A variation of this technique would be to decompose the image into Lens Blur based bi-frequency layers, instead of using the simpler Gaussian Blur command. Lens Blur offers many variables, therefore I have not had the time to explore the different combinations to create optimal results with the Lens Blur PSF in Smart Sharpen.

Decomposing an image into separate Gaussian low frequency and high frequency layers provides targeted processing, allowing one to take into account the separate and unique properties and requirements of each opposing image frequency. This approach reduces noise amplification or distracting halos, while providing the appropriate selection of Radius and Amount for the targeted image frequency. Smarter Sharpen in it's simplest form uses a single bi-frequency decomposition with deconvolution of the LF data, additionally incorporating HF data blending and sharpening. Optionally, one may perform multiple decomposition and sharpening steps at various blur scales in order to fine tune additional frequency ranges.

Conclusion

The Smart Sharpen filter is ignored by many Photoshop users, as processing multiple frequencies using a single "one size fits all" radius and amount setting often provides unpredictable and/or unpleasant results. Using the Smarter Sharpen technique, deconvolution results in predictable sharpening at multiple spatial frequencies while minimising noise amplification and sharpening halos. It is my opinion that the Smarter Sharpen technique is a significant step forward in using the Smart Sharpen filter in Photoshop.

Action Download

Download the Smarter Sharpen version 1.0 action for Photoshop CS2 or later.

The following image swaps illustrate the differences between Unsharp Mask, Smart Sharpen and Smarter Sharpen on low frequency data:

(Please wait for each each image to load)

1. Original image
2. Original image 1, Smart Sharpen: 500 amount, 1 pixel radius, more accurate enabled
3. Gaussian Blur, 1 pixel radius (images 4-6 use this blurred image)
4. Unsharp Mask: 500 amount, 1 pixel radius, zero threshold
5. Smart Sharpen: 500 amount, 1 pixel radius, more accurate disabled
6. Smart Sharpen: 500 amount, 1 pixel radius, more accurate enabled
7. Original image 1 HF data recombined with the blurred + deconvolved LF data from image 6

Notes:
Image 2 = Result of running Smart Sharpen on regular image data
Image 6 = Result of running Smart Sharpen on Gaussian Blurred data, compare to image 1
Image 7 = Intermediate stage of the Smarter Sharpen workflow, compare to images 1 and 2