The effective use of effects
The techniques I've developed for creating icons are based almost exclusively on Photoshop's vector masks and layer effects. By avoiding working with pixel-based images, I can maintain much more flexibility as I tweak my designs and set up different variations of an icon.
Before Photoshop had vector masks, I searched for a tool that would let me work with Bezier curves while giving me a live preview at screen resolution. Illustrator and Freehand don't do that because they're designed for print, not screen. I know some icon designers use them, but I'm too fussy about the final pixel results. I eventually settled on Fireworks, which is what I used to make my Sprightly icons.
Then Photoshop came out with vector layers and layer effects, and I haven't hardly touched Fireworks since. I've developed a lot of techniques with those effects, and I thought I'd share them here.
One of the keys to experimenting with layer effects is to think about what the effect looks like rather than what its name is. For example, I came to use Outer and Inner Glow for outlines much more often than I use Stroke because they give you more control. Trying out different blending modes (Color Burn and so forth) is also useful when you're trying to represent a material that's a little more interesting than plain plastic.
The layer effects
As I mention in my icon design article, shadows should be applied with care. Simply adding a drop shadow to the entire icon can make it look flat instead of more three-dimensional.
Sometimes, especially when you're trying to create a more 3-dimensional effect, you need a shadow that doesn't exactly mimic the shape of another element. This is done by making a separate layer for the shadow, setting the shadow layer's Fill to 0%, and don't forget to uncheck the "Layer Knocks Out Drop Shadow" option.
When you're going for the effect of an object resting on a surface, rather than hanging on a wall, it helps the effect if the shadow is darker in the areas immediately around the object. You can do this with multiple shadow layers: one for the overall shape, and others to fine-tune the dark areas. Those additional shadow layers will usually have a larger spread to help them blend in.
In the example to the left, three layers are used to create the shadow: one for the overall shape; one to darken the edges because the object is translucent, lightening the center; and one around where the object rests on the surface. A gradient mask is also used to lighten the shadow as it gets farther from the object.
The drop shadow effect is also useful for creating a fuzzy or out-of-focus shape. This is one of those cases where you have to think beyond the name of the effect. This is just like the shadow-only layer trick, only you'll probably want to set the Opacity to 100% and the Distance to 0.
You can also use a Drop Shadow to create a line below, next to, or above an object by setting the Size to 0 and the Distance to 1. I mainly use this technique for divider lines when I'm making interface mock-ups.
In addition to the obvious purpose of making a shape look like and indentation or hole, Inner Shadow can also be used for a shadow-only or highlight-only beveling effect. This can be especially useful if you're already using the Bevel and Emboss effect on the layer. Remember you can change the angle to place the "shadow" on any side you need.
As mentioned above, I often use these for an outline instead of the Stroke effect, especially on smaller icon sizes where I need more control. This is where the Spread and Range settings are particularly useful.
Another useful technique with Inner Glow is to combine it with a linear gradient to sort of contain the gradient's coverage. Set the glow to the "background" color of the gradient, and it will keep the "foreground" color away from the edges. See the aqua ball example below.
Bevel & Emboss
Usually this effect is pretty straightforward. The main place where I've gotten fancy is with the curves when I need something like a bevel that's inset a little from the edge. I admit I haven't completely gotten the hang of tweaking curves, but I usually manage to get what I want out of them.
This isn't a very sophisticated effect, but it's useful for when you want to make different color variations of an icon. However, since you can only have one Color Overlay per layer, this is only good for one color variation. If you need more than that, use Color Fill or Hue/Saturation layers and group them to the shape layer (if it isn't already grouped to something else). Then you can easily switch colors by turning layers on and off.
You can get an initial gradient by using it as your layer's content. This effect gives you one more gradient, which is useful in the fairly common case where you have a linear gradient as a base and a radial gradient as a highlight (see the aqua ball example below).
A gradient overlay is also useful when you might be changing the base color of the layer, but still want a gradient for highlighting or shadowing. If you were to use a gradient layer for color and shading, you'd have to change all the colors of the gradient, which can be tedious. A color fill layer with a transparent gradient over it can be tweaked much more easily. Plus, it frees you to experiment with different transfer modes for the gradient overlay.
This is something I tend to use in interface mock-ups more than icons. Interface mock-ups are always straight and flat, while icons often have elements in perspective or at an angle. So when I need a patterned effect in an icon, I tend to recreate it with paths so I can transform it more freely.
To be honest, I haven't thought of a good use for this effect. I'm not even sure why they included it. Why this, and not some other filter effect?
The first trick I developed with gradient layers was before there was a Fill setting in the layers palette. In order to achieve the effect of setting Fill to 0%, I made a gradient that was completely transparent. Note the difference between Fill and Opacity: Opacity affects the transparency of effects, and Fill doesn't.
It's not often you see "affects" and "effects" in the same sentence, is it?
Getting more effects
One of the limitations of layer effects is that you can only use each effect once per layer. You also can't control the order in which the effects are drawn. You can get around this by duplicating the layer with 0% Fill. Since you now have another layer with exactly the same shape, you get another set of layer effects. The problem, of course, comes when you need to change the shape, and you have to make sure your duplicate layer is synchronized.
Getting around the gradient scale limit
You can't set a gradient's scale to more than 150%, and I've had several cases where this wasn't enough. There is another way to make the gradient bigger, though: make your layer bigger. With a vector mask, for example, you can add paths outside the canvas boundaries.
Anatomy of Paper Folder
Several subtle effects went into creating the basic Paper Folder icon. I sometimes forget what some of them were for and have to turn them off and on to remind myself.
The white tab portion is a color fill layer with a drop shadow, an outer glow for outlining, and a subtle bevel and gradient applied.
The base of the front of the folder is another color fill layer with a similar shadow, outer glow, and bevel, plus a gradient overlay to create the light band across the middle. This has several Hue/Saturation layers grouped to it to create color variations, taking advantage of the fact that the gradient overlay gets drawn on top of the grouped layers.
A duplicate layer is used to get a second gradient overlay, which is a radial gradient (set to color burn) to darken the corners.
To give the appearance of a curve at the bottom, there is a gradient layer covering about the bottom fifth of the folder.
The crease is a vector layer created with the line tool. It's a black color fill layer set to 9% Opacity, and a white drop shadow completes the beveling effect.
Anatomy of an aqua ball
The base layer is a three-color gradient. Alternately, as mentioned above, it could be better to start with a Color Fill layer and use gradients to lighten and darken it.
A radial gradient overlay adds to the highlight at the bottom.
Using the duplicate layer technique, two inner shadows add some depth. One at the top deepens the shadowing with a color burn, and one at the bottom gives a better shape to the highlight by bringing the background color back in.
Finally, a slightly narrowed oval with a white-to-transparent gradient adds the shiny highlight.
My Photoshop wish list
These are in rough order of importance to me. If any of my top three items had been in Photoshop CS, I would have upgraded. But since I don't mess around that much with digital photographs, I might as well stick with version 7.
More flexibility in layer effect instances
Photoshop currently has a very limited system for choosing effects: each effect can be turned on or off. You can't have more than one instance of an effect per layer, and you can't reorder the effects. It would be better if you were given a menu of effects to add to your layer, so that you could rearrange and duplicate them at will.
Finer control over values
This mainly applies to settings that are pixel values, like Offset or Size. I often want to set those to a fractional value like 1.5, but Photoshop won't allow it. I would be surprised if there's actually a technical reason why this can't be done. Also, gradients tend to rotate in inordinately large increments on small layers.
Deeper hierarchies of sets and groups
Layer sets and groups can only go one level deep, and I very often find myself wishing for more. This is particularly true with an icon set like Paper Folder, which is all one Photoshop document. One layer set contains the base folder icon, and every badge is in its own set. In some cases I'd like to organize a badge's layers into sets, but you can't have sets inside sets. The same goes for groups.
Change the default light angle
Photoshop defaults to 120° for the global light angle; this ought to be changeable in the Preferences window somewhere. I almost always work with the Mac OS X appearance standards, which means a light angle of 90°. Every time I make a new document I have to change it. I forget to sometimes, especially when I duplicate a layer or layer set into a new document, and I assume that all settings will carry over.
Raise the gradient scale limit
As mentioned above, the limit is 150%. I guess they just wanted to have a reasonable range for the slider, but there's no real reason to reject higher values in the text field.
Layer effects are a good thing. I'd have a much harder time creating realistic-looking icons without them. There are limitations, but I've learned to deal with most of them.
My ideal icon creation tool would have layer effects but, of course, without any of the limitations I've mentioned. I would also want a way to link the visibility of certain layers (including mutual exclusion) to facilitate the creation of variations on a design. Finally, I would like something to help coordinate creating and tweaking the shapes and pieces for the same design at different sizes. These features aren't something I really expect to see in Photoshop, or any other commercial application I can think of; they have other priorities. If I get enough free time I might try doing it myself. We'll see.