The purpose of Layer Masking is to try to separate some parts of a photograph from others, so as to be able to work on them independently. In astrophotography usually stars from background or background from nebulosity.
With the Layers active and on screen (it should show one layer, the image we're working on) which will be on the screen, click on it to remind PS that you are working on it, and click Control A, Control C and Control V. This will select All (the image), Copy it and place it on the image, so you now have two layers with the same image on each. Look then at Layers and you'll see that you have the two layers, both visible, but you can't tell as they are identical. The upper one is active (blue).
Click on it with the Alt. key held down. This will bring the Layer Mask onto the main screen, it will be completely blank. As you still have the original image on the clipboard, click Control V and the image will be put onto the Layer Mask on the screen. It will be in monochrome though, showing that it is a Layer Mask.
The general practice is to use Levels to darken the black and lighten the white to almost completely black and white. The basic rule is Black Protects. With this mask in place you will be able to work on the image knowing that the areas that are black will not be affected by whatever you do, your work will only affect the white areas. This is easy enough in the case of photographs of stars or clusters where there is no nebulosity but much more difficult when faint nebulosity and noise compete.
You should have at this point a blank white background and fully black stars, use Filter/Blur/Gaussian blur to slightly blur the edges of the stars, somewhere near 1 or so pixels should do the trick.
Now that you have a satisfactory Layer Mask, and practice is the only way to be sure of that, you must now click on the original which is to the left of the layer and you'll note the colour in the image is visible, because you're now working on the real image rather than the mask. Here you can use many tools to work on the unprotected parts of your image. If it is stars, as shown in the photograph above, try increasing colour by Hue and Saturation in full RGB or by using the colours individually. You can sharpen the stars without adding noise to the background, the usual result if sharpening the whole image. You can use several tools if you wish, to work on the unprotected parts.
When done flatten the layers.
If this step has been to work on the stars rather than the background for instance, you can now repeat the process of setting up the Layer Mask again, but this time you should Invert the image to work on the background so the stars appear black (protected) and the background is white (unprotected). Again use Levels to a far degree almost fully white and fully black, and proceed as before, by clicking on the original, to the left, and using tools such as Gaussian Blur or Dust and Scratches to smooth the unprotected part, the background, of the image. As always smoothing of the background should be done with care. You may find that you have to re-sharpen the stars a touch if you've used the smoothing to excess. Then flatten as before.
