"Feather" sliders in the left part of the dialog window let you soften the mask edges.
Since this adjustment is utterly non-destructive, later on you will be able to restore the mask sharpness without any loss of quality.
- So, select a masked layer in the Layers pane.
- Open the "Masks" dialog.
- To blur edges of the pixel mask, move the "Feather" slider in the "Pixel Mask" panel.
- To soften the vector mask's edges, use the "Feather" slider in the "Vector Mask" area (screenshot).
- To feather the smart filter's mask, use the respective slider in the "Filter Mask" panel.
- If you need to restore the mask sharpness, open the "Masks" dialog once again and reset the "Feather" value to 100%.