"Vector masks" hide certain area of layers they belongs to. Due to vector nature such masks have sharp clear edges and can be resized without loss of quality.
Photoshop Elements naturally supports this kind of mask for "shape" layers.
"Elements+" allows you to add a vector mask to raster, adjustment, and text layers as well.
Since Photoshop Elements lacks the Pen and other advanced vector tools, you can only create a vector mask from a shape or path.
How to make a vector mask from a shape:
NB Photoshop Elements doesn't allow you to draw on an existing vector mask. To create a mask from several shapes, you draw those on the same shape layer (using the "Add to shape area +" mode) and then convert it to a vector mask.
- Use the Custom Shape, Oval, or any other vector drawing tool to create a shape layer in front of any "maskable" layer*.
*You can't mask the Background layer. Also, make sure the target layer hasn't an existing vector mask yet. If it has a raster (ordinary) mask, the two masks may co-exist.
- In the Effects palette, double-click the "Vector Mask from Shape" pictogram.
- If necessary, adjust the mask size and position (more about editing vector masks).
- If necessary, invert the vector mask as explained here.