Minerals have a crystal structure, yet, crystals are relatively rare. What are the reasons for this?

What are relatively rare are crystals of a size visible to the naked eye, and also showing most of the faces that reveal the internal symmetry of their atomic pattern.

Explanation:

Being crystalline, i.e. having a regularly repeated three-dimensional atomic pattern, does not mean that a mineral necessarily formed under conditions where it could nucleate (i.e. assemble as the tiny cluster of atoms that is the “seed” of a single crystal) and keep growing large flat faces until a regular shape becomes visible to the observer.

To a crystallographer who can seek proof of internal atomic order by X-ray diffraction, the actual size of a solid made of highly ordered matter is irrelevant. Specific techniques (variants of X-ray diffraction methods, or polarizing microscopy) can reveal that a solid material is made of a single crystal (i.e. a uniform atomic pattern is repeated in the same orientation anywhere throughout the solid) or consists of many crystals (the same pattern occurs, but it is oriented differently in what are considered individual crystals regardless of their individual shape or size).

For precision, a crystallographer or a mineralogist will use terms such as “monocrystalline” (the atomic pattern has a single orientation throughout the entire specimen, regardless of shape and size) and “polycrystalline” (the specimen is an aggregate, or collection, of “domains” or “grains” in which the atomic pattern is in an orientation different from its neighbours).

A perfect single crystal of quartz, broken in several chunks, doesn’t lose its internal atomic pattern, only its external “habit” (the overall shape imparted by the flat faces that grew, layer by layer, along directions controlled by the rate of addition to the atomic pattern). Each individual broken chip of quartz is considered “monocrystalline” by the mineralogist, even if none ofo them is the whole original crystal.

Most igneous and metamorphic rocks are polycrystalline, i.e. entirely made of crystals, often tightly packed and interlocked. You may discern individual grains mostly when light reflects off surfaces exposed by breaking along preferred directions within some minerals, or because grains from different minerals contrast in colour or luster. Few of the grains will have a regular geometric shape, despite each one being a single crystal. In the case of an igneous rock, some of the well-formed crystal are typically minerals who grew early from the still-liquid magma. Most of the other minerals simply filled the remaining space. If an igneous magma was “gassy” or “watery”, those volatiles may have remained trapped in the last stages of crystallization and formed late pockets in which a few crystals of exceptional quality grew from the remaining dilute magma and had the space needed to fully develop perfect faces. In many rocks, it is later fractures that provided an “open space” in which crystals could grow larger and with well-developed faces from hydrothermal fluids (overheated ion-rich waters), for the future delight of collectors.