According to the article "Key Characteristics of Laundry Detergents" by Design for the Environment (DFE), an EPA partnership program, optical brighteners may be potentially toxic to humans and "Aminotriazine- or stilbene-based whiteners...may cause developmental and reproductive effects." However, the article also notes that further testing is needed to confirm these concerns.
Questions about skin reactions have been also raised and many studies have shown no sensitization; however, one study that did patch testing noted no sensitivty occurred unless UV light or "abrasive action" was used. So it seems that there is a potential for problems if you wear clothes impregnated with optical brighteners in natural sunlight.
Experiments with bacteria, fungi, and yeast have shown that optical brighteners are absorbed and transported to new cells. Great, just what we need. Mutant microorganisms