For me, the bottom line is "all things in moderation." If you like milk, try using low-fat or nonfat varieties. Whole milk has about 3 percent fat. Moreover, much of that fat is "bad fat" -- saturated fat. Skim milk has essentially no fat. Most stores carry both 2 percent and 1 percent fat milk, as well. So you have four choices.