Whom are you talking to? is not used, largely because anyone formal enough to use whom will usually avoid ending a sentence with a preposition. (This is a highly controversial point, and I really don't want to restart the argument; I am merely making a point about common usage. My own view, for what it's worth, is that the practice is certainly not always wrong, but is confusing and ugly enough to make it worth advising learners to avoid it.)