The Duke asks his listener a rhetorical question: who would actually lower himself and bother to have an argument with the Duchess about her indiscriminate behavior?
He thinks the answer is "nobody."
We don’t think that there is much open and honest communication in this relationship!