Ambitious people shouldn’t mock the hard-work of these poor villagers. their unattractive (homely) joys and destiny were not known of. “All of you who are reading this, don’t give disrespectful smiles, this is the simple and plain annual work of these poor villagers”
Gray is giving a message that us readers should not make fun of these villagers, or think that any person or work is inferior to us.