The demonstrators want Mr Najib to face charges over allegations that billions were looted from his brainchild investment fund, 1MDB.
They were undeterred by a police ban and the arrest of a number of activists ahead of the planned action.
Mr Najib has denied any wrongdoing and says he will not be cowed by protests.
In a statement on his blog, Mr Najib called Bersih - the electoral reformist group which organised the protest - "deceitful". He said the group had become a tool for opposition parties to unseat a democratically elected government.
But his accusations were dismissed by Bersih deputy chair Shahrul Aman Shaari, who told the crowds gathered at the National Mosque: "We are not here to bring down the country. We love this country. We are not here to tear down the government, we're here to strengthen it."
This is the second rally organised by Bersih, which means "clean" in the Malay language, to demand Mr Najib's resignation in 15 months, and comes after weeks of rising tension between the "yellow shirts" and pro-government "red shirts".