What we worry about is... that essentially HMRC will be acting as judge and jury," Mr Garnier told the BBC.
HMRC recently explained how the system will work.
It will only target those who have long-term debts and have received at least four demands for payment and will ensure that at least £5,000 is left in total across all debtor's accounts, including savings accounts, after the unpaid tax is seized.
HMRC will freeze the amount owed in accounts for 14 days to allow time for a debtor to pay before the money is seized.
The Low Incomes Tax Reform Group has called on HMRC to give more concrete assurances about the right to appeal against any seizure.
But the ACCA accountancy body, which after the Budget described the plans as "seriously draconian", now calls them "less fearsome than first thought" after more detail was published.
"On paper, the safeguards look relatively robust, and the reality is it is unlikely that anyone will be left penniless," said Chas Roy-Chowdhury, head of taxation at the ACCA..
The plans are now going through a consultation process. If approved by Parliament, they will take effect in 2015-16.