Mr Selzer said increasing productivity was key to future UK growth
Economist Irwin Stelzer has called a Tory plan to set up an independent body "to hold the government to account" over the public finances "ridiculous".
Mr Stelzer, a Sunday Times columnist, said the party looked like they were saying to voters they did not trust themselves not to spend too much money.
He was speaking at a fringe meeting at the Tory conference in Birmingham.
The Office for Budget Responsibility would be a key part of the Tories proposed revamp of government finances.
It is designed to give the UK "the most credible and responsible fiscal policy framework in the world".
Philip Hammond, shadow chief secretary to the treasury, told the same Politeia fringe meeting that the intention was to "create a rod for our own backs".
He said policy would have to be constrained as a result - but that was necessary because as a nation the UK must live within its means.
Mr Hammond told the meeting that Prime Minister Gordon Brown had "squandered the legacy he inherited" and that the "massive failure" of policy meant that in the medium term the effects of the credit crunch could be more "grave" in the UK than in the US.
Mr Stelzer, a director of the US think-tank the Hudson Institute, said that the long-term key in the UK was to increase productivity - in particular by reducing the size of the welfare state and cutting taxes.
Both Mr Hammond and Mr Stelzer agreed that knee-jerk over-regulation of the financial sector would be a bad thing.
But both believed that there had to be some way of ensuring those who sold financial products had continuing accountability, or as they put it having "skin in the game".
This would stop people, for example, pocketing the fee for a mortgage even if it was defaulted on months later.
Richard Aitken Davies, president of the Association of Chartered Certified Accountants, said there had been bad behaviour in the financial markets and said "some senior people need to be made examples of" - as in the case of Enron where executives were sent to prison.