The Senedd has been the assembly's home since 2006
Welsh Secretary Peter Hain has warned that any early referendum on giving the assembly full law-making powers would be defeated.
He said there was no consensus on a vote and could not see it happening over the four years of this assembly.
He spoke after retired diplomat Sir Emyr Jones Parry became chair of commission on a referendum by 2011.
The assembly government said holding a vote would depend on public support and Mr Hain was "entitled to his views".
A referendum within four years was part of the One Wales deal which led to the formation of the Labour-Plaid assembly government and First Minister Rhodri Morgan said last week that it would be "on or before the next (assembly) election".
But Mr Hain told the BBC's Politics Show Wales that there was not wide political support for a referendum "and I do not see that this assembly term, quite frankly".
Sir Emyr, the UK's former ambassador to the United Nations, was named last week as the man to lead the convention paving the way to a referendum.
His appointment has been criticised by Labour MP Don Touhig, who said the job should have gone to someone with a more up-to-date knowledge of Wales.
Mr Hain called Carmarthen-born Sir Emyr "ideal" for the job, but said some might be entitled to feel that his appointment came "out of the blue".
The Welsh secretary said he backed primary law-making powers for the assembly, but added: "I didn't take the Government of Wales Bill through, nor did MPs vote for it, to be bounced into an early referendum".
'Picking a fight'
In response, an assembly government spokesperson said: "We've always said that holding a referendum would depend on public support being there for further powers for the assembly.
"Peter Hain is entitled to his views, just like anyone else, about where public opinion will be on this during the remainder of this term."
Plaid MEP Jill Evans said the One Wales agreement "clearly states that a referendum on a full parliament will be held during this term".
She said Mr Hain was "obviously picking a fight with Rhodri Morgan and the One Wales government," and asked whether he was trying to increase his support among Labour MPs or "planning to be a road block against the policies of the assembly government".
Mr Hain also criticised Conservative proposals to allow only English MPs to vote on English matters in the wake of devolution.
"So we have an English nationalist party flying under the Conservative flag, a Welsh nationalist party (Plaid Cymru) and a Scottish Nationalist Party (SNP)," he said.
"Labour is the only party that believes in the unity of the United Kingdom."
Dai Davies, the independent MP for Blaenau Gwent, has called for a cut in the number of Welsh MPs from 40 to 22 to reflect the growing influence of the assembly.
Mr Davies said a huge amount of public money was being spent on MPs when "a significant amount of business, even now, has nothing to do with our constituencies or Wales in general. So we must be realistic in how we spend public money."
'Counter-productive and crazy'
But Mr Hain responded: "I cannot understand why we want fewer Welsh MPs of whatever party to fly the Welsh flag in parliament, to decide our Welsh budget, to decide our position on a whole series of crucial issues.
"The Conservatives, now the independent MP Dai Davies, [and] the nationalists, they all want to short-change Wales by having Wales's clout at Westminster reduced.
"That's a very counter-productive and crazy policy to pursue if you're interested in a strong Wales."
On another issue, the possibility that the assembly government might try to outlaw smacking of children, Mr Hain said criminalising parents who gave their child "a small tap" was very worrying.
The idea was ruled out last week by the UK government in both Wales and England, and Mr Hain said the idea that Wales could go ahead with it had "come out of nowhere".
He said First Minister Rhodri Morgan had not discussed it with him, Welsh Labour had no policy on the idea, and added: "People would need to consult and think this through".