Clegg is determined to make the plan happen
The British Olympic Association is pressing ahead with plans to enter a Great Britain football team at future Games, despite opposition.
At a meeting on Thursday, it was agreed that British men's and women's teams will compete at the 2012 London Games.
However, neither the Welsh nor the Scottish Football Associations attended the meeting.
"We will be going ahead anyway," said BOA boss Simon Clegg. "It is the BOA who selects the Games team."
New director of performance Sir Clive Woodward said the BOA had the option of selecting players even if the two FAs fail to agree to the concept of a combined team.
"The Olympics is a very special occasion and if a player from any of the four countries wanted to play, I think it would be a shame if they were not allowed to participate," said Woodward.
Great Britain will play with one team but it is up to them how they do it
Fifa president Sepp Blatter
"If the football teams were put together with strong squads and a strong coach I think they would have a real medal chance."
The Welsh and Scottish FAs fear that involvement with a British Olympic team could threaten their separate status with the European and world football governing bodies.
"The Football Association of Wales will not undertake anything that would jeopardise its position as a separate nation within Fifa and Uefa," said FAW secretary David Collins when the idea was first mooted.
However, Fifa president Sepp Blatter has assured them this would not be the case.
"We have confirmed that the four British associations will not lose their rights and privileges acquired back in 1947," he said.
"Great Britain will play with one team but it is up to them how they do it. It can be a mixed team, it can be from just one of the home nations, whatever they want to do."
A British women's team will also compete at the 2008 Games in Beijing, should they qualify.
England would need to finish as one of the top three European sides at the 2007 World Cup in China. Norway, Sweden, Denmark and Germany are the other four European teams taking part.