Detainees were prevented from talking to the media
Journalists were given limited access to the Dungavel centre near Strathaven, in Lanarkshire, on Monday.
The unit has been at the centre of controversy in recent months over the conditions and the length of time people are being held.
However, when some detainees indicated that they wanted to tell their side of the story they were prevented from speaking to BBC Scotland.
Robina Qureshi, from the refugee help group Positive Action in Housing, said that they had been "gagged from speaking directly to the media about their plight".
Centre bosses said allowing detainees to talk to journalists would breach confidentiality.
And Scotland Office Minister George Foulkes said that those being held at the centre did have access to the media to raise any accusations.
Mr Foulkes also said he was satisfied after making several visits that detainees were being treated humanely and with respect for their cultural and religious needs.
'Extremely happy'
"Some individual detainees have grievances - they may be unhappy with their own cases - but these particular grievances shouldn't be used to denigrate the good work being done by the centre," he said.
The 74 people currently being held in the centre - which has a capacity of 148 - include eight children, the youngest of them six months old.
A report last month by MSPs criticised the length of time people were held at the centre and claimed it would have a "detrimental effect" on their mental health.
It called for families with children to be housed within the community while their asylum applications were dealt with.
However, Dr Susan Lennox, the centre's medical director, said she believed the report was "fraught with inaccuracies" and misrepresented healthcare provision.
Dungavel manager Richard Smalley said detainees were not unhappy with conditions.
He said: "The detainees, almost without exception, are extremely happy with the facilities.
"They may not want to be in detention, which is understandable, but whilst they are this is probably the best place to be."
Mr Foulkes said: "Although it is regrettable, it is necessary to detain families who have children with them because the adults present a risk of absconding or their claims need to be assessed.
"The question is: If the adults need to be detained, is it better to detain them as families or separate them from their children?
"I think it is right that the families ought to be kept together rather than separated."
Meanwhile, the Scottish National Party said the problem lies not with the conditions at the centre, but with the UK Government's current asylum policy.
SNP MSP Linda Fabiani said: "MSPs know, and have often said, that it is not the staff or the conditions at Dungavel which are the issues, but the policies of the London government and the fact that elected representatives of the Scottish people have been refused information about an institution operating in Scotland."