A correspondent for Radio Russia has said the rebels, who crossed from Chechnya on 7 August, were sustaining casualties and were running out of ammunition.
The correspondent said part of the rebel force was observed retreating to Chechen territory.
(Click here to see a map of the region)
On Saturday the defence ministry said its forces had captured commanding heights near two villages held by the rebels.
It said air raids had destroyed an ammunition depot and several vehicles reportedly travelling from neighbouring Chechnya but there has been no independent confirmation of the reports.
Chechen moves
In the Chechen capital Grozny President Aslan Maskhadov said he had called up 15,000 Chechen fighters "to take up arms and be ready to protect the nation's northern borders".
"Events in Dagestan have been provoked to unleash a big war against Chechnya," Mr Maskhadov said.
Between 1994 and 1996 Russian forces conducted a disastrous campaign against separatist rebels inside Chechnya in which tens of thousands of troops were killed.
But President Maskhadov said Chechnya was not involved in the conflict and his republic did not want war with either Russia or Dagestan.
The rebels, who come from the fundamentalist Wahhabi sect - whose adherents include the Saudi Arabian rulers - are led by Shamil Basayev, a Chechen warlord who broke ranks with Mr Maskhadov after the 1994-1996 war.
The rebels say they are more familiar with the rough mountain terrain than the Russian troops and have made much more progress against the larger Russian forces than they originally thought possible.