People living near the studio set in Elstree in Hertfordshire say noise from the house and the rowdy fans who gather outside made their lives unbearable throughout the most recent series.
They are now planning to fight plans to extend the show for another three years and hold corporate events at the house.
Kay Prentice, who lives 100 yards from the house, said: "By the end of nine weeks, we were just thinking, when is it ever going to end?"
She said gangs of youths had gathered on the patch of land separating her home in Borehamwood from the Big Brother House.
"They were hurling eggs, golf balls and bottles at the Big Brother house until the early hours of the morning.
"They were breaking down the hedgerow to try to get a good look.
"We couldn't let our kids play out there, and all the dog walkers stayed away."
Ms Prentice, 43, said the house itself was badly soundproofed and the occupants seemed unaware they were in a residential area.
"You can hear them up to their antics, and even when they get called into the diary room.
"I know it's good for the area because it creates jobs, I don't dispute that.
"But something has got to be done while that show's running."
Ms Prentice and her neighbours from the other 16 houses in her street are putting pressure on Hertsmere Borough Council to insist on better soundproofing and security before it grants planning permission for more shows.
Permission has already been given for a Celebrity Big Brother show at the house in November.
The council will consider two further applications - one to extend the lease on the site until January 2006 and the other to host corporate events there.
Richard Grove, head of planning, said the applications would probably be considered in October.
He said the planning committee would be holding a meeting with environmental health officers and Endemol - the production company behind Big Brother - to discuss how to minimise disturbance to nearby residents.
"There are some serious concerns, but we have not had mass objections," said Mr Grove.