A government review of new 24-hour licensing laws has found what it calls a "mixed picture" of their impact.
The introduction of 24-hour drinking laws proved to be controversial
Crime and alcohol consumption are down overall but some areas have seen a rise in disorder and drink-related violence has increased in the early hours.
Culture Secretary Andy Burnham said councils and police needed to do more to use "the considerable powers" open to them to tackle the problem.
Shadow home secretary David Davis said ministers were "deluding themselves".
He said: "The police, hospitals all say this has failed. Over a million drink-inspired violent crimes against people last year - the highest ever.
"Roughly a 50% increase in anti-social behaviour based on alcohol... So this is not a successful policy."
But Mr Burnham said the introduction of the Licensing Act in 2005 "has not led to the widespread problems some feared".
"Our main conclusion is that people are using the freedoms but people are not sufficiently using the considerable powers granted by the Act to tackle problems," he said.
"And there is a need to rebalance action towards enforcement and crackdown on irresponsible behaviour."
Gordon Brown announced a review of the Licensing Act shortly after becoming prime minister in June last year.
The laws allow pubs and clubs in England and Wales to apply for later - or even 24-hour - licences to serve alcohol.
It was supposed to usher into Britain a continental-style cafe culture.
Fewer than 4% of premises (5,100) have applied for round-the-clock pub opening - and many that have are hotels, stores and supermarkets.
Only 470 pubs, bars and nightclubs are open 24 hours and the average closing time across all licensed premises has got just 21 minutes later.
Ministers point to the latest figures, which suggest a fall of 1% in overall crime and of 10% in violent crime committed since licensing laws changed.
However, Mr Burnham said: "Whilst crimes involving violence may have reduced over the evening and night-time period, the evidence also points to increases in offences, including violent crimes reported between 3am and 6am.
"This represents 4% of night-time offences.
"Similarly, whilst there is no clear picture of whether alcohol related demands on A&E services and alcohol-related admissions have risen, some hospitals have seen a fall in demand, others have reported an increase.
"It is also clear that the overall reduction in alcohol-related disorder we wanted to see across the country has not materialised consistently in all areas."
Measures planned include:
- Far more instant closures of pubs and clubs where disorder has occurred
- Impose a "two strikes and you're out" rule for off-licences caught selling alcohol to people under 18
- Introduce a tougher system of red and yellow cards for landlords and shop keepers breaching licensing laws. It would firstly impose conditions and ultimately revoke licences
- Increase the maximum fine for anyone not obeying an instruction to stop drinking from £500 to £2,500
- Produce a ranking of areas based on the risk their bars and off-licences pose to crime and disorder, public nuisance and children.
But Jan Berry, chairman of the Police Federation, said: "For the last couple of years we haven't seen the decrease in alcohol-related offences.
"We have seen an increase in the number of police officers assaulted."
Sir Simon Milton, chairman of the Local Government Association which represents local authorities, said the idea that late-night licences would end binge drinking had totally failed.
He said policing resources were being stretched further into the night because of people who had been drinking.
Tim Martin, chairman of the Wetherspoon pub chain, told BBC2's Newsnight that 24-hour licensing had not helped counter binge drinking.
"People need to be re-educated about their drinking habits. It's worked really well with drink driving, but you have to persuade people it's a bad idea to get paralytic," he said.
Labour MP Martin Salter, a member of the Commons home affairs committee, blamed binge drinking on those who made alcohol too easily available to young people.
"The problem is the availability of cheap alcohol from supermarkets and the fact that, actually, most teenagers are getting alcohol either from family members or from parents or from irresponsible shopkeepers," he said.
Chris Huhne, the Liberal Democrats' home affairs spokesman, said: "The basic problem is not that we need new powers and penalties - it's that we need to enforce the law that we've already got".