The parliament's rural development committee said Mike Watson's Wild Mammals Bill was too wide-ranging and that it would be difficult to make it workable.
Mr Watson, Labour MSP for Glasgow Cathcart, said the committee findings were "absolutely bizarre".
He said: "I remain confident a majority of MSPs will share my view that the barbaric blood sports of fox hunting and hare coursing have no place in today's Scottish society and will vote in favour of the bill," he said.
He added: "It is absolutely bizarre that the committee's recommendation is
not supported by the main body of the report.
"It also defies logic for members of the committee to say the bill cannot be amended."
Mr Watson insisted he had repeatedly stressed he was willing to see the bill amended and the committee had itself acknowledged he had put forward proposed amendments already.
The member's bill is the first piece of proposed legislation that has failed to be recommended by a Scottish parliamentary committee.
End cruelty
Committee convener Alex Fergusson said: "The committee has come to the conclusion it has because having very thoroughly studied all the evidence over a period of 15 months it was unable to agree that the bill could be amended to do what the proposer wanted, which was to end cruelty."
Mr Fergusson, the Tory MSP for South of Scotland, said the committee had also been unable to clearly define unnecessary cruelty which was central to the bill.
He conceded that the parliament was not obliged to accept the committee's recommendations when it debates the bill after recess.
But he warned that such a move would "place the parliament in an awkward situation" given the importance of the committee system at Holyrood.
Speaking in a personal capacity, Mr Fergusson welcomed the possibility of
seeing a new bill on the issue after the obligatory interim period of at least
six months.
Committee member and Labour MSP Cathy Jamieson, who agreed to the general principles of the bill, recommended that her fellow parliamentarians should still pass it when it comes before parliament.
Ms Jamieson, the MSP for Carrick, Cumnock and Doon Valley, said: "Even though the recommendation came from the committee not to go ahead that still has to be debated in parliament and parliament could still take a different view. It's not all over."
Richard Lochhead, nationalist MSP for north-east Scotland region and one of the minority on the committee who wanted members to agree to the bill's general principles, said it was the most difficult and longest-drawn out committee inquiry in the history of Holyrood.
"On many occasions, I felt I was engaged in trench warfare with battles raging over every letter of every word," he said.
The vice president of the National Farmers' Union of Scotland, John Kinnaird, welcomed the committee's recommendation to reject the bill.
He said that using dogs to control the fox population was "critical" and the
bill would have prevented farmers from protecting their livestock and crops.
He added: "The committee, during the exhaustive evidence collecting process, has clearly recognised the dangers that were posed by this draft legislation and I welcome their decision today."
The Scottish Countryside Alliance said the committee had done "a very thorough job".
"However we will refrain from further comment until the parliament has made its decision," said a spokesman.
Les Ward of the Scottish Campaign against Hunting with Dogs said: "This report found that hunting is cruelty in the name of sport, and yet some of the committee don't want to see that cruelty stopped."