Amec Project Investments has been invited to revise and resubmit its plans
Proposals to build 85 wind turbines in East Ayrshire and Dumfries and Galloway have been rejected on safety grounds.
The Scottish Government said the Kyle wind farm would have had detrimental effects on safety at Prestwick Airport and the surrounding landscape.
Ministers have invited Amec Project Investments to resubmit plans with better shielding to reduce radar and landscape impacts.
The proposal was for 85 turbines with a generating capacity of 255MW.
While the vast majority of turbines would have been built in East Ayrshire, the wind farm would also have extended into Dumfries and Galloway.
The controversial proposal prompted more than 4,200 objections.
Energy Minister Jim Mather said: "The Reporter concluded that the Kyle wind farm would be inconsistent with the East Ayrshire Development Plan because of the effects on air safety at Prestwick Airport and due to the impact on the landscape.
"While I am not bound to accept the Reporter's recommendation, after careful consideration I have decided to do so.
"As our record of approving 14 renewable projects shows, we want to approve good projects, in the right places, with benefits for communities and in harmony with the environment.
"We have therefore invited the applicant to come forward with a new application, with better shielding to reduce radar and landscape impacts."
A spokesman for Vattenfall, which has acquired Amec's wind development portfolio, said the company was "naturally disappointed".
He said: "We intend to work co-operatively with the Scottish Government and local stakeholders and community to consider an alternative proposal."