Fewer than 20% of Department of Health senior civil servants are satisfied with the way it is managed, according to a survey for the government.
More than 80% of the 169 surveyed by an external company think the department is not able to oversee change well.
The Tories described the findings as a "vote of no confidence" in Health Secretary Patricia Hewitt.
But the department said that after much internal and external change last year, it was "learning from its mistakes".
The survey also suggested that confidence in leadership was 37%, compared with 57% across Whitehall.
Shadow health secretary Andrew Lansley said: "If [Ms Hewitt] can't successfully lead the 3,500 people in her department then she can't be trusted to lead the 1.4 million people working in the NHS - none of whom know the direction in which the service is headed.
"Health professionals need to have confidence in the department's capabilities at this time of cutbacks, closures and financial crisis, and these figures won't exactly encourage that."
The survey, carried out in October 2006, also showed that about 16% of those questioned thought the department was managed well as a whole, compared with 55% who said it was not.
About 28% said the top management team provided effective leadership, while 48% disagreed.
The Department of Health described last year as "difficult," but a new NHS chief executive and permanent secretary were providing clear direction.
"Fundamental improvements to the management and governance of the department have been put in place to make us stronger and more efficient in the long-term," a spokeswoman said.