The Conservatives are planning an overhaul of the further education sector in England.
Mr Grayling aims to cut quangos to direct more funding to colleges
They say they will simplify structures so more money goes to colleges.
The shadow spokesman on further education (FE), Chris Grayling, said it was an undervalued "dumping ground" for government initiatives.
The Learning and Skills Council, which funds colleges, said it had improved its efficiency to put the interests of learners and employers first.
'Waste of time'
The Conservative Party intends to publish its FE proposals in the new year, but Mr Grayling has given a hint of their direction.
He said that, at present, senior college staff had to spend too much time in meetings with different but inter-related organisations, talking about similar issues and bidding for money.
"It's a monumental waste of time and these organisations deliver very little in terms of individual skills and training," he told the BBC News website.
"There are too many organisations eating up too much public money and adding little to the skills of the nation."
These include the Learning and Skills Council (LSC), the main conduit for FE funding, the Learning and Skills Development Agency and sector skills councils.
Mr Grayling described the LSC's £9bn budget as "enormous".
"I'm not persuaded we are getting value for money for that."
He declined to go into details at this stage, beyond saying: "We are looking at ways of simplifying the system. It needs to be simple and transparent."
Not only were colleges expected to increase the nation's adult skills base and provide much of the immediate post-16 education and training, they were also now catering for children as young as 14.
FE had been allowed to become "a dumping ground for every initiative we can think of", he said - including the "salvage sector" for the failings in the school system.
"Government say they want this initiative, that initiative and the other initiative and don't resource them.
"There needs to be a linkage between what you ask them to do and what they get in resource terms."
Asked about recent comments by the head of Ofsted, David Bell, that the relatively high number of failing colleges was a "national disgrace", Mr Grayling said the more colleges had to do, the more difficult it was to focus on being excellent.
"We have got some first rate people out there. If you give them more responsibility in how they decide to run their own institutions ... then I think we will see improvements just through that alone," he said.
The LSC's director of strategy and communications, Rob Wye, said it had restructured itself, reducing staff by 800.
"The Learning and Skills Council is improving the efficiency of its own operations constantly to provide value for money for taxpayers.
"We work closely with all our partners to ensure that we get every penny possible to the front line. The interests of learners and employers are paramount."
A spokesperson for the Department for Education and Skills said the FE sector was getting "significant extra resources".
The aim was to cut out unnecessary bureaucracy and introduce lighter touch inspections "so that we create strong and autonomous demand-led institutions".
The department recognised that the skills strategy and 14-19 reforms placed "considerable and challenging new demands" on colleges.
A review of the role of FE colleges had been commissioned, to report next autumn.