The HyShot scramjet engine was to be fired off the back of a rocket, launched from the Woomera Protected Area, a testing ground north of Adelaide, Australia.
Data from the flight box shows that the rocket flew off course, thwarting the test flight.
Project leader Allan Paull from the University of Queensland said in a statement: "Although we didn't achieve all that we set out to achieve, we succeeded in gathering valuable data, and we are encouraged by the fact that the payload survived one hell of a ride."
The test involved the rocket carrying the scramjet into the upper atmosphere before falling back to Earth.
If the experiment had been successful, the scramjet would have started to work under its own power for a few seconds before crashing into the ground.
Scientists had hoped the engine would reach a speed of Mach 7.6, or 7.6 times the speed of sound.
Further test flights have now been put on hold until flight path glitches are fixed.
Passenger jets
Experts say scramjet technology has the potential to reduce the flying time from London to Sydney to two hours, and substantially cut the cost of space launches.
That makes them more efficient than conventional rocket motors, which carry their own oxygen supply on board, adding weight and cutting the potential payload. But the big problem is that scramjets only start to work at speeds greater than five times the speed of sound, or Mach 5.
A few weeks ago, the Defense Advanced Research Projects Agency in the United States launched a scramjet from a gun. The Americans say their experimental engine flew under its own power for a fraction of a second - a world first.
A Nasa scramjet test in June failed when the missile that was intended to bring the engine up to its operating speed went off course.
The engine tested at Woomera was built by an international consortium led by researchers at the University of Queensland and is jokingly referred to as a "scroungejet".
"Ours is a low-cost alternative, and we've had to develop all sorts of ancillary equipment on the cheap," said Allan Paull, speaking before the launch.
"We've bought a lot of bits and pieces off the shelf from automotive shops," he added.