The Haunted Grange of Goresthorpe was never published before and contains characters similar to Holmes and his sidekick Dr Watson.
"The story shows that the idea of a Holmes and Watson story came to him earlier than we had thought," said Owen Dudley Edwards, of the Conan Doyle Society.
There is a reason why the book was never published - it is considered to be a "rough and ready" ghost story, written by Sir Arthur when he was an 18-year-old medical student.
Sir Arthur was born in Edinburgh and studied medicine there before opening shop as a "consulting pathologist."
His first story published was The Mystery of Sasassa Valley in Chambers's Journal in 1879 but he sent The Haunted Grange of Goresthorpe to Blackwood's Edinburgh Magazine sometime earlier.
The story was filed and forgotten, remaining in the Blackwood files until 1942, when they were presented to the National Library of Scotland.
It was not published before as Sir Arthur's family felt it was not up to his usual standard, but following the death of Dame Jean Conan Doyle, the author's only surviving daughter, his family have relented.
Ian McGowan, librarian at the National Library, said the book was discovered on 24 neatly-written sheets of paper.
It was among correspondence of contributors to Blackwoods Magazine including Anthony Trollope, George Eliot, Henry James and Joseph Conrad.
"The story is written with childhood composition in much of its style, and hurls down its ghost with the promptitude of an over-zealous dog showing its prey to an owner on first emergence from a pond," said McGowan.
But he conceded that it was a valuable contribution to studying Holmes, as it "possessed many of the marks of the mature Conan Doyle".
As such, despite the story's lack of literary merit, it will be of great interest to fans of Sherlock Holmes and the many academics who study the life and work of Sir Arthur.
Dripping blood
Though Holmes is undoubtedly his most famous character, some argue that the writer's other work is neglected.
Sir Arthur - who died in 1930 - is considered by some to be the greatest writer of short stories in the last century.
"Most people think of Sherlock Holmes but he wrote many more successful novels," said Dudley Edwards.
"It is exciting to find this and interesting to see how Conan Doyle developed afterwards.
The Haunted Grange of Goresthorpe tells the story of a ghost who appears to a landowner in a Scottish mansion.
"He paints a picture of the house dripping blood," said Dudley Edwards.
The book reveals that the ghost haunting the house is actually terrified by the ghost of the wife he murdered.
"This obviously comes from him being a medical student, he thinks of the house like an open wound."