Doctors are raising a series of concerns about the confidentiality of the electronic care records system.
Electronic records is one part of the £6.8bn IT upgrade
The database containing medical records for up to 50m patients in England is part of the £6.8bn NHS IT upgrade.
A group of doctors has told the British Medical Journal that the system is not secure enough and suggested the money would be better spent on care.
But the officials behind the project said chip and pin technology would ensure security standards were high.
The 10-year programme, run by a government agency called NHS Connecting for Health, is aimed at linking more than 30,000 GPs to nearly 300 hospitals by 2014.
KEY PARTS OF NHS IT UPGRADE
Choose and book - A system to allow patients to book hospital appointments at a place, date and time of their convenience from GP surgeries. Nearly 10m such referrals are made each year
NHS care records service - An electronic database of patient medical records which will allow NHS staff across the country to access information wherever someone is treated
Electronic prescriptions - More than 325m prescriptions are made each year. By 2007 the paper based system will be replaced with an electronic version, which will allow patients to pick up repeat prescriptions from any pharmacy in the country
The new systems include an online booking system, e-prescriptions and fast computer network links between NHS organisations as well as the electronic records system.
The project has been mired in controversy since it started two years ago over the extra cost NHS trusts have to bear to train staff and cope with delays in the implementation of various software.
Confidentiality has also been a long-running source of contention, with the government offering patients the chance to opt out of having their records on the system.
NHS staff across the country will be able to access the details, although there will be different levels of access for different staff.
But doctors said there was a risk that passwords would get shared or staff would inadvertently get access to patients' details.
Michael Foley, a consultant anaesthetist at Middlesbrough's James Cook University Hospital, said: "Many hospitals already have electronic access to laboratory records and radiological images.
"Passwords are sometimes shared, screens left on in open view.
"When the medical history of the whole population becomes available, the potential for loss of confidentiality is obvious"
He added the huge sums of money invested would be better spent improving patient care.
Anthony Winston, a consultant in eating disorders at Warwick University, said patients should have the right to choose whether their medical details are added to the database.
Dr Paul Cundy, IT expert at the Royal College of GPs, said the concerns over confidentiality could undermine the whole project.
"The key is that patients and clinicians must have confidence that information will be secure and shared only with patient consent."
Dr Simon Eccles, national clinical lead for NHS Connecting for Health, said chip and pin technology would be used to "ensure the highest levels of security".
"The reason for introducing this system is so that those caring for the patient will have access to all the right information as they need it.
"Medical staff will only be able to see the information related to the patient they are caring for."