The extent and scale of mass surveillance undertaken by Britain's spy agencies is to be scrutinised in a major inquiry to be formally launched on Thursday.
Parliament's intelligence and security committee (ISC), the body tasked with overseeing the work of GCHQ, MI5 and MI6, will say the investigation is a response to concern raised by the leaks from the whistleblower Edward Snowden.
Sir Malcolm Rifkind, the committee chair, said "an informed and proper debate was needed". One Whitehall source described the investigation as "a public inquiry in all but name".
The announcement comes four months after the Guardian, and leading media groups in other countries, including the New York Times and the Washington Post, began disclosing details of secret surveillance programmes run by Britain's eavesdropping centre, GCHQ, and its US counterpart, the National Security Agency.
The Guardian has been urging a debate about programmes such as GCHQ's Tempora and the NSA's Prism, which allow the agencies to harvest vast amounts of personal data from millions of people intelligence that is routinely shared between the two countries.
In a change from its usual protocol, the normally secretive committee also announced that part of its inquiry would be held in public.
It will also take written evidence from interested groups and the public, as well as assessing secret material supplied by the intelligence agencies. The Guardian will also consider submitting evidence.
Conceding that public concerns had to be addressed, Rifkind, a former foreign secretary, added: "There is a balance to be found between our individual right to privacy and our collective right to security."
The ISC, which has been criticised for being too close to the agencies, has been under pressure to provide more robust scrutiny of the intelligence community. In recent weeks Lord King, a former chair of the committee, Sir David Omand, a former director of GCHQ, and Stella Rimington, a former head of MI5, have all raised concerns about the laws governing the secret services and the amount of scrutiny they are subjected too.
Formally, the committee has decided to broaden an existing inquiry into whether the intelligence laws are "fit for purpose".
Rifkind said: "In recent months concern has been expressed at the suggested extent of the capabilities available to the intelligence agencies and the impact upon people's privacy as the agencies seek to find the needles in the haystacks that might be crucial to safeguarding national security."
Nick Clegg, who had been asking for the oversight regime to be looked at afresh, quickly welcomed the committee's move. A source close to the Liberal Democrat leader said on Wednesday night: "We very much welcome the ISC's decision to broaden the scope of their investigation.
"The deputy prime minister has been an outspoken advocate of the need for us to have a proper debate about these complicated and important issues. He very much backs other voices getting involved in that debate and looks forward to the ISC doing some of it in public."
The admission that legitimate issues have been raised by the Guardian investigation also undercuts those on the Conservative benches demanding that the primary response to the Guardian disclosures should be prosecution of the newspaper for breaking the Official Secrets Act. Those demands surfaced again in parliament on Wednesday.
At prime minister's questions David Cameron criticised the Guardian and urged select committees to hold inquiries, following a question from the former defence secretary Liam Fox asking whether it was double standards to prosecute newspapers that hacked the phones of celebrities, but not those papers that released information that endangered national security.
Responding, Cameron said: "The plain fact is that what has happened has damaged national security and in many ways the Guardian themselves admitted that when they agreed, when asked politely by my national security adviser and cabinet secretary, to destroy the files they had, they went ahead and destroyed those files."
He added: "So they know that what they're dealing with is dangerous for national security. I think it's up to select committees in this house if they want to examine this issue and make further recommendations."
But a spokesperson for the Guardian said: "The prime minister is wrong to say the Guardian destroyed computer files because we agreed our reporting was damaging.
"We destroyed the computers because the government said it would use the full force of the law to prevent a newspaper from publishing anything about the NSA or GCHQ.
"That is called 'prior restraint' and it is unthinkable in the US, where the New York Times and Washington Post have been widely applauded along with the Guardian for reporting on the Snowden files. That reporting has so far led to a presidential review and three proposed bills before Congress."
Shortly after Cameron's intervention, it emerged that the Commons home affairs select committee would mount an investigation into the issues raised by the Guardian disclosures. It will also look into whether the paper has endangered national security and potentially broken the law, as part of a wider current investigation into counterterrorism. Rifkind said the ISC would be seeking contributions from outside the agencies "to ensure that the committee can consider the full range of opinions expressed on these topics. Once it has considered those written submissions it will also hold oral evidence sessions, some of which it expects to hold in public."
Julian Smith, the Tory MP for Skipton who has written to the Metropolitan police calling for the newspaper to be prosecuted under the Official Secrets Act and the Terrorism Act 2000, has been granted a debate in parliament next week in which "he will lay out the reasons why I believe that the Guardian has crossed the line between responsible journalism and seriously risking our national security and the lives of those who seek to protect us".
Elsewhere, privacy campaigners gave a cautious welcome to the intelligence committee's inquiry and said they would be prepared to submit evidence. But they all raised fears the ISC was still too sympathetic to the agencies and lacked the necessary clout to stand up to them.
Nick Pickles, director of Big Brother Watch, said: "This is a welcome step forward given the widespread concern that Britain's laws are not fit for purpose. However, such a debate cannot be allowed to take place behind closed doors and without pressing questions being asked about the legal justification for what we know to be happening presently at GCHQ and elsewhere."
Shami Chakrabarti, director of Liberty, said: "Some will say better late than never, others fear a tactical whitewash to calm public concern. It's certainly significant that the committee feels compelled to dig a little deeper but that's no substitute for much broader public and political debate."