Home Secretary: Do you have what it takes to do Britain's most demanding job?

From terror laws and prisons to ice-cream vans and massage parlours, the range of responsibilities that falls to a Home Secretary is at best daunting, at worst impossible. As John Reid wonders what's hit him, Michael Cockerell reveals how previous incumbents were still haunted by the job long after they'd left office

Dr John Reid has had a new job for every year that New Labour has been in power. But his ninth post, as Home Secretary, is the most hazardous of all. For even though it is one of the glittering Great Offices of State, it can be a poisoned chalice.

Only one of the 23 post-war Home Secretaries has gone on to become Prime Minister: Jim Callaghan. A number of others have had to resign or had their political careers irreparably damaged by their time at the Home Office.

Over the years I have filmed and talked to many Home Secretaries and their officials to try and build up a picture of the job. "It is probably the most dangerous job in government," said Ken Clarke, who emerged unscathed from his 15-month tenure in 1993. Not so his successor, Michael Howard.

"I don't pretend that in four years in that demanding job I made no mistakes," he says. "And it's a matter of luck, usually, whether you make your mistakes in a dark corner with nobody looking, or in the spotlight. There aren't many dark corners in the Home Office."

Jack Straw, who became New Labour's first Home Secretary in 1997, says: "One of my predecessors gave me a gypsy's warning about the Home Office: he said the key thing you've got to remember is that at any one time there will be 50 sets of officials working on projects that will undermine the government and destroy your political career. And the worst thing is that not only do you not know who they are, but they don't know who they are."

The man who was Straw's Sir Humphrey, the Permanent Secretary at the Home Office, (until he became Cabinet Secretary), was Richard Wilson. "When you are in the Home Office you never can absolutely sit back in your desk and say, 'This is all going well,'" says the now Lord Wilson. "You always have to say to yourself, 'so far'. You always have to touch wood. I learned in the Home Office to touch wood in a way I never had to in any other department."

When you become Home Secretary, one of the things that can astonish the appointed person is the extraordinary range of responsibilities that the job carries. As well as the core business of crime and punishment, immigration, asylum, terrorism and the police, the list includes nudist beaches, mad dogs, relations with the Royal Family, GMT, massage parlours, nuclear bunkers and ice-cream vans.

The job of Home Secretary as the keeper of the Queen's Peace goes back more than 200 years. But for all the prestige of running the ancient department of law and order, it can be the least glamorous of all Cabinet posts.

"This is the job in which you get to know your own country most closely," says Douglas Hurd, who was Home Secretary for four years under Margaret Thatcher. "You tend to see the bad bits of it, run-down housing estates, prisons, very complicated legal cases, riots, horrors of all kinds."

It is also the job which is perhaps the most politically exposed in government, for any part of your huge empire may suddenly hit the headlines. "The climate of the Home Office was like that of summer storms blowing up absolutely out of a clear blue sky," said Roy Jenkins, who served two terms as Labour Home Secretary. "And they were often very violent storms." Or as others who have worked there graphically put it: the corridors of the Home Office are paved with dynamite.

Prisons, as both John Reid and Charles Clarke realise only too well, are the most perilous part of the estate. Although most of your manpower and budget goes into managing them, you know that there no votes in prisons - both literally and metaphorically. And you also know that any stage they can cost you your head.

"It's a very dramatic thing to go in to one of these huge Victorian prisons in the late afternoon with the darkness gathering about it," says Lord Hurd. "They are such a striking part of your job. And it's not something you can simply pass by on the other side, saying, 'I'm going to think about this every other Friday.'"

Lord (Kenneth) Baker, John Major's Home Secretary, says: "I think most Home Secretaries cross their fingers and hope that nothing's going to go wrong in prisons while they're Home Secretary - because there's a cigarette paper's difference between success and failure in administering prisons. At any time it may explode."

I put Baker's view to Jack Straw when he was Home Secretary. His frank response astonished me. "Every Home Secretary does not just cross his fingers: you have to pray daily, light a candle and get down on your knees as far as the prisons are concerned. And also make sensible policy decisions. But there is no doubt at all, when it comes to the prison service, prayer has an important role to play."

Any prayers that Straw's predecessor Michael Howard had offered up went unanswered. He presided over a calamitous series of prison escapes and did himself no good by attempting to heap the blame on the head of the Prison Service. Like many recent Home Secretaries, Howard was a lawyer. That can be an advantage in a department that is precedent-laden and casework driven.

But if you seek to fend off attacks on you in Parliament and the media by deploying the worst lawyerly traits of splitting hairs and chopping logic, you will only make things worse for yourself.

Howard was one of a string of Home Secretaries to be mercilessly lampooned - in A Very Open Prison on BBC TV. The accident-prone Henry Brooke - Home Secretary under Harold Macmillan - had been the regular target of David Frost's weekly satirical show TW3. Ted Heath's man, Reggie Maudling, was excoriated by Private Eye and Paul Foot, and most recently David Blunkett was ridiculed in a More 4 satire. Although in the latter two cases their extramural activities involving, respectively, money and sex, had made them irresistible Aunt Sallies.

And the ramshackle Home Office itself, with its 70,000 employees and annual budget of over £14bn, has long been a top target for the media and the public. As Harold Wilson's first Home Secretary, Sir Frank Soskice, once put it in plaintive internal memo: "Poor old Home Office - we aren't always wrong, but we always get the blame."

One of the hardest things about running such a sprawling empire is to keep tabs on everything that is being done in your name. As Shadow Home Secretary, Jack Straw said that he would sometimes turn on the Today programme in the morning and discover things that the Home Office was doing that he knew nothing about. I asked him whether things had changed once he became Home Secretary himself.

Again with disarming candour, he said that there were times when it had not changed. "I turn on the radio in the morning and I hear about some report and think, 'I know I've not seen anything about that in my red box.' But there you go. There's so much going on in the Home Office that there's no way you can know about it all. And if you start fretting that you've got to know everything, in the end you'll know nothing."

Roy Jenkins told me that although much of the casework a Home Secretary has to deal with could seem routine, "it was important to have some sort of antenna that told you when something might be wrong and might potentially blow up into a big thing. And when that happened one agonised over cases a great deal. Because I always regarded the essence of the Home Secretary's job as trying to hold the rather delicate balance between the liberty of the individual and the security of the state."

Perhaps the most difficult of all such judgements are when you have to decide whether to grant early release to a murderer who has received a life sentence. The lifers' cases will come to you after the parole board has carefully considered them and made its recommendation for release.

"Every weekend as Home Secretary, I had a diet of four or five lifers to decide whether to release," says Straw. "You've got to pay very careful attention to these." I put to Lord Merlyn-Rees, who was Jim Callaghan's Home Secretary, that it must be one of the most difficult things if you let out someone who was meant to be serving a term of life imprisonment, and he kills again.

"Yes," replied Merlyn-Rees. "And I can't guarantee that I've not done that. No Home Secretary can." Straw said: "You have to do your best. But the moment you're into the business of releasing life sentence prisoners, people who by definition have murdered once, you can't guarantee absolutely that they won't go out and do it again. What you can do, however, is make the best judgement on the evidence that's available as to the level of risk. But life is a risky business I'm afraid."

What every Prime Minister requires in his Home Secretary is a that most prized of all Whitehall qualities - a safe pair of hands. PMs learn that there is little political advantage to be gained by interfering in Home Office matters. Even Lady T, who was forever seeking to micro-manage, would keep out of Home Office affairs. And when he appointed him for a second time as Home Secretary in 1974, Harold Wilson told Roy Jenkins that the advantage was that he could be a "semi-detached" member of the government at the Home Office. As Home Secretary you learn that you must never overtly seek to use the job as a stepping stone to higher things. "You must be ambitionless," said Lord Rees, "because of the issues that crop up all the time, you'd worry yourself to death if you were thinking, 'I want to get on, I want to become Prime Minister.'"

Jack Straw demurs slightly: "Ambition in politics is not a crime, it's part of the lifeblood. I was ambitious to get the job and keep it. But if you allow extraneous factors to get in the way of the decisions you make, then I think you'll come a cropper."

I asked Roy Jenkins what he regarded as the key qualities for success as a Home Secretary. "You need to have a certain sense of proportion, to know what's an important issue and what isn't, because you can easily get bogged down in pointless detail. You need to know what your priorities are, while at the same time having a certain defensive sense - an instinctive "corner of the eye" view - so that although there are some issues you don't want to concentrate on, you are aware of the threat before the dagger actually goes in your back. And you have to be able to defend your policies better than adequately in the House of Commons and before the public generally."

"It's a job that is a very big challenge for any human being to take on," says the former top Home Office mandarin, Lord Wilson. "It's a marvellous challenge - but a daunting one. It requires super-human qualities of stamina, intelligence, compassion and good sense; and all the great Home Secretaries have those qualities."

But even that combination can be undone if you do not possess perhaps the most important gifts of all - shrewd political judgement, good luck and the chance to quit while you are ahead.

Michael Cockerell is the maker of the BBC documentary, How to Be a Home Secretary

Douglas Hurd 1985-1989

ON THE HOME OFFICE

"It used to be a real fortress of a department, which resented outsiders and erected fortifications to prevent outsiders knowing what the hell was going on inside the castle. There is a sort of ethos, there is a character about the department where you can still see traces of those fortifications."

ON SENTENCING POLICY

"I think [we] went too far. I mean, it looked as if there was a kind of Dutch auction in trying to be tougher and tougher on criminals. I think it is a mistake to get in to an argument about who's tougher on crime. Everybody's against crime, the question is how you effectively deal with it."

ON THE SECRET

SERVICES

"By definition you don't know what everyone is doing every minute of the day. So you need to know what you need to know. I did not feel ever let down on that."

Jack Straw 1997-2001

ON TOUGHER SENTENCING

"I had great confidence that what I was doing was not just wanted by my constituents, in a populist sense, but needed by them."

ON DOING THE JOB

"The way I try to get hold of things is by getting out. If you just sit and read the briefs you'll never get a sense of what's happening. So I go out on patrol with police officers, talk to them confidentially when I get the chance, and when I go around prisons, I depart from the route so I meet people I'm not supposed to. That's more difficult in a prison for obvious reasons."

ON THE TRAVAILS OF MICHAEL HOWARD

"It wasn't his fault, I mean you - you never know. [Prisoners] can suddenly just get it into their heads that they're going to get over the wall. If [this] happens, two or three are going to [go] at the same time."

ON THE SECURITY SERVICES

"It's not as big a deal as people imagine. There aren't spies lurking in the shadows, spying on the normal business of politics. I am pretty happy about the way the service is run, and I believe it is run to high standards of integrity, and it's there for a purpose, which is ensuring that we don't get blown up by terrorists, that our economy is not seriously disrupted, and that we don't suffer even greater problems from drug dealers."

ON NEWS MANAGEMENT

"I turn on the radio and I hear some report and think 'I've not seen anything about that in my box.' But there's no way you can know about it all. If you start fretting that you've got to know everything, in the end you'll know nothing. There's no point in being surprised about anything in this job."

Michael Howard 1993-1997

ON CRIME STATISTICS

"I was shown charts. And they showed crime rising inexorably. And the officials actually said to me, 'This is the pattern of rising crime. It's gone on, with one or two small and inconsequential blips, like this for most of this century. It is going to continue to go up, and the first thing, Home Secretary, that you have to understand, is that there is nothing you can do about it.' "

ON THE SCANDAL AT WHITEMOOR PRISON (where prisoners escaped as the staff played Scrabble)

"I was appalled. Everybody else was appalled, and I was appalled. I was just as appalled as everyone.

It was appalling, it was appalling, and I was absolutely horrified and there then commenced quite a lengthy business of trying to ensure that security really was taken very seriously."

ON THE PARKHURST ESCAPE (where high-security inmates scaled the walls using ropes they'd made in the prison workshop)

"I was then pretty angry, because after the Whitemoor escape I had been given assurance after assurance that measures were in place, that nothing like that could ever happen again, and that everybody was now fully seized of the importance of security, that all the procedures were being followed, that everything was in place. And so when that happened, I was pretty angry.

"I can't [personally] look under the beds in the prisons, and I can't make sure that the doors are fastened, and that prisoners are counted in and counted out when they go from one part of the prison to the next."

ON THE SECRET SERVICES

"I think I knew what I needed to know."

Kenneth Clarke 1992-1993

ON GETTING AWAY WITH IT

"I was a very lucky Home Secretary, I was only there for 15 months. I had one prison escape when the guys got out through a hole in the roof and then it started to rain and they got back through the hole again and got back into the dry."

Kenneth Baker 1990-1992

ON TERRORISM

"I remember one day I was having an early morning meeting at about 8.30 or so, in that big office overlooking St James's Park, and there was an explosion, a bomb had gone off. And I looked out of the window, and I saw a plume arising somewhere near Downing Street. It was the mortar attack on No 10. So immediately the security forces got me over there, got into No 10, it was then all completely shut off, and no one could get anywhere near it, and the Cabinet had moved to the room that the Cabinet does occupy in moments of crises. And as Home Secretary you have to cope with that sort of situation."

David Waddington 1989-1990

ON THE STRANGEWAYS PRISON RIOT

"When all this kerfuffle was going on, somebody said, 'there's an urgent message for you', and I picked up the phone, and the message was to the effect that a riot had started in Strangeways. So I thought, 'Gosh, I've got some problems to deal with now.'

It was a searing experience because I could see criticism mounting as to what was going on. But I was not in the business of vetoing the advice which I had been given and telling people, whatever the risk, you've got to retake the prison, because people are jeering at us as they see on the television people throwing down slates from the roof and the rest of it; that wasn't what I was going to do at all. But on the other hand, I knew perfectly well that the Prime Minister wasn't happy about what was going on. Well, of course, I couldn't sleep for worrying about all this. I could see things weren't going as they ought to be going.

"It was certainly very bad for my own reputation. And I wish things hadn't worked out in the way in which they did. But nobody was killed and no prisoners escaped. And goodness knows what the criticism would have been if the Army had been brought in and a few rioters had been shot. I then really would have been crucified."

Roy Jenkins 1965-1967, 1974-1976

ON THE TORREY CANYON OIL DISASTER

"As Home Secretary I had to deal with this. I knew very little about oil tankers, Cornwall, beaches [or] oil. It dominated the papers for some time, and one had to take all the key decisions oneself. The key decision was whether you bombed the wretched thing and tried and sink it. We did, and it went quite well... [but] looking back, I got too involved in that [decision]."

ON THE HOME OFFICE CULTURE

"I was perfectly clear that I had to bust this system. This involved the most wearing hour's interview I've ever had in my life, and I told [his Permanent Secretary Sir Charles] Cunningham that the system had to go. His eyes filled with tears. It was only later I realised they were tears of rage."

ON THE SECRET SERVICES

"I did not form a very high regard for how they discharged their duties. I also think that living one's life in this sort of spy-bound world gives people a slightly distorted view of things. If you're not careful you get into a sort of Alice in Wonderland world in which truth is falsehood and falsehood is truth, and nothing is in contact with reality."