Iain Duncan Smith has not grown into his job. It is time for him to go

He has not the judgement, the personality, the intellect, the leadership skills or the confidence to lead his party to victory

Iain Duncan Smith has been lucky: the Conservative Party, unlucky. Mr Duncan Smith was doubly fortunate, in the way that the local election results were presented and in the opponents who broke cover after the polls closed. On the face of it, 545 additional council seats sounds a good result, especially as it was well in excess of most predictions. From a Tory point of view, however, the seat gains were a snare and a delusion. They provide no grounds for optimism as to the party's prospects in a general election. There, the crucial indicator is the percentage of the vote, and the Tories won only 34 per cent.

The psephologist Professor Tony King has warned the Tories that the outcome is a matter for relief, not joy. He is wrong. There are no grounds for relief. Thirty-four per cent – four points lower than Mr Hague's score in 2000 – was a terrible mid-term result. An opposition with any hope of winning the next election ought to have been up around 45 per cent. At 34 per cent, the Tories are in serious danger of losing more ground, especially as the Liberals reached 30 per cent, their best ever result, and made further gains along the M3 corridor. Tory MPs who are worried about a Liberal challenge have every reason to stay worried.

Yet that is not how the result was spun. Some of the blame for this must go to Crispin Blunt. Though Mr Blunt is a courageous fellow, he is also over-encouragingly named: a blunt instrument, but without much instrumentality. A former soldier, he was the sort of young officer who would be better at gallantry awards than at strategy. The trouble is, however, that the award would be posthumous. On Thursday night, he charged straight at the machine-guns. Nobody followed him, and he got shot. But Mr Blunt was only saying in public what at least two-thirds of his parliamentary colleagues believe in private.

His premature intervention was not assisted by David Mellor's agreement. A shallow, selfish creature, Mr Mellor used the Tory party when it suited him and occasionally revisits it when he wants some cheap publicity. I doubt if there is a single Tory MP who believes that Mr Mellor has any moral or intellectual credentials to comment on Tory party matters.

Mr Mellor is the sort of critic whom an embattled Tory leader would pray for, especially when abetted by the BBC. The BBC approached the elections much as it did the Iraq war. Then, it had hoped to gloat over Anglo-American reverses; last Thursday, it was salivating at the thought of bad news for the Tories. So blatant was the BBC's partisanship, so shallow its analysis, that it did not even realise that it was right. The Tories were in trouble. A more dispassionate, percentage-based approach would have reached that conclusion. Instead, as the seat gains which the BBC had hoped would not occur did materialise, its commentators were forced into graceless retreats.

It was a bad night for the Tories. It will be a still worse one if it enables IDS to retain the leadership. In 2001, there were two good reasons for supporting Mr Duncan Smith: he was neither Kenneth Clarke nor Michael Portillo. Mr Portillo is an interesting, highly intelligent man on a voyage of self-discovery. This has involved train trips through Europe, television programmes on Wagner and stints as a shelf-stacker in a supermarket. Fascinating stuff, but he could not be allowed to use the leadership as another chapter in his Bildungsroman. Before you take that job, you have to know who you are and what you believe.

Not that Ken Clarke could ever be accused of self-doubt. He knew why he wanted to lead the Tory party: so that he could get stuck in to Tony Blair for failing to abolish the pound. Fortunately, Mr Clarke's opposition to the Iraq war has destroyed his last chance of leading the Tory party; that makes it safe as well as necessary to hold another leadership contest.

Mr Duncan Smith was elected for negative reasons. He has not begun to convert them into positive ones. At the time, some of his supporters – including me – hoped that he might turn into another Truman or Attlee: a first-class leader if not a first-class intellect. But he has not grown into the job. He has shrunk in it, further diminishing his party's élan vital.

The Tory party's problems are easily summarised. Most voters neither know what it stands for nor trust it. Most Tory MPs are aware of this, yet see no way to put things right. That is why Tory MPs' morale has never been lower.

Disraeli told his fellow Tories that unless their party was a national one, it was nothing. For most of the subsequent century and a quarter, it was not only possible for Tories to agree with him; they could feel complacent while doing so. In recent years, however, there has been a growing fear in Tory circles that the bonds which link party and country have been broken. Hence the despairing, self-flagellatory tone which the so-called Tory modernisers adopt, as if a party which had given such service to the nation could only justify its future by apologising for its past.

Such grovellings are as unnecessary as they are self-destructive. Britain is still a profoundly conservative country. This has been acknowledged by much the most formidable analyst of contemporary public opinion, although he has still to embody his work in academic form: one Tony Blair. Mr Blair has based most of his political programme on a theft of Tory themes and an infiltration into Tory territory. Where his instincts place him at odds with Tory ones, most notably over the euro, he has deferred to Tory Britain. It would be absurd for any Tory to be less confident in the electoral appeal of Toryism than the leader of the Labour Party is.

Under good leadership, the Tories' poll ratings would be well up in the forties, especially now that Mr Blair's domestic agenda is in such trouble. The voters want to know when they are going to receive value for their money which the Government has spent on public services. Mr Blair only seems interested in telling them that he has been walking with God. Those of us who always believed that Tony Blair would disappear up his own spin have been underestimating him. It now looks as if he will disappear up his own apotheosis.

Unless IDS rescues him. Even a failed government can retain power if the opposition is led by an implausible alternative prime minister. Iain Duncan Smith has not the judgement, the personality, the intellect, the leadership skills or the self- confidence to lead his party anywhere near victory. Though any of the likely replacements would do better than IDS, the ideal outcome of a leadership contest would be a final two-horse race between William Hague and Oliver Letwin. We could then judge whether Mr Hague might now persuade the public of his merits, and whether Mr Letwin had enough jugularity to lead the Tories out of opposition.

Last October, IDS nearly resigned. He then made the wrong decision, and stayed. Even if he does not face an immediate challenge, he will do so after the next, inevitable crisis, and he will not survive. He should go now, while he can do so with dignity.