Slick customer: Director Paul Thomas Anderson strikes it rich with his gritty new oil saga

After his last film, Paul Thomas Anderson couldn't get arrested in Hollywood. But, as he reveals here, the director has hit paydirt with his forthcoming oil epic

The success of There Will Be Blood has put Anderson at the forefront of a riskier, edgier Hollywood © AP

You can't help but readinto the fact that Paul Thomas Anderson's latest film begins with a solitary man down a deep pit, hacking at rock with a pickaxe: like the oil prospector anti-hero of There Will Be Blood, Anderson seems to be going hell for leather after something that's unusually hard to reach – and unusually rewarding.

And so it turns out. Blood is an intransigently challenging, rich and turbulent work. Which makes the awards and nominations the film has scooped all the more remarkable. A Golden Globe Best Actor for Daniel Day-Lewis has been accompanied by nine Bafta nominations. And now it is up for eight Academy Awards, including Best Picture, Best Leading Actor, Best Director and Best Adapted Screenplay.

Depth and darkness are not qualities generally associated with the young US film-makers usually considered Anderson's peers. Wes Anderson, David O Russell, Sofia Coppola, Spike Jonze – all largely inhabit the daylight, conceiving playful, often dazzling tableaux under brilliantly glazed surfaces.

Paul Thomas Anderson, however, has combined high-energy stylistic swagger with genuine social and psychological finesse. His breakthrough film, Boogie Nights (1997), a panoramic evocation of the 1970s LA porn scene, was an exuberantly retro trash-culture tribute act, yet it also delved into the anxieties of characters who might easily have been lurid cartoons.

His next big ensemble piece owed much to his hero and eventual mentor, the late Robert Altman. Densely woven, exhausting and overwrought, Magnolia (1999) was nevertheless one of the few films of the last decade that can genuinely be called orchestral in conception.

Then Anderson blindsided admirers with his follow-up, Punch Drunk Love, a perverse romantic comedy about a series of mystifying discordances organised around Adam Sandler in an electric blue suit. Less a conventional American auteur film than an expensive piece of conceptual art, it polarised critics, but Anderson cheerfully stands by it. "It's probably my favourite film," he tells me. "I look at that movie and I hope they'll look back and say, 'Well, that's the good one.'"

Blood comes as even more of a surprise – so dark and sober that it's like following up Singin' in the Rain with an Alban Berg opera. Ostensibly inspired by Upton Sinclair's 1927 novel Oil!, it's a historical drama about an empire-building oil man named Daniel Plainview, and is built around a force-of-nature performance by Day-Lewis, speaking in megaphone cadences that suggest a masterly impersonation of the late actor-director John Huston. Hence speculation that Plainview's story is a veiled "prequel" to Roman Polanski's Chinatown, an Anderson favourite, in which Huston played the monstrous magnate Noah Cross.

This is just one of the mysteries of a film in which little is clear, and almost no artistic decisions obvious. For example, there is the starkly ominous orchestral score, by Radiohead guitarist Jonny Greenwood, which at no point seems conventionally to gel with the action: Blood is as daringly counter-intuitive as Stanley Kubrick was in his use of composers such as Penderecki in The Shining. Indeed, having made several Altman-esque works, Anderson has now made one that in some ways feels as close to a posthumous Kubrick epic as is imaginable. "We're all children of Kubrick, aren't we?" Anderson admits over the phone. "Is there anything you can do that he hasn't done?"

You expect a certain seriousness from Anderson, but Blood's raw-boned sobriety is new: it is a film that has little room for levity. But that's not how the director views it. "I see so much humour in this story, it makes me laugh – at least, how absurd it all gets. I remember feeling that we can't just be grim, we have to have some humour in here. We didn't want it just to be dark. I always thought at the very least that it should be uplifting," Anderson laughs.

Given Day-Lewis's notoriously thorough methods for preparing roles, one can imagine him spending two years down a mine to become Plainview, scrabbling at rock with his fingernails (he did recently comment: "In terms of the physical preparation, there wasn't really anything to do but stay fit and start digging holes"). But beyond explaining that he provided his star with copious documentation on oil drilling and Dust Bowl-era audio recordings, Anderson prefers to draw a veil over Day-Lewis's training. "I don't think it's a director's job to peek behind the curtain too much," he says. "The work Daniel did on his own is still a mystery to me, and I like it that way."

And the origins of Plainview's distinctive patrician boom? "Daniel sent me some tape recordings on a Dictaphone he used that must have been made in the 17th century, because everything sounded so broken and old, like a home-grown American aristocracy – John Huston does have a little bit of that."

Born in 1970, Anderson was raised in LA's San Fernando Valley; his father, now dead, was Ernie Anderson, a television actor known for voiceovers and playing "'horror host" Ghoulardi, a goateed hipster character who played bongo skulls while introducing late-night chillers. Anderson named his production company Ghoulardi in tribute to him; it was his dad who gave him his first film lessons, snapping his fingers through His Girl Friday on TV as a lesson in dialogue rhythm.

In his early twenties, Anderson enrolled at NYU's film school, but dropped out after two days to carry on making his own shorts. His first feature, Sydney (1996), was a taut portrayal of an old gambler, his surrogate son and an unreliable waitress-hooker. Little seen, it's ripe for rediscovery, not least because it's one of the few films ever to squeeze some warmth out of Gwyneth Paltrow.

Boogie Nights made Anderson a star auteur at 27. He eventually came under the approving wing of Altman, who repaid Anderson's enthusiastic homages by taking him on as back-up director while shooting his final film, A Prairie Home Companion. It's rumoured that Anderson shot sections of the film, though he claims he didn't have much to do. "I sat in a chair and ate all day and watched Bob work, and once in a while he'd say, 'Tell them to do this,' and I'd run down and back. I was his errand boy. It was so much fun."

Also on Altman's film, in front of the camera, was Anderson's partner, Maya Rudolph, a Saturday Night Live regular noted for her impersonations of Beyoncé and Donatella Versace. Now he and Rudolph have a two-year-old daughter, Pearl, whose birth, he says, has changed his priorities: "As I have got older and become a father, there's less and less time for films," he admits.

Like many American directors, Anderson confesses surprise that suddenly it's become possible to feel optimistic about producing good work in Hollywood. "I could have stood on a soapbox [in 2006] and railed about what was going on, but it's hard to do that now. It's been an amazing year."

The awards success of There Will Be Blood is symptomatic of this change. It's a film that isn't remotely afraid of extremities, ambiguities and downright obscurities – which, of course, Anderson relishes. "It's a gamble you take, the risk of alienating an audience. But there's a theory – sometimes it's better to confuse them for five minutes than let them get ahead of you for 10 seconds."

'There Will Be Blood' opens on Friday

Tycoon terrors: Larger-than-life moguls who filled the big screen

Citizen Kane, Orson Welles, 1941

Inspired by newspaper tycoon William Randolph Hearst, this rags-to-riches tale remains the template for stories of Mammon and melancholia

Giant George Stevens, 1956

The descent of James Dean's poor boy made good in this epic of the oilfields is a cautionary lesson in the dangers of overnight fame

Chinatown, Roman Polanski, 1974

This blackest of LA noirs starred John Huston as Noah Cross, vampiric snatcher of Californian water and dysfunctional patriarch

Melvin and Howard, Jonathan Demme, 1980

An apocryphal rereading of the life of producer/airman/ hermit Howard Hughes, starring Jason Robards as a hitchhiking sage

Eureka, Nicolas Roeg, 1984

It's gold that does for Gene Hackman's Arctic prospector – along with magic, madness, Kabbala and a young Joe Pesci