Shooting in the dark

Vicky Cristina Barcelona, however, is scarcely a comedy. True, it breaks with the awkward skulduggery of his London films — the cycle that began with Match Point (2005) — to return to a style akin to what could be called his "middle period" — think Crimes and Misdemeanours (1989) or Husbands and Wives (1992). Like those films, Vicky Cristina Barcelona has a surface froth: a picturesque setting, jaunty music and an arch voice-over relating the story of our two travelling Americans, Vicky and Cristina. What lies beneath the froth, however, is a bottomless quagmire of pessimism.

Allen has always liked a tragedy. "I like the idea of doing serious films. You know when you do a comedy there is a monster on your shoulder saying, 'Come on, you've gone two minutes without a laugh.' You can't survive it … When I pick out a film to do, I will pick the one that is interesting but also hopefully black and tragic and complicated, because that is fun for me."

Vicky Cristina Barcelona is thus serious business. Sensible Vicky (Rebecca Hall, also nominated for a Golden Globe) and yearning, dissatisfied Cristina (Scarlett Johansson, Woody Allen's current muse) are both transfixed, in their separate ways, by Javier Bardem's Juan Antonio. He is a smooth operator with a textbook Latin chat-up style who appears at their restaurant table to invite them to his country retreat "to see the sights, eat and drink well and make love". This unlikely tryst is thrown into disarray, however, by the arrival of Juan Antonio's former wife, the explosive Marie Elena, played by Penelope Cruz as a woman forever teetering on the edge of something way more explosive than a nervous breakdown.

Allen holds out little hope for any of these characters. Vicky is too scared to take life by the horns; Cristina will never know what she wants or who she wants.

"There's a discomfort in her, there's an anxiety in her that she attaches to every relationship sooner or later, and thinks that it's the relationship when, in fact, the shortcoming is in her." Juan Antonio and Marie Elena are passionately in love, but "the chemistry between them is never going to work", he says. "He'll always feel that she's the greatest woman he's ever known when she was healthy, but she's so rarely healthy. And she'll always be attracted to him because he's such a charismatic personality, but she's too volatile to have it work out. It's sad."

As always, there is a guessing game going on over which character in this film represents Allen himself, especially given that he no longer plays his own leads. Does Vicky speak for him in the voice of reason? Or is he sleazy Juan Antonio? Neither, says Allen, who always maintains that his films do not represent him, literally or emotionally. "Sometimes when I'm feeling my happiest and everything's going well in my life, I'll make my darkest, most depressing kind of thing," he says. The early slapstick comedies, by contrast, coincided with "not a good time in my life".

There are plot elements of his films, however, that so clearly mirror aspects of his own life that the parallels seem undeniable. One of his most successful films, Manhattan (1979), tells the story of a middle-aged writer's relationship with a 17-year-old girl — not that this is in itself portrayed as dubious — played by Mariel Hemingway. At the time, Allen was rumoured to be involved with a high-school student, Stacey Nelkin, who had a bit part in Annie Hall.

Allen's fascination with much younger women remains quite brazen, both on screen and in real life; famously, he is now married to his former partner Mia Farrow's adopted daughter, Soon-Yi Previn, who is more than 30 years his junior. While not an illegal relationship — he never adopted Soon-Yi himself — the match seemed creepily inappropriate, especially when Farrow subsequently accused Allen of abusing the seven-year-old girl they had adopted together. There is an unpleasant frisson, therefore, when he talks about how pleasant it was to come to the set of Vicky Cristina Barcelona each morning to gaze upon Scarlett Johansson, even though that would be true for anybody. He may see his life and his art as separate, but we can't.

There is, however, no disputing the fact that Allen continually revisits lines of inquiry in his films that preoccupy him in real life. Whoever his recent run of confidence tricksters, murderers and ruthless corporates — in Match Point and his subsequent film, Cassandra's Dream (2007) — may or may not represent, for example, he will happily admit to fantasies that feed the murders they commit.

"I could give you a long list of people I think the world would be better without, but I don't have that size of character," he says equably. "I'm a sneering little writer in a room who imagines these things. I don't have the tragic dimensions of someone who would actually go out and perform the act, but I do dream of it."

The doomed romantics in Vicky Cristina Barcelona, meanwhile, embody a theme he has been periodically revisiting since his first "serious" film, the gloomy Interiors (1978). Allen is intrigued by the tragedy of the would-be artist who finds he or she has no talent, represented in this film by Cristina. She has just made a short film she recognises is without merit and now has no idea what to do with her life.

"It is a subject that has been on my mind for many years," Allen agrees. "It always seems so sad to me that there are many people out there full of feelings for things, ideas, about life, about the universe and poetry, and they want so badly to express them and they can't because they have no musical talent, no graphical talent and no literary talent. It's a terrible position to be in."

In a sense, he sees this sense of missing the mark as a malaise endemic to Americans. The fact that he has been able to make his last four films in Europe — three in London and this one in Barcelona — goes some way to fulfilling a lifelong but actually impossible dream of being a European filmmaker, a little Yankee version of a Bergman or a Fellini. To work with foreign actors, to shoot in another language and another country, brings him closer to that idea.

"But the truth of the matter is that there has always been a mystique about Europe for Americans. They always feel it's romantic, it's mysterious, it's free sexually, it's bohemian, it's in advance of the United States. They feel you go to Europe and it's a deeper, more cultural, more artistic life and I think that, for many of them, that's true."

Allen's pessimism about his own existence goes far deeper, however, than a mere cultural cringe. Behind the constant scrabble of work, he sees life as a meaningless void that no amount of art, music, books or romance can fill.

"Everything I've done will come to nothing," he says, "in the same sense that we are born but nobody knows what we're doing here. Everybody lives a life full of anxiety and terror, full of sorrow and problems, even the luckiest ones. And I consider myself one of the luckiest ones. And finally what happens is that you age, you get sick and die. So all your striving and accomplishment: what does it mean? Nothing saves you from your fate, which is annihilation with no point."

But while you cannot escape nothingness, at least you can distract yourself from looking at it.

Woody Allen makes films, he says, because he can. "It's an interesting job and I'm not suited to a lot of jobs. But the other reason is that it's a great distraction: it takes up most of your time for a year.

"And you work in a pretend world. I leave my house and then I could be working with gangsters or cowboys or chorus girls, or it can be charming people who live in very rich places.

"I like to get away from reality, just like the viewer likes to get away from reality for an hour and a half while he watches a film."

Maybe only three of his films are any good, he suggests — or five, or 10, whatever — but those are the ones people remember. "And over the years, a kind of mystique has built up that isn't really based on anything. If you throw a million things at the wall, some of them will stick. That's how you keep your life in films going and keep getting finance."

Ingmar Bergman, his great hero, reached a point where he announced he was making his last film — Fanny and Alexander — then retired to his island off the coast of Sweden. Allen, who spoke to Bergman often on the phone but always declined to visit him, cannot imagine that. His diligence will die with him.

"I could envisage a scenario where I woke up in the morning and couldn't raise finance for a film," he says soberly. "They say, you know, your films are too terrible or you are a has-been, nobody wants to see your films and everybody hates you. But when and if that is the case, I'd still work. I'd write a book.

"I can't picture myself getting up in the morning and just enjoying myself. What would I do? Walk around, go to the museum, have lunch? I'd go crazy if I did that."

Vicky Cristina Barcelona is out on December 26.

-Stephanie Bunbury