World Cup 2018 failure underlines looming crisis in the United States

When I lived in America I came to firmly believe that the US would win the World Cup before England won it again. Only time will tell if I'm right, but it won't be happening in Russia in 2018 after Bruce Arena's side failed to qualify for the finals.

This will be the first World Cup finals the USA have missed since 1986 – a better attendance record than England, who failed to qualify for the 1994 finals (which were, of course, held stateside). The US's cataclysmic 2-1 defeat to Trinidad & Tobago means they, along with their Caribbean hosts, are the two teams automatically knocked out of the final regional qualifying group.

That the US regularly qualify is not entirely unrelated to the preferential treatment their continental region is afforded by Fifa. The Confederation of North, Central American and Caribbean Association Football (Concacaf) is awarded three automatic berths and the opportunity of a fourth via an intercontinental play-off.

Football-mad Africa receives just five finals places and the world's most populous continent, Asia, is only assured of four. Next summer, Fifa's reward for awarding all these places to Concacaf will be appearances by Mexico, Costa Rica and Panama, plus Honduras if they can overcome Australia over two legs in November. Great news for the minnows of central America, bad news for the coffers of world football's governing body.

America's riches and its corporate clout are what tempts Fifa into giving them preferential treatment. Many of the tournament's global sponsors are US-based corporations. Also, there is the huge money to be made via TV rights in the planet's richest sports market.

How America's Fox Sports executives will be kicking themselves now after bidding $400m to snatch the rights to 2018 and 2022 away from rivals ESPN. Soccer has been a growing battlefield in US broadcasting – with the other indigenous sports NFL, MLB, NBA and NHL tied up in long-term rights deals or costing unbelievable amounts of money (the current deal for American football is close to $40bn).

Now Fox will be wondering how it will make back the money it paid for the World Cup finals when the games will be played during America's less-profitable daytime TV hours – and won't even feature the national team

This failure to qualify is also a hammer blow for the kudos of the sport which has, mostly, been on an upward trajectory in recent years.

The sport's domestic league, Major League Soccer, has slowly but surely gained a foothold in the nation's sporting psyche. The game is incredibly popular among elementary school children and the women's game is among the best in the world. But America's youth tends to drift away from soccer at high school level for a wide variety of cultural reasons, and the structure of MLS means that the professional game has failed to truly spread its roots across the country.

The men's and women's teams are the symbol of American soccer power. The men's side have attempted to match the women given Americans the opportunity to support their country on the international sporting stage, something which doesn't happen in the other major US sports. Now that four-yearly opportunity will be missing.

Netherlands and Chile have also failed to qualify for the 2018 World Cup finals and there will doubtlessly be much debate about where they go next. But neither country will be left debating whether they actually care about football.

That soccer in America has an ongoing existential crisis is a sign that it is still the poor relation to the Big Four sports and that memories of the game's sudden rise and fall in the 1970s are still lurking. The MLS is only 21 years old and still suffering growing pains, such as the inability to even establish a team in Miami, despite the desperate attempts of the league and David Beckham to do so.

Jürgen Klinsmann was brought in with a brief to raise the game's profile and give it a stronger, longer-lasting structure. He had many successes during his tenure; victories over Germany, Italy and Holland were celebrated. USA also came fourth in the Copa Centenario and reached the round of 16 in Brazil 2014.

As the USMNT stuttered in its campaign towards the 2018 World Cup Klinsmann was sacked last November and replaced by one of the old hands of American soccer, Arena. There is much symbolism to be found in this managerial change. Klinsmann was a World Cup winner brought in to give the American game a global profile. Arena is a much-respected figure within the American game, but has no experience of the sport outside the US.

After the debacle at the Ato Boldon Stadium Arena declared: "There's nothing wrong with what we're doing. Nothing has to change. To make any kind of crazy changes I think would be foolish. We're building a good system in our professional league."

But the mostly MLS-based team failed to qualify and its brightest star, Christian Pulisic, plays for Borussia Dortmund. The brief era where a raft of American stars lit up the Premier League, most notably Clint Dempsey and Tim Howard, has passed. Is MLS strong enough to produce a team capable of competing on the world stage?

The seemingly never-ending debate about soccer's place in American culture goes on. That may well be underlined nest year when Fifa rules over the destination of the 2026 World Cup. The options are Morocco and a joint Concacaf bid led by USA, Canada and Mexico. It means the country are running out of time to ensure the sport is fully established and that tournament does not go down the whimper.