Something I've noticed with both my own past startup failures and other startups I've known: by far the most common failure reason is "There was no reason for them to be a company in the first place."
By that I mean that either there was no customer demand for what they were building, or there were already lots of other companies that solved the problem just as well and they had no unique angle on the problem, or a key technical assumption they were relying on turned out to be false, or the market was better served by lots of little firms rather than one high-growth startup. In other words, they never found product/market fit, because there was either no market for the product or they couldn't build the product to serve the market.
The problem is that usually you can only determine this in hindsight. If everybody assumed that the only businesses that can work are those that already have a working product and customers, we'd never get any innovation. I've learned to think of "Finding a reason for the company to exist" as the primary job description for a founder, and failure means that you are doing your job but haven't completed it yet.
My hunch after starting a few side projects, and taking the leap in to a full on failed company is that while things must be tried, you should secure some level of real customer interest before diving fully in.
I'm not sure how much it takes, but there should be real evidence.
Most startups that outperform start in areas where the reason to exist isn't obvious. It only becomes obvious later. (Do we need yet another search engine?) This is why it's important to have a lot of low-capital companies searching these markets. Once they find the reason, then it's appropriate to flood them with capital.
A valid question even when Google came to the fore. We had Yahoo, HotBot, Altavista, and then Google came along. And while the others were search engines, Google was superior.
Will people switch to a better internet search engine ?
Sad fact, but true. Most companies simply have no need to have been started, and the failure stats bear this out. But as you also state, in aggregate, this process is necessary or time stands still.
What's interesting thinking about history, would Webvan and Pets.com fall under "no demand", "too soon", or just tried to do too much given the timing? Now we have services that are similar, for instance chewy.com is the modern version of of Pets.com.
"Too soon" is usually another way of saying "A critical piece of infrastructure that my business plan requires doesn't exist yet."
Instacart's founder is fond of saying that Instacart couldn't have existed before 2012. I suspect what he means is that there were a collection of technical & social changes that happened in the early 2010s that let him recast the problem Webvan was solving in an economical way. These were: 1) smartphones allow real-time coordination across thousands of workers, without hiring lots of managers 2) cloud-computing lets you run big-data algorithms to feed those instructions to thousands of workers, without building data centers 3) because of the Great Recession, thousands of workers were unemployed and desperate for some way to earn money 4) increasing urbanization has clustered people together in a city and made them disinclined to drive, which increases the demand for a grocery-shopping service and decreases the cost of servicing them and 5) everybody had Internet access and cellphones.
By contrast, WebVan spent a billion dollars building warehouses, at a time when the total number of Internet users was < 150 million. They bought their own fleet of delivery vans and hired their own drivers, at a time when employment was full and labor costs were high. They had to invest much more capital for a much smaller market, and then had much higher variable costs.
Being a "startup" doesn't repeal the laws of business - you still have to pay for labor, generate returns on capital, generate more value than you charge, and charge more than you spend. But because computers operate millions of times faster than humans and don't require wages, if you setup the business model right you can realize huge efficiencies of scale. "Setup the business model right" is the tricky part - WebVan thought that "Internet ordering" was a crucial part of the business model for online grocery delivery, but it turned out that "use existing supermarket infrastructure" and "coordinate lots of shoppers so they can work very efficiently" were the real keys, and the technology to do that hadn't been invented yet.
As evidence: I can get a 35 pound bag of dog food delivered to my house in about an hour via Amazon Prime Now. That's basically both WebVan's and Pets.com business model but on the back of Amazon's logistics ecosystem.
What makes this work is both that it's not 2001: both in that there are significantly more Internet users, they have higher speed connections, and they're much more comfortable with online purchases.
Totally this. I lived with my aunt in the bay area back when webvan.com was hot, and I remember a time when the two of us saw a commercial of theirs. I remember saying how great of an idea it was, that it could really take off, and I distinctly remember my aunt -- not a very prolific internet user but not a luddite by any means -- saying that it was weird buying groceries online. That really stuck with me, because my aunt is/was the kind of person to try something like that, but even she wasn't comfortable with it.
Plus also webvan.com blew a billion dollars building a delivery infrastructure that they couldn't support.
By that I mean that either there was no customer demand for what they were building, or there were already lots of other companies that solved the problem just as well and they had no unique angle on the problem, or a key technical assumption they were relying on turned out to be false, or the market was better served by lots of little firms rather than one high-growth startup. In other words, they never found product/market fit, because there was either no market for the product or they couldn't build the product to serve the market.
The problem is that usually you can only determine this in hindsight. If everybody assumed that the only businesses that can work are those that already have a working product and customers, we'd never get any innovation. I've learned to think of "Finding a reason for the company to exist" as the primary job description for a founder, and failure means that you are doing your job but haven't completed it yet.