What do these prediction markets produce? Some people are saying "yes" and others are saying "no", and the answer is either "yes" or "no", but why bother spending money on predicting when society can spend money on replicating it? Wouldn't an actual replication attempt be more useful?
> The latter is a perfectly reasonable thing to do - it's not easy for an average person to "pay off some lab", but if they provide liquidity to the prediction market they are giving an explicit subsidy for anyone that can answer the question.
Is any of this liquidity going to actual (replication) research, because if it is not, again: what are these markets tangibly producing? Moving a bunch of numbers around a ledger does not seem very useful.
They're producing the wisdom of the crowd, which is a real and highly accurate piece of information. It's quite difficult and expensive to produce information as fast and reliable by other means. And they don't cost much, it's mostly money being moved around.
edit: I interpreted it as asking wether prediction markets in general produce value. In this specific case I'm 100% with you, they're absolutely useless in predicting wether this finding is going to replicate or not.
BTW probably 100% useless is going to be better than trusting a single reply in a HN thread. Even averaging out a group of replies on HN is going to be pretty bad, probably worse than averaging out a group of replies on Reddit.
The idea of wisdom of the crowd is based on the idea that knowledge about a topic (both false and true) is roughly normally distributed (as many things are in nature), so the averaged result of a large group of answers is likely to be close to the real answer, as long as there are no external factors pushing the whole distribution left or right.
Also, the final result is not going to be the answer if it's gonna replicate, but more the odds of it replicating (i.e. the odds of a paper like this being legit). The odds could be 1 in a million, and it still wouldn't affect the reality of LK-99 being super conductive or not.
The opinion of a crowd is "real and highly accurate"? The opinion of crowds is frequently completely disconnected from reality. Crowds are often an amplifier of individual delusion. As for accuracy, the only thing the opinion of a crowd is accurate about is the opinion of that particular crowd (not even "the crowd" - look at election polling)
The wisdom of the crowds works given a large and diverse sample of independent predictors. People who don't know anything about a topic will vote randomly so their votes effectively cancel each other out, but people who know more about a particular topic will be biased towards correct answers.
This is correct - the overwhelming majority of people did not get involved with crypto. Even for people and companies who did most put a fraction of their money into it.
So practically speaking, what can you do with the fact that X% of fans (because people betting are enthusiasts) think LK-99 will reproduce and Y% think it wont?
You assume all the people betting are enthusiasts. The theory of prediction markets is that rational actors in the market will recognise that a portion of the betters is overhyped and adjust their bets to make use of their irrational behaviour.
If the rational actors are actually effective at making such adjustments I don't know, I bet there's statistics out there on how well prediction markets correlate with reality.
In any case, even if the market was perfect, it wouldn't tell us if LK-99 would reproduce, which I guess is the meat of your question. It would just tell us how likely it is that an experimental result made under those specific circumstances would reproduce. And what you could do with that information depends on what your answer to the question: "How would I be affected if LK-99 would reproduce?" would be.
If you're a big energy business leader, and you want to filter what topics to spend your valuable time on maybe you could set a rule that you only want spend time reading scientific papers that have >10% odds of being legit.
More realistically though, I think things like prediction markets are mostly useful to traders who are trying to arbitrage things like resource markets. What's the price of copper going to do when this turns out to be true? You could adjust your futures based on that.
Basically, making an average of a large number of estimates of an unknown value will (of course) fail if most/none of the estimators have any idea of the actual value being estimated.
I don't think I've seen that before. But the article you linked doesn't make the conclusion you suggest at all, instead they pose a corrected value. If indeed no one estimating had no information at all, the average length of Chinese person's nose would be close to that corrected value.
It's the same with this topic. You won't get an answer to the question "Is this particular paper true or not?" but you'll get an answer to the question "Are papers submitted under these circumstances making claims like this likely to be true?". The crowd will only answer the question they can answer. I think that's from "Thinking fast and slow".
It's not well explained in the version I linked (apologies, I should have looked for a clearer version).
The point of the story when I originally heard it is that no one has SEEN the Emperor's nose. So any statistical function (like averaging) of estimates is totally useless as they are all guesses.
No one has seen 'papers submitted under these circumstances' so no amount of 'crowd wisdom' will make any difference.
Also, as an aside the idea that 'the crowd will only answer the question they can answer' is ... bizarre. People will answer anything you ask them, and you have no way to know if they are just making it up.
How would you construct such a knowledge barrier? Another prediction market?
Also, suggesting that there is such a thing as 'smart' money - presumably due to having more of it? - is amusing. As pointed out elsewhere in this discussion, there has been a lot of smart money acting particularly dumb over the last few years.
A "Do you understand what superconductivity is?" or "Do you have a physics or engineering degree?" barrier?
And the relevant question isn't whether 'smart' money does dumb things: it's whether 'smart' money does dumb things less frequently than a random sample of money.
No one is an oracle, and there are absolutely outlier events that specifically confound experts, but I can't believe that increased expertise is uncorrelated with increased accuracy.
Tetlock’s Superforecasters performed better than experts, though:
>In the Good Judgment Project, "the top forecasters... performed about 30 percent better than the average for intelligence community analysts who could read intercepts and other secret data"
The opinion of a crowd is generally useless in highly technical matters. The people betting on this stuff generally do not have the background to evaluate any claims appropriately, and just react to what other people (who they believe to be more informed) are saying.
There is literally zero wisdom in the crowd about a brand new just discovered material that’s only ever been produced by one small group by definition.
That's not correct. Condensed matter physicists will have a good handle on how plausible this is (but not certain). Other people will vote randomly so their votes cancel out, effectively leaving the final result as biased by the expert opinions. That's how the wisdom of the crowd works.
> What do these prediction markets produce? Some people are saying "yes" and others are saying "no", and the answer is either "yes" or "no", but why bother spending money on predicting when society can spend money on replicating it? Wouldn't an actual replication attempt be more useful?
They are producing predictions of future value. It's not clear when you're only considering a single case, but what if you only have enough money to fund two projects and you have 15 applicants? You could pay a panel of experts to evaluate them and now you can only fund one project, or you can exploit the prediction market and fund the projects that seem to have the best chance of success according to the crowd. So in effect, the crowd is funding projects by freeing up funds that would otherwise go towards bureaucracy.
The wisdom of the crowds works given a large and diverse sample of independent predictors. People who don't know anything will vote randomly so their votes effectively cancel each other out, but people who know more about a particular topic will be biased towards correct answers.
The way resources are allocated is very important, but getting it optimal is very hard. Stock market is one of structures which helps to create long-term incentives to optimize resource allocation.
You can directly see how it works if you compare market-based economies to e.g. a planned economy Soviet Union: a lot of goods produced by Soviet industry were not in demand, especially consumer goods. When Soviet Union was no more a lot of factories were closed because they were producing some utterly irrelevant shit.
> Is any of this liquidity going to actual (replication) research, because if it is not, again: what are these markets tangibly producing?
Many economic concepts work in practice only at scale. E.g. if there's a one-off $1000 incentive, it might not attract people capable of doing that. But if there's an opportunity to make $1000 every second, people might put an effort into taking that opportunity.
Prediction markets create incentives to do particular stuff, as all markets do.
If there was enough money at stake, it could definitely incentivize replication research.
There are two scenarios.
Scenario 1: Suppose you have a lab with all necessary equipment and materials. Normally you would use it for your own research (i.e. research new materials). But if there's e.g. $100M prediction market on replication of a particular result, you might consider redirecting it to replicating that research instead.
If you do it before others, you can sell your replication proof to a hedge fund which will then get a position on prediction market before revealing the proof.
Scenario 2: If there's enough money in replication markets, hedge funds might specifically fund laboratories which replicate stuff.
In what way would the people in the prediction markets fund a replication? That's not something that normal people just do. And if I wanted to do that, I don't even know how.
> The latter is a perfectly reasonable thing to do - it's not easy for an average person to "pay off some lab", but if they provide liquidity to the prediction market they are giving an explicit subsidy for anyone that can answer the question.
Is any of this liquidity going to actual (replication) research, because if it is not, again: what are these markets tangibly producing? Moving a bunch of numbers around a ledger does not seem very useful.