This isn't a new thing. One thing I did at Enron was to try and figure out the construction status of power plants being built I read a lot of small local newspapers, city council minutes, looked for job listing, and called subcontractors etc. The fun side of trading involves getting a view of the market which is beyond newspaper articles and reports.
I've heard of another Enron project where they installed EM devices on land adjacent to power line ROW, to measure the current. If you understand when certain lines are near capacity you can get a better price for your supply over other lines.
That sounds really interesting. How hard do you think it would be to automate that kind of analysis of newspapers and such? Would love to hear what you think. Email's in my profile.
Sentiment analysis is quite easy given enough source material. However, this type of open source intelligence requires skilled researchers. A lot of the relevant sources are not machine accessible, and the investigative process is driven by hunches and incomplete qualitative data. Even distinguishing between relevant and irrelevant source material can be a difficult task for the most skilled researcher.
A lot of newspapers have horrible websites. Aside from aesthetics, search is broken on many of these sites. Another problem is local government, city council minutes and other information is not made available in digital format.
I work at a newspaper that is a controlled by a corporate office across the country. (They know best) I also live in a small town that has been slow to adopt technology.
Given the signals intelligence of small town newspapers I always wondered why there wasn't a company that gave out really cheap 'site build/management' software for these folks with a proviso buried deep deep deep inside the terms of service that gave the hosting company a right to see all the data.
I'd imagine it looks a lot like WPEngine but with a more newsy flair.
