I was wondering about the extent to which blog chatter correlates with information from other sources such as prediction markets. This is pretty difficult to test, because the things that people write on blogs can be hard to decode and categorise. But I think I found one reasonable and straightforward test case: use of the word ‘recession’ in blogs versus the probability of a recession from a prediction market. The theory is that when a recession is more likely, people should blog about it more, and the level of recession chatter should be positively correlated with the probability of a recession predicted by a prediction market.

So I got the daily counts of the number of times ‘recession’ was used in blogs, according to Technorati, and the probability of a US recession in 2008 according to Intrade. The overlapping data for both series covered about the past 90 days. Of course, people could blog about recessions in any country, not just the US, and people could use the word ‘recession’ when they’re saying something like ‘a recession is unlikely in 2008′, so I expect the relationship to be far from perfect. Here’s the basic data that I got, with the count of the word ‘recession’ on the left, and the Intrade probability on the right:

Here’s a scatter plot of the two data series together, with a fitted logarithmic curve (it fits slightly better than a linear relationship):

For those econometricians reading this who may be thinking spurious regression, I did a Phillips-Ouliaris cointegration test on the fitted model above and the p-value was 0.079, so it’s borderline whether this is spurious or not, but I’m willing to believe there’s a meaningful relationship.

An interesting quote from an interview with Hal Varian in the Real Time Economics blog:

I think marketing is the new finance. In the 1960s and 1970s [we] got interesting data, and a lot of analytic fire power focused on that data; Bob Merton and Fischer Black, the whole team of people that developed modern finance. So we saw huge gains in understanding performance in the finance industry. I think marketing is in the same place: now we’re getting a lot of really good data, we have tools, we have methods, we have smart people working on it. So my view is the quants are going to move from Wall Street to Madison Avenue.

(Ok I know the interview is from 3 months ago, but I just found it).

In case you don’t know, Hal is now the chief economist at Google. Not to be outdone, Yahoo! has hired Preston McAfee and a bunch of other smart people. Hal, Preston: If you need more people, please drop me an email :)

Anyway, I agree with Hal. The Internet and e-commerce have generated fantastic volumes of data suitable for demand estimation, which is at the heart of quantitative marketing. The basic data you need for this is information about what people purchased and what they paid. Sites like Amazon must be sitting on an absolute treasure-trove of such data. The analytical tools are getting quite sophisticated too. I’m more of a theory person but recently I’ve been reading up on the ‘BLP method‘ for estimating demand, and judging by how complicated it is, it must be pretty sophisticated. (See also the recent podcast on EconTalk with Ian Ayers, author of Supercrunchers).

It seems inevitable that businsess will become more and more quantitative as such methods for data analysis become more well known. I was thinking about what students should study to get the skills necessary to compete in this quantitative world. I think a graduate degree (’postgraduate’ if you’re European) is probably essential — at least a Master’s degree, if not a PhD. In terms of specific things to study, I came up with the following list

• Economics (of course), especially microeconomics, econometrics and game theory. Study to advanced graduate level.
• Statistics: Study theory to advanced undergraduate level or graduate level, plus some applied courses in data analysis. Get good at using software like Eviews, SPSS, or R.
• Applied mathematics: To at least intermediate undergraduate level. Learn how to use Matlab or Mathematica.
• Marketing: Take some quantitative courses, plus maybe a bit of theory.
• Computer programming: Take some computer science courses to intermediate undergraduate level. Learn one programming language quite well (it doesn’t really matter which one).
• Information systems: You need to know how to converse with databases and write scripts. Basic or intermediate undergraduate level should be enough. Or just play with MySQL and learn by yourself.

Maybe this list sounds a bit tough, but remember: If it’s easy, it won’t make you rich.