Why Statisticians Reject Most of What Media call as analytics?

Why statisticians reject most of what media call as analytics and then re-classify them as reporting? I attended a Big Data Analytics seminar recently with a statistician coworker. He thought it offered nothing new in analytics as compared to what existed in the industry ten years ago. I agreed with him and yet the 200 page slideshow on advanced mining and data visualization kept flashing in my mind.  It’s a fact usage gives a word the meaning. May be it’s time the meaning of the term analytics got .  But what these statisticians are talking about?

Analytics is a fascinating term and the meaning seems to suggest many flavors as more businesses embrace it. A couple of decades back financial services companies led the systematic ways of capturing and storing large volumes of transaction level data. Then they employed quantitative analysts who identified patterns and predicted customer behavior. A well-known application of analytics from those days is a scorecard, for example FICO Score. It predicted the likelihood of a customer walking away without paying his debt and it made sense.   

What really changed is the data landscape. A recent NPR story points out two related facts. The external data storage market is now worth $70 Billion a year and companies are allocating close to 15% of their Information Technology budget to warehouse data.   Data storage is a function now many businesses subcontract to companies like Switch, Amazon and Dell and for example, the size of Switch’s server farm in Las Vegas is as large as seven football fields.  It’s easy  to visualize every click you make in a website, every swipe you do with your credit card, and every ‘like’ you do on Facebook  is  finally finds its way to one of these servers. The unprecedented growth of data results in Big Data challenges we hear every day. Analytics applications that consume these massive data are touted to provide a competitive edge to the businesses that make conscious decisions to store data.

However, existence of analytics goes back to the days before we had computers, internet and quants. Now it’s like the Wright Brother’s invention that got really complicated when the advancements in physics, material science and aero dynamics wrapped around it. Who does not like a Dreamliner (not an endorsement).  Same way, advancements in Information technology, hardware and software alike, transformed the ways businesses capture, store and retrieve volumes of data. And they did it lot cheaper than in the past. Data mining and statistical  techniques, mostly used in academic and research settings, found many applications in in this new found data world and flourished  as analytics. 

Benefits of analytics are often exaggerated by the tools and solution vendors who benefit monetarily when a business decides to take the ‘analytics’ route. I found these vendors and their sales literature adds tremendously to the sloppy use of the term analytics. Let us look at some long words and sentences.

Applications providers, like SAP, emphasize that their work-flow solution is configurable and ready for analytics integration and business intelligence . When database vendor Oracle extends R capabilities, Greenplum, another cool database vendor, is driving the future of big data analytics by integrating Base SAS libraries at database server level.  When we hear partnership with SAS and R, the names long associated with reliable statistical modeling and analysis, it’s quite convincingly implied that they are providing analytics. A world of reporting tools, like MicroStrategy, SAP Business Objects or Hyperion provide data mining, business intelligence and analytics that leverage on multi-dimensional databases and brings out insights to the management in a drill down-roll up fashion.

Once you peddle through the jargons, they, Google analytics included, are still talking about reporting that just got extravagant with all the technological advancements.  It is hard is to define the term analytics without offending a lot of people. A lot of people already claimed their stake in this ‘next big’ thing. Not related, but an MIS division head I know recently changed his title to ‘ Data Scientist’. When I checked what changed, he said they hired a consultant to do Hadoop and Microstrategy for them.

Coming back to the main topic:  what’s core analytics means?  Yes I added ‘Core’ to emphasize. Probing deeper, it sounds like they mean predictive modeling or predictive analytics.  A set of old fashioned test-control- validation exercise using various statistical techniques like logistics regression, survival analysis, classification trees or even machine learning. Wiki does a better job in explaining this.  People who do such work are often called modelers and they want to differentiate themselves from a set of IT or near IT guys who primarily deal with reporting systems.   The seemingly simple issues these people deal with are not solved by the smartest visualization software – Like what is a statistically sound substitution to use for missing values in sample data?   How to derive some performance for the customer we never had? – There are hundreds of articles published in journals on these topics and hundreds were awarded PhDs. But still there is no agreement. Since there is no one rule and generalization is not a possibility, software cannot hide it under a layer. From data side, such analytics are often supplemented with data that’s not available in corporate Hadoop Big Data mine. So they don’t believe an off-the-shelf application sitting on that mine is going to get the things done. There is a difference in deliverables too. The so called generic analytics applications provide reports, tracking dashboards or warning systems.  Core analytics deliverables are sets of rules that sit in an application and acts like an expert.  Say a scoring model that replaces an underwriter.  When humans learn from the new environments and their own mistakes, an expert system pretends that what it had gleaned from the past is still sound. Moody’s AAA rating of junk bonds in 2008, is an example.  (I know genetic algorithms and artificial intelligences counter argue, but they cannot detect human lie).

The paragraph above attests how drawn-out and jargonized this topic is. Liberals have no place in such discussions. 

Then there is an overlap of these two worlds.   Google Analytics for example, it’s possible to setup a test- control strategy to see what works best in real life (A/B Testing).  Many analytics applications can be configured to work dynamically, for example: Amazon recommendations or fraud detection, are trained on the fly but rules behind them still lying in the disputed land.

I counted at least a dozen times the word analytics used in our meetings last week. It almost always meant some numbers to support an idea or argument. A report. With that statement, this topic is open for discussion.

S&P Downgrade of US and my stock

635 points drop in Dow Jones index was not expected though the Friday closed with lot of discussions on European economies and most importantly S&P's downgrading of US debt rating to AA+. S&P dropped the ball so many times so market appeared to know them better. Throughout today, market was buying US treasuries and gold as a safe haven ( read not the treasuries from AAA rated sovereigns) and that made me believe that market is going to stay at a 300 point loss levels. But appears the market sentiments are now driven by something stronger than S&P rating .

So those who stay invested can now watch whats going to unfold in a few days. Bank of America and Citibank are battered for no reasons. May be its better to book the loss tomorrow and hope to buy it back cheaper later.

Health Savings Accounts & Retirement Savings

An article I recently read about the 401K plans ( The Great Stock Myth ; The Atlantic) concludes that betting only on your 401K saving may not be a good idea as the historic returns are not even close to whatever we all have in our minds. With the kind of fluctuations we observe in the stock market(In other words, zero sum game), what makes the 401K attractive is the tax differed contribution and the employer match. Surprisingly Health Savings Accounts(HSA)s also come with similar features and hence provide a good avenue to park your savings for any future medical expenses.

Health Savings Accounts (HSA) are available since 2003 for the Americans who are holding high deductible health plans and more recently companies are encouraging the employees to sign up for HSA by contributing lump-sum dollars on their behalf. Of course it helps the companies to save on the premium they are paying to insurance companies but I think this plan overall helps in reforming the halthcare system ( Since this was introduced in 2003, the credit probably goes to George Bush but Obama did modify this plan to plug some of the misuses)


HSA is not available to everyone and its contributions are always associated with a high deductible health insurance plan . For example, if your health insurance plan requires you to pay-up the initial family medical expenses up to $2,400 (as of 2010) you are eligible to participate in an HSA. By opening an HSA account, you could transfer up to $6,150 a year tax free for meeting your medical expenses in the future. HSA is offered by most of the banks and are just like a savings/investment account with a debit card and check book. There are a few features I found very attractive as compared to a traditional plan.


1. Premium for your health insurance is lower because of the high deductibles and normally the out of pocket maximums are also lower than a traditional plan. You can use your tax advantaged HSA account to meet the deductibles/Out of the pocket maximum. Normally , employers contribute a percentage of the deductibles so there is always some funds to start with.

2. HSA balances are like a savings account and any unspent money is for you to keep for the future. Since young people tend to incur less of medical expenses, this help them to accumulate whatever unused funds from previous years to the future. Even if you switch to a low deductible plan in the future, you could still use these funds for eligible medical expenses for you or your family.

3. Your contributions are tax differed and based on you income this could be up to 30% return. Most of HSA providers allow you to invest these balances in Mutual Funds and other eligible instruments. So it works just like your 401K. In case you make a withdrawal for non medical expenses, you will have to pay a penalty of 10% and the income tax. Normally such withdrawals are made when you have less or no income so you will end up paying less income tax.

The best part I liked in HSA is that it makes you more responsible in spending on health care. For example, I now want to know how much the doctors are charging me . Some of my colleagues are negotiating better rates with physiotherapists and Chiropractors and the money is going from their savings. And of course I have a peace mind that if the expenses are exceeding the out of pocket maximum, insurance is going to take care of that.

Frankly, most of the working people can meet their expenses for primary care but its hard to be prepared for catastrophic medical situations and HSA seems provide just that.

Confused new home buyers

New home buyers first stumble at the decision point of buy or continue to rent and it invariably ends up in an elaborate spreadsheet exercises to compare the tax benefits, cost of renting, expected appreciation and outside the spreadsheet, the thrill of pursuing the American dream. Once decided, the next grinding stage was looking for a dream home -good neighbourhood, good school districts, commuting distance to workplace and daycare , construction quality and the number of bedrooms and a host of other things. But recently the complexity of this process increased due to recession, changed terms of loans and the new foreclosure and other laws. I think the recent shake-ups in the industry got the new buyers confused to the core.

The recent foreclosure mess is a good example for things that added to such confusion. Its true that most of the banks did not do the due diligence before every foreclosure as their status as a mortgage servicing company did have more benefit in foreclosing a loan rather than modifying it . In other words, Obama administration did know that as a result of the securitization process ( Wall Street taking over the mortgage receivables and selling it to varied investors as a packaged investment ) there was less incentives for the mortgage servicing companies to modify the loans and hence offered a $700 Million package for doing the loan modification. The recent fiasco in foreclosure and the fact that all 50 state attorneys suing the major banks would change this incentive system. Now the foreclosure means more paper work and legal process where the cost of this will outweigh the benefit of the foreclosure in comparison with the loan modification. May be its good for the current mortgage holders and industry in general as every foreclosure adds a home to the already surplus inventory of the homes.

New buyers are puzzled at two things
1. are the prices at bottom? Moody's say its not and people are just waiting for a right time. Also, the income levels did not change in last five years so buyers feel that prices will eventually have to correct to justify the income levels in a no growth economy.
2. Will the rates change? There is a feeling that's the interest rates are probably at the bottom now but the mortgage prices are not.

In addition, there are more foreclosed homes that are added to the supply of homes and there is always a better deal in waiting. May be the new foreclosure messs will help remove that confusion.

Healthcare reform bill

I was following the healthcare bill of Obama close but not that excited about it. There is no doubt that the bill helps increase the number of people who are insured and soft on pre-exiting conditions - a thing that only rich countries could afford and US should provide basic health care for all its citizens as a privilege. Apart from that, there is nothing in the bill that tries to fix the current problem. I am for the government proving health care thought its on network of hospitals to act as a benchmark for private sector and also to provide the Medical education free based on the meritocracy. I think if the government wants to fix the problem the should start with doctors by providing free education and these debt free doctors bring a new perspective to the overall health care system.

Toyota

Think about it. When you press the accelerator, the pad gets stuck. Its definitely scary for a regular driver. I always thought accelerating is a straight forward operation but Toyota story now tells us its not. Rather, its so complicated and has become a threat to Toyota's long enjoyed reputation for quality. Halting the production in the US plants only made the consumers more nervous, though it was the right thing to do.

Though Toyota CEO acknowledged that the company was chasing the numbers and compromised on quality, I feel there was also a 'pull' factor from US consumers. Every other person in the US wanted a Toyota and dealers did not mind stocking Toyotas in large numbers. The only way for Toyota to deliver such large numbers were outsourcing parts from across the world and probably that's where the things started going wrong.

Though Honda stands to gain from this mess, I feel it helps Ford better to emerge as a profitable business in 2010. Also, their product lines are more stylish and better quality. Especially when it comes to trucks and SUVs , Ford has some good products American consumers love to take look at.

Volker Rule

Bad part of Volker rule was that I lost a couple of grants in the unexpected stock market crash. I am not alone, everyone in the lunch table today was whining about bad timing Obama chose for this announcement and how the Volker rule and now doubtful Health reform bill will make him a failure. On a side note, I really wonder the President understands the amount of bad mouthing happens when the stock market goes down a couple of points. This one thing is, probably, the most significant, contributing variable in to his approval rating.

Some channels were showing the way stock market tumbling as Obama started speaking and by the time he was finished, it was well below the day's open levels.This reminds me of an old Saturday Night Live show where Mr. George Bush (Will Ferrel ) says that he decided to speak only in the night because whenever he speaks during the day, the stock market goes in to the crapper. I hope that's not the case with Obama as I have no plan to track his public appearances -:)

Jokes aside, I think the 'Volker rule ' is a formal acknowledgement from the Government that the taxpayers were held hostages during the crisis. If implemented right, these measures could bring the credibility back to the US financial markets(of course with a bad hit on their bottomlines). Its not just Paul Volcker, otherwise moderate Geithner also agrees that such regulations are required to minimise the future risks. Volcker Rule was announced just after the "Financial Crisis Responsibility Fee" and I guess that's what pissed the markets off. We will have to wait and see what's the real impact of Volcker Rule on the financial institutions and how diluted it would be when it comes for implementation. I still want to believe that Obama team wants to implement some sensible policies to avoid the future troubles and if it was just for a political mileage, they were better off without this announcement.