Looking backwards over a decade of Microsoft BI

It was in early 2000 if memory serves that I first used OLAP Services in SQL Server 7 and Cognos Novaview 1.0 and ten years later I wondered, as you do at this time of the year, what has changed and what will the future bring? and specifically  is there a future for the BI professional with the dawn of BI in the cloud?

What did/do people like me actually do?

  • All the usual tasks that surround any engineering project form gathering requirements, to testing and training users to get accurate results and insight form the system.
  • Export Transform & Load (ETL), this is a huge area encompassing data quality, integration of data across disparate systems and encoding business logic
  • Possibly design a pack of reports/dashboards/ analytics or provide some user interface to the data.  This ranges from using off the shelf tools and wizards to custom development (e.g. in Silverlight) to create tailored rich experiences for the users.

Even without mentioning the cloud these task have changed over the last ten years I have worked in BI:

  • DTS was fun to use and a swine to debug but the power of integration services made the initial design more important and allowed for a much more robust and flexible solution to load data.
  • OLAP services also got completely overhauled in SQL Server 2005; it got the Kimball bus architecture, and a host of powerful new features like many to many dimensions, but this also took it further away from the business user as did the new BI Design studio which was clearly aimed at the BI professional.
  • Reporting Services arrived in 2002 and this has become more and more user friendly while retaining the power and flexibility to be embedded in applications and deliver rich reporting at scale.

SQL Server 2008 R2 arrived last year and attempted to address the need for the business user (or information worker in Microsoft speak) to design their own analytics.  This new tool PowerPivot introduced a new column based in memory analytical engine (vertipak), which is simple and fast at the expense of the power of  traditional analysis services in such areas as fine grain security control and the development of really complex business logic.  This will be up-scaled in SQL Server vNext (aka Denali), but will exist in parallel with analysis services.

Assume for the moment all of this and more will be available at some point in SQL Azure/Office 365 or something like it – where Microsoft BI is offered as a platform as a service (Note: This is pure conjecture on my part) what will change for the BI Professional?

I don’t see any of the fundamental tasks changing, all of the promise of the cloud is good as I never had to worry about setting up infrastructure on most of my projects, indeed often I was never allowed near it and had to put in change requests to get accounts setup and access to data . And that’s a good point where is the data in this new world?  If the data is on premise it will have to be moved to the cloud, presumably using some sort of cloud based integration services and we could be talking about a lot of data. 

A lot of of the presentation layer is already built on web services so moving that to the cloud will make little or no difference to those, for example SharePoint is in Office 365 although PowerPivot isn’t there yet that can’t be too difficult.

That leaves the design tool which are typically rich clients, like BI Dev Studio and Excel , and these will have to stay on our desktops, it will just be a question of having the right credentials to deploy to the cloud , for example PowerPivot can already load data from SQL Azure and BI Dev studio will allow you to design reports for SQL Azure shortly.

The point about all of this is that the role won’t change that much even if all of the services are available through SQL Azure + Office 365, data will have to be cleansed, and transformed, and users will need support and guidance on how to turn their data into meaningful insights. This means working with the clients and understanding their culture and this in turn means work where they are.  This can mean expensive travel and working away form home, but if it can be done anywhere then anyone can do it remotely, which is one reason why off-shoring has only had a limited impact on the BI services industry in the UK.

The appetite for BI in a tough economy shows no sign of declining with the exception  of the public sector, and even here some large programs may well have been scrapped but administrators at all levels looking for savings  will still need BI to assess the impact of any cuts and to prioritise them. 

So I am pretty sure that the next 10 years will be interesting and possibly disruptive but at the end of the next decade people will still need reports and analytics and so they’ll still be a role for us.