Analytics Time Machine

troype, 01/17/2014

A typical BI Analytics implementation (and here I am talking about classic Data Warehouse, populated via ETL / ELT, and queried with Reported Tools) generates and logs plethora of day-to-day operational information, starting with database, producing workload statistics (Oracle AWR for example), ETL tools, generating tons of logged data in proprietary log file formats as well as their repositories, and Reporting Tools, producing their own logging information. If you were lucky to trace all of it, you would take backups of Informatica session logs (otherwise they would be overwritten by the next daily ETL) and Oracle BI Server Enterprise Edition logs, and increase log retention in ETL repositories as some data may get purged. And then you end up with an impressive pile of logs for parsing and mining for bad symptoms, and / or manual matching with repositories data. From my experience, most of BI Admins keep the latest snapshot only. And I agree, what is the use if you don't have good tools to digest all this information.

And then some major change happens, for example, your business decided to upgrade your primary source of data, an ERP App, which would trigger an avalanche of various upgrades, including BI Analytics Warehouse. How many times have you seen complaints about slow performance after some minor change, and what would you expect from a major upgrade? How can you measure and quantify improvements (promised by your vendors and implementers) or identify the regressions in your Analytics performance (per complaints from executive business users)?

One way is to carry out good benchmarking of your BI Implementation before and after, which is not a simple and straight-forward effort. Another option is to get some partial performance snapshots from your EPM (Oracle Enterprise Manager or AWR at least).

But wouldn't it be nice to have all that logged diagnostic information from ETL and Reporting Tools as well as RDBMS captured, parsed, integrated and conveniently stored in a single place? That's the approach we have taken with Troype APE Diagnostics. Not only does it allow to summarize and present daily detailed analytic information about one's BI implementation (ETL or reporting) and its performance, but also it captures the runtime snapshot of the Analytics environment during ETL and end users running their reports. So, BI Admins can use it as a Time Machine to compare the state of various components, drill into details, track the trends, and quickly quantify any variations in performance before and after major or minor changes in their BI Applications Implementations.

Refer to the following screenshots for more details.