We are developing tools and algorithms to automate complex data analysis, reducing costs and increasing results. In December 2018, we released the first version of AdEvaluator™, a program that scientifically evaluates whether and to what extent advertising boosts sales and profits using the actual sales data from accounting programs such as QuickBooks.
Complex data analysis is a multi-billion dollar business. Major data analysis tool makers alone report revenues totaling over $4 billion per year: SAS Institute ($3.2 Billion), IBM SPSS ($0.3-1.0 Billion), MathWorks ($850 Million), Wolfram Research (at least $40 million), and a number of less well known smaller firms. Medical businesses, financial firms, and science and engineering organizations spend billions of dollars per year on these tools and the salaries of the analysts, scientists, and engineers performing the analyses.
Complex data analysis increasingly determines the approval of new drugs and medical treatments, medical treatment decisions for individual patients, investment decisions for banks, pensions, and individuals, important public policy decisions, and the design and development of products from airplanes and cars to smart watches and children’s toys.
State-of-the-art complex data analysis is labor intensive, time consuming, and error prone — requiring highly skilled analysts, often Ph.D.’s or other highly educated professionals, using tools with large libraries of built-in statistical and data analytical methods and tests: SAS, SPSS, MATLAB, Mathematica, the SciPy ecosystem (numerical and scientific extensions to Python), the R statistical programming language, Excel and similar tools. Salaries and overhead for these analysts range from $40/hour to $200/hour (using a 25 % overhead rate), sometimes even more.
Total Cost of Analyses ($50/hour)
| Analysis Duration | Hours | Total Cost |
|---|---|---|
| 2 weeks | 80 | $4,000 |
| 2 months | 320 | $16,000 |
| 6 months | 960 | $48,000 |
Results often take months or even years to produce, are often difficult to reproduce, difficult to present convincingly to non-specialists, difficult to audit for regulatory compliance and investor due diligence, and sometimes simply wrong, especially where the data involves human subjects or human society. Many important problems in business and society remain unsolved despite modern computer-intensive data analysis methods.
A widely cited report from the McKinsey management consulting firm suggests that the United States may face a shortage of 140,000 to 190,000 such human analysts by 2018: http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-data-the-next-frontier-for-innovation
We are developing tools and algorithms to automate complex data analysis, reducing costs and increasing results.
The Mathematics Recognition Problem (about 14 minutes)
Introduction to Automating Complex Data Analysis (about 18 minutes)
Automating Complex Data Analysis Presentation to the Bay Area SAS Users Group (BASAS) on August 31, 2017 (about 32 minutes)
Automating Complex Data Analysis White Paper
Our in-depth white paper discusses complex data analysis, presents a case study, and reports some preliminary results with a prototype of an automated system for data analysis. It covers the same material as our presentation to the Caltech Alumni Entrepreneurship Group (CAEG) in Palo Alto (below).
Automating Complex Data Analysis Presentation to Caltech Alumni Entrepreneurship Group (about 1 hour, 45 minutes)
We are developing tools and algorithms to automate complex data analysis, reducing costs and increasing results.
(C) 2017-2018 by John F. McGowan, Ph.D.