Home

We are developing tools and algorithms to automate complex data analysis, reducing costs and increasing results.

Complex data analysis is a multi-billion dollar business. Major data analysis tool makers alone report revenues totaling over $4 billion per year: SAS Institute ($3.2 Billion), IBM SPSS ($0.3-1.0 Billion), MathWorks ($850 Million), Wolfram Research (at least $40 million), and a number of less well known smaller firms. Medical businesses, financial firms, and science and engineering organizations spend billions of dollars per year on these tools and the salaries of the analysts, scientists, and engineers performing the analyses.

Complex data analysis increasingly determines the approval of new drugs and medical treatments, medical treatment decisions for individual patients, investment decisions for banks, pensions, and individuals, important public policy decisions, and the design and development of products from airplanes and cars to smart watches and children’s toys.

State-of-the-art complex data analysis is labor intensive, time consuming, and error prone — requiring highly skilled analysts, often Ph.D.’s or other highly educated professionals, using tools with large libraries of built-in statistical and data analytical methods and tests: SAS, SPSS, MATLAB, Mathematica, Scientific Python, the R statistical programming language, Excel and similar tools.  Salaries and overhead for these analysts range from $40/hour to $200/hour (using a 25 % overhead rate), sometimes even more.

Total Cost of Analyses ($50/hour)

Analysis Duration Hours Total Cost
2 weeks 80 $4,000
2 months 320 $16,000
6 months 960 $48,000

Results often take months or even years to produce, are often difficult to reproduce, difficult to present convincingly to non-specialists, difficult to audit for regulatory compliance and investor due diligence, and sometimes simply wrong, especially where the data involves human subjects or human society. Many important problems in business and society remain unsolved despite modern computer-intensive data analysis methods.

A widely cited report from the McKinsey management consulting firm suggests that the United States may face a shortage of 140,000 to 190,000 such human analysts by 2018: http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-data-the-next-frontier-for-innovation

We are developing tools and algorithms to automate complex data analysis, reducing costs and increasing results.

Introduction to Automating Complex Data Analysis (about 18 minutes)

Automating Complex Data Analysis Presentation to the Bay Area SAS Users Group (BASAS) on August 31, 2017 (about 32 minutes)

Automating Complex Data Analysis White Paper

Automating Complex Data Analysis Presentation to Caltech Alumni Entrepreneurship Group (about 1 hour, 45 minutes)

 

We are developing tools and algorithms to automate complex data analysis, reducing costs and increasing results.

 

(C) 2017 by John F. McGowan, Ph.D.