Co-Authors: Brian Skinn, Eric Kelly
Every tech era has buzzwords and trends that are billed as worthy of saving the
world or positioned as if they'll one day take it over. In the 1990s we had the
“information superhighway,” the 2000s gave us “web services,” and in the 2010s
“Big Data” and “machine learning” were everywhere.
Monthly phrase counts retrieved from Google Books Ngram Exports
Machine learning is one field that has proved to be far more than a passing fad,
however. Instead, it has grown into an established academic discipline with
significant value and staying power, and is often the first tool that data
analysis folks reach for to solve their problems. To be sure, machine learning
has significant strengths, especially when large amounts of clean, consistent
data are available, and it’s providing dramatic benefits across fields as
diverse as medical imaging, product recommendation, language translation,
self-driving cars, and traffic analysis.
There are numerous areas where machine learning is not necessarily the optimal
tool, though, especially where data is lacking or of poor quality, or where
visibility into the model’s internal decision-making processes is critical. In
these situations, modeling techniques that make use of the processes underlying
the data, informed by domain-specific expertise, are often able to outperform
machine learning methods. Such ‘domain-directed’ methods are a great addition to
your toolbox—they can be effective when machine learning fails or reaches its
limits, and they often provide superior predictive performance in addition to
improved model interpretability.
Defining “Algorithmic” vs “Domain-Directed” Modeling Approaches
A little over two decades ago, Leo Breiman published
Statistical Modeling: The Two Cultures, where he
outlined two ways statisticians go about modeling data. One approach seeks, with
the help of domain expertise, to explicitly model the processes by which the
observed data was generated, while the second approach focuses on prediction and
forecasting based on algorithmic processing of just the available data itself,
without reference to an underlying model. Stated another way, these contrasting
approaches explore a fundamental question: Do we care about understanding, or is
simply predicting future data good enough?
The latter approach, called algorithmic modeling by Breiman, has become the
dominant paradigm used everywhere from classical domains like time-series
forecasting to modern machine learning methods. But with recent advances in both
deterministic statistical libraries such as scipy.stats,
Pingouin, and statsmodels, and also
probabilistic programming libraries including Stan,
PyMC, Bean Machine, and
NumPyro, we are seeing a revival in the more classic
domain-directed approach, albeit with advanced, modern tools. Deterministic
tools represent one class of domain-directed methods, which treat input data as
a set of discrete values and carry out statistical analysis and modeling of that
input data. Probabilistic programming libraries are another type of
domain-directed analysis tools that explore the full distribution of possible
inputs and outputs for a system of interest. In particular, probabilistic
programming tools are able to treat input parameters as complete statistical
distributions, rather than just discrete values.
Quansight has significant experience with these libraries, and also contributes
regularly to SciPy, Bean Machine, and PyMC. We are therefore able to apply them
readily to our clients’ projects where appropriate. In this post, we’ll show
some of the advantages of domain-directed approaches, to help you decide if
they’re right for your problem.
Exploring Domain-Directed Approaches
So, first of all, what’s the appeal of domain-directed approaches?
One good way to illustrate the appeal is with a story from the history of
astronomy. For many centuries, we predicted the paths of planetary motions using
recorded tables of distances of the planets from the Earth. These predictions
were based on geocentric assumptions that planets orbit around the Earth, and
had noticeable errors. Attempts to resolve these errors spurred the development
of the theory of epicycles, where planets are assumed to have
invisible orbits within orbits. These epicycles had lower prediction error but
were very cumbersome to compute, especially in the era before computers. They
also relied on a physical model that we know today—and that, despite apocryphal
accounts, they also knew at the time—to be false.
Eventually, starting with the work of Copernicus and ultimately culminating with
the work of Kepler and Newton, we moved to heliocentric models of planets in
elliptical orbits around the Sun. This greater understanding led to models that
were not just more physically sensible, but that were also simpler and provided
more accurate predictions.
The figure below illustrates how a more-complex, less-realistic epicycle model
can be made to match the observed distance over time between two planets in
elliptical heliocentric orbits approximating those of Earth and Mars. In the
epicycle model on the left, Earth (blue) is stationary and Mars (red) orbits in
a circle that itself is orbiting the Earth. In the heliocentric model, both
planets are orbiting the Sun (yellow). The center panel plots the Earth-Mars
distance for both the epicycle (dashed line) and heliocentric (solid line)
models, showing how the epicycle model provides a reasonable but inexact match
to the more physically realistic heliocentric model.
In this way, domain-directed approaches, including probabilistic programming
techniques, specifically focus on a better understanding of the processes that
lead to the creation of the observed data. This intrinsically causal approach
allows us not only to ask both direct and counterfactual questions, but also to
challenge our own assumptions. A causal model offers an opportunity to answer
not just what might happen in the future, but why it would do so.
This especially comes in handy when we are entering a new domain we don’t yet
understand in depth. Even though it requires more effort to establish a
domain-driven model than an algorithmic one, the domain-driven model provides a
foundation of understanding that can be extremely helpful as we develop our
analysis. For example, as our understanding improves, we are better informed
about which assumptions our prediction algorithms can safely make and which
assumptions are inconsistent with our understanding of the world.
Machine Learning Models Can Provide Substandard Performance Due to the Absence of Domain Knowledge
Unfortunately, modern data science workflows often incorporate few or no
elements of domain-directed understanding. Ultimately, when data scientists go
about their work, we are conducting statistical analysis and modeling of some
sort, whether simple or complex. In many cases, this work begins with generating
simple summary statistics and creating preliminary visualizations of the data.
Then, when we are still only just beginning to understand the domain, we often
stop exploring the underlying system, leap into generating algorithmic model
predictions, and tune the model so that our evaluation metrics are optimized.
In part, this leap is driven by the high-quality off-the-shelf prediction
algorithms available in popular libraries like
scikit-learn and prophet. With these
kinds of solutions, we might obtain an algorithmic model with relatively good
predictive power, but it’s reasonably likely that we will observe substandard
performance and have little understanding of why. Further, it will be difficult
for us to improve the model without modifying or expanding the dataset. In these
situations, the only way to overcome the limitations of algorithmic approaches
may be to switch to a domain-directed modeling paradigm, with the choice between
deterministic and probabilistic methods being made based on the details of the
system under study.
The Need for Better Uncertainty and Model Quality Estimates
While all data modeling approaches can be adapted to offer an uncertainty
estimate along with their predicted outputs, many off-the-shelf solutions do
little better than a mean estimate and some uncertainty bounds on that mean
estimate. In many domains, that can lead to very deceptive results.
As one simple demonstration, the figure below shows
Anscombe's quartet, a set of very different datasets
that all yield identical x and y summary statistics (mean, variance, etc.) and
linear fit parameters (slope, intercept, and correlation/R-squared value). While
it’s easy to see that a linear fit with uniform error is a suitable model for
only one of these four two-dimensional datasets, with high-dimensional data it
is extremely difficult to visualize the quality of fit of an algorithmic model.
Without underlying domain-driven guidance as to a good shape for our model, we
can’t easily trust whether the uncertainty and quality of fit parameters we
derive from our summary statistics are large or small.
This principle was taken to an extreme with the Datasaurus Dozen
dataset, where strikingly different visual patterns of data are shown to have
nearly identical one-dimensional mean and standard deviation values, as well as
identical correlation coefficients. Again, without an underlying domain-driven
model telling us whether we expect star-shaped or dinosaur-shaped data, we
really can’t tell whether the uncertainty and quality of fit represented by
these summary statistics are large or small.
The above examples demonstrate how we are best served during data analysis by
looking at our data directly with the guidance of a specific model, rather than
just at its summary statistics. Similarly, we are best served during inspection
of the inferences and predictions of our models if we bring to bear well
established and understood statistical models and analytical tools.
For example, the following histogram visualizes
recent Matura exit-exam scores for a sample of high-school
students in Poland. If we simply assume that exam performance should follow a
normal distribution and only calculate a simple mean estimate with uncertainty
bars, it would completely obfuscate the real story behind this dataset. First,
the data clearly shows a deviation from a normal distribution, with an
unexpectedly high number of students receiving scores at and just over the
passing mark of 30, and an unexpectedly low number receiving scores just under 30. This deviation is visible in the
probability plot of the dataset (inset), in the form
of the “knot” in the data, as marked by the red arrow.
Further, the probability plot reveals that a normal distribution is not actually
the correct distribution for this dataset, as shown by the deviation of both
high- and low-end tails from the best-fit line. This is because the normal
distribution assumes an unbounded domain for the data, whereas these data are
bounded between 0 and 100, and a significant fraction of the data lies close to
these bounds. A better choice for this bounded dataset might be something like a
beta distribution, in combination with an additional
model able to capture the discontinuity near a score of 30.
The use of domain-directed tools thus gives us access to more relevant models and their associated distributions, and the enhanced structure and insight they provide.
Probabilistic Programming Libraries As Domain-Directed Tools
These sorts of challenging statistical questions are usually better answered
with a more bespoke model motivated by domain knowledge. For example, are we
expecting our data to be linear versus curved? Star-shaped versus
dinosaur-shaped? Binomially-distributed versus beta-distributed?
This is because there are typically relationships and interactions underlying
the dataset that are hard to explicitly capture with standardized algorithmic
machine learning methods. Probabilistic programming tools offer an approach to
capture many of these complex relationships, by explicitly specifying the
probabilistic model best suited for the domain at hand. They do this by
providing a programmatic interface designed to mimic the notation statisticians
have used for decades to describe their models.
For example, suppose you wanted to describe a probabilistic model of a linear
relationship between X and Y. Statisticians might use the following
Here, the tilde (~) operator indicates that a variable is defined as being drawn
from a distribution, rather than by a discrete equation. For example, in this
system ɑ is drawn from a normal distribution with a mean of 0 and a standard
deviation of 10, and σ is drawn from a half-Cauchy distribution (a Cauchy
distribution only defined for positive values, since σ is the standard deviation
of the distribution for Y and so cannot be negative) with a scale parameter of 5. Conversely, the intermediate quantity μ is defined as a direct combination of
X and the two probabilistic quantities ɑ and β.
The following code sample is written to use functionality provided by
PyTorch and Bean Machine, and
describes the same relationships as the equations above:
The colored underlines in both of the figures are paired, to highlight the
similarities between the definitions in code and the related mathematical
representations. As you can see, the PyTorch/Bean Machine code expresses these
statistical concepts in a form very similar to what we would write in standard
mathematical notation, allowing us to easily craft a domain-directed model for
Just as there is no shortage of machine learning prediction libraries, there is
no shortage of probabilistic programming frameworks we can use. Nearly all of
these libraries have robust communities with deep expertise. Some popular
choices at Quansight include Bean Machine,
PyMC, Stan, and NumPyro and, in
fact, multiple Quansight staff are contributors to both Bean Machine and PyMC.
For more technical background on probabilistic programming, see the 2022 PyData
London talk by Chris Fonnesbeck, which demonstrates
the PyMC package, and either
Bayesian Modeling and Computation in Python by Martin, Kumar &
Probabilistic Programming & Bayesian Methods for Hackers
by Cameron Davidson-Pilon, both of which are free and open-access.
Domain-Driven Modeling Outcomes Are Easier to Explain and Justify
Often algorithmic machine learning methods are optimized to produce the best
predictions by adjusting the model to minimize prediction errors over the
training dataset. But in the process of developing high-quality predictions in
this fashion, they yield a trained model that is not very understandable and is
challenging to interpret. Especially in settings where people carry legal or
financial responsibility for the final decisions, predictions originating from
poorly-understood models are hard to trust. Further, legislation is emerging in
some jurisdictions requiring that companies be able to provide clear insight
into software-determined financial and legal decisions, making some algorithmic
models infeasible there.
In contrast, in a more domain-directed approach, every component of a model can
be endowed with an interpretation that is meaningful to a domain expert. For
example, in a time-series model, like the one below for the Mauna Loa monthly
mean carbon dioxide data from 1970–2010 below, we can capture
long-term trends (e.g., overall slope), medium-term trends (e.g., local
deviation from the overall slope), and seasonal effects (e.g., cyclic annual or
monthly variations). When such a model is fitted, each parameter is meaningful
and something a stakeholder can consider when making decisions.
In addition, in some cases an underlying domain-driven model further allows us
to perform causal inference from the dataset, as opposed to just identifying
correlation, and potentially answer questions of significant practical value
like: Did this marketing campaign cause this purchase? Did this drug treatment
cause the patient to get better?
Domain-Directed Models Are More Robust to Noisy and/or Low-Volume Data
Algorithmic approaches really shine when you have relatively clean data, and a
lot of it. But more often than not, the data is less than perfect. With enough
data, modern machine learning and deep learning methods can often overcome these
issues. But when you are just starting to understand your data and want to make
predictions, you often won’t have enough data for robust algorithmic analysis.
When there is not much data, finding good modeling assumptions can be the
difference between successfully extracting the signal from the data and losing
it to noise.
With the domain-directed approach, modeling possible sources of noise in your
data will allow you to start making reasonable inferences early on about those
sources. These inferences can even help guide subsequent data collection
processes, which will help to improve the quality of future data. Even if you
eventually move to a more algorithmic approach, you will have benefitted from
insights obtained using the domain-directed approach.
In this way, domain-directed modeling can also be undertaken as an intermediary
step between more basic data analysis and more prediction-focused machine
learning methods, which you might use when you have more data available. The
insights gained in those early modeling steps are essential to the development
of future solutions.
There are many settings where domain-directed approach will shine, but both
modeling approaches have their place with their assorted strengths and
weaknesses. Here are a few takeaways to help you decide which approach will work
best for you.
Strengths of Algorithmic Approaches
Standardized, with assumptions and shortcomings internalized by the community
Easier to deploy
Require less tweaking
Work well when assumptions in the data align with assumptions in the algorithm
Work better on large, clean datasets, with uniform confidence/importance across all data points
Unavoidable for datasets where no clear domain-directed model can be identified
Strengths of Domain-Directed Approaches
Easier to generate insights for decision makers
Better communicate uncertainty
Work better when datasets are small or medium sized (fit in memory on a single machine)
Easy to adapt to complex interconnected datasets
A domain-directed approach is not the best solution to every problem, but it is
a very good technique for most problems. Even if a simple model can be
confidently assumed for key parameters—for example, a normal, log-normal,
exponential, or other distribution—you can gain the benefits of using a
domain-directed modeling approach.
No matter what problem you’re trying to solve with your data, Quansight has
expertise that can help. Whether you are just getting started or have
established and deep expertise and need extra help to reach your goals faster,
reach out to us for an initial consultation. We’ll be glad to help you make more
of your data—contact us at email@example.com.