This meta-study deals primarily on evaluating the large number of studies on algorithm visualization to fill the gap on AV effectiveness (ie. by combining the mixed results studies can we make a conclusion on whether or not AV is useful in a learning environment?)
Scenarios of use within CS education are: Lectures, Assignments, Class Discussion, Labs, Study, Office Hours, and Tests.
AV research techniques are: Anecdotal (intuitive based off instances of system use), Programmatic (assessing the actual programs within the system), Analytic (effectiveness, while avoiding the overhead of empirical data collection), and Empirical (collection of data on humans involved in tasks with AV systems).
Theories of effectiveness: there are four broad theoretical camps AV studies fall into, Epistemic Fidelity (humans carry around the idea of symbolic models of the physical world that are the basis of reasoning/action, which graphical representations try to convey), Dual-coding (cognition consists of two symbolic systems: one encodes words, the other pictures. Providing both allows for facilitation of knowledge easier through dual representation), Individual differences (measurable differences in human abilities/styles will lead to measurable performance differences in AV use), and Cognitive Constuctivism (individuals knowledge is based off their subjective experiences, therefore AV technology must be actively engaged with in order to benefit).
According to the meta-study, Cognitive Constructivism has the most consistent empirical support. On this view, AV tech is educationally effective to the extent that it actively engages learners.
Differences in what knowledge is measured: two types of knowledge, conceptual/declarative (understanding of the abstract properties of an algorithm) and procedural (understanding of the step-by-step behavior of an algorithm). Most visualizations focus on the procedural as it is much more sensitive to AV benefits, so most studies measure it alone. The measure of knowledge acquisition vs. improvement is also a major difference.
The study concludes that AV technology is educationally effective. It is not effective if students merely view the visualizations, but if used as a vehicle for actively engaging students, AVs are very effective (compared to not using them).