Ifft P,
Lebedev MA, Li Z, & Nicolelis MAL.
(2012, October) Bimanual
brain-machine interface. (Poster
presentation) Society for Neuroscience 2012 Annual
Meeting, New
Orleans, Louisiana.
Brain-machine
interfaces (BMIs) - devices that connect brain areas
to external actuators - strive to restore limb mobility and sensation to
patients suffering from paralysis or limb loss. Here we report a novel BMI that controls two virtual arms simultaneously.
The development of BMIs for bimanual control is
important because even the most basic daily movements such as opening a jar or
buttoning a shirt require two arms. We for the first time have designed and
implemented a bimanual BMI where activity of multiple cortical areas is
translated in real-time into center-out reaching movements performed by two
virtual arms. Eight multielectrode arrays, a total of 768 electrode channels,
were implanted in the primary motor (M1), sensory (S1), supplementary motor (SMA),
dorsal premotor (PMd), and
posterior parietal (PP) cortices of both hemispheres of a rhesus monkey.
Movement kinematics of each arm were extracted from
the same ensemble of 400 neurons using a Wiener filter and an unscented Kalman
filter (UKF). Typically, a single neuron contributed to the movements of both
left and right arms. Movements were enacted by arms of a virtual rhesus monkey
avatar on a computer screen presented in first-person to the monkey. On each
trial, the virtual arms moved their central locations to peripheral targets
presented simultaneously on the right and left sides of the computer screen. Peri-event time histograms and linear discriminant
analysis revealed a highly distributed encoding scheme, with movement
directions of both limbs represented by both ipsilateral
and contralateral areas. Furthermore, movements were
represented by multiple cortical regions, including both primary and
non-primary motor areas which have been previously identified areas important
for bimanual coordination. Over the course of several weeks of real-time BMI control, the monkey’s performance clearly
improved both when the monkey continued to move the joystick and when the
joystick was removed. These results support the feasibility of
cortically-driven clinical neural prosthetics for bimanual operations.
Li
Z, Tate AJ, Lebedev MA, & Nicolelis MAL. (2011, November) Software-based scalable multichannel
spike acquisition. (Poster
presentation) Society for Neuroscience 2011 Annual
Meeting, Washington, D.C.
Given the distributed
nature of brain processing, recording from more neurons is essential for
improving accuracy in next-generation brain-machine interfaces (BMIs). To facilitate channel counts in the thousands while
keeping hardware requirements low, a new approach to large-scale multichannel spike acquisition is needed. Fortunately, with
the steady improvement in computational power available from desktop computers,
the signal processing tasks previously requiring custom-built hardware can now
be performed in software. We have developed a software suite for large-scale
spike acquisition using commonly-available signal acquisition cards. Our system
offers scalability by networking spike-acquisition computers, a central server
computer, and BMI computers through the TCP and UDP protocols. Multiple
spike-acquisition computers, each with one or more analog input signal
acquisition cards, filter, threshold, and extract spike waveforms from voltage
traces recorded from preamplifiers or high-gain headstages.
The extracted spike waveforms are sent to the central server for online
spike-sorting, logging, and redistribution to users of neuronal data (computers
executing decoding algorithms). Workstations for monitoring data acquisition
and setting spike-sorting parameters can also connect to the central server.
Acquisition computers can also send full voltage traces on selected channels to
spike-sorting workstations for visualization and sonification.
We complement this recording infrastructure with a custom-built graphical spike-sorting
interface. Our software is written in C++ with GUI elements from the Qt
library. When combined with electrode implant arrays, high-gain headstages, signal acquisition cards, and desktop
computers, our software offers a full multichannel
spike acquisition system capable of facilitating real-time BMI experiments with high channel counts. Our
networked system allows incremental expansion, remote control of acquisition
and sorting over internet or local network, and multiple, remote users of
neuronal data. We have tested operation with up to 128 channels using signal
acquisition hardware from National Instruments and ADLINK Technology. This
system integrates directly with our BMI software suite and offers a solution for a range of BMI and electrophysiological tasks.
[Poster PDF]
O’Doherty
JE, Lebedev MA, Li Z, &
Nicolelis MAL.
(2011, November) Towards a
brain-machine-brain interface with virtual active touch using randomly
patterned intracortical microstimulation.
(Poster presentation) Society for
Neuroscience 2011 Annual Meeting, Washington, D.C.
We have recently
shown that intracortical microstimulation
(ICMS) of the primary somatosensory cortex (S1) can
be used for delivering somatosensory feedback during
active exploration tasks performed under brain-machine interface (BMI) control. We have hypothesized that ICMS
could be used to mimic the perception of surface texture in future neuroprosthetic systems and suggested that spatial or spatiotemporal
patterns of ICMS could be used for such encoding. Notwithstanding previous work
on temporally patterned ICMS of S1 by us and others, the applicability of
temporal patterns of ICMS to neuroprosthetic
sensation remains poorly understood. In particular, the minimum discriminable difference between temporally modulated ICMS
patterns remains unknown. To examine discrimination threshold for temporal ICMS
patterns, we trained rhesus monkeys in an active exploration task for which
they discriminated periodic trains of ICMS pulses (200 Hz bursts at a 10 Hz
secondary frequency) from aperiodic ICMS trains with
the same average pulse rate, but distorted periodicity (200 Hz bursts at a
variable instantaneous secondary frequency). Pulse trains were drawn from a
gamma distribution of inter-burst intervals with a constant mean, but different
coefficients of variation (CV). The monkeys discriminated aperiodic
pulse trains with CVs of 0.25 or greater. To probe BMI control while aperiodic
ICMS trains were delivered, movement kinematics were
extracted from the activity of neuronal populations recorded in the sensorimotor cortex. The accuracy of reconstructions
improved when the recording intervals affected by ICMS artifacts were removed
from analysis. These results suggest that temporally patterned ICMS can be used
to provide a graded tactile sense for neuroprosthetic
limbs.
[Poster PDF]
Li
Z, O'Doherty JE, Lebedev MA, & Nicolelis MAL. (2010, November) Closed-loop adaptive decoding using
Bayesian regression self-training. (Poster presentation) Society for Neuroscience
2010 Annual Meeting, San Diego, California.
Neuronal tuning
to arm and hand movements during normal motor control and to movements of an
artificial actuator during brain-machine interface (BMI) operation is known to vary over time.
Thus, BMI decoders for controlling prosthetic
devices must adapt to these changes if they are to operate over long time
spans. We propose the Bayesian regression self-training method for updating the
parameters for an unscented Kalman filter decoder. This method uses the
unscented Kalman filter's output to periodically update the filter's neuronal
tuning model parameters in a Bayesian linear regression. To allow updates on
subsets of neurons and to allow addition of newly-discovered neurons, we
approximate the probability distribution on the tuning model parameters using a
factorized distribution and use variational Bayes to compute the Bayesian
regression solution. We performed closed-loop experiments during which a Rhesus
monkey implanted with cortical multielectrode arrays in the primary motor (M1)
and primary sensory (S1) areas controlled a cursor using the unscented Kalman
filter with Bayesian regression self-training. Over the course of 29 days,
Bayesian regression self-training maintained control accuracy better than
decoding without updates. The updates did not require example hand movements
from the BMI user or any assumptions about the desired
movements. These results indicate that Bayesian regression self-training can
maintain BMI control accuracy over long time periods,
making clinical neuroprosthetics more viable.
[Poster PDF]
Han Z, Li Z, O’Doherty, JE, Lebedev MA, &
Nicolelis MAL
(2010, November) Decoding self-timed
motor behavior with hidden Markov models. (Poster presentation) Society for Neuroscience
2010 Annual Meeting, San Diego, California.
Neuronal
mechanisms of temporal structuring of motor behavior are a topic of interest in
both basic neuroscience and the field of brain-machine interfaces. To elucidate
temporal encoding by motor cortical ensembles, we examined modulations in motor
and premotor cortices of rhesus monkeys that
performed self-timed button presses. Populations of cortical neurons exhibited
modulations during the delays, as well as movements. The temporal structure of
this motor behavior was decoded from simultaneously recorded ensembles of
cortical neurons using hidden Markov models (HMMs).
We modeled the time-course of temporal patterns in population activity prior
to, during, and after button presses as chains of brain states, corresponding
to states in the HMMs. The HMMs
modeled spiking activity in each state as a multivariate normal distribution on
binned spike counts with mean and covariance fitted from training data. We
explored several ways to build time-flexible HMMs to
model button presses of different durations. One approach was to set the
transition model between states so that some states may transition to
themselves; another approach was to set the transition model so that some
transitions may bypass one or more states along the chain. We found that our
models could accurately decode button presses and that there was a trade-off of
accuracy versus HMM complexity. Furthermore, we found that modeling the entire
ensemble as a whole by taking into account the covariance of neuronal activity
was important for decoding accuracy. We conclude that properly-designed HMMs can be used to model the temporal dynamics in neuron
populations and can predict motor behavior for applications such as BMIs.
[Poster PDF]
Li Z, O'Doherty JE, Lebedev MA, & Nicolelis MAL. (2009, October) Simultaneous BMI
decoding and tuning model update using Bayesian regression. (Poster
presentation) Society for Neuroscience 2009 Annual Meeting, Chicago, Illinois.
Probabilistic
filtering methods such as the Kalman filter and unscented Kalman filter enable
decoding of movement commands in brain-machine interfaces (BMIs).
In most prior work, the tuning models defining the relationship between desired
movements and neural recordings have been held fixed after initial model fitting.
However, several studies of neuronal variability have shown that the modulation
patterns of many neurons change over time, making filtering with fixed models
sub-optimal. Updating tuning models without access to the desired movements (or
assumed proxies for them) for re-fitting is a difficult machine learning
problem and has not been previously demonstrated for real neural data. In this
study, we show that a Bayesian tuning model update method built upon the
unscented Kalman filter can address this problem. Our method periodically
updates tuning model parameters using its own filter output as training data
("self-training"). Our periodic updates are computationally light
enough so that when amortized as a background computation, they can keep up
with changing tuning in real time. To improve the accuracy of the filter output
used to update the tuning model, our method smooths
filter output using the Rauch-Tung-Striebel smoother.
We keep track of the uncertainty in our tuning model parameters and perform Bayesian
updates. Our update method was tested on neuronal ensemble data recorded in two
monkeys that performed joystick reaching tasks. The monkeys were chronically
implanted with multielectrode arrays in multiple cortical areas, including
primary motor cortex, dorsal premotor cortex, and
supplementary motor area. We used recordings collected over 4-6 months to
perform off-line reconstructions. Our update method significantly improved
filtering accuracy versus an analogous filter with a fixed tuning model. The
results suggest that our update method can be used to improve BMI performance for neuronal populations with
time-varying properties.
[Poster PDF]
Li
Z, O'Doherty JE, Hanson TL, Lebedev MA, Henriquez CS,
& Nicolelis MAL. (2008, November) Unscented Kalman filter for brain-machine
interfaces. (Poster
presentation) Society for Neuroscience 2008 Annual
Meeting, Washington, D.C.
Selection of
appropriate decoder is key to the success of any brain-machine
interface (BMI). Ideally, the decoder should accurately
represent the relationship between neuronal modulations and behavioral
parameters and capture the specifics of the required behavioral output. To meet
these goals in a BMI
that enacts reaching movements, we developed a decoding algorithm called the n-th order unscented Kalman filter for BMIs.
This decoder improves the common Kalman filter by allowing estimation of
desired hand movements under non-linear models of neural tuning to hand movements.
It employs a quadratic neural tuning model in which the instantaneous firing
rates of movement-tunned neurons are modeled by the
sum of a baseline firing rate, terms for the x- and y-axis components of
position and velocity, and terms for the velocity magnitude and distance from
the center position of the workspace. This quadratic neural tuning model
predicts instantaneous firing rates significantly more accurately than a linear
neural tuning model in position and velocity. The algorithm also augments the
Kalman filter state with a history of hand kinematic variables in the n-1
previous time steps, allowing an n-th order
auto-regressive hand movement model. This n-th order
AR model produces better predictions of the future hand kinematic variables
before incorporating the neural activity input and allows the neural tuning
model to capture relationships between neural activity and hand movement at
different time offsets. The algorithm was tested in experiments in which two
rhesus macaques implanted with multielectrode arrays in cortical motor areas
performed reaching tasks first manually, then through a BMI. We demonstrated superior performance of
the new algorithm compared to other commonly used decoders in both off-line
reconstruction accuracy and on-line, closed-loop BMI-controlled task performance.
[Poster PDF]
Grant BD, Li Z, Hanson TL, O'Doherty JE, Lebedev
MA, & Nicolelis MAL.
(2008, November) Multipurpose,
expandable suite for brain-machine interfaces. (Poster presentation) Society for Neuroscience
2008 Annual Meeting, Washington, D.C.
We have developed
a custom brain-machine interface (BMI) suite using Microsoft Visual C++ that implements an
array of BMI tasks. This BMI suite can be used for biological inputs
of single-unit and multiunit extracellular
recordings, local field potentials (LFPs), electromyograms (EMGs) and
limb-movement parameters. Extracellular recordings are automatically
spike-sorted using waveform templates generated by a modified classification
expectation maximization(CEM) algorithm and real-time sorting by Multichannel
Acquisition Processors (Plexon, Inc.). Template generation for two hundred
units takes approximately five minutes. Inputs can be selected via custom
graphical user interface (GUI), allowing arbitrary combinations to be selected
for BMI decoding. Decoding algorithms are modular
and interact using a common interface with the BMI software suite, allowing for easy
addition of new decoding algorithms. The output of the decoding algorithms is
sent to the experiment manager which handles stimuli, feedback, and
experimental procedure logic. The behavioral task, another modular component,
can be switched in the experiment manager without affecting the operation of
the rest of the BMI
suite because the suite uses a standardized interface that smoothly links with
all tasks. LAN and internet communication via the UDP
protocol allow the suite to send kinematic data to other BMI workstations or collaborators around the
globe in real-time. Kinematic variables and algorithm predictions can be
visualized in real-time using hardware accelerated 3D graphics. All
experimental data is recorded via a unified logging system that automatically
saves all pertinent variables every time they change. Finally, the system
incorporates a feedback loop based on spatiotemporal microstimulation
through multielectrode arrays implanted in brain sensory areas.
[Poster PDF]
Li Z, O'Doherty JE, Hanson TL, Lebedev MA,
Henriquez CS, & Nicolelis MAL.
(2007, November) N-th
order Kalman filter improves the performance of a brain machine interface for
reaching. (Poster presentation) Society for Neuroscience 2007 Annual Meeting, San Diego, California.
Brain machine
interfaces (BMI) offer the possibility of restoring
normal functions, such as limb mobility, to persons with paralysis. Here, we
report a novel algorithm for decoding neural activity into commands for
prostheses. This algorithm, n-th order Kalman filter,
offers a better BMI
performance compared to previously used signal processing and machine learning
methods. We implemented this algorithm in a BMI that processed the activity of cortical
neuronal ensembles recorded in monkey (Macaca mulatta) engaged in a reaching
task and converted the neuronal signals into the predictions of the actuator
movements. The algorithm was optimized to utilize the abundant statistical
information contained in the movement profile and to detect stereotypical
patterns of movements. The n-th order Kalman filter
differs from the standard Kalman filter by augmenting the state variables with
a short history of the n-1 past state variables. This allowed better prediction
of the next state even before incorporating the neural activity input. The
comparison of the prediction accuracy obtained with the n-th
order Kalman filter versus the standard Kalman filter and Weiner filter showed
a significant improvement in performance for several tracking and reaching
tasks. This improvement was achieved because the algorithm settled to a set of
stereotypical movement profiles instead of exploring an infinite number of
possible movements - the property of both normal voluntary movements and
practical neuroprosthetic applications.
[Poster PDF]
Grant BD, Li Z, Hanson TL, O'Doherty JE, Lebedev
MA, & Nicolelis MAL.
(2007, November) Automated
spike sorting of multiunit data for brain-machine interface applications.
(Poster presentation) Society for Neuroscience 2007 Annual Meeting, San Diego, California.
With the rapid
development of large scale, multi-unit recordings and their utilization in brain
machine interfaces (BMIs), algorithms for sorting
neuronal waveforms gained the utmost importance. We have developed
spike-sorting software for our BMI suit that records multichannel
neuronal inputs and performs real-time decoding of behavioral parameters from
these neural data. The quality of decoding hinges directly on the quality of
the sorted units extracted from neuronal ensemble activity. Utilizing a
classification expectation maximization(CEM) algorithm written in C++, we can sort
128 channels of neuronal data in real time. MySql is
utilized in order to distribute the computation over multiple computers for
optimal speed. In addition to the CEM, our algorithm automatically filters the data to
remove mechanical and electrical artifacts. After the units are sorted, the
corresponding sorted templates are sent utilizing MySql
back to main CPU. The template waveforms are then written into Plexon settings
and utilized in Plexon's recording server to sort
units in real time. The templates can be updated by repeating this procedure
during the recording session. Our spike-sorting implementation is fast and
flexible. Utilizing 5 minutes of data to use for the template initiation, 128
channels are sorted in approximately 10 minutes. These operations save more
than 2 hours typically required to sort such amount of multineuronal
data manually. Moreover, we have combined the sorting software with the
algorithm that selects the neurons which are best tuned to behavioral
parameters of interest. Using these neuronal subpopulations improves the
decoding by minimizing overfitting. The algorithm can
also adjust the clusters of sorted waveforms to improve the decoding accuracy.
[Poster PDF]
Li Z & Pizlo Z.
(2003, November) The
Role of Mental Representation in Problem Solving. (Poster presentation) 44th Annual Meeting
of the Psychonomic Society, Vancouver, Canada.
Our previous work
on the Traveling Salesman Problem and on the 15-puzzle suggested that humans
use pyramid representations involving hierarchical clustering to solve
problems. We tested this suggestion
by manipulating mental representation of the M+M puzzle, a sliding tile puzzle.
M+M puzzle can be presented to a subject in two, quite different ways. The results indicate that these two
presentations of the same problem were indeed treated as different problems by
the subjects. One presentation
allowed the subjects to solve the problem quickly, whereas the other made it
very difficult, if possible at all, to solve the problem. Simulation experiments showed that the
two presentations of the M+M puzzle lead to different pyramid representations,
which in turn lead to different solutions.
The presentation that was easy for the subjects was also “easy” for the
pyramid algorithm. These results
provide direct evidence for the role of mental representation in problem
solving.
[Poster PDF]
Li Z & Pizlo
Z. (2003, July) Pyramid model of human
problem solving. 36th Annual Meeting of the Society for
Mathematical Psychology, Weber State University, Ogden, Utah.
Last year we
presented results from experiments with human subjects, who were asked to solve
15-puzzle. We found that subjects
do not use distances among the states; instead they use “direction”. We conjectured that direction to the
goal is an emergent property of a pyramid representation of a problem. The pyramid representation is obtained
by hierarchical clustering in the problem space. The clustering does not assume the
existence of a global metric in the graph representing the problem, but only
existence of local neighborhoods on several levels of scale. This, in turn, assumes the existence of
an ordinal scale for distances.
Recently, we applied this model to 15-puzzle and demonstrated its
psychological plausibility. We also
showed the ability of the model to solve the classical problem of finding a
path around obstacles. Finally, we
applied this model to the M+M puzzle.
This puzzle has two different presentations (versions). Each version leads to a different
pyramid representation, and hence, to different solutions. We also tested human subjects on both
versions of this puzzle. The
results show a strong effect of the problem presentation on human
performance. This effect is similar
to that obtained in the simulation experiments with the pyramid algorithm.