TY - JOUR
T1 - Classification of fNIRS Finger Tapping Data With Multi-Labeling and Deep Learning
AU - Sommer, Natalie M.
AU - Kakillioglu, Burak
AU - Grant, Trevor
AU - Velipasalar, Senem
AU - Hirshfield, Leanne M
N1 - Publisher Copyright:
© 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
PY - 2021/11/1
Y1 - 2021/11/1
N2 - Studying the relationship between the brain and finger tapping motions can contribute towards an improved understanding of neuromuscular impairment. Furthermore, acquiring brain data signals non-intrusively during finger tapping exercises, and building a robust classification model can aid in the field of human computer interaction. In this paper, we present a promising approach for spatially descriptive multi-labeling of spatiotemporal functional Near Infrared Spectroscopy (fNIRS) data to autonomously detect different finger tapping levels in different regions of the brain simultaneously. Our multi-class multi-labeling technique assigns labels to the left and right index fingers, and a given label describes one of three different finger tapping frequencies (rest, 80bpm, and 120bpm) to be monitored in the corresponding contralateral spatial location in the brain’s motor cortex. We train a CNN/LSTM-based network to classify the aforementioned finger tapping levels spatially and simultaneously. The evaluation, based on simultaneous multi-label predictions for two brain regions, is performed with a metric commonly used in multi-labeling, Hamming Loss, along with confusion matrix-based measurements. Promising testing results are obtained with an average Hamming Loss of 0.185, average F-Score of 0.81, and average Accuracy of 0.81. Moreover, we explain our model and novel multi-labeling approach by generating Shapley Additive Explanation values and plotting them on an image-like background, which represents the fNIRS channel layout used as data input. Shapley values help to add interpretability to our deep learning model and by confirming expected results, offer a pathway to the future development of complex deep learning models that attempt to predict social-cognitive-affective states.(Figure
AB - Studying the relationship between the brain and finger tapping motions can contribute towards an improved understanding of neuromuscular impairment. Furthermore, acquiring brain data signals non-intrusively during finger tapping exercises, and building a robust classification model can aid in the field of human computer interaction. In this paper, we present a promising approach for spatially descriptive multi-labeling of spatiotemporal functional Near Infrared Spectroscopy (fNIRS) data to autonomously detect different finger tapping levels in different regions of the brain simultaneously. Our multi-class multi-labeling technique assigns labels to the left and right index fingers, and a given label describes one of three different finger tapping frequencies (rest, 80bpm, and 120bpm) to be monitored in the corresponding contralateral spatial location in the brain’s motor cortex. We train a CNN/LSTM-based network to classify the aforementioned finger tapping levels spatially and simultaneously. The evaluation, based on simultaneous multi-label predictions for two brain regions, is performed with a metric commonly used in multi-labeling, Hamming Loss, along with confusion matrix-based measurements. Promising testing results are obtained with an average Hamming Loss of 0.185, average F-Score of 0.81, and average Accuracy of 0.81. Moreover, we explain our model and novel multi-labeling approach by generating Shapley Additive Explanation values and plotting them on an image-like background, which represents the fNIRS channel layout used as data input. Shapley values help to add interpretability to our deep learning model and by confirming expected results, offer a pathway to the future development of complex deep learning models that attempt to predict social-cognitive-affective states.(Figure
KW - CNN
KW - LSTM
KW - fNIRS
KW - finger tapping
KW - hamming loss
KW - machine learning
KW - multi-labeling
KW - spatially descriptive
KW - spatiotemporal
UR - http://www.scopus.com/inward/record.url?scp=85115742730&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85115742730&partnerID=8YFLogxK
U2 - 10.1109/JSEN.2021.3115405
DO - 10.1109/JSEN.2021.3115405
M3 - Article
AN - SCOPUS:85115742730
SN - 1530-437X
VL - 21
SP - 24558
EP - 24568
JO - IEEE Sensors Journal
JF - IEEE Sensors Journal
IS - 21
ER -