V i l e l m i n i   K a l a

Ph.D THESIS PROJECT

Co-adaptive multi-sensory interactions

Lab: Sensory-Motor Integration Lab, Rutgers University, New Brunswick, NJ
Supervisor: Professor Elizabeth B. Torres

In my doctorate thesis, I present new data-types, analytics, and human-computer interfaces as a platform to enable a new type of co-adaptive-behavioral analyses to track neuroplasticity. I exhibit seven different works, all of which are steps in creating an interface that collaborates in a closed-loop formula with the sensory-motor system in order to augment existing or substitute lost sensations. Such interfaces are beneficial as they enable the systems to adapt and evolve based on the participants rate of adaptation and preferences in evolution, ultimately steering the system towards favorable regimes.

I started by trying to address the question: "how does our novel sensory-motor system learn and adapt to changes?". In a pointing task, subjects had to discover and learn the sequence of the points presented on the screen (which was repetitive) and familiarise themselves with a non-predicted event (which occurred occasionally). In this very first study, I examined the learnability of motor system across seven individuals, and I investigated the learning patterns of each individual.

Then, I explored how other bodily signals, such as temperature, are affecting movement. At this point, I conducted two studies. In the first one, I looked into the impact of the temperature range in the quality of the performed movement. This study was conducted in 40 individuals, 20 Schizophrenia patients, known to have temperature irregularities, and 20 controls. I identified the differences between the two populations in the range of temperature and the stochastic signatures of their kinematic data. To have a better look into the relation of movement and temperature, I conducted a second study utilizing data of a preprofessional ballet student recorded during her 6h training and her follow up sleep. For this study, I designed a new data type that allows us to examine movement as a function of temperature and see how each degree of temperature impacts the uctuations in movement.

This new data structure could be used for the integration of any bodily signal. Next, I identified the need to build visualization tools that could picture in real-time sensory information extracted from the analysis that would be informative to the participant. Such tools could be used in a vision-driven co-adaptive interface. For this reason, I designed an in-Matlab avatar that enables us to color-code sensory information to the corresponding body parts of the participant. In our next study, I examined two college-age individuals (a control and an Asperger syndrome) under sensory modalities and preferences. I built methods to extract for each individual the preferred sensory modality from the motor stream, selectivity, and preferences of the particular modality that motivate the system to perform at its best, preferability. These two parameters were critical to finally close the loop by letting the system decide upon the individual preferences.

Therefore, I moved from the open-loop approach, to which all the so-far described studies belong to, into the closed loop approach. Firstly I study a natural closed-loop interface established by the dyadic interaction of two ballet dancers while rehearsing. In this natural paradigm, the closed-loop coadaptation happens through the touches and the pushes that dancers apply on each other in order to co-ordinate, kinesthetic adaptation. Therefore, I applied network connectivity metrics and extracted information such as underlying synergies, leading, and lagging body-parts to name a few. Such tools could be used in a vison-driven co-adaptive interfaces to evaluate the interaction between the participant and the displayed avatar. Finally, I built an artificial audio-driven co-adaptive interface which can track the adaptation and progress of the individual and intelligently steer the system towards the preferred and motivational conditions of the participant. For this conducted study, I utilized the heart rate of a salsa dancer to adjust the tempo of the music. The study showed that such a system can steer the stochastic signatures even of the heart (autonomic signal) creating strong evidence that I can guide the human system towards desire regimes.

UNDERGRADUATE THESIS PROJECT

Implementation and design of one-button augmentative and alternative communication system

Lab: Lab Of Distributed Multimedia Information Systems And Applications, Technical University of Crete, Chania, Grecce
Supervisor: Professor Starvros Christodoulakis

My undergraduate thesis project was awarded the “Student’s Entrepreneurship Grand” for the academic year 2011-2012 by the Center for Innovation and Entrepreneurship of the Technical University of Crete.

The project consists of the implementation and the design of a single button augmentative and alternative communication system for users with very limited kinetic ability (for example they are only able to use a single button) and severe speech impairments (they are not able to communicate orally). It is specially designed to be accessed with a single button; and its functionality focuses on offering a user the opportunity to communicate and express himself, using phrases or building his own sentences, and on using basic multimedia application, such as PDF reader, movie and music player, photo viewer, with a single button.

The final result of my thesis project is both the implementation of the application, which has been briefly described above, and the decision of the equipment it will be used with to be a feasible entrepreneurial idea.

Popularity and newspapers

Recently my undergraduate thesis project has been published in one on the biggest newspapers in Greece, Ethnos. I had the happiness to see myself and the girl, Stella, who inspired me on implementing this project, on the cover of the newspaper. After that, several other magazines and newspapers published our story too.

Other publications have been initiated by the Center for Innovation and Entrepreneurship of the Technical University of Crete. All projects awarded by the program have been published on the book (in Greek language) of the center, as well as some newspapers.