Multimodal Learning Experiences
New technologies continue to emerge in an ever-­increasing pace. While none or at best few of these technologies are directly aimed at education or learning they do have a strong impact on society and therewith on education.

In the sense of Marc Weiser’s vision of ubiquitous computing today we already live in a world of augmented physical objects and in which daily activities are tracked and used for personalisation and contextual interaction (Weiser, 1999).

While the upcoming of new technologies is often associated with great enthusiasm and potential, there is only little empirical evidence on the effects of ubiquitous and ambient learning support and the role of new forms of interaction on the processes of learning and knowledge building. New forms of multimodal, embodied, tangible interaction, augmented human-­object interaction and situated displays researched within this theme aim to have an impact on motivation, meta-­cognitive skills and self-­efficacy in learning. In that sense this theme looks at the micro-­loop of human-­computer interaction and researches the effects of different forms of interaction and their design parameters as also their effects on primary and secondary variables important for learning and teaching.

Rosmalen, Peter van


The educational background of this research comes from research on feedback (Hattie & Timperley, 2007; Mory, 2004), awareness and situation‑awareness (Endsley, 2000), self‑efficacy and self‑directed learning (Butler & Winne, 1995) and instructional design of immersive and augmented media (Van Merrienboer & Kirschner, 2007, Specht, Ternier, & Greller, 2011). According to the work of Hattie & Timperley (2007) effective feedback is one of the most powerful influences in learning. Awareness is one of the key concepts of informal learning support (Syvanen, Beale, Sharples, Ahonen, & Lonsdale, 2005) that can be used as instrument to acquire information relevant (e.g. about tasks, concepts, or the workspace) for the learner within the ubiquitous learning environment (Ogata, 2009). The instructional perspective for learning experiences considers the concept of situational awareness (Endsley, 2000). Endsley defines situational awareness as “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future”. Context‑aware computing builds on this paradigm and uses the “context of use” (Oppermann & Specht, 2000) to help users in filtering information, sequencing activities, and support interaction in context. Computer Science background of this theme comes from Human‑Computer‑Interaction (Norman, 2002, Heer & Shneiderman, 2012), adaptive educational systems (Brusilovsky, 1996), context‑aware computing (Zimmermann, Specht, & Lorenz, 2005), multi‑modal interfaces, sensor­‐based interaction, and ambient displays for learning (Börner et al., 2013). Likewise language technologies are accepted as technology to enhance interaction or to analyse text, its role in education ought to be investigated.

Embodied and augmented interaction in mixed reality

Physical and virtual objects are increasingly being enriched with content and functionalities thus becoming service interfaces for digital media (Sterling, 2005). Towards an Internet of Things (Dodson, 2003) these links are used to integrate physical and virtual objects into existing networks of people or even create networks of things by giving the objects an identity. Carrying this idea further even leads to a possible fusion of physical objects with digital information. This notion of blending the real and digital world is connected to the concept of mixed reality, where physical and digital objects co‑exist, interact and enhance each other. In a world where information is widely distributed and highly contextualised, ambient systems incorporating the mixed reality concept can be used to enable the access to digital content that is available in a real world context, building on the links between people, objects, and data.

Towards multi‑modal and embedded learning experiences

The constant change of interaction modalities is closely connected to the ongoing technical development and the related computational models. An interaction approach that goes beyond conventional graphical user interfaces for personal computing is the use of ambient media in the periphery of the user. Associated with a more tangible and social interaction corresponding systems make use “of the entire physical environment as an interface to digital information. Instead of various information sources competing against each other for a relatively small amount of real estate on the screen, information is moved off the screen into the physical environment” ((Dourish, 2001, Wisneski et al. 1998).

From another point of view this more embodied interaction and the rather situated than individualised design approach triggered by embedding information technology into the physical world extends the digital world beyond the desktop, thus becoming an “ambient social infrastructure” (McCullough, 2005). This aspect goes hand in hand with the call for engaging user experiences, “where technology is designed to enable people to do what they want, need or never even considered before by acting in and upon the environment” (Rogers, 2006).


  • WEKIT (experience capturing) aims to develop an open Technology platform for Augmented Reality experience based on open standards and licenses and to research how we identify, acquire and exploit skills valued by industry. The project will develop and evaluate a conceptual framework for capturing workplace experience, combining it with technical documentation and to augment training in situ with live expert guidance, a tacit learning experience and a re-enactment of the expert.
  • TESLA (trust-based e-assessment): The TeSLA project provides to educational institutions, an adaptive trust e-assessment system for assuring e-assessment processes in online and blended environments. It will support both continuous and final assessment to improve the trust level across students, teachers and institutions The system will be developed taking into account quality assurance agencies in education, privacy and ethical issues and educational and technological requirements throughout Europe.
  • METALOGUE (intelligent tutoring), The main goal of the Metalogue project is to design a dialogue system with metacognitive capabilities that is based on natural spoken language and multimodal interaction and evaluate the developed technologies in multiperspective educational dialogue setting. 
  • LACE (sensor infrastructure based on xAPI). The LACE projects collect and evauates multimodal data for Learning Analytics support. It is a community effort to demonstrate best practices and added value of Learning Analytics in different educational sectors.


Key publications:

  • Börner, D., Kalz, M., & Specht, M. (2013). Beyond the channel: A literature review on ambient displays for learning. Computers & Education, 60(1), 426–435. doi:
  • Specht, M., Ternier, S., & Greller, W. (2011). Mobile Augmented Reality for Learning: A Case Study. Journal of the Research Center for Educational Technology, 7(1), 117–127. Retrieved from
  • Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015). Augmenting the senses: a review on sensor-based learning support.Sensors, 15(2), 4097-4133.
  • Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015, November). Presentation Trainer, your Public Speaking Multimodal Coach. InProceedings of the 2015 ACM on International Conference on Multimodal Interaction (pp. 539-546). Acm.
  • Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015). Stand Tall and Raise your Voice! A Study on the Presentation Trainer. In Design for Teaching and Learning in a Networked World (pp. 311-324). Springer International Publishing.
  • Schneider, J., Börner, D., Van Rosmalen, P., Specht, M. (2016). Enhancing public speaking skills - an evaluation of the Presentation Trainer in the wild. In EC-TEL’16 Proceedings of the eleventh European Conference on Technology enhanced learning