KARVIMIO Project

KARVIMIO (German: (KI-basierte Augmented Reality Visualisierung von Bedienungsanleitungen medizinischer Instrumente im Operationssaal) is a three-year BMBF funded project with a total volume of 1.73 Million Euros.

The overall goal of KARVIMIO is the research and development of an Artificial intelligence (AI) based Augmented Reality (AR) assistance system to support the operating room (OR) team in the hospital in order to increase process and treatment quality and patient safety. AI-based object recognition and tracking through static dynamic camera sensors will be used to automatically detect the medical instrument in focus and its condition. Through multimodal interaction with AR glasses, the medical staff will be able to quickly visualize the relevant operating instructions, in order to have the necessary information available at the point of use. This project aims to solve a common problem in the OR, that are difficulties in instrument handling. These typically require the gathering of necessary information in paper or PDF based manuals, an in turn a pause during the procedure that causes reduced patient care and safety. KARVIMIO therefore aligns with the goal of the WHO Global Patient Safety and Action Plan 2021-2030 to keep necessary instrument instructions always at spot.

Internal Kick Off 01.08.2022

Today we are very happy to internally kick off our new project KARVIMIO which is founded by the BMBF. Our HEX Lab team will lead the project in collaboration with the medical experts from the LMU Klinikum, medtec specialists from Medability, and our colleagues from IAT who focus on participatory, ethical and security aspects.

Internal Kick Off
Internal Kick Off

Official Kick Off 29.11.2022

We’re happy to officially kick off our project together with our project partners and all associated partners and collaborators. We are looking forward to working with all of you!

Official Kick Off
Official Kick Off

Co-Creation Workshop @ LMU 31.01.2023

To get the perspectives of the different stakeholders (such as surgeons, surgeon’s assistant, medtech industry and clinical IT) we discussed in two focus groups how intercation possibilities can and should look like and what technical and administrative framework conditions are. We thank the 22 participants from different disciplines very much for their valueable feedback and are excited to incorporate all this valuable information into the further development process!

Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU
Co-Creation Workshop @ LMU

Capital Congress Berlin 14.-16.06.2032

The Institute for Work and Technology IAT (research focus Health & Quality of Life) successfully represented the KARVIMIO project at the Capital Congress in Berlin from 14.06.23 to 16.06.23. In the exhibition area of the Network of German Health Regions “W.I.R. – Wissen.Innovation.Region” Elena Cramer and Alexander Kucharski presented the project to interested visitors.

Capital Congress Berlin
Capital Congress Berlin

XR Day Nuremberg 06.07.2023

The FAU team showcased KARVIMIO at the public XR Day 2023 (organized by XR Hub Nürnberg, one of the central XR Hubs of the XR HUB Bavaria initiative) and presented it to the State Ministry for Digitalization represented by MRin Dr. Armbruster as well as Dr. Michael Fraas, Economics and Science Officer of the City of Nuremberg. We would like to thank the XR-Hub in Nuremberg for having us! The XR-Hub connects innovative start-ups, industry and research.

XR Day Nuremberg
XR Day Nuremberg

XR Day Siemens Healthineers 12.07.2023

During his Keynote on on #AI driven XR interfaces (and all sorts of realities #asor) at the at the #XR day of Siemens Healthineers, Daniel Roth presented the KARVIMIO project to an interested audience of healthcare professionals and researchers. Thank you for the stage and the interesting talks!

XR Day Siemens Healthineers
XR Day Siemens Healthineers
XR Day Siemens Healthineers
XR Day Siemens Healthineers
XR Day Siemens Healthineers
XR Day Siemens Healthineers

Bavarian Business Night 17.07.2023

We were very pleased to present KARVIMIO to stakeholders of the economy yesterday at the Bavarian business night (vbw – Vereinigung der Bayerischen Wirtschaft e. V.). Thank you for many great conversations and lively exchanges and also many thanks for the organization XR HUB Bavaria and the support Bavarian State Ministry for Digital Affairs.

Bavarian Business Night
Bavarian Business Night
Bavarian Business Night
Bavarian Business Night
Bavarian Business Night
Bavarian Business Night
Bavarian Business Night
Bavarian Business Night
Bavarian Business Night
Bavarian Business Night
Bavarian Business Night
Bavarian Business Night

ShiftMedical 20.-22.09.2023

ShiftMedical is the perfect place to network within academia, practitioners, and industry regarding medical AR and VR applications. It was a pleasure to participate and present the KARVIMIO project! We met many old and new friends around our presentation and demo! Images are © Shift Medical GmbH/Patrick Paetz

ELSI Workshop @ IAT 29.09.2023

After a little more than one year in the project, the workshop is a good opportunity to review the current status and next plans together from a legal, ethical and social perspective. Many thanks to the IAT for the organization and all partners’ participants for the successful collaboration!

ELSI workshop
ELSI workshop

NRW Cluster Conference 19.03.2024

One of the main topics of the cluster conference of the Medicine. NRW cluster in Düsseldorf was the translation of research results into application. In addition to interesting workshops and networking, we were able to present our KARVIMIO project and take away interesting suggestions and impulses. Many thanks for that!

CARS Conference 18.06.2024

During a lecture at the CARS conference in Barcelona at the University Pompeu Fabra, the Institute for Work and Technology (Research department: Health Economics and Quality of Life) presented the KARVIMIO project to a broad audience. Thank you for the opportunity and the interesting talks!

Publications

Graph-Based Object Tracking for AR Assembly Guidance @IEEE VR Conference 18.03.2024

We are happy to present our paper “GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance” by Shiyu Li, Hannah Schieber, Niklas Corell, Bernhard Egger, Julian Kreimeier, and Daniel Roth at IEEE VR 2024 in Orlando. Beside augmented reality assembly guidance, tracking objects without markers and in different assembly states is a crucial part of the KARVIMIO project. We had inspiring discussions with many colleagues and look forward to future work and next year’s conference!

Requirement Analysis for an AI-based AR Assistance System for Surgical Tools in the Operating Room: Stakeholder Requirements and Technical Perspectives @IJCARS

We are happy to share our insights on stakeholders’ requirements and technical perspectives in this open-access journal article. This was a group effort from the whole KARVIMIO team, and we hope other researchers and practitioners can also utilize this knowledge.

Project Partners

Technische Universität München/Klinikum rechts der Isar (Project Lead)

Prof. Dr. Daniel Roth

Dr. Julian Kreimeier

Shiyu Li

Medability GmbH

Medability is a leading German company specializing in the development and production of high-end surgical simulators. Our innovative simulators empower medical professionals to enhance their skills and build confidence in using the latest surgical instruments. Our simulators are designed to facilitate professional education and sales, providing a valuable resource for surgeons seeking to advance their knowledge and expertise. With a commitment to excellence and a focus on innovation, Medability is dedicated to delivering the highest quality simulators to customers.

Philipp Stefan | philipp.stefan@medability.de
Medability GmbH
Geretsrieder Strasse 10A
81379 Munich
www.medability.de

LMU Klinikum München

LMU Hospital is a leading healthcare institution known for its advanced research capabilities. Collaborating with global institutions, they continuously contribute to the advancement of healthcare and the improvement of countless lives. One of its areas of expertise is orthopedic and trauma surgery research, where it has made significant strides in improving surgical techniques and patient outcomes. Their dedicated team of experts focuses on studying the latest advancements in trauma care and implementing innovative approaches in the operating room.

Additionally, LMU Hospital is actively involved in augmented reality research, exploring the potential applications of this technology in the medical field. Through cutting-edge simulations and studies, they are investigating how augmented reality can enhance surgical procedures, medical training, and patient education.\

Dr. med. Sebastian Andreß

Dr. Andreß is a fourth year orthopedic trauma resident at the LMU Hospital. Furthermore, he is taking part in an advanced training as a medical informatics specialist for the last two years. In his doctoral thesis, he researched the improvement of surgical treatment of acetabular fractures using 3D printers. During research stays at the John Hopkins University in Baltimore, Maryland, USA and the University of British Columbia, Canada, he developed medical augmented reality and deep learning computer vision applications.

Julia Högl

Julia Högl graduated in Medicine at Ludwig Maximilian University in 2022. She did her elective part of the intern year at the LMU Hospital in Orthopedics and Trauma Surgery and now is working as a resident in surgery. In her doctoral thesis, she researches in endovascular repair in complex aortic pathologies. She is passioned about technical development in surgery and improving patients‘ outcomes based on clinical analysis.

Cindy Walk

Cindy Walk is a experienced surgical assistant with 4 years in orthopedic and trauma surgery at LMU Hospital. Currently in her 5th semester of medical studies, her profile embodies a fusion of hands-on expertise and academic dedication.

Felix Merkl

Felix Merkl is a research assistant and software developer and joined the LMU Hospital 2022. He finished his Master of Science at the Technische Universität München in Biomedical Computing. His research interests are medical augmented and mixed reality applications.

Institut für Arbeit und Technik der Westfälischen Hochschule

IAT

The Institute of Work and Technology (IAT) is a central scientific institution of the Westphalian University of Applied Sciences (WH) in cooperation with the Ruhr University Bochum. The focus “Health Economy & Quality of Life” combines research, development and testing of innovative design solutions in the health economy with the aim of creating a demand-oriented supply design, identifying and overcoming obstacles to innovation and demonstrating sustainable design options. In order to achieve these goals, the project is being conducted on several levels, from which three thematic areas can be derived: 1 Health care – health care on a national and international level in the outpatient and inpatient sectors; 2 Individual health – health development trajectories, health behavior, personal responsibility for health and health competence; 3 Living conditions – demographic change, products and services for the elderly, but also forms of living and forms of life-world-oriented (self-)organization.
The three thematic areas are not to be understood as separate research units; rather, there are a number of interfaces that are reflected in the activities of the research focus. These interfaces are demographic change digitalization, social inequality and economic efficiency. In the area of digital health and assistive technologies, the IAT can draw on a broad range of experience from projects dealing with ELSI issues in.

Dr. Peter Enste

Dr. Peter Enste has been director of the research focus Health Economy & Quality of Life (G&L) at the Institute for Work and Technology (IAT) since 2020. He studied social sciences at the Ruhr University Bochum and completed his doctorate on the topic of “Health-related personal responsibility in the context of the lifespan”. Dr. Enste’s research focuses on social and economic issues related to development trends in the health care industry. The research focuses on cross-sectional topics such as individual development trajectories, digitalization and social inequalities.

Alexander Kucharski

Alexander Kucharski has been a research associate of the research focus Health Economy & Quality of Life (G&L) at the Institute of Work and Technology (IAT) since 2021. He studied social science at the Ruhr-Universität Bochum in both his Bachelor’s and Master’s degree programs (Health Systems and Health Economics). In his own research and in project work, he dealt with participatory design possibilities of digital health, assistance and care applications as well as their use by different target groups. In addition, his research focuses on the implementation conditions of digital care structures and the social implications of technical innovations in the healthcare sector.

Elena Cramer

Elena Cramer has been a research associate of the research focus Health Economy & Quality of Life (G&L) at the Institute for Work and Technology (IAT) since 2020. She studied physiotherapy at the Hochschule für Gesundheit Bochum. She then completed her Master of Science in Evidence-based Health Care with a focus on scientific work/research methods, health care delivery, data analysis and applied physiology in health care. In addition to her studies, she worked recurrently as a physical therapist in neurorehabilitation and orthopedics. From 2017 to 2020, Ms. Cramer worked as a research assistant in the interdisciplinary research project “RehaBoard”. Elena Cramer’s research focuses on digitalization in healthcare, care analyses and health literacy research.

Funding agency: Bundesministerium für Bildung und Forschung (BMBF)