Radiation Visualization using Augmented Reality For neoadjuvant radiation therapy treatment for Sarcoma, radiation oncologists prepare radiation dose distribution maps based off of CT scans. After administration of radiation therapy, often a tumor resection is performed for sarcoma patients who are in a later stage of cancer. To mitigate tissue necrosis following tumor resection surgery, surgical oncologists would benefit from viewing the radiation dose distribution on the anatomy of the patient. Through prescribing feature points and the camera input from an augmented reality headset, the virtual model of the patient’s radiation dose distribution can be overlayed onto their physical anatomy in the operating room via means of an augmented reality application. During the augmented reality application runtime, the alignment of the virtual hologram to the physical anatomy is automatic, in real-time, and stable across multiple viewing angles. In addition, users can manipulate aspects of the virtual model with thresholds of radiation, transparency, and toggleable regions.
Robust Eye Tracking for da Vinci Surgical System Eye gaze tracking can enable numerous exciting applications in robotic surgery such as hands-free instrument control, skill assessment, and cooperative surgical procedures. However, existing methods in gaze estimation either rely on complex hardware installations on the surgeon console or are sensitive to head movements post calibration. We demonstrate the effectiveness of wearable eye tracking glasses on the da Vinci Research Kit (dVRK). With a stereo calibration procedure and head tracking technology, we quantify gaze estimation error and decalibration events, such as head movement and glasses slippage. We show that these are significant sources of error in wearable gaze tracking technology. Future work includes compensating for these decalibration events for robust eye tracking in robotic surgery.
Sensorless Force Estimation on da Vinci Research Kit Most current surgical robots lack force feedback to the surgeon during surgery, which can lead to excessive force applied to tissue. We hypothesize that adding force feedback would create a more immersive surgical environment than is currently available. A typical procedure in which force feedback can play an important role is anastomosis. Anastomosis is a critical part of most reconstructive surgeries with dire consequences in the event of an anastomotic leak. Despite being a key risk factor for anastomotic leaks, currently, there is no objective method of measuring force intraoperatively. Unfortunately, estimating tissue interaction forces at the tips of robotically controlled surgical instruments has proven challenging. Sensor-based force feedback comes at a high cost as force sensors make instrument sterilization difficult and can limit the life of an instrument. We propose a sensorless method to estimate external forces on the da Vinci Research Kit. We will further apply these methods to build in-vivo force metrics for anastomosis and investigate the universality of these methods by deploying them on other surgical robots.
Surgical Training and Skill Assessment through Augmented Reality The current apprenticeship model for surgical training in the operating room is insufficient to meet the growing demand for more surgeons. Many endoscopic procedures are taught in the operating room, limiting trainees’ opportunities for experimentation and feedback. Current training simulations for minimally invasive surgeries require costly equipment and expert surgeons for feedback. Alternatively, Augmented reality has the potential to increase flexibility and effectiveness in endoscopic surgical training, but additional research is critical to understanding the needs of surgical trainees so that such systems are designed to meet their needs. Thus, we work together with surgical trainees and experts by conducting semi-structured interviews with co-design activity to integrate their designs into the AR applications.
Vision based Navigation in Endoscopic Kidney Surgeries Endoscopic surgery is the standard of care for the treatment of diseases of the renal collecting system such as stones and tumors. By using small endoscopes, urologists can perform natural orifice procedures, minimizing morbidity for patients. However, current endoscopes used in urological surgery have a limited depth and field of view, leading to impaired visibility and complicating accurate localization within the intrarenal anatomy. Furthermore, surgeons need to rely on mental mapping between preoperative CT scans and surgical field, requiring higher expertise and increasing the mental workload. We are working on navigation technologies for such surgeries, using the methods such as Structure from Motion, SLAM, CT segmentation, and registration. We aim to close the gap between the preoperative scans and endoscopic video streams, and provide better visualization and navigation during surgeries, improving both surgeon and patient experiences. We aim to expand the project to other surgeries, and develop better methods for navigation.