Carto is a 2017-2018 Sling Health team lead by Dr. Jimmy Xu, an interventional radiology resident at the Mallinckrodt Institute of Radiology. His team is creating smart mapping tools for image guided interventions using augmented reality (AR). They are prototyping a head mounted display that displays images dynamically. In addition, there will be a suite of smart tools using their proprietary machine learning model to guide the physician to a safer and better outcome for the patient. He is lucky to have recruited an all star team of developers, hackers, and builders to bring this project to fruition.
Here is a Q & A with Jimmy Xu:
Q. Why AR in radiology?
A. Radiology is in love with screens. Enter any radiology reading room and you'll see up to 5 large screens at any workstation. Since imaging transitioned over to digital PACS there has been little change in how radiologists read studies. This is the same for image-guided procedures. Ultrasound machines still use screens and fluoroscopy machines in the angiography suites may be the biggest offenders with screens over a meter long attached to the ceiling. Using augmented reality in radiology is a long awaited update now that the technology is mature enough for consumers. There are obvious ergonomic factors that would be improved with dynamic screens that can be placed anywhere, but AR can help operators perform procedures better with smart guidance. We are creating a device that would incorporate live imaging with gesture and voice control as well as a suite of tools to make the procedure faster, safer, and ultimately better for the patient.
Jimmy Xu, MD. Radiology Resident.
Q. Where did you get the idea from?
A. I thought of this idea during my radiology rotations in medical school when I was struggling (still am struggling) to make sense of the imaging. During a procedure the patient is lying on a table in front of you and there is a screen hanging above the patient. The orientation of this screen is in a different plane from the patient. So, when we do interventions we often are looking away from the patient at a screen and then extrapolating the position of the instruments and their relationship to the lesion or task at hand. This is not ideal. This would be the equivalent of a surgeon performing a robotic surgery with the screen rotated 90 degrees. AR was the obvious choice to help solve this problem.
Q. Tell me about your team members. Where did you find them?
A. The team really speaks to the amazing infrastructure and resources of Sling Health. After I spoke to Kavon about my idea, within days I received dozens of really solid resumes and within a couple of weeks was interviewing and recruiting those who best fit the project. WashU has such a plethora of talent from all the different scientific and humanities disciplines. It’s an HR director’s dream. For example, the first person I recruited was Peng, a BME PhD student, who is our lead on machine learning and image processing. He will help build tools that will analyze images to help guide procedures. Austin, a physics and BME student, leads our hardware integration. He has experience with 3D printing and physical computing. Irene is our MBA student helping with market research and business development. I was able to recruit three Computer Science majors, Tara, Devin, Lucas, through Todd Sproul’s Mobile Applications Class. I pitched them a simple project to overlay images onto the patient using Apple’s ARKit. They demoed their app on Project Demo Day and I was lucky to be able to convince them to join the team and lead the software development effort. Last, but definitely not least, is Nina, a sophomore computer science major who built a really fun and addicting game called Dots (which you can buy on the app store).
Q. How did Sling Health help your project development?
A. Sling Health has been vital in many ways in developing this project. First, the team would not exist were it not for Sling Health’s infrastructure, which we talked about earlier. Second, its curriculum is so important because it provides focus and direction that propels the project forward. Last, its network of mentors, subject matter experts, and investors make building, validating, and funding a project that much easier.
Team Carto at their weekly meeting. From left:
Peng Hu, Tara Gildersleeve, Devin Ryan, Nina Woythaler, Lucas Drummond, Jimmy Xu, Austin Sloop, Danqing Irene Li.
Q. What are the struggles when working on your project?
A. The implementation of our system is challenging not because of the limited options, rather the opposite, because there are so many ways to accomplish this task. A quick review found many augmented reality projects using widely different technologies, all with their pros and cons. There are optical see through headsets that project a light image onto a lens, which allows the operator to better see their physical environment, but depth perception of the light object, resolution, and transparency is limited. With the proliferation of mobile phones and VR there are other systems that use a camera pass through system so that the projection of objects onto the screen is more believable, but there may be issues with latency and sensing the physical environment.
At the same time, the technology is still so early that there haven’t really been any well-executed AR image guidance systems beyond academic circles. Right now, simply deciding which platform to pursue beyond the prototype stage is difficult because there are so many different platforms coming out right now.
We are also still trying to determine the best clinical application. In other words, what nail do you want to attack with this big hammer of AR. We’ve identified several areas within interventional radiology but you could go many ways – training vs surgical guidance etc.
Last, developing a beautiful user experience is difficult because the VR/AR vernacular has not been fully established. How exactly do you interact with these virtual objects? Do you trend toward skeuomorphism (think fake shutter sounds on digital cameras) or do you try to develop a completely new design language? These are the sort of questions we are struggling with in trying to create a seamless, thoughtful, and intuitive experience for our user.
Prototype of Carto. The power of AR in manipulating the image.
Q. What can be the potential challenges in the future for Carto?
A. There are many challenges in any technology product. But the biggest challenge is understanding the end user and ultimately end user adoption. Healthcare has two diametrically opposed views toward innovation. On one end innovation is necessary in Healthcare, and we've seen it in the discovery of the X-ray, invention of vaccines, and proliferation of public health, which has saved millions of lives. On the other hand, it remains extremely stubborn to change once a framework is in place. At Barnes Jewish Hospital, a top US News World and Report Honor Roll Hospital, paper notes continue to be written. So, to place our product into that context, the product not only needs to be so good that everyone wants to use it, we must be able to implement it in a simple and economical fashion which is difficult. My background is in healthcare IT. I worked at Epic, a healthcare EMR company where we did large-scale projects with large health systems. I found that the sticking point was seldom the technology (although many times they thought it was), it was the change management to get everyone on board. Like I said earlier, radiology is in love with its screens and is also facing massive technological change with the advent of artificial intelligence. There’s a saying that the future of radiology is bright, but the future for radiologists is uncertain. Our success will depend on how radiologists adapt to this changing landscape and how we can shape that change.
Q. What is your prototype ?
We are using ARKit to build our prototype. Previously systems like HoloLens and Meta required equipment costs of from several hundred to several thousand. But now it is possible to prototype an app for way less which greatly decreases the barrier to entry. What makes us most excited is the advent of accessible machine learning tools that we can use to guide the operator in recognizing anatomic structures, lesions, and complications much more quickly.
You can follow Carto at their website: https://www.augmentedinterventions.com/
Jimmy also welcomes any feedback. email@example.com
Peng demonstrated the power of machine learning in image analysis.
- No labels