Modern medical imaging devices are increasingly capable of providing detailed anatomical and functional information that is essential in areas such as diagnostics, screening for disease prevention and surgical planning. Advances in high-field MRI and multislice CT have resulted in greatly decreased scan times and an increase in resolution, which in turn has resulted in a significant size increase of the acquired datasets.
In this work we show a distributed interactive Augmented Reality (AR) application for the co-located visualization of large volumetric datasets. The method combines intuitive interaction techniques, advanced display devices and high performance visualization on Grids into a device that has many applications.
In the early 1930's Erwin Schrödinger published what became known as "Schrödinger's Cat Paradox". The idea consists of putting a cat into a closed box together with a device that would release a poisonous gas inside the box, as soon as one atom of a contained radioactive substance decays. The radioactive substance and it's amount are chosen such, that within one hour, the chances that an atom decays are 50%. After one hour, the chances of the cat being still alive, would thus be 50%. But according to the understanding of quantum mechanics, the cat would be both dead and alive at the same time, unless one would open the box and look inside, to ascertain the state of the cat.
In this work, we proposed an alternative to opening the box of Schrödinger's Cat: using our Window in Virtual Reality to take a look inside the box.
The closest we could get to a cat was a little panther from the Zoological Museum Amsterdam, shown in the following picture.
Photograph of our small panther (by Robert Belleman, UvA Amsterdam).
Our little panther is conserved in Ethanol, and we've been told, that
it is much older than it looks: apparently about one century!
We put the panther into both an MRI scanner as well as a CT scanner. The MRI scan was done on July 13th, 2004, by Erik Akkerman (AMC, Amsterdam) on a Philips Intera 3T scanner.
Photograph of our panther while it is scanned in the AMC MRI scanner.
(Photograph by Robert Belleman)
The CT scan was done on July 15th, 2004, by Marcel van Straten (AMC, Amsterdam) on a Philips MX8000 multislice scanner.
Closeup of the panther in the CT scanner.
(Photograph by Marcel van Straten)
Below are two animations produced by Marcel van Straten using Philips' visualization software. Both animations are from the same data but with different transfer tables.
This research investigates the design, implementation and application of an intuitive display method for the visualization of volumetric medical data sets. The co-located visualization of a dataset together with physical objects with which they are associated (i.e. a patient) provides significant new possibilities. Examples of these are found in image assisted minimally invasive surgery and non-destructive sample analyses. The method that was developed in this project uses object tracking techniques, high performance visualization algorithms and advanced display techniques to create the illusion that the display has opened a virtual window into a real object.
Using object tracking techniques, the display method is "aware" of its location and orientation. This information is used to create an accurate representation of the data with respect to the viewer and his environment. Using this "context", the display method can also be used to provide additional information on the environment, overlaying real objects with computer generated images.
The display device we used is an HP Tablet PC.
As you can see from the above image, there are markers attached to the tablet PC (those shiny spheres on top). These markers are used to track the tablet's position and orientation. This is done with the tracking system delivered by ART GmbH. The system uses two cameras, which emit infra-red light flashes and record the reflections of this light within their scope.
The markers are specially designed to well reflect said light emissions, and are thus tracked by the cameras. The cameras are connected to a host PC, which uses the images of the cameras to compute the position of each marker in 3D space. The software that comes with the system, allows to configure so called "bodies", configurations of 4 or more markers, which can be tracked with six degrees of freedom, meaning that not only their position, but also their orientation can be computed by the tracking system. We use such "bodies" to track three things: the display, the user's head and a wand.
The head of the user is tracked by means of a cap, which the user must wear. On this cap, markers are attached, forming another so called "body". The tracking software reports exact position and orientation of the cap, which allows us to create an appropriate projection.
Additionally, the user can also use a wand-like device which provides intuitive interaction with a cutting plane, that allows to slice through the data, as well as a probing plane, to probe values on the cutting plane.
Finally, we should mention the visualization host. All images displayed on the display are computed in real-time on an SGI Onyx 4 located at SARA, Amsterdam. The tablet, as well as the tracking system, are connected to the Onyx 4 over network. Furthermore, the Onyx 4 uses a VANier board to read out the contents of the rendering pipe in hardware, and send it to the tablet PC over the Network.
To visualize the volumetric data, we developed a program based upon Kitware's VTK toolkit, version 4.5, using SGI's OpenGL Volumizer library version 2.7 for the volume rendering. The VTK program was adapted to receive the tracking information from the tracking system, and render images according to that. The images themselves are than transfered from the Onyx 4 to the tablet PC using SGI's OpenGL Vizserver version 3.4. The server runs on the Onyx 4, while the client is used on the tablet PC to connect to it. During a session, we used a 16:1 compression scheme.
Applications for the proposed display method are multitude; already mentioned were medical diagnostics, image assisted minimally invasive surgery and non-destructive sample analyses. The method also has great potential for training and educational purposes. One of the reasons why 3D visualization techniques are currently not routinely used in hospitals is because 3D has not been integrated into medical training. Apart from medical science, volumetric datasets are also generated in other scientific areas, including physics, biology and computational science. Additional application areas are expected within these areas.
The real-time volumetric visualization of these data sets is often beyond the capabilities of local computing resources. The graphics computing resources required to display medical data volumes at interactive rates will not always be present on site. Although modern accelerated graphics cards have increased in performance enormously over the last decade, they still lack the necessary features and power provided by professional solutions. Unfortunately, the cost of these professional solutions prohibits the introduction of these systems in medical organizations. Moreover, the cost-effectiveness of these systems suffers if they are under-utilized.
Grid computing provides methods that enable access to computational resources across organizational domains. Amongst others, Grids allow access to resources that are too specialized for in-house acquisition.
To achieve its goal, the design of this interactive display uses this Grid paradigm to access graphics computing resources like high throughput graphics rendering pipelines, high performance computing devices and high capacity storage systems. Based on the current position and orientation of the display, a volumetric rendering is computed on a remote computing site from a previously obtained volumetric scan of the object. The rendered image is sent back to the display over the optical testbed so that it appears as if the display has opened a window into the virtual object.
We presented our work on the exhibition floor at SC2004 in Pittsburgh, PA. The dataset we used, stored on the Onyx 4 in Amsterdam, was 172 by 246 by 167 voxels, each voxel being represented by a two-byte value. The display and the tracking system were connected to the Onyx 4 over a 10 Gigabit network. The roundtrip time of the network between Pittsburgh to Amsterdam was around 110 ms. The network was "only" 7 hops from the exhibit floor at Pittsburgh to the Onyx in Amsterdam. It was specially routed for the Dutch exhibit booth at SC2004 over Abilene to New York and from there trans-atlantic over Surfnet, directly to Amsterdam. This resulted in a performance of about 5 frames per second. Please note that the bottleneck in this setup was the network connection. We measured frame rates of over 50 frames per second locally on the Onyx 4 with the same dataset.
In the following diagram, the network topology of the demo given at the Super Computing Conference 2004 in Pittsburgh, PA, is shown. During the demo, the display and the tracking system were displayed on the exhibition floor in Pittsburgh, while the rendering was performed across the ocean, on an SGI Onyx 4 located in Amsterdam.
Currently, we are looking into various things. We want to improve our demo (both code and interaction mechanisms), but we are also looking into getting some more interesting data sets to apply our new visualization method to. Furhtermore, we are also looking into other application areas.
This work is performed within the Virtual Laboratory for eScience project (VL-e), the Netherlands. We also acknowledge the help and support of the following organizations:
Below you can find additional media related to this project.
