KEYWORDS: Cameras, Machine learning, Education and training, 3D modeling, Optical flow, Endoscopes, Visualization, 3D image processing, Support vector machines, Surgery
Endoscopic examination has the advantage of being minimally invasive, but the field of view is narrow and the shape of the gastrointestinal tract creates blind spots that may cause lesions to be overlooked. It is thought that these problems can be solved by performing 3D reconstruction from images using Visual SLAM (Simultaneous Localization and Mapping) and grasping the structure in three dimensions. However, when this method, which requires matching between images, is applied inside the gastrointestinal tract, the similar environment in the gastrointestinal tract continues repeatedly, which prevents appropriate matching and appears as noise during 3D reconstruction. Therefore, we propose a method that evaluates the reliability of feature points using epipolar constraint equations and optical flow for matching points of images obtained from a monocular camera, and classifies whether they are correct matching points by machine learning. The specific methods are 1) image matching, 2) calculating epipolar constraint formulas and distances between matching points, and 3) classifying whether matching points are correct by machine learning. 4) Perform 3D reconstruction using only correct matching points. In order to demonstrate the effectiveness of this method, we conducted experiments using simulation images with known three-dimensional structures. For machine classification, K-means clustering method and nonlinear SVM (Support Vector Machine) were used for comparison. We also conducted a similar experiment with a real object. These results suggest that the method can perform correct 3D reconstruction even in the gastrointestinal tract and contribute to the identification of lesions such as early esophageal cancer.
One of the minimally invasive cardiac surgery procedures is valvuloplasty for mitral regurgitation. Valvuloplasty uses an artificial tendon cord to replace the torn tendon cord and a valvuloplasty ring to correct the enlargement of the valve ring. The appropriate positions for embedding the artificial tendon cord and the annuloplasty ring are marked with dyes during surgery, but the endoscopic view is narrow and may be obscured by surgical instruments. In this study, we propose a system that estimates and displays the position of hidden markers using the coordinates of the center of gravity of each marker and the positional relationship between markers during the surgery. First, the spectral reflectance is estimated from color images obtained from a stereo endoscope and the marker areas are extracted. Next, the 3D information of the marker area is estimated using the stereo method, and the position of the center of gravity is calculated for each marker. Then, inter-frame matching is performed using the contours of the markers to detect hidden markers. Finally, the relationship between the calculated center-of-gravity positions for each marker is used to estimate the center of gravity positions of the hidden markers. The effectiveness of the proposed method was confirmed by estimating the center of gravity of markers in the blind spot using an image of a marker drawn on a pig's heart fragment, which was designed to look like a valve. The proposed system can compensate for the blind spots by estimating the position of the hidden markers during surgery.
We propose an autostereoscopic 3D display with an increased aperture ratio of the parallax barrier. The problem with high aperture ratio is that subpixels are observed simultaneously by both eyes, resulting in high crosstalk ratio. To overcome this problem, we propose the image processing method that suppresses the crosstalk by displaying crosstalk subpixels in black. Thus, the necessary pixels can be observed as much as possible. Using the prototype system, we confirmed that a wide viewing area with a crosstalk ratio of less than 10% and high quality 2D images with less image quality degradation can be obtained.
In mitral valve reconstruction, an annuloplasty band is sutured the patient's valve to restore valve function. The size of the band is determined intraoperatively by inserting a measuring device. However, the insertion of the measuring device is difficult in minimally invasive surgery with small incisions, which has become popular in recent years. We previously proposed a system that superimposes a virtual image of a valve ring on an endoscopic image using AR (Augmented Reality) technology to assist in surgery. However, the system was limited to superimposing a virtual image of a ring on an endoscopic image taken from a single viewpoint. In this study, therefore, a more accurate simulation that can be checked from various directions based on 3D shape data. First, the 3D information of the marker area is estimated using stereo method. Next, the spectral reflectance is estimated from color images obtained from a stereo endoscope and marker areas are extracted. Finally, the annuloplasty band automatically selected from 3D positions of the markers is displayed on the 3D simulation. To evaluate this method, we performed the spectral reflectance estimation and 3D simulation using a pig heart marked with surgical maker and confirmed its effectiveness. This system enables size selection without inserting measuring instruments during surgery and is expected to shorten the operation time.
Because of the narrow viewing angle of the endoscope, it is difficult to grasp the entire image of the digestive tract at once. Therefore, blind spots may occur due to the shape of the gastrointestinal tract, which may result in missed lesions. Virtual endoscopy using CT is the current standard method for obtaining an overall view of the digestive tract. However, since virtual endoscopy detects surface irregularities, it cannot detect lesions without irregularities, including early cancers. In our previous study, we proposed a method of endoscopic entire 3D image acquisition of the digestive tract using a stereo endoscope. However, stereo endoscopes increase the burden on the patient due to the larger endoscope tube. Therefore, in this paper, we propose method of endoscopic entire 3D image acquisition of the digestive tract using a monocular endoscope. The method is as follows: 1) Move the endoscope to capture a series of images of the digestive tract, 2) Estimate the position of the endoscope in each frame by image analysis, 3) acquire 3D points using the continuous images and 4) Reconstruct an entire image of the digestive tract from the 3D point cloud. To confirm the effectiveness of this method, an experiment was conducted using a pattern placed inside a straight tube. The results suggest that this method of endoscopic entire 3D image acquisition of the digestive tract using a monocular endoscope may be able to determine the 3D location of lesions such as early-stage esophageal cancer.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.