This paper investigates the use of Reinforcement Learning (RL) to train an Artificial Intelligence (AI) humanoid opponent to play a Virtual Reality (VR) table tennis game. A self-play RL algorithm is implemented to train an AI opponent through competing against itself in Unity environment. The simulation environment replicates key physics of table tennis, and the agent controls racket movements to hit the ball. Experimental result indicates that the agent progressively enhances its table tennis skills according to rising ELO rating and optimization of aggressive gameplay strategies. This research provides a valuable framework for utilizing RL to overcome limitations of scripted AI opponents in physics-based sports games. RL opens new possibilities for human-like AI that can provide dynamic and adaptive experiences in VR games.
There are many approaches to paint line drawings. However, they are limited in dynamic color update while drawing editing. This curtails the editability of drawings and also hampers the auto-painting of animated drawings. We propose a stroke-based painting representation that associates colors with strokes. A painted drawing is represented purely with strokes, based on which region colors are resolved. With this representation, users can freely modify painted drawings with colors being updated automatically. Color stroke is further extended to auto-paint 2D animated drawings. Experimental results demonstrate the advantages of our method in painting drawings and managing colors.
We present a pilot study on expressive B-spline curves (XBSC), an extension of disk B-spline curves (DBSC). XBSC facilitates expressive drawings in terms of shape and color. For shape, colors on both sides of XBSC strokes are defined independently instead of using a single parameter for both sides as in DBSC. We perform coloring by considering the envelopes of XBSC as diffusion curves. Our results show that XBSC can be used to easily draw a wide range of images with fewer number of primitives compared to previous methods.
The objective of this research is to develop a computer tool to perform body motion sensing, dynamics analysis, and realtime visualization based on a virtual reality system. The aimed system can enable effective motion detection, monitoring and instant visual highlights involving muscle contractions during one’s exercises. Experimental results are presented in projecting one’s movement onto a virtual skeleton by applying inverse kinematics.
We present a novel approach to movability assessment on physiotherapy for shoulder periarthritis via fine-grained 3D Residual Networks (R3D) deep learning. The unique deep neural networks is able to automatically extract the spatiotemporal features from the RGB-D videos. In our preliminary studies, we have a set of VR sports games customized for the immersive and interactive sports environment, to regulate the patient’s rehabilitation exercises. In this way, acquisition of RGB-D action videos can be more specific to the subject and defined movements; and fine-grained feature discrimination of the same subject can be better achieved from the longitudinal study, to increase the accuracy of therapeutic assessment.
Two key research issues are addressed: (i) A semantic representation and interpretation framework by using a lightweight self-supervised learning approach, namely the Context-Free Grammar and Push-Down Automaton; and (ii) A mobile phone App implementation of B-mode medical ultrasound imaging with a handheld probe, which can make use of the learned semantic features of scanned images for future home-based health screening.
We propose CAVRoom, a multiplatform virtual shared space framework for multiplayer sandbox interaction. Participants using various devices such as PCs, Macs, mobile tablets, and mobile phones can interact together in the virtual shared space. CAVRoom aims to provide accessible solution for enhanced remote interactions beyond services offered by existing videoconference solutions. We demonstrate the framework in a virtual classroom setting in which students play a Heart Attack card game.
This VR system simulates a table tennis game for adoption in training. The virtual environment allows a real player to play with a virtual AI opponent, with an update and display of the score they gained. Technical innovations are made in modelling of the multi-variant collision between the moving bat and ball with adjustable colliders, and determination of the ball flight path. We present the conceptualization of VR techniques, the detailed implementation of the system and comparative advantages.
In the scenario of a VR hunting game, behaviour and autonomy of in-game animals are crucial in minimizing users’ uncomfortableness and maximizing their experience while gaming. In this study, forward and backward reaching inverse kinematics (IK) is extended and used as the IK solver to handle the locomotion of the autonomous animals. The IK model presented is seamlessly combined with the steering behaviour algorithms so that the animals can perform all sorts of autonomous actions efficiently, supported with realistic and smooth animations. The algorithms presented in this research can also be used for motion behaviour and control of other quadrupedal and multipedal objects.
We propose a scalable AR (Augmented Reality) multiplayer robotic platform, which enables multiple players to control different machines (a drone and a robot) in shared environments, i.e virtual and real environments. We use state-of-theart visual SLAM (Simultaneous Localization and Mapping) algorithms for tracking machine poses based on camera and IMU (Inertial Measurement Units) inputs. Players will observe consistent AR objects between them thanks to our backend system, which synchronizes the AR objects between players. Moreover, the system is scalable in term of hardware (e.g. IMU, camera, machine type) and software (SLAM algorithm) as we utilize ROS for communication between modules. We demonstrate our system on a game developed in Unity, a robust and widely used popular game engine. We present some statistics of the game such as its frames-per-second performance.
The goal of this VR system is to simulate a puzzle / challenge for adaption in cognitive rehabilitation training. Development of the VR system is inspired by a 1st person puzzle-platform game where the player must navigate and complete through a series of puzzle rooms with each room being more difficult than the last. The unique features in this work include the use of “portals” and “portal gun”. The portal gun allows the player to shoot two separate “portals” on walls that will allow anything to be teleported from one portal to the other. Implemented on the Unity engine and SteamVR, the VR Toolkit is employed in modeling and script development. Technical innovations are made in modeling the animated and self-collision detectable spider; upon being collided with a weapon (bullet or blade) it uses a special dissolve shader to give the effect of disappearing gradually from the game. We present the specific tasks of cognitive rehabilitation, conceptualization of VR techniques and the detailed implementation of the system.
The goal of this VR system is to simulate a bowling game for adaption in muscular rehabilitation training. The virtual environment allows the user to pick up a bowling ball and hit the pins, followed by an update and display of the score they gained; and the players can alternate between each other to have a competition. Implemented on the Unity engine and SteamVR, the VR Toolkit is employed in modeling and script development. Technical innovations are made in generation of the grabbing and releasing controllers with adjustable colliders, and the respawn detector triggered when the ball hits the back of the bowling alley in the game. We present the specific tasks of muscular rehabilitation, conceptualization of VR techniques and the detailed implementation of the system.
We present an efficient resource scheduling scheme for out-of-core dynamic streaming of a 3D scene. The entire scene is stored in the cloud and relevant scene data are streamed to a client mobile device in real time based on a user-selected path in the 3D scene. We analyze the path data in order to yield efficient streaming of 3D urban objects. We compare streaming scheduling based on the user-selected path and on only the user’s current location (i.e. without path lookahead) in terms of number of loaded objects, rendering performance, and storage. The client application is implemented in Unity game engine and we perform experiment on an Android mobile device.
Most Virtual Reality (VR) headsets come with cable connections to their computing units. This is inconvenient for deployment in immersive interactive environments. A wireless VR solution is a promising approach for improving the user experience. The Mobile Edge Computing (MEC) provides high bandwidth and low latency, which enhances VR wireless solution for supporting high definition and low latency applications. In this paper, we propose a VR wireless solution based on MEC to meet the requirement of future VR applications with high resolution display and low latency. Compared to existing wireless VR solutions, our solution is superior in improving the video quality and enhancing the quality of the user's experience.
Identifying the potential factors that contribute to an aircraft engine’s durability is a big challenge. In this paper, we propose an interactive system to link the engine maintenance history data, flight route history data, and volcanic ash eruption data, and then visualize the relations among them. Several kinds of visualization forms are used to present the datasets. We implement the React-Redux framework to manage the visual components in the frontend, along with a backend to handle the database. This highly tailored system shows how datasets of different sources can be linked and visualized with interactivity. Data analysts and developers can benefit from full-stack development process from data to intuition, meeting the requirements of customized visualization.
Fiber-wireless (FiWi) access networks are typical hybrid access technologies that combine the high bandwidth of optical access and the flexibility and ubiquitous coverage of wireless access. Mobile edge computing (MEC) provides low latency services and cloud computing capabilities at the edge networks. The convergence of MEC and FiWi access networks can improve network performance and QoS (Quality of Service) of augmented reality (AR) services. Therefore, in this paper, we study the problem of integrating MEC with FiWi to enhance AR services. We first present the integration scheme of MEC with FiWi for AR applications. We then propose our AR applications enhanced by the integration of MEC and FiWi access, which can provide multiple low-latency AR services. Performance evaluation results demonstrate that the integration of MEC with FiWi can support long-distance AR applications with low delay.
A critical effect found in noninvasive in vivo endomicroscopic imaging modalities is image distortions due to sporadic movement exhibited by living organisms. In three-dimensional confocal imaging, this effect results in a dataset that is tilted across deeper slices. Apart from that, the sequential flow of the imaging–processing pipeline restricts real-time adjustments due to the unavailability of information obtainable only from subsequent stages. To solve these problems, we propose an approach to render Demons-registered datasets as they are being captured, focusing on the coupling between registration and visualization. To improve the acquisition process, we also propose a real-time visual analytics tool, which complements the imaging pipeline and the Demons registration pipeline with useful visual indicators to provide real-time feedback for immediate adjustments. We highlight the problem of deformation within the visualization pipeline for object-ordered and image-ordered rendering. Visualizations of critical information including registration forces and partial renderings of the captured data are also presented in the analytics system. We demonstrate the advantages of the algorithmic design through experimental results with both synthetically deformed datasets and actual in vivo, time-lapse tissue datasets expressing natural deformations. Remarkably, this algorithm design is for embedded implementation in intelligent biomedical imaging instrumentation with customizable circuitry.
Phase-shifting profilometry using binary patterns with projector defocusing has been widely used for high-speed 3D measurement. Recently, a ternary Gray-code based phase unwrapping method has been proposed, which enables to accurately unwrap the phase but reduces the required binary patterns. This paper presents a comparison between the ternary and the traditional binary Gray code-based phase unwrapping methods.
Oral lesions are conventionally diagnosed using white light endoscopy and histopathology. This can pose a challenge because the lesions may be difficult to visualise under white light illumination. Confocal laser endomicroscopy can be used for confocal fluorescence imaging of surface and subsurface cellular and tissue structures. To move toward real-time "virtual" biopsy of oral lesions, we interfaced an embedded computing system to a confocal laser endomicroscope to achieve a prototype three-dimensional (3-D) fluorescence imaging system. A field-programmable gated array computing platform was programmed to enable synchronization of cross-sectional image grabbing and Z-depth scanning, automate the acquisition of confocal image stacks and perform volume rendering. Fluorescence imaging of the human and murine oral cavities was carried out using the fluorescent dyes fluorescein sodium and hypericin. Volume rendering of cellular and tissue structures from the oral cavity demonstrate the potential of the system for 3-D fluorescence visualization of the oral cavity in real-time. We aim toward achieving a real-time virtual biopsy technique that can complement current diagnostic techniques and aid in targeted biopsy for better clinical outcomes.
Oral lesions are conventionally diagnosed using white light endoscopy and histopathology of biopsy samples. Oral
lesions are often flat and difficult to visualize under white light illumination. Moreover, histopathology is timeconsuming
and there is a need to develop minimally invasive optical biopsy techniques to complement current
techniques. Confocal laser endomicroscopy holds promise for virtual biopsy in disease diagnosis. This technique enables
fluorescence imaging of tissue structures at microscopic resolution. We have developed a prototype real-time 3-
dimensional (3D) imaging system using a laser endomicroscope interfaced with embedded computing. A Field-
Programmable Gate Array computing platform has been programmed to synchronize cross-sectional image grabbing and
Z-depth scanning, as well as automate acquisition of confocal image stacks. A PC was used for real-time volume
rendering of the confocal image stacks. We conducted pre-clinical and pilot clinical studies to image the murine and
human oral cavity. High quality volume renderings of the confocal image stacks were generated using 3D texture slicing.
Tissue morphology and 3D structures could be visualized. The results demonstrate the potential of the system for
diagnostic imaging of the oral cavity. This paves the way toward real-time virtual biopsy of oral lesions, with the aim to
achieve same-day diagnosis in a clinical setting.
Oral cancers are currently diagnosed using white light endoscopy and histopathology. However, oral tumours are mostly
superficial and can be difficult to visualise. Here we present the use of hypericin with fluorescence endoscopy and laser
confocal fluorescence endomicroscopy interfaced with embedded computing for the diagnosis of oral cancers.
Fluorescence imaging of oral lesions was carried out in the clinic using a fluorescence endoscope. The images were
analyzed to extract the red to blue (R/B) ratios to discriminate between tissue types. The results showed that the R/B
ratio is a good image parameter to discriminate between normal, hyperplastic and malignant oral tissue. We are also
developing an embedded, real-time computing system interfaced to a fluorescence endomicroscope for 3D visualization
of tumors, where synchronization of cross-sectional image grabbing and Z-depth scanning is realized through
programming a Field-Programmable Gate Array. In addition to the programming task, a proprietary control circuit has
been developed for the automated 3D reconstruction of fluorescence sections; and preliminary results from fluorescent
samples have demonstrated the potential of this system for real-time in vivo 3D visualization of tumours. This will
ultimately enable same-day clinical diagnosis to be achieved and further enhance the clinical usefulness of fluorescence
diagnostic imaging.
In recent years, many algorithms were proposed for fringe pattern analysis and phase unwrapping including median filter,
Fourier transform, windowed Fourier transform and wavelet transforms. However most of them are computationally
expensive, which may be a barrier for real-time analysis for fringe pattern. In this paper an FPGA-based system and
optimization framework for fringe pattern processing is presented. Median filter and average filter are taken as examples
to show the performance of FPGA system over normal PC in real-time fringe pattern processing. The algorithms have
been implemented on Celoxica RC340 FPGA development board using Handel-C - a C-like language with inherent
parallelism making the full use of FPGA hardware. Furthermore, evaluation and comparison for computation speed are
given between FPGA-based system and normal PC to demonstrate high potential of our hardware system in fringe
pattern analysis.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.