In this article, we survey the techniques for image-based rendering. In contrast with the conventional approach of using 3-D models for creating new virtual views, image-based rendering techniques require only the original images to produce new virtual views. We have identified four basic categories for image-based rendering techniques, namely non- physically based image mapping, mosaicking, interpolation from dense samples, and geometrically-valid pixel reprojection. This division is based primarily on the nature of the pixel indexing scheme. We discuss the characteristics of each of these categories and present representative techniques. From this survey, it is interesting to note that image-based rendering has evolved to be an area that has generated active interest from both computer vision and computer graphics communities.

The paper deals with concentric image capturing and its use for mapping and visualization purposes. The work is based on a photogrammetric approach in composing hemispheric images from concentric image sequences.

We show that we can effectively fit a very complex facial animation model to uncalibrated video sequences, without benefit of targets, structured light or any other active device. Our approach is based on regularized bungle-adjustment followed by least-squares minimization using a set of progressively finer control triangulations. It takes advantage of three complementary sources of information: stereo data, silhouette edges and 2-D feature points. In this way, complete head models can be acquired with a cheap and entirely passive sensor, such as an ordinary video camera. They can then be fed to existing animation software to produce synthetic sequences.

In this paper, we show that, given video sequences of a moving person acquired with a multi-camera system, we can track joint locations during the movement and recover shape information. We outline techniques for fitting a simplified model to the noisy 3-D data extracted from the images and a new tracking process based on least squares matching is presented. The recovered shape and motion parameters can be used to either reconstruct the original sequence or to allow other animation models to mimic the subject's actions. Our ultimate goal is to automate the process of building complete and realistic animation models of humans, given a set of video sequences.

This paper proposes a basic feature for quantitative measurement and evaluation of group behavior of persons. This feature called 'dominant region' is a kind of sphere of influence for each person in the group. The dominant region is defined as a region in where the person can arrive earlier than any other persons and can be formulated as Voronoi region modified by replacing the distance function with a time function. This time function is calculated based on a computational model of moving ability of the person. As an application of the dominant region, we present a motion analysis system of soccer games. The purpose of this system is to evaluate the teamwork quantitatively based on movement of all the players in the game. From experiments using motion pictures of actual games, it is suggested that the proposed feature is useful for measurement and evaluation of group behavior in team sports. This basic feature may be applied to other team ball games, such as American football, basketball, handball and water polo.

Man-made objects often have a polyhedral shape. For polyhedral objects it is advantageous to use a line-photogrammetric approach, i.e. lines are observed in the images instead of points. A novel line-photogrammetric mathematical model is presented. This model is built from condition equations with image line observations and object parameters in the form of the coordinates of object points and the parameters of object planes. The use of plane parameters significantly simplifies the formulation of geometric constraints. Object line parameters are not included in the model. The duality of the point and plane representation in space is exploited and leads to linear equations for the computation of approximate values. Constraints on the parameters are used to eliminate the rank deficiency and to enforce geometric object constraints. The exterior orientation of the images is assumed to be approximately known. The rotation matrix is parameterized by a unit quaternion. The main advantages of the presented mathematical model are the use of image lines as observations and the way in which it facilitates the incorporation of all types of geometric object constraints. Furthermore, the model is free of singularities through a combination of over- parametrization and constraints. The least squares adjustment allows rigorous assessment of the precision of the computed parameters and allows for statistical testing to detect possible errors in the observations and the constraints. Examples demonstrate the advantages of the proposed mathematical model and show the effects of the introduction of geometric constraints.

This paper considers the problem of automatic registration of stereo images with little or no occlusion. The chosen strategy is based on multiresolution (MR) representations of both the images and the unknown, locally-varying warping map, and uses a coarse-to-fine refinement strategy in which successively more image detail is processed to obtain successively finer- scale corrections to an initial, global estimate of the warping map. The definition of an 'optimal' MR registration algorithm under this strategy is set out in terms of maximizing the chances of fast convergence to the true disparity. We set out the equations for a specific version of this strategy in which spline models of different orders are used for both MR representations. The resulting algorithm, which requires hardly any prior parameter adjustment, shows both robustness and flexibility in tests on both synthesized and real image pairs. Indeed, its performance suggests that it comes close to meeting the previously defined optimality criterion.

We propose a new approach for the representation of objects. It is based on the decomposition of contours in different features: convex or concave curves, convex or concave corners, segments. The decomposition is performed according to the properties of the contour profile, which allows overlapping of primitives. Primitives are then connected according to specific relations, in particular adjacency (not in a strict sense), proximity, similarity, etc., based on the Gestalt Theory. The two sides of the contours are distinguished in order to process faces separately. In this representation, an object is defined by a hypergraph in which it is possible to extract sub-hypergraphs corresponding to all visible faces. Graph matching methods are used for the recognition of objects. If the objects are partially occluded, the search for the maximal generalization between hypergraphs makes it possible to recognize parts of them.

In high precision close-range applications, images of signalizing targets on object(s) are acquired from multiple view points. Sub-pixel locations (image points) of target images are computed. In order to estimate 3-D locations, image point in each view that represent the same 3-D object point are identified. This identification process is referred to as establishment of image point correspondences and is the topic of this paper. The development of a multiple view image point correspondence algorithm based on rectification principles is discussed. The algorithm uses parallel epipolar lines for the establishment of image point correspondences hence, it is termed the Parallel Epipolar Line (PEL) algorithm. The PEL algorithm is suitable for applications where real-time multiple view image point correspondences are required.

This paper presents two integrated solutions for realistic 3D model acquisition and reconstruction; an early prototype, in the form of a push trolley, and a later prototype in the form of an autonomous robot. The systems encompass all hardware and software required, from laser and video data acquisition, processing and output of texture-mapped 3D models in VRML format, to batteries for power supply and wireless network communications. The autonomous version is also equipped with a mobile platform and other sensors for the purpose of automatic navigation. The applications for such a system range from real estate and tourism (e.g., showing a 3D computer model of a property to a potential buyer or tenant) or as tool for content creation (e.g., creating 3D models of heritage buildings or producing broadcast quality virtual studios). The system can also be used in industrial environments as a reverse engineering tool to update the design of a plant, or as a 3D photo-archive for insurance purposes. The system is Internet compatible: the photo-realistic models can be accessed via the Internet and manipulated interactively in 3D using a common Web browser with a VRML plug-in. Further information and example reconstructed models are available on- line via the RESOLV web-page at http://www.scs.leeds.ac.uk/resolv/.

A number of recent projects have demonstrated the utility of Internet-enabled image databases for the documentation of complex, inaccessible and potentially hazardous environments typically encountered in the petrochemical and nuclear industries. Unfortunately machine vision and image processing techniques have not, to date, enabled the automatic extraction geometrical data from such images and thus 3D CAD modeling remains an expensive and laborious manual activity. Recent developments in panoramic image capture and presentation offer an alternative intermediate deliverable which, in turn, offers some of the benefits of a 3D model at a fraction of the cost. Panoramic image display tools such as Apple's QuickTime VR (QTVR) and Live Spaces RealVR provide compelling and accessible digital representations of the real world and justifiably claim to 'put the reality in Virtual Reality.' This paper will demonstrate how such technologies can be customized, extended and linked to facility management systems delivered over a corporate intra-net to enable end users to become familiar with remote sites and extract simple dimensional data. In addition strategies for the integration of such images with documents gathered from 2D or 3D CAD and Process and Instrumentation Diagrams (P&IDs) will be described as will techniques for precise 'As-Built' modeling using the calibrated images from which panoramas have been derived and the use of textures from these images to increase the realism of rendered scenes. A number of case studies relating to both nuclear and process engineering will demonstrate the extent to which such solution are scaleable in order to deal with the very large volumes of image data required to fully document the large, complex facilities typical of these industry sectors.

Selecting the appropriate 3-D capture and modeling technologies to create the contents of a virtual environment (VE) remains a challenging task. One of the objectives of the environment-modeling project in our laboratory is to develop a design strategy for selecting the optimum sensor technology and its configuration to meet the requirements for virtual environment applications. This will be based on experimental evaluation of all performance aspects fa several systems for creating virtual environments. The main problems include: practical sensor calibration, the registration of images particularly in large sites in the absence of GPS and control points, the complete and accurate coverage of all details, and maintaining realistic-look at real-time rate. This paper focuses on the evaluation of the performance of some approaches to virtual environment creation using specifically designed laboratory experimentation. By understanding the potentials and limitations of each method, we are able to select the optimum one for a given application.

Video cameras and digital cameras have become a more and more attractive data source in airborne remote sensing. The problem has been the huge number of images needed to cover areas of interest in practical applications. Therefore automated methods for patching the many images mosaics are essential. The mapping of the third dimension, the height of the terrain has become a more and more relevant issue in practical applications. 3D information is needed for planning purposes, analysis and visualization. Especially the creation of virtual worlds demands full information. In this paper the first experiences of using a global object reconstruction method for automatic creation of image mosaics and digital terrain or surface models using airborne digital camera data are presented. The global object reconstruction method has been tested on a few sets of digital camera data. The largest data sets have consisted of about 150 images. The results show that the method works well. There are practically no visible geometric errors in the resulting mosaics. Also the radiometric quality of the mosaics is good. The seams between the individual images are therefore mostly invisible.

Camera system calibration and related problems are considered as least squares parameter estimation problems based on error- in-variables regression models. The interrelationship between different methods for solving these problems is investigated. In addition, the origin and interrelationship of some approximate methods is also discussed.

Presently there is a growing demand for fast and precise 3D computer vision systems for a wide variety of industrial applications like reverse engineering, quality control and industrial gauging. One important aspect of any vision system is the data acquisition. If the principle of triangulation is used the correspondence problem is to be solved. The coded light approach offers a fast way to overcome this problem and to provide dense range data. In order to get high accuracy range images the system needs to be calibrated. In this paper, we compare two calibration techniques: polynomial depth calibration and photogrammetric calibration. We have carried out both methods independently. To obtain results about the accuracy in object space, we measured the surface of a plane- table.

This paper describes the practical aspects of CCD-line set up and calibration for a 3D Objectscanner based on shape from stereo which was designed and produced at our Institute. This calibration is important if one wants to achieve best results with a scanner. We present an imaging system based on a single CCD-linear sensor and we provide a two-step self-calibration procedure. The results of our calibration method allows to measure object points of an accuracy better than 0.1 mm in 3D space. The self-calibration is repeatable and is suitable to be performed at the users site by using the 3D Objectscanner and the 3D calibration target.

In this paper we consider robust techniques for estimating structure from motion in the uncalibrated case. We show how information describing the uncertainty of the data may be incorporated into the formulation of the problem, and we explore the situations in which this appears to be advantageous. The structure recovery technique is based on a method for self-calibrating a single moving camera from instantaneous optical flow developed in previous work of some of the authors. The method of self-calibration rests upon an equation that we term the differential epipolar equation for uncalibrated optical flow. This equation incorporates two matrices (analogous to the fundamental matrix in stereo vision) which encode information about the ego-motion and internal geometry of the camera. Any sufficiently large, non- degenerate optical flow field enables the ratio of the entries of the two matrices to be estimated. Under certain assumptions, the moving camera can be self-calibrated by means of closed-form expressions in the entries of these matrices. Reconstruction of the scene, up to a scalar factor, may then proceed using a straightforward method. The critical step in this whole approach is therefore the accurate estimation of the aforementioned ratio. To this end, the problem is couched in a least-squares minimization framework whereby candidate cost functions are derived via ordinary least squares, total least squares, and weighted least squares techniques. Various computational schemes are adopted for minimizing the cost functions. Carefully devised synthetic experiments reveal that when the optical flow field is contaminated with inhomogeneous and anisotropic Gaussian noise, the best performer is the weighted least squares approach with renormalization.

Digital Close-Range Photogrammetry has made tremendous improvements over the last years. Many of the elements contributing to those improvements were the advances in algorithms and their implementation into commercial systems. It has become customary that production personnel without photogrammetric know-how use systems. Another area of major change is the use of Digital Close-Range Photogrammetry systems in machine control. These applications require totally automated systems without human supervision. While the algorithmic performance of such systems is a difficult task other issues, such as ultra high-resolution cameras suitable for the rough environment and their response time, are critical elements for the acceptance of such systems. Cameras with much higher resolutions and better performance have been appearing on the market over the last years. There are Cameras with CCD-sensors containing 4kx4k pixels are available. While these cameras offer a larger sensor their performance is not dramatically better than that of the widely used cameras using 3kx2k sensors. Actually the price/performance of these cameras is poorer. A new camera series offering both 3kx2k and 7kx4k sensors with a custom optics and a high-performance CCD-sensor read-out was designed to break current accuracy barriers. Innovations included in these cameras include custom optics. This allows the optics to be tuned to the application and not just general photography needs. They take advantage of the full dynamic range offered by the CCD-sensor, i.e. use of a 12 bit analog-to-digital converter and 16 bit per pixel in order to take full advantage of the CCD-sensor. They have an optical/mechanical design to assure an extreme geometric stability of the camera. Finally they include an 'on-board' processor to perform all image computations within the camera.

Photogrammetrists have generally made use of wide or super wide angle rectilinear lenses that are designed to reproduce straight lines in the object space as straight lines in the image space. This paper investigates the use of fish-eye optics as an alternative to the conventional super wide-angle lens for vision metrology. The fish-eye design, whilst still retro-focus in construction, rejects the design constraints of rectilinear imaging in exchange for more even illumination across the image format and the expectation of reduced lens aberration effects. By means of a series of practical comparisons, this paper will investigate the target image performance and practical usage of two off-the-shelf 35 mm camera mount lenses, namely an 18 mm focal length super wide- angle rectilinear lens and a 16 mm quasi fish-eye lens. Investigations include: an image quality assessment of retro- target images of differing sizes; an analysis of the variation in retro-target image quality with lens aperture; the influence of the different lenses on the internal consistency of highly over-determined self-calibrating multi-station photogrammetric networks and; an accuracy test between networks obtained with each lens type against measurement of the same targets using an industrial theodolite system.

Data processing, in the field of information technology, requires new tools, involving discrete mathematics, like data compression, signal enhancement, data classification and understanding, hypertexts and multimedia (considering educational aspects too), because the mass of data implies automatic data management and doesn't permit any a priori knowledge. The methodologies and procedures used in this class of problems concern different kinds of segmentation techniques and relational strategies, like clustering, parsing, vectorization, formalization, fitting and matching. On the other hand, the complexity of this approach imposes to perform optimal sampling and outlier detection just at the beginning, in order to define the set of data to be processed: rough data supply very poor information. For these reasons, no hypotheses about the distribution behavior of the data can be generally done and a judgment should be acquired by distribution-free inference only.

The evaluation of the suitability of image compression techniques to 3D PTV images is motivated by a conflict between the bandwidth of data transfer and storage systems and the requirements concerning spatial and temporal resolution of 3D PTV data. In this paper the possibilities of lossless compression are outlined and the influence of lossy JPEG and Wavelet algorithm to PTV particle images is displayed. It is shown that JPEG and Wavelet compression technique have almost the same impact to particle images; both techniques can be used up to a compression ratio of 10 in special cases up to 20. For the lossless case a compression ratio of 5 can be reached with image preprocessing.

This study investigates the extension of scale space techniques for manipulating image sequences for extracting and classifying useful information for further vision processes toward image understanding and sequentially facilitating the acquisition phase of Geographic Information Systems. More precisely, the basic concepts, axioms and the implementational issues of multi-resolution approaches will be discussed for treating continuous monotonic scale changes in images taken in a dynamic environment as in the case when a mobile mapping system captures images at constant time intervals. Moreover, discussion will be devoted to the scale variations and resolution issues, in a stereo-pair and within a sequence, and consequently to the use of an operator with adaptable size for tracking and interpreting spatial scale changes with respect to the registered object space. The analysis of a set of images representing the same object space and by using scale space tracing techniques supported with scale changes in temporal basis results to the characterization of features in which their precise localization is extracted by applying a multiple image matching framework, satisfying a set of criteria and taking into account the geometric and radiometric dissimilarities for accommodating the scale differences. Finally, the above framework can be applied to the localization and to the archiving of traffic signs, reaching high levels of automation in real time in a controlled environment, to the registration of the road centerline and to the post-processing of sequences for producing DEMs and Orthoimagery.

This paper describes a moving object detection method using H.263 coded data. For video surveillance systems, it is necessary to detect unusual states because there are a lot of cameras in the system and video surveillance is tedious in normal states. We examine the information extracted from H.263 coded data and propose a method of detecting alarm events from that information. Our method consists of two steps. In the first step, using motion vector information, a moving object can be detected based on the vector's size and the similarities between the vectors in one frame and the two adjoining frames. In the second step, using DCT coefficients, the detection errors caused by the change of the luminous intensity can be eliminated based on the characteristics of the H.263's DCT coefficients. Thus moving objects are detected by analyzing the motion vectors and DCT coefficients, and we present some experimental results that show the effectiveness of our method.

Today's consumer market digital camcorders offer features which make them appear quite interesting devices for virtual reality data capture. The paper compares a digital camcorder with an analogue camcorder and a machine vision type CCD camera and discusses the suitability of these three cameras for virtual reality applications. Besides the discussion of technical features of the cameras, this includes a detailed accuracy test in order to define the range of applications. In combination with the cameras, three different framegrabbers are tested. The geometric accuracy potential of all three cameras turned out to be surprisingly large, and no problems were noticed in the radiometric performance. On the other hand, some disadvantages have to be reported: from the photogrammetrists point of view, the major disadvantage of most camcorders is the missing possibility to synchronize multiple devices, limiting the suitability for 3-D motion data capture. Moreover, the standard video format contains interlacing, which is also undesirable for all applications dealing with moving objects or moving cameras. Further disadvantages are computer interfaces with functionality, which is still suboptimal. While custom-made solutions to these problems are probably rather expensive (and will make potential users turn back to machine vision like equipment), this functionality could probably be included by the manufacturers at almost zero cost.

The purpose of this work is to quantitate and to evaluate the 'Space Activity.' The Space activity is a kind of quantitative evaluation of influence for moving objects and is evaluated comprehensively, which features are measured quantitatively as elements within the measurement space. In this paper, I present a sequence analysis system of the person's behavior on a vehicle-free promenade applying the space activity. This feature is called the 'covered angle.' The covered angle is defined as the angle covering the other objects which the observation point see. It shows the 'motion density' in the measurement area, the movement of the observer, the frequency of the visible area in the walking space and the visual experience of the observer. The sequence analysis system consists of two parts, the motion analysis and the space activity evaluation. In the first part, pedestrians are extracted from the motion images and tracked. In the second part, I measure each pedestrian's movement which is extracted from the first part, analyze pedestrian's moving behavior, the group behavior and the movement area using previously calculated features, and finally evaluate this space activity. The results are visualized graphs and quantitative images in 2D and 3D animations.

Recently, deformable contour models have lately been used extensively for the detection and localization of boundaries for facilitating the image segmentation problem, and also for the extraction of man-made structures such as roads and buildings from gray level imagery. This study tackles the problem of embedding color image information, coming from different channels in deformable models of contour-type for the extraction and localization of road structures of small width that appear in color imagery mainly of R,G,B content and generally for supporting the verification and revision stages of the mapping process. More precisely, two models will be discussed together with their mathematical modeling and their implementational issues. One transforms the original R,G,B image content into color intensity and color contrast energy terms consisting of the external energy term which is formulated as a regularized-type function within the model which finally unevenly contributing to the radiometric content of the road structure. The other model is based on an optimization method that minimizes the given initial description of the road structure as the energy flow on the color edge field coming from the fusion of the R,G,B content. The solution is taken after reaching global minimum and determining the optimum 'color weighted path' executed in iterative fashion. In both the solution presents the contour, shifted and repositioned, expressing the centerline of the road structure which is localized in sub-pixel means taking into account the peculiarities and characteristics of certain spatial extent. Experimentation in semi-urban, in forested areas using medium scale imagery, and SPOT satellite images show the performance of the above models.

This paper outlines an experience of development of an expert system. The necessity of an expert system which can recognize and track any dynamic object in the field of view a vision system, was the main reason of development of the expert system. For achieving the goals of an expert system, some priorities must be considered. Dialogue is the first priority and it should be understandable by program. Otherwise, if a small error is given to computer, it will result in a large failure of knowledge and the wrong processing. Another priority is the knowledge-base, and this includes the procedural and the declarative memories. The paper will explain the application of above priority in a project of real time object tracking. The obtained results of tracking were compared with the results of surveying intersection so that the ability of the expert system was tested.

In 1977, a paper was presented at the SPIE conference in Reston, Virginia, detailing the computer enhancement of the Zapruder film. The forensic value of this examination in a major homicide investigation was apparent to the viewer. Equally clear was the potential for extracting evidence which is beyond the reach of conventional detection techniques. The cost of this technology in 1976, however, was prohibitive, and well beyond the means of most police agencies. Twenty-two years later, a highly efficient means of image enhancement is easily within the grasp of most police agencies, not only for homicides but for any case application. A PC workstation combined with an enhancement software package allows a forensic investigator to fully exploit digital technology. The goal of this approach is the optimization of the signal to noise ratio in images. Obstructive backgrounds may be diminished or eliminated while weak signals are optimized by the use of algorithms including Fast Fourier Transform, Histogram Equalization and Image Subtraction. An added benefit is the speed with which these processes are completed and the results known. The efficacy of forensic image enhancement is illustrated through case applications.