Stereo Vision

Autonomous vehicles stand at the forefront of technological advancement, promising to alter fundamentally how we navigate, with implications for everything from inter city logistics to daily commuting. The expected benefits of this technology include advancements in safety protocols, streamlining of traffic flow, and superior comfort for those onboard. However, the path to widespread integration is paved with signifi cant barriers that need to be overcome. This article provides a thorough review of the historical development of autonomous driving technologies, beginning with the early vehicles powered by internal combustion engines and extending to the present excite ment over Apple’s Project Titan. Rumors suggest this particular project involves the creation of an electric vehicle fully capable of operating without traditional manual controls, including steering wheels and seats. The article concludes by spotlighting the potential for autonomous vehicles to enable disruptive changes in the transport sector, while also considering the myriad of challenges that must be surmounted for their adoption on a grand scale.

With the advent of image and video representation of visual scenes in digital computer, subsequent necessity of vision-substitution representation of a given image is felt. The medium for non-visual representation of an image is chosen to be sound due to well developed auditory sensing ability of human beings and wide availability of cheap audio hardware. Visionary information of an image can be conveyed to blind and partially sighted persons through auditory representation of the image within some of the known limitations of human hearing system. The research regarding image sonification has mostly evolved through last three decades. The paper also discusses in brief about the reverse mapping, termed as sound visualization. This survey approaches to summarize the methodologies and issues of the implemented and unimplemented experimental systems developed for subjective sonification of image scenes and let researchers accumulate knowledge about the previous direction of researches in this domain.

Manual palpation of tissue is frequently used in open surgery, e.g., for localization of tumors and buried vessels and for tissue characterization. The overall objective of this work is to explore how tissue palpation can be performed in Robot-Assisted Minimally Invasive Surgery (RAMIS) using laparoscopic instruments conventionally used in RAMIS. This thesis presents a framework where a surgical tool is moved teleoperatively in a manner analogous to the repetitive pressing motion of a finger during manual palpation. We interpret the changes in parameters due to this motion such as the applied force and the resulting indentation depth to accurately determine the variation in tissue stiffness. This approach requires the sensorization of the laparoscopic tool for force sensing. In our work, we have used a da Vinci R © needle driver which has been sensorized in our lab at CSTAR for force sensing using Fiber Bragg Grating (FBG). A computer vision algorithm has been developed for 3D surgi...

Non-linear optimization on constraint graphs has recently been applied very successfully in a variety of SLAM backends. We combine this technique with a principled way of handling non-Euclidean spaces, 3D orientations in particular, based on manifolds to build a generic and very flexible framework, the Manifold Toolkit for Matlab (MTKM). We show that MTKM makes it particularly easy to solve non-trivial multi-sensor calibration problems while remaining generic enough to handle a very different class of problems, namely SLAM, as well: After an introductory example on single camera calibration we apply MTKM to calibration of stereo vision and IMU w.r.t. the kinematic chain of a service robot, RGB-D and accelerometer calibration of a Microsoft Kinect, stereo calibration on a Nao soccer robot, and several SLAM benchmark data sets illustrating MTKM's versatility. MTKM and all presented examples are available as open source from .

This paper presents a new fingertip detection method that is based on the top-hat transform. A hand in an image is assumed to be segmented in advance. The palm of the hand is obtained using the morphological opening operation. Fingertips are then obtained as the opening residue, namely, the difference between the input image and the palm image. The performance of this approach is compared with three other techniques: (1) the convex hull algorithm, (2) the Kanade-Lucas-Tomasi (KLT) feature tracker, and (3) the SUSAN corner detector. Simulation results show that the method (1) is ineffective for closed hands. The methods (2) and (3) tend to respond falsely to many non-fingertip points. By contrast, the proposed method can detect fingertips with more than 90% success rates even for closed hands where the fingers are in contact with each other. Keywords: Fingertip detection; Top-hat transform; Convex hull; KLT feature tracker; SUSAN corner detector.

Medical imaging tasks, such as segmentation, 3D modeling, and registration of medical images, involve complex geometric problems, usually solved by standard linear algebra and matrix calculations. In the last few decades, conformal geometric algebra (CGA) has emerged as a new approach to geometric computing that offers a simple and efficient representation of geometric objects and transformations. However, the practical use of CGA-based methods for big data image processing in medical imaging requires fast and efficient implementations of CGA operations to meet both real-time processing constraints and accuracy requirements. The purpose of this study is to present a novel implementation of CGA-based medical imaging techniques that makes them effective and practically usable. The paper exploits a new simplified formulation of CGA operators that allows significantly reduced execution times while maintaining the needed result precision. We have exploited this novel CGA formulation to re-design a suite of medical imaging automatic methods, including image segmentation, 3D reconstruction and registration. Experimental tests show that the re-formulated CGA-based methods lead to both higher precision results and reduced computation times, which makes them suitable for big data image processing applications. The segmentation algorithm provides the Dice index, sensitivity and specificity values of 98.14%, 98.05% and 97.73%, respectively, while the order of magnitude of the errors measured for the registration methods is 10 -5 .

Adequate interfaces are an essential part of any Decision Support System (DSS) since they can contribute to a more efficient use of that system. Also, a user-friendly design can facilitate the acceptance and utilization of the software by its end users. This work presents the development process of the interfaces designed for a DSS that is being developed in order to help decision-making in a context of Sensory Analysis. That process began with the construction of mock-ups and, for that purpose, the tool chosen was Lumzy, a web-based wireframing tool for rapid user-interface prototyping.

Abslrod-Minimally iovasive robotic surgery systems has entered daily practice in dedicated clinical centers. Espe- cially heart surgery profits from this new technique, due to a higher accuracy compared to conventional endoscopic interventions. Nevertheless some drawbacks have restricted a broader acceptance of these devices. The most urgent issues are lack of haptic feedback and prolonged operation time. Our research project tackles both topics, on the one hand by adding high fidelity force-feedhack, on the other hand hy automating recurrent manipulation tasks. These features have been integrated into the Endo[PAlR (Endoscopic Partlally-Autonomous Robot) system, an open evaluation platform for robotic surgery. The setup eompri-two low-payload robots equipped with sensorized surgical instruments and a third robot carrying a s t e m camera system. Trocar kinematics, enabling surgical manipulation through small incisions, has been implemented for all robotic arms. I n order to ensure operation close to reality, a thorax and heart phantom for surgical training was used in the experiments. Stereo vision is provided via a head-mounted display and force-reflective input de%,ices are employed for user interaction. The system was evaluated by surgeons and it was capable of performing autonomous knot-tying.

R esum e Le d eveloppement d e l a n o r m e mpeg-4 met en avant l'importance des manipulations de vid eo bas ees objet telles que la composition de sc ene vid eo 2D. Le d eveloppement d'algorithmes semi-automatiques permettant d e r ealiser ce type d'applications n ecessite l'utilisation d'informations pertinentes sur le comportement des objets vid eo dans la s equence originale. Pour cela, cet article propose une m ethode pour d ecouper une s equence en segments temporels o u c haque segment correspond a une orientation objet cam era constante ou non. Ensuite, une vue clef peut être extraite pour chaque segment. Cette d ecomposition d'une s equence d'objet vid eo est bas ee sur l' etude et la repr esentativit e d e c hacun des param etres d'un mod ele a ne de mouvement.

In this paper, a depth detection based on monocular vision coupled with the motion was used to predict the possible collisions on pedestrian crossings. A video taken while in motion is extracted into frames and processed with skin color detection methods to identify the face blobs of pedestrians. The distance to the pedestrian was determined using the difference in size of the face blobs in the consecutive image frames. The field tests show that the size variation does not depend on the speed of the vehicle but fewer steps are ...

In this paper we present a panoramic depth imaging system. The system is mosaic-based which means that we use a single rotating camera and assemble the captured images in a mosaic. Due to a setoff of the camera's optical center from the rotational center of the system we are able to capture the motion parallax effect which enables stereo reconstruction. The camera is rotating on a circular path with a step defined by the angle, equivalent to one pixel column of the captured image. The equation for depth estimation can be easily extracted from the system geometry. To find the corresponding points on a stereo pair of panoramic images the epipolar geometry needs to be determined. It can be shown that the epipolar geometry is very simple if we are doing the reconstruction based on a symmetric pair of stereo panoramic images. We get a symmetric pair of stereo panoramic images when we take symmetric pixel columns on the left and on the right side from the captured image center column...

This article deals with the fusion of two initially independent systems. The first system utilizes panoramic images and rotational geometry to create stereo image pairs and recover depth information using a single camera. The second system utilizes dense depth estimates in order to drive a process, which improves edge detection by factoring in the distance (scale) from the camera at each point. Distance data is fed to a smoothing pre-processor, which splits the image into layers based on pixel distance and independently smoothes each one, utilizing a different sigma. This variation of sigma across image depth ensures that noise is eliminated with minimal loss of overall detail. Significant improvements to the results of subsequent edge detection operations are observed for images with great depth variations. The methods utilized by both systems are analysed and their benefits and drawbacks are exposed. The process of integration is also detailed, outlining problems and solutions that were adopted. The applications and constraints of the system are also discussed.

Mobile robot navigation in unknown environments requires the concurrent estimation of the mobile robot localization with respect to a base reference and the construction of a global map of the navigation area. In this paper we present a comparative study of the performance of the localization and map building processes using two distinct sensorial systems: a rotating 2D laser range nder, and a trinocular stereo vision system.

In this paper we discuss an approach towards passively observing students in a driving simulator. The goal is to enhance the learning experience for students taking lessons in this simulator. To this end, a virtual driving instructor is provided with added information consisting of the gaze behavior of its student. The gaze behavior is defined by estimated head locations and orientations. The learning experience for the student is enhanced by providing added feedback to the student based on his observed behavior.

A new approach for the stereovision problem is presented, aiming to increase the accuracy of stereo reconstruction. The proposed method is edge-based and consists of the correlation of left and right contours, detected with sub-pixel accuracy. The steps of the stereo matching process are: segmentation of each contour into basic contours (strongly- and weakly-curved), detection of correspondences between left-right image basic contours, and computation with sub-pixel accuracy of the pairs of corresponding left-right edge points. By consequence, one achieves the highest correlation and 3D reconstruction accuracy. 3D lane detection, based on a clothoidal model, is evaluated with the proposed stereo algorithm on synthetic and real world images

Gaze control represents an important issue in the interaction between a robot and humans. Specifically, deciding who to pay attention to in a multi-party conversation is one way to improve the naturalness of a robot in human-robot interaction. This control can be carried out by means of two different models that receive the stimuli produced by the participants in an interaction, either an on-center off-surround competitive network or a recurrent neural network. A system based on a competitive neural network is able to decide who to look at with a smooth transition in the focus of attention when significant changes in stimuli occur. An important aspect in this process is the configuration of the different parameters of such neural network. The weights of the different stimuli have to be computed to achieve human-like behavior. This article explains how these weights can be obtained by solving an optimization problem. In addition, a new model using a recurrent neural network with LSTM...

This paper presents an approach for road detection based on image segmentation. This segmentation is resulted from merging 2D and 3D image processing data from a stereo vision system. The 2D layer returns a matrix containing pixel's clusters based on the Watershed transform. Whereas the 3D layer return labels, that are classified by the V-Disparity technique, to free spaces, obstacles and non-classified area. Thus, a feature's descriptor for each cluster is composed with features from both layers. The road pattern recognition was performed by an artificial neural network, trained to obtain a final result from this feature's descriptor. The proposed work reports real experiments carried out in a challenging urban environment to illustrate the validity and application of this approach.

In this paper we describe a perceptual interface which allows to navigate in a virtual D environment by interpreting the movements of the face of human users. The core of the proposed approach consists of a face tracker, which is based on the computation and analysis of integral projections. Applying some predefined heuristics, the location and pose of the face are first estimated and, then, transformed in control signals for the navigation in the virtual world. The system has been implemented using OpenCV and DirectX 9. Some experimental results are shown, proving the feasibility of the proposed technique.

This article presents an innovative probabilistic approach for building volumetric maps of unknown environments with autonomous mobile robots, which is based on information theory. Each mobile robot uses an entropy gradient-based exploration strategy, with the aim of maximizing information gain when building and improving a 3-D map upon measurements yielded by an on-board stereo-vision sensor. The proposed framework was validated through experiments with a real mobile robot equipped with stereo-vision, in order to be further used on cooperative volumetric mapping with teams of mobile robots.

Mobile robotic systems have evolved to include sensors capable of truthfully describing robot status and operating environment as accurately and reliably as never before. This possibility is challenged by effective sensor data exploitation, because of the cognitive load an operator is exposed to, due to the large amount of data and time-dependency constraints. This paper addresses this challenge in remote-vehicle teleoperation by proposing an intuitive way to present sensor data to users by means of using mixed reality and visual aids within the user interface. We propose a method for organizing information presentation and a set of visual aids to facilitate visual communication of data in teleoperation control panels. The resulting sensor-information presentation appears coherent and intuitive, making it easier for an operator to catch and comprehend information meaning. This increases situational awareness and speeds up decision-making. Our method is implemented on a real mobile robotic system operating outdoor equipped with on-board internal and external sensors, GPS, and a reconstructed 3D graphical model provided by an assistant drone. Experimentation verified feasibility while intuitive and comprehensive visual communication was confirmed through an assessment, which encourages further developments.

This paper 1 presents a method for Simultaneous Localization and Mapping (SLAM) relying on a monocular camera as the only sensor which is able to build outdoor, closedloop maps much larger than previously achieved with such input. Our system, based on the Hierarchical Map approach [1], builds independent local maps in real-time using the EKF-SLAM technique and the inverse depth representation proposed in [2]. The main novelty in the local mapping process is the use of a data association technique that greatly improves its robustness in dynamic and complex environments. A new visual map matching algorithm stitches these maps together and is able to detect large loops automatically, taking into account the unobservability of scale intrinsic to pure monocular SLAM. The loop closing constraint is applied at the upper level of the Hierarchical Map in near real-time. We present experimental results demonstrating monocular SLAM as a human carries a camera over long walked trajectories in outdoor areas with people and other clutter, even in the more difficult case of forward-looking camera, and show the closing of loops of several hundred meters.

Place recognition is a challenging task in any SLAM system. Algorithms based on visual appearance are becoming popular to detect locations already visited, also known as loop closures, because cameras are easily available and provide rich scene detail. These algorithms typically result in pairs of images considered depicting the same location. To avoid mismatches, most of them rely on epipolar geometry to check spatial consistency. In this paper we present an alternative system that makes use of stereo vision and combines two complementary techniques: bag-of-words to detect loop closing candidate images, and conditional random fields to discard those which are not geometrically consistent. We evaluate this system in public indoor and outdoor datasets from the Rawseeds project, with hundred-metre long trajectories. Our system achieves more robust results than using spatial consistency based on epipolar geometry.

We propose a place recognition algorithm for simultaneous localization and mapping (SLAM) systems using stereo cameras that considers both appearance and geometric information of points of interest in the images. Both near and far scene points provide information for the recognition process. Hypotheses about loop closings are generated using a fast appearance-only technique based on the bag-of-words (BoW) method. We propose several important improvements to BoWs that profit from the fact that, in this problem, images are provided in sequence. Loop closing candidates are evaluated using a novel normalized similarity score that measures similarity in the context of recent images in the sequence. In cases where similarity is not sufficiently clear, loop closing verification is carried out using a method based on conditional random fields (CRFs). We build on CRF matching with two main novelties: We use both image and 3-D geometric information, and we carry out inference on a minimum spanning tree (MST), instead of a densely connected graph. Our results show that MSTs provide an adequate representation of the problem, with the additional advantages that exact inference is possible and that the computational cost of the inference process is limited. We compare our system with the state of the art using visual indoor and outdoor data from three different locations and show that our system can attain at least full precision (no false positives) for a higher recall (fewer false negatives).

Mobile robot navigation in unknown environments requires the concurrent estimation of the mobile robot localization with respect to a base reference and the construction of a global map of the navigation area. In this paper we present a comparative study of the performance of the localization and map building processes using two distinct sensorial systems: a rotating 2D laser range nder, and a trinocular stereo vision system.

This paper provides an introduction to the Simultaneous Localization and Mapping (SLAM) problem and the extensive research conducted over the years to solve the problem. SLAM is the process by which a mobile robot builds a map of its surroundings and also determines is location at the same time using the map. Also, this paper introduces the algorithms used to solve the problem, their performance and their advantages and disadvantages over each other.

This paper presents a novel architecture to generate a world model in terms of mesh from a continuous image stream with depth information extracted from a robot's ego-vision camera. We propose two algorithms for planar and non-planar mesh generation. A Cartesian grid-based mesh fitting algorithm is proposed for mesh generation of planar objects. For mesh generation of non-planar objects, we propose a Self Organization Map based algorithm. The proposed algorithm better approaches the boundary and overall shape of the objects compared to State-Of-the-Art (SOA). Extensive experiments done on three public datasets show that our method surpasses SOA both qualitatively and quantitatively.