Remote Drone Inspection with Gaussian Splat Inspection Platform and VR Teleoperation

On January 11, 2025

Researcher: Max Midwinter, Kay Han, Raza Rizvi

Description: Over the past several months, the CViSS Lab has advanced remote infrastructure inspection by leveraging Gaussian Splatting technology. The TowerEye AI web inspection tool allows users to intuitively inspect structures like towers by specifying drone navigation points remotely. This project involved developing a custom application for the DJI drone controller using the DJI SDK and ROS2, enabling seamless control of drone operations. While this work is ongoing, we welcome collaborations on the DJI-ROS2 app or contributions to the TowerEye platform.

Interactive 3DGS Bounding Box Extraction by 2D Segmentation Prior

On January 11, 2025

Researcher: Huibin Li

Description: This study proposes an interactive scheme to reconstruct Gaussian Splatting (GS) scenes and export their 3D oriented bounding boxes using a point prompt by users. By comparing 2D instance-level segmentation with pseudo ground truth generated by the Segment Anything Model (SAM), we can synthesize novel views and their semantic segmentations to differentiate objects in 3D scenes. A trained GS model reconstructs the 3D scene from posed RGB images as an input, assigning a 32-dimensional feature to each Gaussian primitive, assuming each ellipsoid belongs to one object. Through ray sampling, similarity calculations, and feature refinement, a 3D GS is trained to represent instance features, enabling object extraction.

Zero-Shot Instance Segmentation of Cell Tower Components

On January 11, 2025

Researcher: Ali Lesani

Description: This research proposes a zero-shot instance segmentation pipeline for automated infrastructure inspection, addressing the challenges of traditional human inspections and supervised learning approaches that require extensive labeled data. The pipeline first uses saliency object detection to separate foreground from background elements, followed by depth map analysis to recover any lost foreground information. A multimodal object detection model then generates bounding boxes around specific components of interest in the foreground. These bounding boxes serve as prompts for SAM (Segment Anything), enabling precise segmentation of assets. This approach eliminates the need for extensive labeled training data typically required for fine-tuning deep learning models, while maintaining accuracy in complex real-world scenarios with challenging backgrounds. 

Introducing F3: A Robust Solution for Automated First-Floor Height Estimation

On January 11, 2025

Researcher: Fuad Hasan

Description: Discover the future of flood risk assessment with F3, a cutting-edge approach for automated first-floor height estimation. This video explains how F3 handles occlusion and real-world challenges using deep feature fusion and advanced height estimation techniques.

Web Inspection Tool with Gaussian Splats

On Feburuary 23, 2024

Researcher: Max Midwinter

Description: The objective is to create a simple and real-time framework to use Gaussian Splatting to advance visual inspection applications (https://github.com/MACILLAS/GS_Stream).

Development of Community Data Collection Platform

On March 22, 2023

Researcher: Huaiyuan Weng

Description: We have created a bike scanning system that captures high-resolution images and point cloud data of buildings. The collected data will be utilized to analyze building features and assess potential losses caused by natural hazards.

Human-machine Collaborative and Distributive Inspection

On December 03, 2022

Researcher: Zaid Abbas Al-Sabbag

Description: To modernize how inspections are performed, we have developed a high-tech solution which allows robots and inspectors to collaborate more effectively to perform inspections using mixed reality headsets.

Defect Detection and Quantification for Visual Inspection

On May 13, 2022

Researcher: Rishab Bajaj, Max Midwinter, Zaid Abbas Al-Sabbag

Description: We propose an unsupervised semantic segmentation method (USP), based on unsupervised learning of image segmentation inspired by differentiable feature clustering coupled with a novel outlier rejection and stochastic consensus mechanism for mask refinement. Also, based on the segmentaion region, damage regions are reconstructed in 3D for quantitative evaluation.

Interactive Defect Quantification through Extended Reality

On September 15, 2021

Researcher: Zaid Abbas Al-Sabbag

Description: A new visual inspection method that can interactively detect and quantify structural defects using an Extended Reality (XR) device (headset) is proposed. The XR device, which is at the core of this method, supports an interactive environment using a holographic overlay of graphical information on the spatial environment and physical objects being inspected. By leveraging this capability, a novel XR-supported inspection pipeline, called eXtended Reality-based Inspection and Visualization (XRIV), is developed. Key tasks supported by this method include detecting visual damage from sensory data acquired by the XR device, estimating its size, and visualizing (overlaying) information on the spatial environment.

Project page: Github

Scale Estimation

On June 18, 2020

Researcher: Ju An Park

Description: Computer vision-based inspection solutions used for detection of features, such as structure components and defects often lack methods to determine scale information. Knowing image scale allows the user to quantitatively evaluate regions-of-interest to a physical scale (e.g. length/area estimations of features). To address this challenge, a learning-based scale estimation technique is proposed. The underlying assumption is that the surface texture of structures, captured in images, contains enough information to estimate scale for each corresponding image (e.g., pixel/mm). In this work, a regression model is trained to establish the relationship between surface textures, captured in images, and scales. A convolutional neural network is trained to extract scale-related features from textures captured in images. Then, the trained model can be exploited to estimate scales for all images that are captured from a structure’s surfaces with similar textures.

Project page: Github

Autonomous Image Localization

On September 28, 2018

Researcher: Chul Min Yeum

Description: A novel automated image localization and classification technique is developed to extract the regions-of-interest (ROIs) on each of the images, which contain the targeted region for inspection (TRI). ROIs are extracted here using structure-from-motion. Less useful ROIs, such as those corrupted by occlusions, are then filtered effectively using a robust image classification technique, based on convolutional neural networks. Then, such highly relevant ROIs are available for visual assessment. The capability of the technique is successfully demonstrated using a full-scale highway sign truss with welded connections.

Project page: Github

Vision-Based Automated Crack Detection

On June 13, 2015

Researcher: Chul Min Yeum

Description: A new vision-based visual inspection technique is proposed by automatically processing and analyzing a large volume of collected images from unspecified locations using computer vision algorithm. By evaluating images from many different angles and utilizing knowledge of a fault’s typical appearance and characteristics, the proposed technique can successfully detect faults on a structure.

Project page: Github