IEEE SMARTCOMP 2023

Camouflaged object detection is a challenging task in computer vision that involves identifying objects that are intentionally or unintentionally hidden in their surrounding environment. Vision Transformer mechanisms play a critical role in improving the performance of deep learning models by focusing on the most relevant features that help object detection under camouflaged conditions. In this paper, we utilized a vision transformer (VT) in two phases, a) By integrating VT with the deep learning architecture for efficient monocular depth map generation for camouflaged objects and b) By embedding VT in the architecture of the multiclass object detection model with multimodal feature input (RGB with RGB-D) that increases the visual cues and provides more representational information to the model for performance enhancement. Additionally, we performed an ablation study to understand the role of the vision transformer in camouflaged object detection and incorporated GRAD-CAM on top of the model to visualize the performance improvement achieved by embedding the VT in the model architecture. We deployed the model in the resource-constrained edge devices for real-time object detection to realistically test the performance of the trained model.

IoT sensors, especially video cameras, are ubiquitously deployed around the world to perform a variety of computer vision tasks in several verticals including retail, health- care, safety and security, transportation, manufacturing, etc. To amortize their high deployment effort and cost, it is desirable to perform multiple video analytics tasks, which we refer to as Analytical Units (AUs), off the video feed coming out of every camera. As AUs typically use deep learning-based AI/ML models, their performance depend on the quality of the input video, and recent work has shown that dynamically adjusting the camera setting exposed by popular network cameras can help improve the quality of the video feed and hence the AU accuracy, in a single AU setting. In this paper, we first show that in a multi-AU setting, changing the camera setting has disproportionate impact on different AUs performance. In particular, the optimal setting for one AU may severely degrade the performance for another AU, and further the impact on different AUs varies as the environmental condition changes. We then present Elixir, a system to enhance the video stream quality for multiple analytics on a video stream. Elixir leverages Multi-Objective Reinforcement Learning (MORL), where the RL agent caters to the objectives from different AUs and adjusts the camera setting to simultaneously enhance the performance of all AUs. To define the multiple objectives in MORL, we develop new AU-specific quality estimator values for each individual AU. We evaluate Elixir through real-world experiments on a testbed with three cameras deployed next to each other (overlooking a large enterprise parking lot) running Elixir and two baseline approaches, respectively. Elixir correctly detects 7.1% (22,068) and 5.0% (15,731) more cars, 94% (551) and 72% (478) more faces, and 670.4% (4975) and 158.6% (3507) more persons than the default-setting and time-sharing approaches, respectively. It also detects 115 license plates, far more than the time-sharing approach (7) and the default setting (0).

Road damages such as potholes may occur on roads due to aging, which may affect the traffic. Periodic inspections of road damages are difficult due to the high cost of road surveys, and we tend to overlook road damages which is therefore considered to be a problem in a long-term. The development of a system that automatically detects potholes and other road damages from dash cam images can allow inexpensive roadside inspections, and can overall improve the problem of the long-term oversight of road damages. Last year, we conducted a demonstration experiment in Edogawa City, Tokyo, using an existing image-based road damage detection method. From that experiment, we confirmed that the detection of potholes on actual roads often causes false detections due to the presence of shadows and manholes. In this study, we propose a method to reduce false positives in pothole detection which was considered to be a problem through the demonstration experiment, and evaluate its performance. Since we believe that the evaluation based on a pothole-only dataset is not valid, we reconstruct a dataset for evaluation by adding shadow and manhole images for validation. Our method consists of two main components: data expansion by image generation and ensemble of classification mechanisms for object detection models. As a result of the validation on the reconstructed pothole dataset, the average precision (AP), which is a measure to evaluate false positives, was improved by 0.172 compared to the existing method. In addition, the reduction of AR (average Recall), which is a trade-off with AP, was suppressed to 0.07. Since our method is not dependent on the domain of potholes, it is expected to be an effective pipeline in tasks and situations where false positives are more problematic than false negatives due to the high incidence of false positives.

The annotation load for a new dataset has been greatly decreased using domain adaptation based semantic segmentation, which iteratively constructs pseudo labels on unlabeled target data and retrains the network. However, realistic segmentation datasets are often imbalanced, with pseudo-labels tending to favor certain head classes while neglecting other tail classes. This can lead to an inaccurate and noisy mask. To address this issue, we propose a novel hard sample mining strategy for an active domain adaptation based semantic segmentation network, with the aim of automatically selecting a small subset of labeled target data to fine-tune the network. By calculating class-wise entropy, we are able to rank the difficulty level of different samples. We use a fusion of focal loss and regional mutual information loss instead of cross-entropy loss for the domain adaptation based semantic segmentation network. Our entire framework has been implemented in real-time using the Robotics Operating System (ROS) with a server PC and a small Unmanned Ground Vehicle (UGV) known as the ROSbot2.0 Pro. This implementation allows ROSbot2.0 Pro to access any type of data at any time, enabling it to perform a variety of tasks with ease. Our approach has been thoroughly evaluated through a series of extensive experiments, which demonstrate its superior performance compared to existing state-of-the-art methods. Remarkably, by using just 20% of hard samples for fine-tuning, our network has achieved a level of performance that is comparable (≈ 88%) to that of a fully supervised approach, with mIOU scores of 60.51% in the In-house dataset.