Publications

2022

1. IV

   Infra Sim-to-Real : An efficient baseline and dataset for Infrastructure based Online Object Detection and Tracking using Domain Adaptation

   Shyam, Pranjay, Mishra, Sumit, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In 2022 IEEE Intelligent Vehicles Symposium 2022

   Abs

   Increasing usage of traffic cameras provides an opportunity to utilize them for smart city applications. However, the efficacy of such systems is determined by their ability to detect and track objects of interest from diverse viewpoints accurately. This is challenging due to the diverse viewpoints, elevations, and distinct properties of camera sensors. Thus, to ensure robust performance, the training dataset should cover many variations, including viewpoints, illumination changes, and diverse weather conditions. However, constructing such a dataset is expensive in terms of data collection and annotation. This paper proposes an unsupervised domain adaptation approach wherein a synthetic dataset is generated using a simulator and subsequently used to ensure performance consistency of multi-object-tracking (MOT) algorithms across a diverse range of manually annotated natural scenes. Towards this end, we emphasize achieving domain invariant object detection by combining image stylization and class-balancing augmentation. Furthermore, we extend the robust detection algorithm to track detected objects across a large time scale using feature embeddings generated by the detector. Based on qualitative and quantitative results, we demonstrate the viability of such a system that is invariant to illumination, weather, viewpoint, and scene changes while providing a baseline for future research.

2. arXiv

   Holisitc Scene Understanding from Disjoint Labels via Multi-Task Stereo Perception

   Shyam, Pranjay, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In arXiv 2022

   Abs

   The perception system of an autonomous vehicle relies on the computation of multiple attributes to mathematically represent surrounding information. Presently these attributes are computed independently using task-specific networks, which result in high computational costs. While the alternative of using a multi-task formulation is highly desirable given its ability to reduce the computational cost, a significant caveat restricting the deployment of such algorithms is their requirement of aligned multi-task labels corresponding to each image. This paper explores the realm of training a multi-task network using pseudo labels generated by task-specific networks using disjoint labels from (1) same and (2) different distributions. Specifically given a task with the labeled dataset, we train teacher models and generate aligned pseudo labels for an unlabelled dataset. These pseudo labels are used to train a multi-task network in a supervised manner. Given this training framework, we evaluate the performance of a baseline multi-task stereo network for performing scene representation using 6 tasks comprising coarse (2D detection) and dense predictions (depth estimation, semantic segmentation, road attribute segmentation, instance segmentation, and panoptic segmentation). We empirically demonstrate this approach to outperform teacher networks highlighting strong scene understanding capability supported by learning a common feature representation. We also demonstrate such networks to demonstrate domain invariant performance, presenting a promising direction to train multi-task learning algorithms using an unlabelled training dataset.

3. arXiv

   Enhancing Self-Supervised Monocular Depth Estimation via Feature Enhancement and Geometric Constraints

   Shyam, Pranjay, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In arXiv 2022

   Abs Code

   Self-Supervised learning provides an alternative for training monocular depth estimation networks without requiring high-density ground truth. However, performance bottlenecks within networks trained using such a formulation are observable as blurry artifacts and poor estimation around edges. To reduce these artifacts and improve predictions around edges without increasing computational complexity, we revisit the design space of underlying networks and propose modifications to improve feature aggregation and representation. We highlight inefficient feature utilization from encoders, sub-optimal upsampling, and lack of geometric constraints to restrict the fidelity of depth estimations. To overcome these, we propose (a) replacement of feature extractor within the encoder to better leverage features across multiple scales, (b) improving feature aggregation and representation within decoder via sub-pixel convolutions, and (c) integrating a lightweight attention mechanism to capture long-range dependencies. Furthermore, to enforce geometric constraints in the form of scene semantics, we propose integrating semantic segmentation as an auxiliary decoder branch and ensure feature sharing using bidirectional-cross-task attention mechanism (BCTAM). We empirically demonstrate the efficacy of the proposed formulation multiple datasets and report state-of-the-art (SoTA) performance vis-a-vis prior approaches.

4. arXiv

   Multi-Source Domain Alignment for Robust Segmentation in Unknown Targets

   Shyam, Pranjay, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In arXiv 2022

   Abs

   Semantic segmentation provides scene understanding capability by performing pixel-wise classification of objects within an image. However, the sensitivity of such algorithms towards domain changes requires fine-tuning using an annotated dataset for each novel domain, which is expensive to construct and inefficient. We highlight that irrespective of the training dataset, structural properties of scenes remain the same hence domain sensitivity arises from training methodology. Thus, in this paper, we propose a domain alignment approach wherein multiple synthetic source domains are used to train an underlying segmentation network such that it performs consistently in unknown real target domains. Towards this end, we propose a pixel-wise supervised contrastive learning framework that enforces constraints in latent space resulting in features belonging to the same class being clustered closely and away from different classes. This approach allows for better capturing of global and local semantics while providing domain invariant properties. Our approach can be easily incorporated into prior semantic segmentation approaches without the significant computational overhead. We empirically demonstrate the efficacy of the proposed approach on GTAV \to Cityscapes, GTAV+Synthia \to Cityscapes, and GTAV+Synthia+Synscapes \to Cityscapes scenarios and report state-of-the-art (SoTA) performance without requiring access to images from the target domain.

5. CVPR

   GIQE: Generic Image Quality Enhancement via N^th Order Iterative Degradation

   Shyam, Pranjay, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022

   Abs

   Visual degradations caused by motion blur, raindrop, rain, snow, illumination, and fog deteriorate image quality and, subsequently, the performance of perception algorithms deployed in outdoor conditions. While degradation-specific image restoration techniques have been extensively studied, such algorithms are domain sensitive and fail in real scenarios where multiple degradations exist simultaneously. This makes a case for blind image restoration and reconstruction algorithms as practically relevant. However, the absence of a dataset diverse enough to encapsulate all variations hinders development for such an algorithm. In this paper, we utilize a synthetic degradation model that recursively applies sets of random degradations to generate naturalistic degradation images of varying complexity, which are used as input. Furthermore, as the degradation intensity can vary across an image, the spatially invariant convolutional filter cannot be applied for all degradations. Hence to enable spatial variance during image restoration and reconstruction, we design a transformer-based architecture to benefit from the long-range dependencies. In addition, to reduce the computational cost of transformers, we propose a multi-branch structure coupled with modifications such as a complimentary feature selection mechanism and the replacement of a feed-forward network with lightweight multiscale convolutions. Finally, to improve restoration and reconstruction, we integrate an auxiliary decoder branch to predict the degradation mask to ensure the underlying network can localize the degradation information. From empirical analysis on 10 datasets covering rain drop removal, deraining, dehazing, image enhancement, and deblurring, we demonstrate the efficacy of the proposed approach while obtaining SoTA performance.

6. arXiv

   DGSS : Domain Generalized Semantic Segmentation using Iterative Style Mining and Latent Representation Alignment

   Shyam, Pranjay, Bangunharcana, Antyanta, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In arXiv preprint arXiv:2202.13144 2022

   Abs arXiv

   Semantic segmentation algorithms require access to well-annotated datasets captured under diverse illumination conditions to ensure consistent performance. However, poor visibility conditions at varying illumination conditions result in laborious and error-prone labeling. Alternatively, using synthetic samples to train segmentation algorithms has gained interest with the drawback of domain gap that results in sub-optimal performance. While current state-of-the-art (SoTA) have proposed different mechanisms to bridge the domain gap, they still perform poorly in low illumination conditions with an average performance drop of - 10.7 mIOU. In this paper, we focus upon single source domain generalization to overcome the domain gap and propose a two-step framework wherein we first identify an adversarial style that maximizes the domain gap between stylized and source images. Subsequently, these stylized images are used to categorically align features such that features belonging to the same class are clustered together in latent space, irrespective of domain gap. Furthermore, to increase intra-class variance while training, we propose a style mixing mechanism wherein the same objects from different styles are mixed to construct a new training image. This framework allows us to achieve a domain generalized semantic segmentation algorithm with consistent performance without prior information of the target domain while relying on a single source. Based on extensive experiments, we match SoTA performance on SYNTHIA \to Cityscapes, GTAV \to Cityscapes while setting new SoTA on GTAV \to Dark Zurich and GTAV \to Night Driving benchmarks without retraining.

7. RS

   Robotic Mapping Approach under Illumination-Variant Environments at Planetary Construction Sites

   Hong, Sungchul, Shyam, Pranjay, Bangunharcana, Antyanta, and Shin, Hyuseoung

   Remote Sensing 2022

   Abs paper

   With the recent discovery of water-ice and lava tubes on the Moon and Mars along with the development of in-situ resource utilization (ISRU) technology, the recent planetary exploration has focused on rover (or lander)-based surface missions toward the base construction for long-term human exploration and habitation. However, a 3D terrain map, mostly based on orbiters’ terrain images, has insufficient resolutions for construction purposes. In this regard, this paper introduces the visual simultaneous localization and mapping (SLAM)-based robotic mapping method employing a stereo camera system on a rover. In the method, S-PTAM is utilized as a base framework, with which the disparity map from the self-supervised deep learning is combined to enhance the mapping capabilities under homogeneous and unstructured environments of planetary terrains. The overall performance of the proposed method was evaluated in the emulated planetary terrain and validated with potential results.

2021

1. AAAI

   Towards Domain Invariant Single Image Dehazing

   Shyam, Pranjay, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In Proceedings of the AAAI Conference on Artificial Intelligence 2021

   Abs PDF

   Presence of haze in images obscures underlying information, which is undesirable in applications requiring accurate environment information. To recover such an image, a dehazing algorithm should localize and recover affected regions while ensuring consistency between recovered and its neighboring regions. However owing to fixed receptive field of convolutional kernels and non uniform haze distribution, assuring consistency between regions is difficult. In this paper, we utilize an encoder-decoder based network architecture to perform the task of dehazing and integrate an spatially aware channel attention mechanism to enhance features of interest beyond the receptive field of traditional conventional kernels. To ensure performance consistency across diverse range of haze densities, we utilize greedy localized data augmentation mechanism. Synthetic datasets are typically used to ensure a large amount of paired training samples, however the methodology to generate such samples introduces a gap between them and real images while accounting for only uniform haze distribution and overlooking more realistic scenario of non-uniform haze distribution resulting in inferior dehazing performance when evaluated on real datasets. Despite this, the abundance of paired samples within synthetic datasets cannot be ignored. Thus to ensure performance consistency across diverse datasets, we train the proposed network within an adversarial prior-guided framework that relies on a generated image along with its low and high frequency components to determine if properties of dehazed images matches those of ground truth. We preform extensive experiments to validate the dehazing and domain invariance performance of proposed framework across diverse domains and report state-of-the-art (SoTA) results.

2. BMVC

   Lightweight hdr camera isp for robust perception in dynamic illumination conditions via fourier adversarial networks

   Shyam, Pranjay, Yoon, Kuk-Jin, Sengar, Sandeep Singh, and Kim, Kyung-Soo

   In The 32nd British Machine Vision Conference, BMVC 2021

   Abs PDF

   The limited dynamic range of commercial compact camera sensors results in an inac- curate representation of scenes with varying illumination conditions, adversely affecting image quality and subsequently limiting the performance of underlying image process- ing algorithms. Current state-of-the-art (SoTA) convolutional neural networks (CNN) are developed as post-processing techniques to independently recover under-/over-exposed images. However, when applied to images containing real-world degradations such as glare, high-beam, color bleeding with varying noise intensity, these algorithms amplify the degradations, further degrading image quality. We propose a lightweight two-stage image enhancement algorithm sequentially balancing illumination and noise removal us- ing frequency priors for structural guidance to overcome these limitations. Furthermore, to ensure realistic image quality, we leverage the relationship between frequency and spatial domain properties of an image and propose a Fourier spectrum-based adversar- ial framework (AFNet) for consistent image enhancement under varying illumination conditions. While current formulations of image enhancement are envisioned as post- processing techniques, we examine if such an algorithm could be extended to integrate the functionality of the Image Signal Processing (ISP) pipeline within the camera sensor benefiting from RAW sensor data and lightweight CNN architecture. Based on quanti- tative and qualitative evaluations, we also examine the practicality and effects of image enhancement techniques on the performance of common perception tasks such as object detection and semantic segmentation in varying illumination conditions

3. IJCAI-W

   Evaluating COPY-BLEND Augmentation for Low Level Vision Tasks

   Shyam, Pranjay, Sengar, Sandeep Singh, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In arXiv preprint arXiv:2103.05889 2021

   Abs paper

   Image restoration and enhancement algorithms aim to improve image quality for ensuring consistent performance of high level vision tasks such as object detection, semantic segmentation, image classification, etc in different adverse conditions. State-of-the-art (SoTA) approaches have relied on brute force combination of large scale datasets, deep convolutional neural networks (CNNs) and data augmentation techniques to achieve a peak performance. However a critical question that remains unanswered is the amount of paired training dataset actually needed to reach peak performance, this is extremely critical as capturing a paired dataset corresponding to different degradations is time consuming. In this paper we focus on region modification based data augmentation techniques as well as discriminator augmentation to examine if these can improve performance of low level vision tasks such as image enhancement, dehazing, deraining, deblurring, super resolution and denoising. The primary objective being in ensuring that an underlying algorithm localizes, identifies quantum of recovery and subsequently recovers affected regions resulting in increased perceptual quality of a recovered image. From extensive experiments we summarize intriguing properties such as improved higher performance from lower amounts of training data without any modification to baseline algorithms

4. ICCV-W

   Weakly Supervised Approach for Joint Object and Lane Marking Detection

   Shyam, Pranjay, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops Oct 2021

   Abs PDF Code

   Understanding the driving scene is critical for the safe operation of autonomous vehicles with state-of-the-art (SoTA) systems relying upon a combination of different algorithms to perform tasks for mathematically representing an environment. Amongst these tasks, lane and object detection are highly popular and have been extensively researched independently. However, their joint operation is rarely studied primarily due to the lack of a dataset that captures these attributes together, resulting in increased redundant computations that can be eliminated simply by performing these tasks together. To overcome this, we propose a weakly-supervised approach wherein, given an image from the lane detection dataset, we use a pretrained network to label different objects within a scene, generating pseudo bounding boxes used to train a network that jointly detects objects and lane lines. With an emphasis on inference speed and performance, we utilize prior works to construct two architectures based on Convolutional Neural Networks (CNNs) and Transformers. The CNN-based approach uses row-based pixel classification to detect and cluster lane lines alongside a single-stage anchor free object detector while sharing the same encoder backbone. Alternatively, using dual decoders, the transformer-based approach directly estimates bounding boxes and polynomial coefficients of lane lines. Through extensive qualitative and quantities experiments, we demonstrate the efficacy of the proposed architectures on leading datasets for object and lane detections and report state-of-the-art (SoTA) performance per GFLOPs.

5. ICRA

   Adversarially-trained Hierarchical Feature Extractor for Vehicle Re-identification

   Shyam, Pranjay, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In 2021 IEEE International Conference on Robotics and Automation (ICRA) 2021

   Abs paper

   Vehicle Re-identification (Re-ID) aims to retrieve all instances of query vehicle images present in an image pool. However viewpoint, illumination, and occlusion variations along with subtle differences between two unique images pose a significant challenge towards achieving an effective system. In this paper, we emphasize upon enhancing the performance of visual feature based ReID system by improving feature embedding quality and propose (1) an attention-guided hierarchical feature extractor (HFE) that leverages the structure of a backbone CNN to extract coarse and fine-grained features and (2) to train the proposed network within a hard negative adversarial framework that generates samples exhibiting extreme variations, encouraging the network to extract important distinguishing features across varying scales. To demonstrate the effectiveness of the proposed framework we use VERI-Wild, VRIC and Veri-776 datasets that exhibit extreme intra-class and minute inter-class differences and achieve state-of-the-art (SoTA) performance.

2020

1. ICCAS

   Retaining Image Feature Matching Performance Under Low Light Conditions

   Shyam, Pranjay, *Bangunharcana, Antyanta, and Kim, Kyung-Soo

   In 2020 20th International Conference on Control, Automation and Systems (ICCAS) 2020

   Abs paper

   Poor image quality in low light images may result in a reduced number of feature matching between images. In this paper, we investigate the performance of feature extraction algorithms in low light environments. To find an optimal setting to retain feature matching performance in low light images, we look into the effect of changing feature acceptance threshold for feature detector and adding pre-processing in the form of Low Light Image Enhancement (LLIE) prior to feature detection. We observe that even in low light images, feature matching using traditional hand-crafted feature detectors still performs reasonably well by lowering the threshold parameter. We also show that applying LLIE algorithms can improve feature matching even further when paired with the right feature extraction algorithm.

2. ICRA

   Dynamic Anchor Selection for Improving Object Localization

   Shyam, Pranjay, Yoon, Kuk-Jin, and Kim, Kyung-Soo

   In 2020 IEEE International Conference on Robotics and Automation (ICRA) 2020

   Abs paper

   Anchor boxes act as potential object localization candidates allow single-stage detectors to achieve real-time performance, at the cost of localization accuracy when compared to state-of-the-art two-stage detectors. Therefore, correct selection of the scale and aspect ratio associated with an anchor box is crucial for detector performance. In this work, we propose a novel architecture called DANet for improving the localization performance of single-stage object detectors, while maintaining real-time inference. The proposed network achieves this by predicting (1) the combination of aspect ratio and scale per feature map based on object density and (2) localization confidence per anchor box. We evaluate the proposed network using the benchmark dataset. On the MS COCO dataset, DANet achieves 30.9% AP at 51.8 fps using ResNet-18 and 45.3% AP at 7.4 fps using ResNeXt-101.