Improving Image Matching using an Ensemble of Local Descriptors and Hardware Design
| dc.contributor.author | Ghaffari, Sina | |
| dc.contributor.supervisor | Li, Kin Fun | |
| dc.contributor.supervisor | Capson, David W. | |
| dc.date.accessioned | 2023-04-12T19:20:09Z | |
| dc.date.copyright | 2023 | en_US |
| dc.date.issued | 2023-04-12 | |
| dc.degree.department | Department of Electrical and Computer Engineering | |
| dc.degree.level | Doctor of Philosophy Ph.D. | en_US |
| dc.description.abstract | Image matching is one of the fundamental problems in computer vision, and has many applications such as object recognition, structure from motion, and 3D reconstruction. In this work, we aim to accelerate image matching algorithms, and improve their accuracy by proposing approaches that have a minimal impact on speed. With this in mind, we focus on handcrafted descriptor and matching algorithms. Our contributions in this dissertation are twofold. The first set of contributions are related to the acceleration of descriptor algorithms and the reduction of resource utilization for image matching by proposing novel circuits based on Field Programmable Gate Arrays (FPGAs). We use FPGAs as a platform due to their features such as parallel processing, low-power computing, and flexibility in design. This work presents a comprehensive analysis of FPGA-based implementations of the Histogram of Oriented Gradients (HOG) algorithm. A novel hardware-software co-design of the HOG algorithm is introduced to accelerate the execution of this descriptor algorithm. We propose methods such as logarithm-based bin assignment, approximate normalization, and a time-sharing protocol for sequential histogram generation for increasing the speed. A novel task allocation to optimize resource utilization on the hardware platform, in addition to acceleration of the HOG algorithm, is also presented. Next, we focus on binary descriptors and present a novel hardware implementation of the Binary Robust Invariant Scalable Keypoints (BRISK) algorithm. BRISK is faster than non-binary descriptors but is computationally expensive with respect to other binary descriptor algorithms. A new sampling pattern for the BRISK algorithm is proposed to facilitate the hardware implementation of BRISK in multiple scales. Our proposed design reduces FPGA resource utilization while maintaining the image matching accuracy. Furthermore, the proposed fully pipelined design achieves a frame rate of 78 fps on images with full HD resolution. The second set of contributions is related to improving image matching accuracy while maintaining performance in terms of computations. For this purpose, the focus is on handcrafted descriptor algorithms which are known to be more computationally efficient than deep learning based algorithms. We analyze and propose fusion of descriptor algorithms which extracts complementary information to attain higher accuracy. To this end, three ensemble methods are proposed. The first method (weighted-fusion) combines a non-binary and a binary descriptor using their weighted distance metrics. The second method (binary fusion) combines a non-binary and a binary descriptor by converting the non-binary descriptor to a binary descriptor using a learned threshold. The third method (non-binary fusion) combines a non-binary and a binary descriptor by transforming the binary descriptor to a non-binary descriptor using a learned scaling factor. Comprehensive experiments on benchmarks from the HPatches, Brown (Photo tourism) and Oxford Affine Covariant Regions datasets are provided. The experimental results and analysis demonstrate a higher mean Average Precision (mAP) of the fusion methods in comparison with the baseline algorithms. The next contribution for accuracy improvement is adding convolutional neural network (CNN) prefiltering to images prior to keypoint detection. The addition of a shallow CNN as the first step of a handcrafted algorithm to improve accuracy is proposed. The CNN is trained to filter the raw input images to achieve higher mAP. Experimental results indicate an improvement of accuracy using this method on the HPatches dataset. Finally, we demonstrate our proposed approaches on a practical application. This application is relevant to environmental research by providing the basis for a tool for the automated identification of wildlife in tracking habitat (in this case, badgers). The proposed methods outperform the commonly-used handcrafted algorithms on identifying individual badgers in multiple images using their facial characteristics. This is done without fine-tuning the algorithms on the target badger identification dataset, which shows the generality of our proposed methods. | en_US |
| dc.description.scholarlevel | Graduate | en_US |
| dc.identifier.bibliographicCitation | S. Ghaffari, P. Soleimani, K. F. Li, and D. Capson, “FPGA-based Implementation of HOG Algorithm: Techniques and Challenges,” in 2019 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM), 2019, pp. 1–7. doi: 10.1109/PACRIM47961.2019.8985056. | en_US |
| dc.identifier.bibliographicCitation | S. Ghaffari, P. Soleimani, K. F. Li, and D. W. Capson, “Analysis and Comparison of FPGA-Based Histogram of Oriented Gradients Implementations,” IEEE Access, vol. 8, pp. 79 920–79 934, 2020. doi: 10.1109/ACCESS.2020.2989267. | en_US |
| dc.identifier.bibliographicCitation | S. Ghaffari, P. Soleimani, K. F. Li, and D. W. Capson, “A Novel Hardware–Software Co-Design and Implementation of the HOG Algorithm,” Sensors, vol. 20, no. 19, 2020, issn: 1424-8220. doi: 10.3390/s20195655. [Online]. Available: https://www.mdpi.com/1424-8220/20/19/5655. | en_US |
| dc.identifier.bibliographicCitation | S. Ghaffari, D. W. Capson, and K. F. Li, “A Fully Pipelined FPGA Architecture for Multiscale BRISK Descriptors With a Novel Hardware-Aware Sampling Pattern,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 30, no. 6, pp. 826–839, 2022. doi: 10.1109/TVLSI.2022.3151896. | en_US |
| dc.identifier.uri | http://hdl.handle.net/1828/14940 | |
| dc.language | English | eng |
| dc.language.iso | en | en_US |
| dc.rights | Available to the World Wide Web | en_US |
| dc.subject | image matching | en_US |
| dc.subject | FPGA | en_US |
| dc.subject | BRISK | en_US |
| dc.subject | SIFT | en_US |
| dc.subject | HOG | en_US |
| dc.subject | Histogram of Oriented Gradients | en_US |
| dc.subject | Hill Climbing | en_US |
| dc.subject | badger identification | en_US |
| dc.subject | hardware implementation | en_US |
| dc.subject | hardware software co-design | en_US |
| dc.subject | Field Programmable Gate Arrays | en_US |
| dc.subject | Computer Vision | en_US |
| dc.subject | Image processing | en_US |
| dc.subject | Local Descriptor | en_US |
| dc.subject | Keypoint | en_US |
| dc.subject | fusion | en_US |
| dc.title | Improving Image Matching using an Ensemble of Local Descriptors and Hardware Design | en_US |
| dc.type | Thesis | en_US |