Design and Implementation of a new Visualization Aided Anomaly Detection Framework
| dc.contributor.author | Farag, Ahmed | |
| dc.contributor.supervisor | Traore, Issa | |
| dc.contributor.supervisor | Yousef, Waleed | |
| dc.date.accessioned | 2023-08-21T17:46:51Z | |
| dc.date.available | 2023-08-21T17:46:51Z | |
| dc.date.copyright | 2023 | en_US |
| dc.date.issued | 2023-08-21 | |
| dc.degree.department | Department of Electrical and Computer Engineering | |
| dc.degree.level | Master of Engineering M.Eng. | en_US |
| dc.description.abstract | In today's data-driven world, the identification of unusual patterns or anomalies in data sets has become increasingly vital, especially in the realm of security data where the detection of these atypical patterns can preempt security threats. This is the juncture where our work, as an extension to UNAVOIDS (Unsupervised and Nonparametric Approach for Visualizing Outliers and Invariant Detection Scoring), becomes instrumental. UNAVOIDS is a distinctive model that integrates specialized techniques for both detection algorithms and visualization methods, operating within a unique space known as the Neighborhood Cumulative Distribution Function (NCDF) space. In this two-dimensional space, each data point is transformed into a unique 2D curve, facilitating visual identification and examination. A salient feature of UNAVOIDS is its fully unsupervised nature, which requires neither prior training nor specific data inputs, eliminating the need for parameter selection or tuning. Another feature is its assignment of a deviation score to each unusual data point, offering a clear gauge of its abnormality. In this study, we successfully deployed UNAVOIDS across four platforms: the Python Package Index (PyPI), a Restful API, a software named VAAD—which integrates UNAVOIDS with the Data Visualization Platform (DVP)—, and a custom Microsoft PowerBivisual. Two main challenges were tackled in this implementation. First, handling large datasets within the RESTful API posed an ongoing challenge. To address this, we adopted compression over file streaming, enabling the efficient transmission of data within the API constraints. Second, creating an interactive visual representation presented a significant challenge due to the unique nature of the data, where each observation is mapped to a 2D curve. We overcame this challenge by mapping curve indices and implementing a reflection mechanism for interactivity between selected curves and other visuals. Our study contributes to the practical implementation and effectiveness of UNAVOIDS, and all these implementations along with their documentations are accessible from the official repository of the ISOT lab. These implementations, catering to users from various sectors including research and development, provide the versatility and effectiveness of UNAVOIDS in diverse environments. | en_US |
| dc.description.scholarlevel | Graduate | en_US |
| dc.identifier.uri | http://hdl.handle.net/1828/15275 | |
| dc.language.iso | en | en_US |
| dc.rights | Available to the World Wide Web | en_US |
| dc.subject | anomaly detection | en_US |
| dc.subject | UNAVOIDS | en_US |
| dc.subject | outliers | en_US |
| dc.subject | data visualization | en_US |
| dc.title | Design and Implementation of a new Visualization Aided Anomaly Detection Framework | en_US |
| dc.type | project | en_US |