Combination of Thermal and sRGB imaging Techniques for Advanced Surveillance System

K. Martin Sagayam, J. Jenkin Winston, Mohd Helmy Abd Wahab, Bharat Bhushan, Radzi Ambar and Hazwaj Mhd Poad, "Combination of Thermal and sRGB Imaging Techniques for Advanced Surveillance System”, Annals of Emerging Technologies in Computing (AETiC), Print ISSN: 2516-0281, Online ISSN: 2516-029X, pp. 27-33, Vol. 5, No. 5, 20th March 2021, Published by International Association of Educators and Researchers (IAER), DOI: 10.33166/AETiC.2021.05.003, Available: http://aetic.theiaer.org/archive/v5/v5n5/p3.html. Review Article


Introduction
In recent scenarios, the surveillance is one of the most predominant usages in all environments for keeping watching and gives alerts to the users [1] [2]. In most of the places such as industries, malls, hospitals, etc. are monitored with the help of security camera [3]. Keeping eye on all the spaces at all times is not possible. It is significantly having thermal and standard Red, Green, Blue (sRGB) imaging property in the required system [4]. Nowadays the security cameras are more advanced, smaller with advanced features which are also reasonable. If the captured images from the camera are combined with sRGB, regions of interest (ROI) and thermal information of the surrounding environment can be ignored [5].

Literature Survey
In a factory, there are various types of machines which are automated and human movement is restricted [6]. In these cases, there are a lot of possibilities that a mishap can occur, maybe a malfunction of the machine or a fire or any such emergency [7]. This can be monitored efficiently with our advanced surveillance system which gives us the enhanced infused output of a thermal and sRGB stream [8]. www.aetic.theiaer.org The advanced surveillance is not limited to an industrial environment but also to forest observations, movement detection and other applications also [9]. The variation of light that is already present in the scene is one of the most problems faced by many surveillance systems.

Existing Technology
Conventional surveillance system that uses digital security cameras is great for domestic residences safe from vandalism, theft, and unwanted intruders [10]. In such scenario, the footage from the surveillance system needs to be examined by spectators and such system usually needs human observers to analyze the footage from the surveillance system [11]. Even though trained employees are assigned for such jobs, it's worth to note that the human ocular attention goes below the threshold level at times. The traditional surveillance systems have such shortcomings, which demands new updated ones [12] [13]. The research work uses Eigen face approach to detect the human emotions. These emotions are noticed from facial expressions [14]. It is also applied in the thermal imaging scenario to find the facial expression from the human emotion [15] [16]. The smart sensing based on the fruit surface temperature using clustering algorithm-based segmentation approach [17] [18]. High quality camera with the better features of RGB and thermal-infrared image is used to map in complex environments even in the darkness [19].

Proposed Work
This research work comprises of providing a superior surveillance with zero blind spot coverage over unseen areas. The system is composed of the thermal imaging system and a high resolution sRGB camera, have obtained the results proving to be a better solution for the futuristic technology in surveillance systems. A system comprising of fixed camera is used for detection and tracking of pedestrian. This work is disintegrated overall processing techniques into sequential steps as follows: a) Video data acquisition from thermal imaging camera module and sRGB camera module as frames has been processed depending upon the frame rate of each video data. b) Each frame of the video data is done pre-processing to reduce noise in image with various filters. c) After Pre-Processing the frame images are implied segmentation algorithms for pattern recognition and object detection process. d) The image frames which is Pre-Processed and segmented from two different video input are then further processed to feature extraction to separate the significant areas from the ordinary one. e) The image frames are further enhanced in detail through Mean, Median filters to increase saturation and for better visual appearance. f) The image frames which are obtained and processed separately from distinct inputs are now fused together into single image frames. g) The image frames from sRGB camera and Thermal camera is nor recreated into a playable video without any motion lag. h) The fused image frames is also converted into a playable video this gives more detailed and zero blind spot output as we get the best of both video cameras. i) Finally, we will get three different and distinct output of image and video processing technique. This can be displayed simultaneously or as an individual.

Methodology
The work flow of the project is described in a pictorial view as you can see in figure 1. www.aetic.theiaer.org

Image Processing
Thermal image is obtained in the form of thermo grams that shows the object with normal temperature in grayscale and as the temperature of the object increases, it changes to pseudo colors. [3] These are an indication that there are variations in wavelength represented by distinct colors [5]. sRGB image or Normal color image is obtained by conventional camera which is also passed through preprocessing and it is reduced to interpretable data for further process. Here the pixel intensity value is used for the manipulation of the data [6,7].

Image Pre-processing
Image pre-processing technique are commonly used technique for enhancing the picture quality eliminating the noise and upgrading the image for further processing techniques [8]. This method preliminarily extracts the vital information from normal video frames and thermal video frames which are the original Video-Image footage of the scenario. This is then pre-processed using Gaussian and morphological filters [9,10].
• Gaussian filter -Gaussian smoothing filter which is used to blur images to remove certain details that are not necessary and ordinary noise present in image as shown in equation 1.
• Morphological filter -this method of filtering is based on shape which processed to simplify the image data and to eliminate irreverence as shown in equation 2. (2) Where, f -Mathematical function, g -Structuring element.

Image Segmentation
This technique is implied as partitioning image into different set of pixel levels which makes easier to analyse the Video frames, commercially used in object detection and certain region of interest can be extracted through the following algorithms [11]. The Pre-Processed image is obtained and processed using filters of edge detection and thresholding technique, and also the object detection technique is applied by using GWBR filter [12,13].
• GWBR filter -It is used for background reconstruction of video image frames used to isolate the environment and the targeted area as shown in equation 3. ( Where, k -Number of frames, p -Order of GWBR filter in binary with mean 0 or 1.

Feature Extraction and Object Detection
This process involves in reducing the amount of resource of the data and will focus on extracting only what is significant for the process [14]. The feature extraction of both normal and thermal video image frames can be used for object detection and recognition [15]. In some cases, image size will be quite large. In such cases, if we want to do image matching and retrieval quickly, we prefer the method of feature extraction. This method can effectively represent major parts of an image which is in the form of compact feature vector. This leads to dimensionality reduction. Thermal signature extraction is based on k-NN (k-nearest neighbour) algorithm for extracting the feature points. The track belief m represents the conditional probability of a detected object A gives the new target data Dn of n frame.
Where, P(Dn) represent the probability of target data. P (Dn) = P (A) · P (Dn|A)+ P (A c ) · P (Dn|A c ) (5) Where A c represent the non-animal objects. P(A) is represent the priori probability of target object being the data Dn have been observed, at n − 1.
P (A) = BelA,m(n − 1) (6) The conditional probability P (Dn|A) is represents as the probability of the function gA(Dn) of kNN is nothing but the ratio of kA to k. (7) where kA is the kNN samples, e.g., if kA = 6 (majority voting), the probability is . Equation (2) is substituted by the equations (3)-(5), to updates the parameter of newly detected scheme. (8)

Data acquisition Video frames processing
The FLIR thermal datasets is used in this experiment for analysing for the surveillance system. The Video data input from sRGB and Thermal Imaging camera module is obtained and spliced into 30 frames per second as shown in figure 2 and 3. This condition for 5 subjects (day, night, auto gain, heavy rain, and snowing) which leads to a total of 150 features, with the different orientation of 00, 450, 900 and 1800 so totally 600 features is considered in this experiment.

Pre-processing
The technique is used to filter the unwanted frequency component from the original image which gives a better quality of the image. It is widely used in the effects of the graphical data, to reduce the image noise and dimensionality of the input information.  Object detection technique can be employed in real life scenarios in this we have applied facial recognition system which will be effective in identifying personals in a crowdie scenarios as shown in figure 6. It is implemented with k-NN algorithm for training the feature points from the input data. During the testing phase, it validates with the source and target information. The confusion matrix is the required term for classification procedure; from this parameter the performance metrics are computed. The evaluation of the experiments is made using quantitative performance metrics [15]. The performance measures are determined by using the expression of detection rate (DR) and false alarm rate (FAR) as given below: The Recall (DR) or true positive is significant to find the sensitivity of the system. FAR is equivalent to the value of 1 -p, where 'p' represents specificity of the system. The valuated experimental results are shown in Table 1. This result shows the significant method for the proposed surveillance systems. The proposed system shows the novel results than the other state of art methods.

Conclusion
This work presented a new design of surveillance system, shown that the enhanced output gives the extracted feature from both Thermal imaging as well as the sRGB image. The design is efficient comparatively to other normal surveillance security systems, taking the coverage, blind spots and visibility factors in count. This is also a future blooming design with lot of scope for further development and design for the betterment of the mentioned efficiency factors.
Finally considering various scenarios and the downsides of today's technology we present this design that overcame the obstacles of current technology and paving a way for futuristic advance surveillance security system, providing a safer environment.