DALF: An AI Enabled Adversarial Framework for Classification of Hyperspectral Images

Hyperspectral image classification is very complex and challenging process. However, with deep neural networks like Convolutional Neural Networks (CNN) with explicit dimensionality reduction, the capability of classifier is greatly increased. However, there is still problem with sufficient training samples. In this paper, we overcome this problem by proposing an Artificial Intelligence (AI) based framework named Deep Adversarial Learning Framework (DALF) that exploits deep autoencoder for dimensionality reduction, Generative Adversarial Network (GAN) for generating new Hyperspectral Imaging (HSI) samples that are to be verified by a discriminator in a non-cooperative game setting besides using a classifier. Convolutional Neural Network (CNN) is used for both generator and discriminator while classifier role is played by Support Vector Machine (SVM) and Neural Network (NN). An algorithm named Generative Model based Hybrid Approach for HSI Classification (GMHA-HSIC) which drives the functionality of the proposed framework is proposed. The success of DALF in accurate classification is largely dependent on the synthesis and labelling of spectra on regular basis. The synthetic samples made with an iterative process and being verified by discriminator result in useful spectra. By training GAN with associated deep learning models, the framework leverages classification performance. Our experimental results revealed that the proposed framework has potential to improve the state of the art besides having an effective data augmentation strategy.


INTRODUCTION
Hyperspectral imaging (HSI) is one of the remote sensing techniques that captures electromagnetic spectrum of different wavelength ranges. The sensors used for HSI give several hundreds of spectral bands of give surface. In HSI imagery, each pixel be considered as a high-dimensional vector with data corresponding to spectral reflectance in certain wavelength. In HSI, it is possible to identify subtle spectral differences which make it suitable for many real-world applications such as estimation of crop LAI, estimation of soil salinity, mineral mapping, fusion of images and sparse un-mixing in spatial domain to mention few. Classification of hyperspectral images is one such widely used application [1], [2], [3], [4], [5]. From the recent studies such as [6], [7], [8], HSI classification is found very significant research area that attracted remote sensing community. HSI classification determines the class of each pixel depending on the pixel's spectral characteristics. HSI images are high-dimensional in nature. Classification of HSI imagery throws certain challenges such as curse of dimensionality, limited training samples and large spatial variability. The challenge of having limited training samples decreases generalization capability of classification methods. The large spatial variability is caused by factors linked to changes in temporal, atmospheric, environmental and illumination conditions. Literature reflects the use of machine learning and deep learning methods for HSI classification. Semisupervised learning-based approaches are widely used for such activity [1], [3], [8], [9], [10], [11], [8], [12],  [13] as it helps in using available training samples as well. Deep learning based convolutional approaches are used in [6], [8], [14], [15]. Apart from Convolutional Neural Network (CNN), most of the studies used Generative Adversarial Network (GAN) models in order to deep with shortage of training samples in HSI classification approaches. A typical GAN model consists of a generator and discriminator who play noncooperative game to leverage quality in the given HSI task. GANs are widely used in the research of HSI classification. While single discriminator is used in [1], [2], [3], [16], [17], a multi-discriminator-based GAN is found in [18] to ascertain the impact of each discriminator on the results of classification. Several flavors of GAN models came into existence. They include cascade conditional model [17], Conditional Random Fields (CRFs) based model [19], Adaptive drop-back enhanced model [7], Markov Random Fields (MRFs) based model [14], quality aware model [20] and Functional Network Connectivity (FNC) based model [21]. While many of the GAN models do not use an explicit classifier, we understood that giving random noise and class label to generator and using classifier module with GAN could improve classification performance. Out contributions in this paper are as follows. 1. We proposed a framework known as Deep Adversarial Learning Framework (DALF) for HSI classification based on GAN architecture that is made up of generator (G), discriminator (D) and classifier (C). 2. An algorithm named Generative Model based hybrid approach for HSI Classification (GMHA-HSIC) which drives the functionality of the proposed framework. 3. A prototype is built using Python data science platform to evaluate the DALF and its underlying algorithm besides comparing the results with the state of the art.
The remainder of the paper is structured as follows. Section 2 reviews literature on many existing generative models for HSI classification. Section 3 presents the proposed framework while Section 4 provides evaluation methodology. Section 5 focuses on the presentation of empirical results. Section 6 provides conclusions and possible scope for future work in the area of HSI classification.

RELATED WORK
Literature has rich information of GAN usage for HSI classification as it could synthesize new samples. Zhan et al. [1] proposed a semi supervised approach based on GAN for HSI classification. It was named as HSGAN which could provide better performance over traditional approaches. Zhu et al. [2] proposed two GAN architectures namely 1D-GAN and 3D-GAN and analyzed the potential different and the new opportunities in HSI classification. They observed that samples generated by GAN could improve performance of the classification. He et al. [3] proposed a filter mechanism to extract spectral-spatial features that are used in the GAN architecture. With limited number of training samples, their method could perform effectively. They intend to improve it further using graph-based learning method. Jiang et al. [16] proposed HADGAN which detects anomalies and exploit reconstruction constraints. It learns discriminative reconstruction in the background by using anomaly targets is the focus of it towards hyperspectral anomaly detection. Liu et al. [17] focused on hyperspectral sample generation using cascade conditional generative adversarial nets. A two-stage GAN named C 2 GAN is proposed for generating samples.
Zhong et al. [19] used Conditional Random Fields (CRFs) along with GAN for realizing GAN based architecture for HSI classification. It is named GAN-CRF which has a generator and discriminator with semisupervised learning besides having probabilistic graphical models. Xue [5] proposed a 3D capsule network for spectral-spatial classification. The discriminator could provide better performance with generated labelled samples and existing training samples. In presence of scarcity in training samples, it could provide better performance. Wu et al. [6] focused on temporal imaging with Temporal Fourier Transformation (TFT) which is reconfigurable. This could be used further in imaging and vision applications. Jiang et al. [22] proposed HSRGAN that learns spectral and spatial features. They defined a loss function that could provide to improve performance while generating images. Paoletti et al. [23] on the other hand investigated deep learning classifiers used with HSI classification.
From the literature, it is found that many of the GAN models do not use an explicit classifier. However, we understood that giving random noise and class labels to a generator and using a classifier module with GAN could improve classification performance. It is achieved with the proposed framework in this paper.

DEEP ADVERSARIAL LEARNING FRAMEWORK 3.1 The Framework
We proposed a deep learning-based framework named Deep Adversarial Learning Framework (DALF). The framework exploits GAN architecture. Right from the introduction of GAN architecture in [24], it is widely used in different computer vision applications. The rationale is to generate new samples to leverage input space to the system. In most of the existing GAN models, target distribution is limited to certain data points. In this paper, we use CNN as generator where it creates data pints related to the underlying distribution. Towards this end, generator is trained to use an adversarial objective function to approximate the distribution. Generator gets this function by invoking the discriminator network. Both generator and discriminator form a non-cooperative game to improve performance in HSI classification. The generator learns to create new samples while the discriminator learns and finds whether a sample belongs to fake or real distribution. The proposed GAN also has a classifier module that classifies the HSI samples based on training (with class label) spectra samples. The aim of the GAN in general is to solve the min-max problem expressed in Eq. 1.
Where G and D denote generator and discriminator components. Random noise input is denoted as z while G(z) is the fake data. The D is aimed at maximizing log(D(x)) while G is meant to minimize log(1-D(G(z)). The expectation operator is denoted by E. The proposed framework for HSI classification is provided in Figure 1. Generator is given not only noise as input but also real class labels. The G is expected to produce fake spectra samples denoted as G(z). The classifier is any machine learning technique such as Support Vector Machine (SVM) to classify spectra samples based on the labelled real spectra. The objective function of the proposed GAN is divided into two parts as expressed in Eq. 2 and Eq. 3.
Where denotes the likelihood of correct input source while denotes the likelihood of correct class labels. The optimization of D and optimization of G are achieved by maximizing Ls+Lc and Ls-Lc respectively. Both generator and discriminator are implemented using the CNN as used in [25]. The generator produces fake spectra data based on the input noise and class labels. The discriminator takes fake spectra and real spectra as input (both are unlabelled) and discriminates real samples from fake ones. The classifier takes fake spectra and labelled spectra as input and produces classification results.

Proposed Algorithm
An algorithm named Generative Model-based Hybrid Approach for HSI Classification (GMHA-HSIC) which drives the functionality of the proposed framework is proposed. The success of DALF inaccurate classification is largely dependent on the synthesis and labelling of spectra on regular basis. The synthetic samples made with an iterative process and being verified by discriminator result in useful spectra. By training GAN with associated deep learning models, the framework leverages classification performance. As presented in Algorithm 1, it takes noise z, real class labels C, real unlabelled spectra X1, real labelled spectra X2 as input and produces HSI classification results with the help of the proposed generative adversarial network model. For each noise sample given in the form of z. There is an iterative process in which the discriminator learns using the CNN model and produces fake spectra data by taking noise samples and class labels as input. Each time fake data is created, it is used by the discriminator along with unlabelled real spectra samples to classify the samples into real/fake. It is done by using CNN model by the discriminator. Thus the discriminator gives its criticism (feedback) to a generator which is crucial in the non-cooperative game involved in realizing GAN architecture. With the given feedback is used, the discriminator gains more knowledge and produces more optimal fake samples. In every iteration, Ls and Lc are computed to optimize G and D for improving performance. The classifier used is either SVM or NN which is meant for producing classification results using given fake samples and real labelled spectra.

Parameter Tuning for SVM and NN
Training parameters tuned for SVM on real data are C and kernel which are used as svm_C and svm_kernel respectively in the Python code as per Scikit-Learn library. The hyperparameter C is meant for adding penalty for misclassified instances. If C influences decision boundary affecting classification accuracy. If C is large, the classifier tries to minimize the number of misclassified instances. Though we did experiments with 0.1, 1, 10,100 and 1000 as C value, we found highest performance in terms of accuracy when C value is 1000. Therefore, C is fixed at 1000 and kernel used is "linear" as it is found faster. While a trained SVM is applied on fake data, a Pipeline object is created and used with different tunable parameters. As presented in Table 1, the parameters used are found relatively good for synthetic data. With respect to neural network, MLPClassifier is used in the implementation. Its parameter details are as given in Table 2. The parameters provided are used in order to tune the performance of the NN classifier. NN classifier is used in the proposed GAN based system for classification of hyperspectral images.

Deep Autoencoder
The proposed framework uses an autoencoder which is adapted from [26]. It is an unsupervised learning-based technique for dimensionality reduction. It exploits neural networks for representation learning. The autoencoder is made up of three layers. A visible layer, a hidden layer and a reconstruction layer with d inputs, L units and d units respectively. The training procedure has two steps as expressed in Eq. 4 and Eq. 5.
Where f(.) is an activation function while bh and by denote bias of hidden and output units respectively. The waits associated with input and hidden layers respectively are denoted as wh and wy. Euclidean distance between x and y is used to compute reconstruction error.

EVALUATION METHODOLOGY
Experiments are made using the implementation made with Python data science platform using Google Colab. The CNN framework explored in [25] is used for building generator and discriminator. The classifier is made up of either SVM or NN. Dataset is collected from Pavia university dataset [27]. The Pavia University dataset is a hyperspectral image dataset which gathered by a sensor known as the reflective optics system imaging spectrometer (ROSIS-3) over the city of Pavia, Italy. The image consists of 610×340 pixels with 115 spectral bands. The image is divided into 9 classes with a total of 42,776 labelled samples, including the asphalt, meadows, gravel, trees, metal sheet, bare soil, bitumen, brick, and shadow. Prior to making experiments with the proposed GAN architecture, the ground truth is obtained from the dataset as presented in Figure 2. The ground truth is generated and presented for Pavia university dataset [27] which is used to evaluate the performance of the proposed algorithm and compare it with the state of the art. Based on the ground truth, the performance of the proposed framework in generating spectra samples and HSI classification with SVM and NN is evaluated.
Based on the confusion matrix the performance metrics such as precision, recall and F1-score derived from number of true positives, number of true negatives, number of false positives and number of false negatives. As presented in Table 3, the metrics result in a value between 0 and 1. The value 1 indicates best performance.

EXPERIMENTAL RESULTS
After experimentation, there are two spectra sample distributions. Evaluation is made to know whether fake distribution is as good as real distribution. After generating synthetic spectra, it is evaluated with two machine learning classifiers such as SVM and NN. The performance of the proposed framework is compared against state of the art as well.

Real Data
Synthetic Data Generated by GAN As presented in Figure 3, the results of generator of the proposed DALF framework for 9 samples are presented. To ascertain the utility of the synthetic spectra, they are evaluated with two classification techniques such as SVM and NN. In both real and synthetic samples' results, the horizontal axis shows the number of iterations while the vertical axis shows the value for mean_spectrum -std_spectrum (blue dotted line at the bottom), mean_spectrum (orange line at the middle) and mean_spectrum + std_spectrum (green dotted line at the bottom). As presented in Table 4, the HSI classification performance with SVM is provided in terms of precision, recall and F1-score for different real spectra samples. As presented in Table 5, the HSI classification performance with SVM is provided in terms of precision, recall and F1-score for different synthetic spectra samples. As presented in Table 6 the HSI classification performance with NN is provided in terms of precision, recall and F1-score for different augmented spectra samples As presented in Figure 5, the real spectra samples are provided in horizontal axis and the HSI classification performance in terms of precision, recall and F1-score is provided in vertical axis. More in precision, recall and F1-score refers to more in performance. It is observed from the results that each spectra sample exhibited different level of performance with respect to SVM classification. Both sample 5 and sample 9 showed 100% performance. As presented in Figure 5, the synthetic spectra samples are provided in horizontal axis and the HSI classification performance in terms of precision, recall and F1-score is provided in vertical axis. More in precision, recall and F1-score refers to more in performance. It is observed from the results that each spectra sample exhibited different level of performance with respect to SVM classification. Sample 7 showed least F1score with 0.77. Figure 6. HSI classification performance of NN on real samples As presented in Figure 6, the real spectra samples are provided in horizontal axis and the HSI classification performance in terms of precision, recall and F1-score is provided in vertical axis. More in precision, recall and F1-score refers to more in performance. It is observed from the results that each spectra sample exhibited different level of performance with respect to NN classification. Sample 3 showed least F1score with 0.8127. As presented in Figure 7, the augmented spectra samples are provided in the horizontal axis and the HSI classification performance in terms of precision, recall and F1-score is provided in vertical axis. More in precision, recall and F1-score refers to more in performance. It is observed from the results that each spectra sample exhibited the different level of performance with respect to NN classification. Sample 3 showed the least F1-score with 0.8125. Table 5. HSI classification performance on real spectra As presented in Table 5 the HSI classification performance of SVM and NN with real, synthetic and augmented data, in terms of F1-score is provided. Figure 8. HSI classification performance of two models in terms of F1-Score As presented in Figure 8, the HSI classification performance of SVM and NN are compared for the real, synthetic and augmented (training is given on real and synthetic samples) spectra samples. The results revealed that each sample showed different possibilities in terms of classification performance. For each sample, the classification performance is evaluated. The performance of NN is better than that of SVM on real data. When it comes to synthetic data used with SVM and augmented data used with NN, SVM showed better performance. As synthetic data cannot capture class boundaries well, it is observed that augmented data with NN showed less performance. However, with GAN based framework, it is an encouraging fact that augmented data is working as good as original data at least in the context of weakly supervised HSI classification.

CONCLUSION AND FUTURE WORK
The cold start problem with insufficient training samples for hyperspectral image classification and the problem of leveraging classification efficiency is considered in this paper. Towards this end, we proposed an Artificial Intelligence (AI) based framework named Deep Adversarial Learning Framework (DALF) that exploits deep autoencoder for dimensionality reduction, Generative Adversarial Network (GAN) for generating new Hyperspectral Imaging (HSI) samples that are to be verified by a discriminator in a non-cooperative game setting besides using CNN for classification. The success of DALF inaccurate classification is largely dependent on the synthesis and labelling of spectra on regular basis. The synthetic samples made with an iterative process and being verified by discriminator result in useful spectra. By training GAN with associated deep learning models, the framework leverages classification performance. Our experimental results revealed that the proposed framework has the potential to improve the state of the art besides having an effective data augmentation strategy. In future, we intend to improve the DALF and evaluate with hyperspectral imaging samples of different domains. REFERENCES IJEEI ISSN: 2089-3272 