Preview

Journal of «Almaz – Antey» Air and Space Defence Corporation

Advanced search

Millimetre-wave radar image recognition using neural networks

https://doi.org/10.38013/2542-0542-2022-3-48-58

Contents

Scroll to:

Abstract

The article describes an experimental software and hardware system for invisible security screening, using a millimetre-wave radar signal and a recognition module that employs a deep convolutional neural network as a classifier.

The software and hardware system was developed using the methods of deep learning of neural networks, in particular, the transfer learning technology, which make it possible to adapt the previous recognition models to solving new tasks.

The hardware and software system can potentially enable invisible security screening of people and classification of carried objects without impeding the flow of people. In comparison with the traditional methods, this solution allows to significantly increase the security screening rate.

This system can be adapted to recognize various sets of objects of interest depending on particular customer’s requirements in various fields of application.

The scientific novelty of the work consists in the development of methods for classifying millimetre-wave images based on the transfer of a learning model in deep neural networks.

The deep machine learning technology allows to solve the problem of the quality of recognition, which was limiting the application of such systems for a long time.

For citations:


Dudikhin V.V., Ivanov A.S., Mezhuev I.Yu., Shokov A.V., Yakupov I.Yu. Millimetre-wave radar image recognition using neural networks. Journal of «Almaz – Antey» Air and Space Defence Corporation. 2022;(3):48-58. https://doi.org/10.38013/2542-0542-2022-3-48-58

Introduction

Today, the issues related to radar-based recognition and detection of concealed objects, including those hidden under clothes, invoke a great interest [1][2]. Solving this problem allows to protect various critical social infrastructure facilities from people carrying potentially dangerous objects. One of the main disadvantages of millimetre- wave security screening systems for detection of concealed objects is low definition of object images.

This disadvantage is associated with a low depth of field due to the systems’ narrow screening area [1][3] and with a low spatial resolution of the wavelength band [4]. In this paper, we propose possible solutions to these problems. Firstly, in order to enlarge the screening area, the described system comprises an automatic aiming and focusing subsystem that allows to adjust the screening area depending on the actual position of the object being screened. Secondly, neural network-based image recognition methods are implemented to solve the problem of threat image identification.

However, security screening systems of this class still have some drawbacks. Thus, such a system is not able to detect objects inside the human body or those hidden by metal objects, including on person’s clothes, or by wet clothes. Damp clothes are also a factor that reduces the probability to detect and recognize a concealed object.

Recognition of such objects enables invisible screening without impeding the flow of people, thus considerably accelerating the security screening rate. Another advantage is the possibility to reduce the human factor associated with traditional methods of monitoring involving a metal detector.

The problem of radar object recognition based on a neural network generally comes down to the problem of classification, i.e. the estimation of probabilities to classify objects as those belonging to a particular class.

The objective of the study is to develop appropriate methods for radar image acquisition using a millimetre-wave radar signal, as well as the methods of classification of complex shaped objects based on neural networks.

Test rig description

During experiments, we created an experimental software and hardware system comprising receivers and transmitters, a set of software modules intended for image preprocessing and image recognition with the help of a pre-trained neural network, hardware control and user interface implementation.

Figure 1 shows the functional diagram of an experimental software and hardware system for invisible security screening. The transmitting equipment comprises 5 radiation sources generating microwaves with the wavelength of 3 mm. These sources are installed into special racks numerically designated 1, 2, and 3, as shown in the diagram. The rack allows to change the position of radiation sources to gain more uniform radiation distribution over the object to be screened.

Fig. 1. Functional diagram

The receiving equipment comprises a matrix installed on an positioning element. The radar image is focused on the matrix with the help of a special fluoroplastic lens. The positioning element allows to acquire a sufficiently sharp radar image irrespective of the distance between the matrix and the object to be screened.

A software-based noise reduction function helps enhance the quality of radar images. After noise reduction, a signal corresponding to the acquired radar image is sent to the input of a neural network which solves the problem of classification. Recognition results are displayed by a software interface module developed specially for the purpose of the study.

For the development of the test rig, radiation parameters are selected with consideration given to the following factors.

  • Use of millimetre waveband allows to reduce overall dimensions of a radio system, which means considerably smaller dimensions of antennas.
  • Unlike optical and infrared radiation waves, millimetre waves have higher penetrability to see through most non-metal objects. Use of IR radiation cannot ensure a sufficient level of reflection intensity after penetrating clothes.
  • Radio waves are classified as non-ionizing radiation. In particular, human skin almost prevents millimetre waves from penetration and inflicting any damage to the internal organs of the human body [5].
  • Normally, person’s clothes are not visualized and become invisible on radar images captured in the millimetre waveband, but human body outlines can be clearly distinguished, as well as some person-borne objects (coins, buttons, pens, keys, etc.). Therefore, weapons, explosives, narcotic substances, or other contraband items hidden under clothes can be detected as they are visualized immediately [6].

3-mm wavelength continuous point sources are used as the system’s transmitting elements. Total power of wave sources is 400 mW. Wave source spectrum width is not less than 50 MHz. Spatial position of sources provides uniform illumination of the object to be screened. Due to short-term coherence of radiation sources (<20 ns) and long-time accumulation of the detector matrix (~50 ms), interference distortions in the received signal are lower than the noise level. Thus, the position of receivers affects the quality of the received signal only in terms of geometrical optics.

A TeraSense imaging sensor matrix is currently used as the receiving element. This matrix consists of detectors made based on a GaAs high-mobility heterostructure. The imaging sensor is manufactured on a single wafer. Such process ensures high homogeneity and reproducibility of the detector parameters (pixel-to-pixel deviation responsivity is within a 20-percent range). Sensor resolution is 64 × 64 pixels. Detector pixel size is 1.5 mm. Each detector unit has a room-temperature (25 °C) responsivity up to 50 kV/W with read-out circuitry and noise-equivalent power of 1 nW/√Hz in the frequency range of 10 GHz – 1 THz. The detection mechanism is based on excitation of plasma oscillations in a two-dimensional electron system with subsequent rectification performed on special defects made in the electron system [7].

Control system

Under this protect, the detector matrix positioning algorithm has been developed. A special fluoroplastic lens is used to get an object image on the matrix. To get an matrix focus image, the matrix shall be placed at a certain distance away from the lens. The distance is calculated by the thin lens formula

1 / f + 1 / d = 1 / F,

where d – distance from lens to object, f – desired distance from lens to matrix, F – lens focal length, in this case it is equal to 40 cm. For this purpose, the test rig has a positioning element based on the delta mechanism. To acquire the data indicating the distance to the unit under test, a lidar with special software was used for selecting its field of vision.

Data exchange with the test rig’s control system hardware is enabled via a common USB interface. For data exchange with the lidar, there is a special software library written in the Python programming language (PL). The lensto- object distance measurement diagram is shown in Figure 2.

Fig. 2. Lens-to-object distance measurement diagram
1 – receiver with lens, 2 – lidar, 3 – principal axis of receiver,
4 – screening area, 5 – initial points captured by lidar,
6 – point with averaged coordinates, d – distance from lens to object

The set of coordinates of objects points (5) in its own polar coordinate system serves as initial data received from the lidar (2). Initial data are used to select points in a given region that corresponds to the screening area (4), whose coordinates are averaged. The target range is determined by projecting the averaged point (6) on the pre-calibrated axis (3) of the receiver (1).

Interaction with the positioning element is implemented by means of G-code [8]. Depending on the distance, the matrix position changes on the basis of lidar data, indicating the position of the object to be screened. Besides, the detector matrix is moved by the positioning element in the plane parallel to the lens plane in such a way that the object’s radar image is in the centre of the frame.

Data acquisition and processing system The detector matrix is connected to a computer via a COM port. The library for the Python PL is used for data acquisition. The software module that uses the library enables asynchronous framing and multi-thread processing of frames. Examples of radar images are shown in Figure 3.

Fig. 3. Examples of radar images of a gun (a, b) and a knife (c, d)

Noise contamination of radar images directly received from the matrix degrades the quality of object recognition by a classifying neural network. To solve this problem, we use a noise reduction system based on signal separation by the minimum level and on elimination of the basic immanent noise-induced flare spot which does not depend on the position of light sources.

In order to identify objectless images, filtering by the threshold of the total sum of all matrix pixels is applied. Such images are not allowed to be transferred to a neural network for further processing. This approach makes it possible to improve the classification accuracy and increase the system performance.

Based on classification results generated by a neural network, equalization is employed to improve the stability of classification of blurred images. Fine tuning of equalization parameters is available as well. In this case, the target value is calculated on the basis of probabilities of object identification as the object of a particular class or as the most frequent object observed in n previous frames.

Application of neural networks

Problem-oriented research papers describe examples of neural network applications for radar image recognition [9][10]. The peculiarity of this study is that it involves deep learning methods with the transfer of models of pre-trained neural networks (transfer learning).

During experiments, different variants of neural network configurations were analysed as possible options for solving related problems. Based on the analysis, three different variants of the classifier architecture were selected, trained and tested on test images, using the transfer learning technology with pre-trained neural networks such as MobileNet, ResNet and Inception. The resulted structures allow to determine the probabilities for radar image classification as a certain class of potential threats.

To implement architectures of neural networks, we used the Keras library proved to be an efficient solution for operations with deep learning of neural networks. Keras is written in Python PL and compatible with popular software libraries such as TensorFlow and Theano.

The basic algorithm for creating applicable neural networks is based on the deep learning principle [11]. For algorithm implementation, we developed a database with radar images of real objects, detection of which was defined as the project objective. Radar images for the database were recorded by the test rig at different camera angles.

To increase the size of the dataset employed for neural network training, we applied an augmentation method that enabled synthesis of extra images. In this case, the augmentation method includes:

  • rotation of a real object within 40°;
  • movement of a real object along two axes within 0.1 of the image size;
  • rotation with spreading within 40°;
  • object zoom within 0.1 of the image size;
  • reflection of an object on two axes;
  • sampling data randomization.

We used a specially developed software module for radar image transformation. This solution allowed to increase the database capacity up to 8019 radar images. Therefore, this approach ensures sufficiently balanced data sampling (with approximately equal class sizes), avoiding the effect of the bias of an estimator.

To implement machine learning technology, we formed three data sets such as training, validation and test data sets taken from the general radar imagery database. 8019 initial radar images were divided as follows: 70 % (5613 images) for training data sample and 15 % (1203 images) for validation data sample and test data sample, respectively.

Under the project, 4 models of classifiers were developed and tested. The first model is based on the pre-trained convolutional neural network MobileNetV2 to solve the problem of binary classification only. The model architecture implements feature extraction via a pre-trained neural network and uses a trainable perceptron on the output. This converges to a single neural processing element in the output layer with the sigmoid activation function.

The other three models are intended to solve the problem of classification with three classes. The models’ architectures comprise classifiers based on more massive (with a larger amount of neurons) perceptrons which have three neural processing elements with the softmax activation function on the output. They are also implemented on the basis of various pre-trained convolutional neural networks included in the Keras library.

Comparative analysis of classifiers Below are the characteristics of neural networks under consideration:

  • MobileNetV2, 3,045,955 parameters in total, faster but less accurate (preferable solution for real-time problem solving, provided the desired quality is assured);
  • ResNet152V2, 59,512,835 parameters in total, slower but more accurate (supposed to be the most accurate solution);
  • InceptionV3, 22,983,971 parameters in total, quite fast and accurate (supposed to be a compromise solution).

To test the models, we developed five different tests that reflected various possible modes of operation of classifiers.

Test 1. Includes 1203 images from the test sample after passing through a jitter generator similar to that used during training and validation.

Test 2. Sampling data including 33 images of 3 classes with poor quality and/or inappropriate camera angle, without jitter.

Test 3. Includes 1203 images from the test sample without jitter.

Test 4. 1203 images from the test sample after passing through a jitter generator with less aggressive parameters in comparison with those used for training and validation:

  • rotation within 20°;
  • movement along two axes within 0.05 of image size;
  • rotation with spreading within 20°;
  • zoom within 0.05 of image size;
  • reflection on two axes.

Test 5. Includes 1203 images from the test sample after passing through a jitter generator with more aggressive parameters in comparison with those used during training and validation:

  • rotation within 80°;
  • movement along two axes within 0.1 of image size;
  • rotation with spreading within 80°;
  • zoom within 0.1 of image size;
  • reflection along two axes.

The results of five tests on three models are summarized in Table 1 and shown in Figure 4.

Table 1

Test results, accuracy measure, %

Upon completing the theoretical phase including configuration, training and testing of different variants of neural networks in the static mode on pre-acquired images, we ran a few experiments for real-time testing of the developed models on real equipment.

Figures 5 and 6 show the examples of test rig operation in the binary mode (danger/safe). During simulation experiments, test objects were hidden on a dummy. We conducted six series of experiments, each including 40 measurements. Depending on the camera angle at which a radar image was acquired, the recognition system generated the correct answer, indicating the level of danger of a concealed object with an accuracy of 0.75 to 0.9.

Fig. 5. Recognition of a concealed dangerous object

Fig. 6. Recognition of a concealed non-dangerous object

Conclusion

Our comprehensive study proves that the application of neural network-based classifiers is a feasible and reasonable solution to recognize radar images of potential threats hidden on a person.

As a conclusion, the detector matrix resolution shall be increased because the current 64 × 64 pixel resolution reaches its performance limitations.

The developed software package that uses neural networks has the scalability property. If the detector matrix resolution is increased, the existing software can be easily customized to be able to recognize a considerably larger amount of object classes.

This is due to the fact that the developed classifiers employ structures pre-trained on a large amount of labelled data taken from the ImageNet database which contains 14 million images divided into 22 thousand classes.

Recognition of a larger amount of objects takes a lot of high-quality radar images for training the existing neural networks. The technology enables fine tuning of the recognition system for a specific set of objects the customer is interested in.

The amount of matrices required for object radar image recognition depends on recognition accuracy requirements. Moreover, the accuracy depends on the quality of the data set used for training. The issue related to the prediction of recognition accuracy is currently under study.

References

1. Солдатов Д. П., Гладун В. В., Павлов Р. А., Пирогов Ю. А., Тищенко Д. А. Обнаружение потенциально опасных предметов с помощью системы пассивного радиовидения миллиметрового диапазона. Ученые записки физического факультета Московского университета, Московский государственный университет имени М. В. Ломоносова, 2012. № 1. – С. 67–72.

2. Сердюцкая Д. А., Максимова А. А. Моделирование процессов обнаружения оружия на теле человека при его импульсном облучении, Моделирование оптимизация и информационные технологии: Воронежский институт высоких технологий. № 3 (14), 2016.

3. Гладун В. В., Котов А. В., Криворучко В. И. Система ближнего пассивного радиовидения 3-мм диапазона // Журнал радиоэлектроники. 2010, №7.

4. H. Zhong, J. Xu, X. Xie, T. Yuan, R. Reightler, E. Madaras, and X.-Ch. Zhang, Nondestructive defect identification with terahertz time-of-flight tomography, IEEE Sensors Journal, vol. 5, no. 2, pp. 203–208 (2005).

5. Hitchcock, R. Timothy. Radio-frequency and Microwave Radiation. – American Industrial Hygiene Assn., 2004. – P. 1–3.

6. Sheen, D. M., McMakin, L. & Hall, T. E. 2001 Three-dimensional millimeter-wave imaging for concealed weapon detection. IEEE Trans. Microwave Theory Techn. 49, P. 1581–1592.

7. Terahertz imaging cameras. TeraSense Group, Inc. URL: https://terasense.com/products/sub-thz-imaging-cameras/ (дата обращения: 21.03.2022).

8. Oberg, Erik; Jones, Franklin D.; Horton, Holbrook L.; Ryffel, Henry H. (1996), Green, Robert E.; McCauley, Christopher J. (eds.), Machinery’s Handbook (25th ed.), New York: Industrial Press, – P. 1284–1287

9. Разевиг В.В., Ивашов С.И., Бугаева А.С., Нейросетевой метод восстановления радиоголограмм, успехи современной радиоэлектроники, Том: 74. № 12, 2020, – C. 5–16.

10. Назаров А. В., Благодыренко Е. В. Алгоритм идентификации радиоизображений, представленных многомерными групповыми точечными объектами, с использованием карты Кохонена – современные проблемы создания и эксплуатации вооружения, военной и специальной техники, III Всероссийская научно-практическая конференция. Военнокосмическая академия имени А. Ф. Можайского. Санкт-Петербург, 2016.

11. Николенко С., Архангельская Е., Кадурин А. Глубокое обучение. Погружение в мир нейронных сетей, Питер, 2020 г.


About the Authors

V. V. Dudikhin
STC-T, NPK “NIIDAR” JSC
Russian Federation

Dudikhin Viktor Vladimirovich – Candidate of Engineering Sciences, Head of Laboratory.
Science research interests: information systems and technologies, artificial intelligence systems, neural networks, methods of analytical data processing, business and competitive intelligence (Open Source Intelligence, OSINT), data mining and text mining.

Moscow



A. S. Ivanov
STC-T, NPK “NIIDAR” JSC
Russian Federation

Ivanov Aleksandr Sergeyevich – Engineer.
Science research interests: machine learning, computer vision.

Moscow



I. Yu. Mezhuev
STC-T, NPK “NIIDAR” JSC
Russian Federation

Mezhuev Igor Yurievich – Candidate of Engineering Sciences, Head of Department.
Science research interests: software product lines design, development of adaptive software, development of scientific experiment control systems.

Moscow



A. V. Shokov
STC-T, NPK “NIIDAR” JSC
Russian Federation

Shokov Aleksandr Vladimirovich – Engineer.
Science research interests: machine learning, computer vision.

Moscow



I. Yu. Yakupov
STC-T, NPK “NIIDAR” JSC
Russian Federation

Yakupov Igor Yurievich – Head of Laboratory.
Science research interests: radio-wave imaging, control software development, research systems automation.

Moscow



Review

For citations:


Dudikhin V.V., Ivanov A.S., Mezhuev I.Yu., Shokov A.V., Yakupov I.Yu. Millimetre-wave radar image recognition using neural networks. Journal of «Almaz – Antey» Air and Space Defence Corporation. 2022;(3):48-58. https://doi.org/10.38013/2542-0542-2022-3-48-58

Views: 1046


Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.


ISSN 2542-0542 (Print)