Behavioral Anomaly Detection in IoT Networks Using Artificial Intelligence

Karla Fišić; Mila Matijević; Matej Kvesić

doi:https://doi.org/10.47960/3029-3200.2026.2.2.27

Research Article

DOI: https://doi.org/10.47960/3029-3200.2026.2.2.27

Citations (Crossref, OpenAlex): …

Karla Fišić* , Mila Matijević and Matej Kvesić

*CA	karla.fisic@fsre.sum.ba
	mila.matijevic@fsre.sum.ba
	matej.kvesic@fsre.sum.ba

Full text:

In Press

This article belongs to Vol. 2 No. 2, 2026

K. Fišić, M. Matijević, and M. Kvesić, “Behavioral Anomaly Detection in IoT Networks Using Artificial Intelligence,” International Journal of Innovative Solutions in Engineering, vol. 2, no. 2, pp. 27–37, doi: 10.47960/3029-3200.2026.2.2.27.

pages 27-37

Download a citation file:

Vol. 2 No. 2 Article 20 Download

Preview and download a citation file in BibTex format that can be imported by citation management software, including Mendeley, EndNote, ProCite, RefWorks, and Reference Manager.

This article is archived in Zenodo

Abstract

IoT networks are difficult to secure; devices are resource-constrained and heterogeneous, and they generate traffic volumes that make manual monitoring impractical. Because of this, it remains challenging to tell normal behavior apart from malicious activity. In this work, we focus on detecting anomalies in IoT network traffic using an autoencoder-based approach. The model is trained only on normal network behavior and learns to recognize typical patterns. Any significant difference from these patterns is then treated as a potential anomaly. The experiments were performed on a public dataset that included more than 86000 normal network flows alongside 16696 MQTT brute-force attack samples. The model detected 87.73% of attacks. However, a subset of attack flows was not detected – specifically those whose statistical properties closely resembled normal traffic, which the model had no basis to flag as suspicious. Overall, the results show that reconstruction-based detection is effective in practice, but has clear limitations. Using reconstruction error alone is insufficient to detect subtle or well-disguised attacks, so additional methods should be explored in future work.

Keywords

Internet of Things, Anomaly Detection, Autoencoder, MQTT Protocol

ijise ID

20

URI

https://ijise.ba/article/20/

Publication Date

In Press

References

Statista, “Number of Internet of Things (IoT) connected devices worldwide from 2019 to 2030”, 2025. Accessed: Jan. 05, 2026. [Online]. Available: https://www.statista.com/topics/2637/internet-of-things/
A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, and M. Ayyash, “Internet of Things: A Survey on Enabling Technologies, Protocols, and Applications,” IEEE Commun. Surv. Tutorials, vol. 17, no. 4, pp. 2347–2376, 2015, doi: https://doi.org/10.1109/COMST.2015.2444095.
M. Antonakakis et al., “Understanding the Mirai Botnet”, in Proceedings of the 26th USENIX Security Symposium, 2017, pp. 1093-1110. Accessed: Jan. 05, 2026. [Online]. Available: https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/antonakakis
D. Bank, N. Koenigstein, and R. Giryes, “Autoencoders,” 2020, arXiv. doi: https://doi.org/10.48550/ARXIV.2003.05991.
J. Lin, W. Yu, N. Zhang, X. Yang, H. Zhang, and W. Zhao, “A Survey on Internet of Things: Architecture, Enabling Technologies, Security and Privacy, and Applications,” IEEE Internet Things J., vol. 4, no. 5, pp. 1125–1142, Oct. 2017, doi: https://doi.org/10.1109/JIOT.2017.2683200.
A. Banks and R. Gupta, “MQTT Version 3.1.1”, OASIS Standard, 2014. Accessed: Jan. 06, 2026. [Online]. Available: https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.pdf
A. Sivanathan et al., “Classifying IoT Devices in Smart Environments Using Network Traffic Characteristics,” IEEE Trans. on Mobile Comput., vol. 18, no. 8, pp. 1745–1759, Aug. 2019, doi: https://doi.org/10.1109/TMC.2018.2866249.
V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM Comput. Surv., vol. 41, no. 3, pp. 1–58, Jul. 2009, doi: https://doi.org/10.1145/1541880.1541882.
M. Ahmed, A. Naser Mahmood, and J. Hu, “A survey of network anomaly detection techniques,” Journal of Network and Computer Applications, vol. 60, pp. 19–31, Jan. 2016, doi: https://doi.org/10.1016/j.jnca.2015.11.016.
R. Sommer and V. Paxson, “Outside the Closed World: On Using Machine Learning for Network Intrusion Detection,” in 2010 IEEE Symposium on Security and Privacy, May 2010, pp. 305–316. doi: https://doi.org/10.1109/SP.2010.25.
C. Zhou and R. C. Paffenroth, “Anomaly Detection with Robust Deep Autoencoders,” in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax NS Canada: ACM, Aug. 2017, pp. 665–674. doi: https://doi.org/10.1145/3097983.3098052.
G. E. Hinton and R. R. Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks,” Science, vol. 313, no. 5786, pp. 504–507, Jul. 2006, doi: https://doi.org/10.1126/science.1127647.
M. Sakurada and T. Yairi, “Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction,” in Proceedings of the MLSDA 2014 2nd Workshop on Machine Learning for Sensory Data Analysis, Gold Coast Australia QLD Australia: ACM, Dec. 2014, pp. 4–11. doi: https://doi.org/10.1145/2689746.2689747.
D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” 2013, arXiv. doi: https://doi.org/10.48550/ARXIV.1312.6114.
P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, and G. Shroff, “LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection,” 2016, arXiv. doi: https://doi.org/10.48550/ARXIV.1607.00148.
N. Shone, T. N. Ngoc, V. D. Phai, and Q. Shi, “A Deep Learning Approach to Network Intrusion Detection,” IEEE Trans. Emerg. Top. Comput. Intell., vol. 2, no. 1, pp. 41–50, Feb. 2018, doi: https://doi.org/10.1109/TETCI.2017.2772792.
H. Hindy, C. Tachtatzis, R. Atkinson, E. Bayne and X. Bellekens, “MQTT-IoT-IDS2020: MQTT Internet of Things Intrusion Detection Dataset”, IEEE DataPort, Jun. 23, 2020, doi: 10.21227/bhxy-ep04 Accessed: Dec. 02, 2025. [Online]. Available: https://ieee-dataport.org/open-access/mqtt-iot-ids2020-mqtt-internet-things-intrusion-detection-dataset
D. Dinculeană and X. Cheng, “Vulnerabilities and Limitations of MQTT Protocol Used between IoT Devices,” Applied Sciences, vol. 9, no. 5, p. 848, Feb. 2019, doi: https://doi.org/10.3390/app9050848.
F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation Forest,” in 2008 Eighth IEEE International Conference on Data Mining, Dec. 2008, pp. 413–422. doi: https://doi.org/10.1109/ICDM.2008.17.
B. Schölkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, and J. C. Platt, “Support Vector Method for Novelty Detection”, in Advances in Neural Information Processing Systems (NIPS 1999), pp. 582-588, 1999. Accessed: Apr. 06, 2026. [Online]. Available: https://papers.nips.cc/paper_files/paper/1999/hash/8725fb777f25776ffa9076e44fcfd776-Abstract.html
Y. Mirsky, T. Doitshman, Y. Elovici, and A. Shabtai, “Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection,” 2018, arXiv. doi: https://doi.org/10.48550/ARXIV.1802.09089.
S. Chesney and K. Roy, “AI Empowered Intrusion Detection for MQTT Networks,” in 2022 International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD), Durban, South Africa: IEEE, Aug. 2022, pp. 1–6. doi: https://doi.org/10.1109/icABCD54961.2022.9856124.
M. H. Bhuyan, D. K. Bhattacharyya, and J. K. Kalita, “Network Anomaly Detection: Methods, Systems and Tools,” IEEE Commun. Surv. Tutorials, vol. 16, no. 1, pp. 303–336, 2014, doi: https://doi.org/10.1109/SURV.2013.052213.00046.