Electronic and Cyber Defense

Electronic and Cyber Defense

Dark Web Text Classification using BERT’s Language Model

Document Type : Original Article

Authors
1 Master's student,, Shahrood University of Technology, Shahrood, Iran
2 Assistant Professor, , Shahrood University of Technology, Shahrood, Iran
Abstract
The hidden nature and limited access of the dark web has led to the proliferation of many criminal activities, including cyber threats, arms sales, drug sales, and the sale of illegal tools. The emergence of large language models has created the hope that it will be possible to analyze the content on the dark web with proper accuracy. In this regard, the use of mass cyber data available in the dark web will be very useful and effective to prevent cyber threats and train language models. The technology of large language models requires a lot of high-quality data for better training and to achieve sufficient accuracy, and this is the challenge that researchers in the field of cyber security face due to the contamination of the data available on the dark web. Most of the researches in this field have been focused on all the characteristics of the dark web dataset and low-quality data and have not been able to achieve high accuracy. In this thesis, we presented a new language model based on the BERT-based language model, which was trained on the data extracted from the dark web. The proposed model is a transformer-based text model that uses a two-way encoder of transformers for a learning approach and we evaluated it on a high - quality dataset, without repetitive data, free of unknown words, all in English and specifically on hacking and security data. Finally, by analyzing the evaluated values ​​of the proposed model with the previous models, it was found that the proposed model was able to have better accuracy in data classification due to the injection of quality data compared to the previous models.
Keywords
Subjects

[1]     Aghaei, Ehsan, and Ehab Al-Shaer. "ThreatZoom: Neural Network for Automated Vulnerability Mitigation." In Proceedings of the 6th Annual Symposium on Hot Topics in the Science of Security, 2019, pp. 1-3. https://doi.org/10.1145/3314058.3318167
[2]     Tann, Wesley, Yuancheng Liu, Jun Heng Sim, Choon Meng Seah, and Ee-Chien Chang. "Using Large Language Models for Cybersecurity capture-the-flag Challenges and Certification Questions." 2023,arXiv:2308.10443.
https://doi.org/10.48550/arXiv.2308.10443
[3]     Devlin, Jacob. "Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding." 2018, arXiv:1810.04805.
[4]     Jin, Youngjin, Eugene Jang, Jian Cui, Jin-Woo Chung, Yongjae Lee, and Seungwon Shin. "DarkBERT: A Language Model for the Dark Side of the Internet." 2023, arXiv:2305.08596.
https://doi.org/10.48550/arXiv.2305.08596
[5]     Choshen, Leshem, Dan Eldad, Daniel Hershcovich, Elior Sulem, and Omri Abend. "The Language of Legal and Illegal Activity on the Darknet." 2019, arXiv:1905.05543.
https://doi.org/10.48550/arXiv.1905.05543
[6]     Al Nabki, Mhd Wesam, Eduardo Fidalgo, Enrique Alegre, and Ivan De Paz. "Classifying Illegal Activities on Tor Network Based on Web Textual Contents." In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, 2017, pp. 35-43.
[7]     Liao, Xiaojing, Kan Yuan, XiaoFeng Wang, Zhou Li, Luyi Xing, and Raheem Beyah. "Acing the Ioc Game: Toward Automatic Discovery and Analysis of Open-source Cyber Threat Intelligence." In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security,2016, pp. 755-766. https://doi.org/10.1145/2976749.2978315
[8]     Bradbury, Danny. "Unveiling the Dark web." Network security, 2014, no. 4: 14-17. https://doi.org/10.1016/S1353-4858(14)70042-X
[9]     Faizan, Mohd, and Raees Ahmad Khan. "Exploring and Analyzing the Dark web: a New Alchemy." First Monday, 2019. https://doi.org/10.5210/fm.v24i5.9473
[10]     Dingledine, Roger, Nick Mathewson, and Paul F. Syverson. "Tor: The Second-generation Onion Router." In USENIX security symposium, vol. 4,2004, pp. 303-320.
[11]     Jin, Youngjin, Eugene Jang, Yongjae Lee, Seungwon Shin, and Jin-Woo Chung. "Shedding New Light on the Language of the Dark web." 2022, arXiv:2204.06885.
https://doi.org/10.48550/arXiv.2204.06885
[12]     Samtani, Sagar, Weifeng Li, Victor Benjamin, and Hsinchun Chen. "Informing Cyber Threat Intelligence Through Dark Web Situational Awareness: The AZSecure Hacker Assets Portal." Digital Threats: Research and Practice (DTRAP) 2, 2021, no. 4: 1-10.
https://doi.org/10.1145/3450972
[13]     Törnberg, Petter. " How to Use Large-Language Models for Text Analysis", SAGE Publications Ltd, 2024.
[14]     Rajaraman, Nived, Jiantao Jiao, and Kannan Ramchandran. "Toward a Theory of Tokenization in LLMs."2024, arXiv:2404.08335.
https://doi.org/10.48550/arXiv.2404.08335
[15]     Chang, Yupeng, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen et al. "A Survey on Evaluation of Large Language Models." ACM Transactions on Intelligent Systems and Technology 15, 2024, no. 3: 1-45. https://doi.org/10.1145/3641289
[16]     Shanahan, Murray. "Talking About Large Language Models." Communications of the ACM 67,2024, no. 2: 68-79.
https://doi.org/10.1145/3624724
[17]     Koroteev, Mikhail V. "BERT: a Review of Applications in Natural Language Processing and Understanding." 2021, arXiv:2103.11943 (2021).
https://doi.org/10.48550/arXiv.2103.11943
[18]     Biryukov, Alex, Ivan Pustogarov, Fabrice Thill, and Ralf-Philipp Weinmann. "Content and Popularity Analysis of Tor Hidden Services." In 2014 IEEE 34th International Conference on Distributed Computing Systems Workshops (ICDCSW),2014, pp. 188-193.
[19]     Moore, Daniel, and Thomas Rid. "Cryptopolitik And the Darknet." Survival 58,2016, no. 1: 7-38.
[20]     Wolf, Thomas, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac et al. "Huggingface's Transformers: State-of-the-art Natural Language Processing."2019, arXiv:1910.03771.
[21]     Mohammed, Athar Hussein, and Ali H. Ali. "Survey of Bert (bidirectional encoder representation transformer) Types." In Journal of Physics: Conference Series, vol. 1963, no. 1,2021, pp. 012173.
[22]     Loshchilov, Ilya, and Frank Hutter. "Decoupled Weight Decay Regularization."2017, arXiv:1711.05101.
[23]     Townsend, James T. "Theoretical Analysis of an Alphabetic Confusion Matrix.",1971, Perception & Psychophysics 9: 40-50. https://doi.org/10.3758/BF03213026
[24]     Magnus, Amy L., and Mark E. Oxley. "Theory of confusion." In Applications and Science of Neural Networks, Fuzzy Systems, and Evolutionary Computation IV, vol. 4479, 2001, pp. 105-116. https://doi.org/10.1117/12.448337
[25]     Grandini, Margherita, Enrico Bagli, and Giorgio Visani. "Metrics for Multi-class Classification: an overview."2020, arXiv:2008.05756.
https://doi.org/10.48550/arXiv.2008.05756
[26]     Sokolova, Marina, Nathalie Japkowicz, and Stan Szpakowicz. "Beyond Accuracy, F-score and ROC: a family of Discriminant Measures for Performance Evaluation." In Australasian joint conference on artificial intelligence,2006, pp. 1015-1021.
[27]     Liu, Yang, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. "Datasets for Large Language Models: A Comprehensive Survey."2024, arXiv:2402.18041.
https://doi.org/10.48550/arXiv.2402.18041
[28]     Wang, Jiahao, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu. "A Survey on Data Selection for LLM Instruction Tuning."2024, arXiv:2402.05123.
https://doi.org/10.48550/arXiv.2402.05123
[29]     Ketkar, Nikhil, Jojo Moolayil, Nikhil Ketkar, and Jojo Moolayil. "Introduction to Pytorch."2021, Deep learning with python: learn best practices of deep learning models with PyTorch : 27-91.
[30]     Salton, Gerard, and Clement T. Yu. "On the Construction of Effective Vocabularies for Information Retrieval." Acm Sigplan Notices 10, 1973, no. 1: 48-60. https://doi.org/10.1145/951787.951766
[31]     Ali, Mehdi, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug et al. "Tokenizer Choice For LLM Training: Negligible or Crucial?."2023, arXiv:2310.08754.
https://doi.org/10.48550/arXiv.2310.08754
[32]     Jiang, Ming, Jennifer D’Souza, Sören Auer, and J. Stephen Downie. "Improving Scholarly Knowledge Representation: Evaluating bert-based Models for Scientific Relation Classification." In Digital Libraries at Times of Massive Societal Transition: 22nd International Conference on Asia-Pacific Digital Libraries, ICADL 2020, Kyoto, Japan, Proceedings 22, November 30–December 1,2020, pp. 3-19.
c.ir. (in Persian)
[3]    M. Alaei and F. Yazdanpanah, “A Method for Energy and Delay Aware Routing in Wireless Multimedia Sensor Networks,” J ELECTRONICAL & CYBER DEFENCE, vol.12, no. 47, 2024. https://ecdj.ihu.ac.ir. (in Persian)
[4]    H. Zhou and J. Li, “Sleep Scheduling for Enhancing the Lifetime of Three-Dimensional Heterogeneous Wireless Sensor Networks,” In book: Computer Supported Cooperative Work and Social Computing, pp.365-375, 2023. https://doi.org/ 10.1007/978-981-99-2356-4_29.
[5]    M. Alaei and F. Yazdanpanah, “ZOGLO: A Scheme of Zoning and Data Gathering for Lifetime Optimization in Wireless Sensor Networks,” Journal of Soft Computing and Information Technology (JSCIT), vol. 7, Issue 2, pp. 71-80, 2019. (in Persian)
[6]    A. Seyyedabbasi, G. Dogan, and F. Kiani, “HEEL: A New Clustering Method To Improve Wireless Sensor Network Lifetime,” IET WIRELESS SENSOR SYS, vol. 10, no. 3, pp. 130-136, 2020. https://doi.org/10.1049/ietwss.2019.0153.
[7]    S. Jadhav and S. Jadhav, “KPSO: K-Mean and PSO Based Clustering Algorithm for Wireless Sensor Network,” 6th IEEE International Conference On Computing, Communica-tion, Control and Automation (ICCUBEA), 2020. https://doi.org/10.1109/ICCUBEA54992.2022.10011024.
[8]    A. Joseph, R. Asaletha, V. J. Manoj, and R. Nishanth, “Enhancing the Network Lifetime of a Wireless Sensor Network using Modified K-Means Firefly Optimization,” J PHYS, vol. 2466, no. 012019, 2023. https://doi.org/10.1088/ 1742- 6596/2466/1/012019.
[9]    F. Elfouly, A. Khedr, M. H. Sharif, E. Alreshidi, K. Yadav, H. Kusetogullari, and R. Ramadan, “ERCP: Energy-Efficient and Reliable-Aware Clustering Protocol for Wireless Sensor Networks,” SENSORS,  vol. 22, no. 8950, pp. 1-17, 2022. https://doi.org/10.3390/s22228950.
[10]  M. Alaei and F. Yazdanpanah, “EELCM: An Energy Efficient Load-Based Clustering Method for Wireless Mobile Sensor Networks,” MOBILE NETW APPL, vol. 24, no. 5, pp.1486-1498, 2019. https://doi.org/10.1007/s11036-019-01270 -2.
[11]  B. Sarangi and B. Tripathy, “Outlier Detection Technique for Wireless Sensor Network Using GAN with Autoencoder to Increase the Network Lifetime,” INT J COMP NETW INF SEC, vol. 15, pp. 26-38, 2023. https://doi.org/10.4018/ IJBDCN.286705.
[12]  S. Z. Majidian and M. Shirmohammadi, “Clustering and Routing in Wireless Sensor Networks Using Multi-Objective Cuckoo Search and Game Theory,” J ELECTRONICAL & CYBER DEFENCE, vol. 10, no. 3, 2023. DOR:20.1001 .1.23224347.1401.10.3.2.0, https://ecdj.ihu.ac.ir.  (in Persian)
[13]  D. S. Sultana, D. Bordoloi, C. Singh, D. Srivastava, N. Thiyagarajan, and N. Chinthamu, “A Comparative Approach on Enhancing Lifetime of Wireless Sensor Networks,” 5th IEEE International Conference on Contemporary Computing and Informatics, 2023. https://doi.org/10.47750/pnr.2022.13 .S07.189.
[14]  R. Subha and A. Haldorai, “Improved EPOA Clustering Protocol for Lifetime Longevity in Wireless Sensor Network,” SENSORS,  vol. 3. no. 100199, 2022. https://doi.org/10.1016/j.sintl.2022.100199.
[15]  D. Agrawal, S. Pandey, P. Gupta, and M. K. Goyal, “Optimization of Cluster Heads Through Harmony Search Algorithm in Wireless Sensor Networks,” J INTELL FUZZY SYST, vol. 39, no. 6, pp. 8587-8597, 2020, https://doi.org/ 10.3233/ JIFS-189175.
[16]  L. Jawad and A. Idrees, “Integrated Encoding and Scheduling Protocol for Improving the Lifetime in Wireless Sensor Networks,” INT J COMPUT APPL T, vol. 69, no. 334, pp. 334-343, 2022. https://doi.org/ 10.1504/IJCAT. 2022.10054564.
[17]  V. Narayan  and A. K. Daniel, “Energy Efficient Protocol for Lifetime Prediction of Wireless Sensor Network using Multivariate Polynomial Regression Model,” J SCI IND RES INDIA, vol. 81, no. 12, pp. 1297-1309, 2022. https://doi.org /0.56042/jsir.v81i12.54908.
[18]  K. Debasis, L. Sharma, V. Bohat, and R. Bhadoria, “An Energy-Efficient Clustering Algorithm for Maximizing Lifetime of Wireless Sensor Networks using Machine Learning,” MOBILE NETW APPL, vol.28, no.2, pp. 1-15, 2023. https://doi.org/ 10.1007/s11036-023-02109-7.
[19]  S. Phommasan, Widyawan, and I. W. Mustika “Cluster Selection Technique with Fuzzy Logic-Based Wireless Sensor Network to Increase the Lifetime of Networks,” 5th International Conference on Research of Information Technology and Intelligent Systems (ISRITI), 2022. https://doi.org/ 10.1109/ISRITI56927.2022.10052871.
[20]  A. Taha, H. Abouroumia, S. Mohamed, and L. Amar, “Enhancing the Lifetime and Energy Efficiency of Wireless Sensor Networks Using Aquila Optimizer Algorithm,” FUTURE INTERNET, vol. 14. no. 365, pp. 1-17, 2022. https://doi.org/ 10.3390/fi14120365.
[21]  C. D. Tran, N. Tam, N. Quy, and H. Binh, “An Energy-Efficient Scheme for Maximizing Data Aggregation Tree Lifetime in Wireless Sensor Network,” J AMB INTEL HUM COMP, vol. 14, 2023. https://doi.org/s12652-023-04621-w.
[22]  A. J. Yuste-Delgado, J. C. Cuevas-Martinez, and A. Trivino-Cabrera, “A Distributed Clustering Algorithm Guided by the Base Station to Extend the Lifetime of Wireless Sensor Networks,” SENSORS, vol. 20, no. 8, ID: 2312, pp. 1-18, 2020. https://doi.org/10.3390/s20082312.
[23]  A. M. Jubair, R. Hassan, A. Aman, H. Sallehudin, Z. Al-Mekhlafi, B. Mohammed, and M. Alsaffar, “Optimization of Clustering in Wireless Sensor Networks: Techniques and Protocols,” APPL SCI, vol. 11, no. 23, 11448, pp. 1-30, 2021. https://doi.org/10.3390/app112311448.
[24]  M. Mirzasadeghi and H. Bakhshi, “A New Method for Clustering Wireless Sensor Networks to Improve the Energy Consumption,” J COMMUN ENG, vol. 5, no. 2, pp. 136-149, 2016. https://doi.org/10.22070/jce.2017.1614.
[25]  M. Khan, J. Khan, K. Mahmood, I. Bari, H. Ali, N. Jan, and R. Ghoniem, “Algorithm for Increasing Network Lifetime in Wireless Sensor Networks Using Jumping and Mobile Sensor Nodes,” ELECTRONICS, vol. 11, no.2913, pp. 1-15, 2022. https://doi.org/ 10.3390/electronics11182913.
[26]  R. Medeiros, J. M. Villanueva, and E. Macedo, “Lifetime Increase for Wireless Sensor Networks Using Cellular Learning Automata,” WIRELESS PERS COMMUN, vol. 123, pp. 3413–3432, 2022. https://doi.org/10.1007/s11277-021-09295-1.
[27]  R. Marappan, P. Vardhini, G. Kaur, S. Murugesan, M. Kathiravan, N. Bharathiraja, and R. Venkatesan, “Efficient Evolutionary Modeling in Solving Maximization of Lifetime of Wireless Sensor Healthcare Networks,” SOFT COMPUT, vol. 27, Issue. 16, PP. 11853-11867, 2023. https://doi.org/ 10.1007 /s00500-023-08623-w.
[28]  M. Alaei, P. Sabbagh, and F. Yazdanpanah, “A QoS-aware Congestion Control Mechanism for Wireless Multimedia Sensor Networks,” WIREL NETW, vol. 25, no. 3, pp. 4173-4192, 2019. https://doi.org/10.1007/s11276-018-1738-8.
[29]  S. Misra and R. Kumar, “An Analytical Study of LEACH and PEGASIS Protocol in Wireless Sensor Networks,” International Conference on Innovations in information Embedded and Communication Systems (ICIIECS), 2017, https://doi.org/10.1109/ICIIECS.2017.8276118.
[30]  R. Chang and C. Kuo, “An Energy Efficient Routing Mechanism for Wireless Sensor Networks,” 20th International Conference on Advanced Information Networking and Applications (AINA’06), 2006. https://doi. org/10.1109/AINA.2006.86.
[31]  W. B. Heinzelman, A.P. Chandrakasan, and H. Balakrish-nan, “An Application-specific Protocol Architecture for Wireless Microsensor Networks,” IEEE T WIREL COMMUN, vol. 1, Issue 4, 2002. https://doi.org/10.1109/ TWC.2002.804190.
Volume 13, Issue 4 - Serial Number 52
Winter
Winter 2026
Pages 63-76

  • Receive Date 19 September 2025
  • Revise Date 08 November 2025
  • Accept Date 28 November 2025
  • Publish Date 22 December 2025