Efficient TinyML Architectures for On-Device Small Language Models: Privacy-Preserving Inference at the Edge

Mangesh Pujari ; Anshul Goel ; Anil Kumar Pakina
International Journal Science and Technology (IJST... 📅 2024 Vol 3, No 3 🔗 DOI: 10.56127/ijst.v3i3.1958
Abstract
Deploying small language models (SLMs) on ultra-low-power edge devices requires careful optimization to meet strict memory, latency, and energy constraints while preserving privacy. This paper presents a systematic approach to adapting SLMs for Tiny ML, focusing on model compression, hardware-aware quantization, and lightweight privacy mechanisms. We introduce a sparse ternary quantization technique that reduces model size by 5.8× with minimal accuracy loss and an efficient federated fine-tuning method for edge deployment. To address privacy concerns, we implement on-device differential noise injection during text preprocessing, adding negligible computational overhead. Evaluations on constrained devices (Cortex-M7 and ESP32) show our optimized models achieve 92% of the accuracy of full-precision baselines while operating within 256KB RAM and reducing inference latency by 4.3×. The proposed techniques enable new applications for SLMs in always-on edge scenarios where both efficiency and data protection are critical.
Keywords
TinyML Edge AI Small Language Models On-device Inference Ai Preserving Privacy Compression of Neural Networks Quantization and Microcontrollers.
How to Cite

Mangesh Pujari, et al. (2024). Efficient TinyML Architectures for On-Device Small Language Models: Privacy-Preserving Inference at the Edge. International Journal Science and Technology (IJST), 3(3). https://doi.org/10.56127/ijst.v3i3.1958

Mangesh Pujari; Anshul Goel; Anil Kumar Pakina, "Efficient TinyML Architectures for On-Device Small Language Models: Privacy-Preserving Inference at the Edge," International Journal Science and Technology (IJST), vol. 3, no. 3, 2024.

Mangesh Pujari; Anshul Goel; Anil Kumar Pakina. "Efficient TinyML Architectures for On-Device Small Language Models: Privacy-Preserving Inference at the Edge." International Journal Science and Technology (IJST), vol. 3, no. 3, 2024.

Mangesh Pujari; Anshul Goel; Anil Kumar Pakina. "Efficient TinyML Architectures for On-Device Small Language Models: Privacy-Preserving Inference at the Edge." International Journal Science and Technology (IJST) 3, no. 3 (2024).

Mangesh Pujari, et al. (2024) 'Efficient TinyML Architectures for On-Device Small Language Models: Privacy-Preserving Inference at the Edge', International Journal Science and Technology (IJST), 3(3). doi: 10.56127/ijst.v3i3.1958.

Mangesh Pujari; Anshul Goel; Anil Kumar Pakina. Efficient TinyML Architectures for On-Device Small Language Models: Privacy-Preserving Inference at the Edge. International Journal Science and Technology (IJST). 2024;3(3).

Artikel Terkait

Perception and Behavior of Farmers Towards the Use of Organic Fertilizers in Rice Plants in Kewayuhan Village, Pejagoan District, Kebumen Regency

Furqon, Ngafiful; Aprilia, Rennanti Lunnadiyah

Heart Disease Prediction Using Logistic Regression and K-Nearest Neighbor: A Comparative Study of Classification Algorithm Performance

Sururi, Yan Risa Aspi; Ghozi, M. Asadullah Al

Integrated Technical, Thermal, and Techno-Economic Assessment of Hot-Oil and Electric Heating Systems for 15–22 MT Asphalt Storage Tanks

Arifuddin, Arifuddin; Wiseno, Elbi

Generative AI Risk Governance in Organizational Information Systems: A Conceptual Model Integrating Security, Ethics, and Digital Trust

Nurdiyanto Yusuf

Design of a SCADA-Based Control System for Hybrid Power Generation Integrating a Screw Turbine and Solar Panels

Mekel, Jonah Alfred; Bawano, Franklin; Mekel, Alfred Noufie; Saroinsong, Tineke

Performance Evaluation of an N₂ Membrane Generator under Different Operating Temperatures

Pradito, Okttu Kristi; Korawan, Agus Dwi; Mawarni, Drajat Indah
Tren Sitasi Jurnal