Differential Privacy-Preserving Algorithms for Secure Training of Machine Learning Models
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V6I2P110Keywords:
Differential Privacy, Privacy-Preserving Machine Learning, Secure Training, Federated Learning, DP-SGD, Deep Learning, Privacy BudgetAbstract
Since ML is used more widely in data-driven apps, issues about data privacy and protection are becoming more common. It provides a method for assessing and controlling the privacy of individuals in datasets used for machine learning (ML) training. This paper examines DP-preserving algorithms designed for the safe training of machine learning models. We study centralized, local, and distributed methods for applying differential privacy to the training of logistic regression, support vector machines, and deep neural networks. Next, we investigate the foundations of differential privacy, including privacy budgets, the concept of sensitive data, and noise addition, and examine how they impact the accuracy and reliability of the model. We apply DP-SGD, examine its effects on utility and privacy, and study models that combine federated learning with secure multi-party computation. We utilize the MNIST, CIFAR-10, and Adult Income datasets in a comprehensive experiment to evaluate the accuracy, privacy loss, convergence, and runtime of our system. While training a DP model incurs costs in utility, our testing shows that selecting the right parameters and utilizing a combination of privacy approaches can yield secure and high-performing results. Our research aims to inform machine learning (ML) research on privacy issues and provide guidance on implementing differential privacy in ML applications
References
[1] Sweeney, L. (2002). k-anonymity: A model for protecting privacy. International journal of uncertainty, fuzziness and knowledge-based systems, 10(05), 557-570.
[2] Machanavajjhala, A., Kifer, D., Gehrke, J., & Venkitasubramaniam, M. (2007). L-diversity: Privacy beyond k-anonymity. Acm transactions on knowledge discovery from data (tkdd), 1(1), 3-es.
[3] Li, N., Li, T., & Venkatasubramanian, S. (2006, April). T-closeness: Privacy beyond k-anonymity and l-diversity. In 2007 IEEE 23rd International Conference on data engineering (pp. 106-115). IEEE.
[4] Dwork, C. (2006, July). Differential privacy. In International colloquium on automata, languages, and programming (pp. 1-12). Berlin, Heidelberg: Springer Berlin Heidelberg.
[5] Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016, October). Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security (pp. 308-318).
[6] McSherry, F., & Talwar, K. (2007, October). Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) (pp. 94-103). IEEE.
[7] Geyer, R. C., Klein, T., & Nabi, M. (2017). Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557.
[8] McMahan, H. B., Ramage, D., Talwar, K., & Zhang, L. (2017). Learning differentially private recurrent language models. arXiv preprint arXiv:1710.06963.
[9] Shokri, R., & Shmatikov, V. (2015, October). Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security (pp. 1310-1321).
[10] Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., ... & Seth, K. (2017, October). Practical secure aggregation for privacy-preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (pp. 1175-1191).
[11] Truex, S., Baracaldo, N., Anwar, A., Steinke, T., Ludwig, H., Zhang, R., & Zhou, Y. (2019, November). A hybrid approach to privacy-preserving federated learning. In Proceedings of the 12th ACM workshop on artificial intelligence and security (pp. 1-11).
[12] Aono, Y., Hayashi, T., Wang, L., & Moriai, S. (2017). Privacy-preserving deep learning via additively homomorphic encryption. IEEE transactions on information forensics and security, 13(5), 1333-1345.
[13] Gentry, C. (2009). A fully homomorphic encryption scheme. Stanford University.
[14] Zhao, Y., Zhao, J., Yang, M., Wang, T., Wang, N., Lyu, L., ... & Lam, K. Y. (2020). Local differential privacy-based federated learning for the Internet of Things. IEEE Internet of Things Journal, 8(11), 8836-8853.
[15] Papernot, N., Song, S., Mironov, I., Raghunathan, A., Talwar, K., & Erlingsson, Ú. (2018). Scalable private learning with pate. arXiv preprint arXiv:1802.08908.
[16] Du, M., Wang, K., Xia, Z., & Zhang, Y. (2018). Differential privacy-preserving training model in wireless big data with edge computing. IEEE transactions on big data, 6(2), 283-295.
[17] Wu, X., Zhang, Y., Shi, M., Li, P., Li, R., & Xiong, N. N. (2022). An adaptive federated learning scheme with differential privacy-preserving. Future Generation Computer Systems, 127, 362-372.
[18] Zhao, J., Chen, Y., & Zhang, W. (2019). Differential privacy preservation in deep learning: Challenges, opportunities, and solutions. IEEE Access, 7, 48901-48911.
[19] Li, X., Chen, Y., Wang, C., & Shen, C. (2022). When Deep Learning Meets Differential Privacy: Privacy, Security, and More IEEE Network, 35(6), 148-155.
[20] El Ouadrhiri, A., & Abdelhadi, A. (2022). Differential privacy for deep and federated learning: A survey. IEEE Access, 10, 22359-22380.
[21] Wei, K., Li, J., Ding, M., Ma, C., Yang, H. H., Farokhi, F., ... & Poor, H. V. (2020). Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15, 3454-3469.
