TY - GEN
T1 - Large-Scale News Classification using BERT Language Model
T2 - 6th International Conference on Sustainable Information Engineering and Technology, SIET 2021
AU - Nugroho, Kuncahyo Setyo
AU - Sukmadewa, Anantha Yullian
AU - Yudistira, Novanto
N1 - Publisher Copyright:
© 2021 ACM.
PY - 2021/9/13
Y1 - 2021/9/13
N2 - The rise of big data analytics on top of NLP increasing the computational burden for text processing at scale. The problems faced in NLP are very high dimensional text, so it takes a high computation resource. The MapReduce allows parallelization of large computations and can improve the efficiency of text processing. This research aims to study the effect of big data processing on NLP tasks based on a deep learning approach. We classify a big text of news topics with fine-tuning BERT used pre-trained models. Five pre-trained models with a different number of parameters were used in this study. To measure the efficiency of this method, we compared the performance of the BERT with the pipelines from Spark NLP. The result shows that BERT without Spark NLP gives higher accuracy compared to BERT with Spark NLP. The accuracy average and training time of all model's using BERT is 0.9187 and 35 minutes while using BERT with Spark NLP pipeline is 0.8444 and 9 minutes. The bigger model will take more computation resources and need a longer time to complete the tasks. However, the accuracy of BERT with Spark NLP only decreased by an average of 5.7%, while the training time was reduced significantly by 62.9% compared to BERT without Spark NLP.
AB - The rise of big data analytics on top of NLP increasing the computational burden for text processing at scale. The problems faced in NLP are very high dimensional text, so it takes a high computation resource. The MapReduce allows parallelization of large computations and can improve the efficiency of text processing. This research aims to study the effect of big data processing on NLP tasks based on a deep learning approach. We classify a big text of news topics with fine-tuning BERT used pre-trained models. Five pre-trained models with a different number of parameters were used in this study. To measure the efficiency of this method, we compared the performance of the BERT with the pipelines from Spark NLP. The result shows that BERT without Spark NLP gives higher accuracy compared to BERT with Spark NLP. The accuracy average and training time of all model's using BERT is 0.9187 and 35 minutes while using BERT with Spark NLP pipeline is 0.8444 and 9 minutes. The bigger model will take more computation resources and need a longer time to complete the tasks. However, the accuracy of BERT with Spark NLP only decreased by an average of 5.7%, while the training time was reduced significantly by 62.9% compared to BERT without Spark NLP.
KW - BERT language model
KW - distributed NLP architectures
KW - Large-scale text classification
KW - Spark NLP
UR - https://www.scopus.com/pages/publications/85118848885
U2 - 10.1145/3479645.3479658
DO - 10.1145/3479645.3479658
M3 - Conference contribution
AN - SCOPUS:85118848885
T3 - ACM International Conference Proceeding Series
SP - 240
EP - 246
BT - Proceedings of 2021 International Conference on Sustainable Information Engineering and Technology, SIET 2021
PB - Association for Computing Machinery
Y2 - 13 September 2021 through 14 September 2021
ER -