Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers

Wataya, Tomohiro; Miura, Azusa; Sakisuka, Takahisa; Fujiwara, Masahiro; Tanaka, Hisashi; Hiraoka, Yu; Sato, Junya; Tomiyama, Miyuki; Nishigaki, Daiki; Kita, Kosuke; Suzuki, Yuki; Kido, Shoji; Tomiyama, Noriyuki

doi:https://doi.org/10.1007/s11548-024-03316-7

Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers

Wataya, Tomohiro; Miura, Azusa; Sakisuka, Takahisa et al.

International Journal of Computer Assisted Radiology and Surgery, 2025

アクセス数:1,658件（2025-08-17 06:13 集計）

固定URL: https://hdl.handle.net/11094/100326

閲覧可能ファイル

ファイル	フォーマット	利用条件	サイズ	閲覧回数	利用開始日	説明	information
IntJComAssist_s11548-024-03316-7	pdf	なし	4.19 MB	36	2025.02.14

Creative Commons : 表示

論文情報

ファイル出力 EndNote Basic出力 Mendeley出力

タイトル	Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers
著者	NRID 1000090983023 1000090983023 ORCID 0000-0002-1224-6601 0000-0002-1224-6601 researchmap permalink tomohiro-wataya tomohiro-wataya 著者名 Wataya, Tomohiro
	researchmap permalink azusamiura azusamiura 著者名 Miura, Azusa
	著者名 Sakisuka, Takahisa
	著者名 Fujiwara, Masahiro
	NRID 1000040294087 1000040294087 ORCID 0000-0002-6148-0672 0000-0002-6148-0672 researchmap permalink read20200825 read20200825 著者名 Tanaka, Hisashi
	著者名 Hiraoka, Yu
	著者名 Sato, Junya
	著者名 Tomiyama, Miyuki
	ORCID 0000-0002-7006-3408 0000-0002-7006-3408 著者名 Nishigaki, Daiki
	ORCID 0000-0002-5825-3348 0000-0002-5825-3348 researchmap permalink kosuke-kita kosuke-kita 著者名 Kita, Kosuke
	著者名 Suzuki, Yuki
	NRID 1000090314814 1000090314814 ORCID 0000-0001-5293-4110 0000-0001-5293-4110 researchmap permalink read0196068 read0196068 Scopus Author ID 7004389358 7004389358 著者名 Kido, Shoji
	NRID 1000050294070 1000050294070 researchmap permalink 3430 3430 Scopus Author ID 7006024942 7006024942 著者名 Tomiyama, Noriyuki
キーワード等	Active learning
	Bidirectional encoder representations from transformers
	Medical safety
	Natural language processing
	Radiology report
内容	The version of record of this article, first published in International Journal of Computer Assisted Radiology and Surgery, is available online at Publisher’s website: https://doi.org/10.1007/s11548-024-03316-7.
抄録	Purpose: Systems equipped with natural language (NLP) processing can reduce missed radiological findings by physicians, but the annotation costs are burden in the development. This study aimed to compare the effects of active learning (AL) algorithms in NLP for estimating the significance of head computed tomography (CT) reports using bidirectional encoder representations from transformers (BERT). Methods: A total of 3728 head CT reports annotated with five categories of importance were used and UTH-BERT was adopted as the pre-trained BERT model. We assumed that 64% (2385 reports) of the data were initially in the unlabeled data pool (UDP), while the labeled data set (LD) used to train the model was empty. Twenty-five reports were repeatedly selected from the UDP and added to the LD, based on seven metrices: random sampling (RS: control), four uncertainty sampling (US) methods (least confidence (LC), margin sampling (MS), ratio of confidence (RC), and entropy sampling (ES)), and two distance-based sampling (DS) methods (cosine distance (CD) and Euclidian distance (ED)). The transition of accuracy of the model was evaluated using the test dataset. Results: The accuracy of the models with US was significantly higher than RS when reports in LD were < 1800, whereas DS methods were significantly lower than RS. Among the US methods, MS and RC were even better than the others. With the US methods, the required labeled data decreased by 15.4–40.5%, and most efficient in RC. In addition, in the US methods, data for minor categories tended to be added to LD earlier than RS and DS. Conclusions: In the classification task for the importance of head CT reports, US methods, especially RC and MS can lead to the effective fine-tuning of BERT models and reduce the imbalance of categories. AL can contribute to other studies on larger datasets by providing effective annotation.
出版者	Springer Science and Business Media Deutschland GmbH
収録物名	International Journal of Computer Assisted Radiology and Surgery
刊行年月	2025-01-08
言語	英語
ハンドルURL	https://hdl.handle.net/11094/100326
PISSN	18616410
eISSN	18616429
関連情報 (isIdenticalTo)	DOI (出版社版) https://doi.org/10.1007/s11548-024-03316-7
アクセス権	open access
権利情報	This article is licensed under a Creative Commons Attribution 4.0 International License.
出版タイプ	Version of Record
カテゴリ	学術雑誌論文 Journal Article

資源タイプ	学術雑誌論文
ローカル資源タイプ	学術雑誌論文
DCTERMS.bibliographicCitation	International Journal of Computer Assisted Radiology and Surgery. 2025
DC.title	Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers
DC.creator	Wataya, Tomohiro
	Miura, Azusa
	Sakisuka, Takahisa
	Fujiwara, Masahiro
	Tanaka, Hisashi
	Hiraoka, Yu
	Sato, Junya
	Tomiyama, Miyuki
	Nishigaki, Daiki
	Kita, Kosuke
	Suzuki, Yuki
	Kido, Shoji
	Tomiyama, Noriyuki
DC.publisher	Springer Science and Business Media Deutschland GmbH
DC.language' scheme='DCTERMS.RFC1766	英語
DCTERMS.issued' scheme='DCTERMS.W3CDTF	2025-01-08
DC.identifier	https://doi.org/10.1007/s11548-024-03316-7
DC.subject	Active learning
	Bidirectional encoder representations from transformers
	Medical safety
	Natural language processing
	Radiology report
DC.description	The version of record of this article, first published in International Journal of Computer Assisted Radiology and Surgery, is available online at Publisher’s website: https://doi.org/10.1007/s11548-024-03316-7.
DCTERMS.abstract	Purpose: Systems equipped with natural language (NLP) processing can reduce missed radiological findings by physicians, but the annotation costs are burden in the development. This study aimed to compare the effects of active learning (AL) algorithms in NLP for estimating the significance of head computed tomography (CT) reports using bidirectional encoder representations from transformers (BERT). Methods: A total of 3728 head CT reports annotated with five categories of importance were used and UTH-BERT was adopted as the pre-trained BERT model. We assumed that 64% (2385 reports) of the data were initially in the unlabeled data pool (UDP), while the labeled data set (LD) used to train the model was empty. Twenty-five reports were repeatedly selected from the UDP and added to the LD, based on seven metrices: random sampling (RS: control), four uncertainty sampling (US) methods (least confidence (LC), margin sampling (MS), ratio of confidence (RC), and entropy sampling (ES)), and two distance-based sampling (DS) methods (cosine distance (CD) and Euclidian distance (ED)). The transition of accuracy of the model was evaluated using the test dataset. Results: The accuracy of the models with US was significantly higher than RS when reports in LD were < 1800, whereas DS methods were significantly lower than RS. Among the US methods, MS and RC were even better than the others. With the US methods, the required labeled data decreased by 15.4–40.5%, and most efficient in RC. In addition, in the US methods, data for minor categories tended to be added to LD earlier than RS and DS. Conclusions: In the classification task for the importance of head CT reports, US methods, especially RC and MS can lead to the effective fine-tuning of BERT models and reduce the imbalance of categories. AL can contribute to other studies on larger datasets by providing effective annotation.
DC.rights	This article is licensed under a Creative Commons Attribution 4.0 International License.
DC.format	100326
citation_title	Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers
citation_author	Wataya, Tomohiro
	Miura, Azusa
	Sakisuka, Takahisa
	Fujiwara, Masahiro
	Tanaka, Hisashi
	Hiraoka, Yu
	Sato, Junya
	Tomiyama, Miyuki
	Nishigaki, Daiki
	Kita, Kosuke
	Suzuki, Yuki
	Kido, Shoji
	Tomiyama, Noriyuki
citation_publisher	Springer Science and Business Media Deutschland GmbH
citation_language	英語
citation_date	2025-01-08
citation_journal_title	International Journal of Computer Assisted Radiology and Surgery
citation_doi	https://doi.org/10.1007/s11548-024-03316-7
citation_public_url	https://hdl.handle.net/11094/100326

Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers

閲覧可能ファイル

論文情報

論文詳細を表示