Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers

Wataya, Tomohiro; Miura, Azusa; Sakisuka, Takahisa; Fujiwara, Masahiro; Tanaka, Hisashi; Hiraoka, Yu; Sato, Junya; Tomiyama, Miyuki; Nishigaki, Daiki; Kita, Kosuke; Suzuki, Yuki; Kido, Shoji; Tomiyama, Noriyuki

doi:https://doi.org/10.1007/s11548-024-03316-7

Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers

Wataya, Tomohiro; Miura, Azusa; Sakisuka, Takahisa et al.

International Journal of Computer Assisted Radiology and Surgery, 2025

Number of Access:1,657（2025-08-14 04:32 Counts）

Identifier to cite or link to this item: https://hdl.handle.net/11094/100326

Link to primary information

File	Format	Terms of use	Size	Views	Date.Available	Description	information
IntJComAssist_s11548-024-03316-7	pdf	None	4.19 MB	36	2025.02.14

Creative Commons : Attribution

Item Information

Output File Export EndNote Basic Export Mendeley

Title	Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers
Creator	NRID 1000090983023 1000090983023 ORCID 0000-0002-1224-6601 0000-0002-1224-6601 researchmap permalink tomohiro-wataya tomohiro-wataya CreatorName Wataya, Tomohiro
	researchmap permalink azusamiura azusamiura CreatorName Miura, Azusa
	CreatorName Sakisuka, Takahisa
	CreatorName Fujiwara, Masahiro
	NRID 1000040294087 1000040294087 ORCID 0000-0002-6148-0672 0000-0002-6148-0672 researchmap permalink read20200825 read20200825 CreatorName Tanaka, Hisashi
	CreatorName Hiraoka, Yu
	CreatorName Sato, Junya
	CreatorName Tomiyama, Miyuki
	ORCID 0000-0002-7006-3408 0000-0002-7006-3408 CreatorName Nishigaki, Daiki
	ORCID 0000-0002-5825-3348 0000-0002-5825-3348 researchmap permalink kosuke-kita kosuke-kita CreatorName Kita, Kosuke
	CreatorName Suzuki, Yuki
	NRID 1000090314814 1000090314814 ORCID 0000-0001-5293-4110 0000-0001-5293-4110 researchmap permalink read0196068 read0196068 Scopus Author ID 7004389358 7004389358 CreatorName Kido, Shoji
	NRID 1000050294070 1000050294070 researchmap permalink 3430 3430 Scopus Author ID 7006024942 7006024942 CreatorName Tomiyama, Noriyuki
Subject	Active learning
	Bidirectional encoder representations from transformers
	Medical safety
	Natural language processing
	Radiology report
Description	The version of record of this article, first published in International Journal of Computer Assisted Radiology and Surgery, is available online at Publisher’s website: https://doi.org/10.1007/s11548-024-03316-7.
Abstract	Purpose: Systems equipped with natural language (NLP) processing can reduce missed radiological findings by physicians, but the annotation costs are burden in the development. This study aimed to compare the effects of active learning (AL) algorithms in NLP for estimating the significance of head computed tomography (CT) reports using bidirectional encoder representations from transformers (BERT). Methods: A total of 3728 head CT reports annotated with five categories of importance were used and UTH-BERT was adopted as the pre-trained BERT model. We assumed that 64% (2385 reports) of the data were initially in the unlabeled data pool (UDP), while the labeled data set (LD) used to train the model was empty. Twenty-five reports were repeatedly selected from the UDP and added to the LD, based on seven metrices: random sampling (RS: control), four uncertainty sampling (US) methods (least confidence (LC), margin sampling (MS), ratio of confidence (RC), and entropy sampling (ES)), and two distance-based sampling (DS) methods (cosine distance (CD) and Euclidian distance (ED)). The transition of accuracy of the model was evaluated using the test dataset. Results: The accuracy of the models with US was significantly higher than RS when reports in LD were < 1800, whereas DS methods were significantly lower than RS. Among the US methods, MS and RC were even better than the others. With the US methods, the required labeled data decreased by 15.4–40.5%, and most efficient in RC. In addition, in the US methods, data for minor categories tended to be added to LD earlier than RS and DS. Conclusions: In the classification task for the importance of head CT reports, US methods, especially RC and MS can lead to the effective fine-tuning of BERT models and reduce the imbalance of categories. AL can contribute to other studies on larger datasets by providing effective annotation.
Publisher	Springer Science and Business Media Deutschland GmbH
Source Title	International Journal of Computer Assisted Radiology and Surgery
Date of Issued	2025-01-08
Language	English
Handle URL	https://hdl.handle.net/11094/100326
PISSN	18616410
eISSN	18616429
Relation.isIdenticalTo	DOI (Publisher Version) https://doi.org/10.1007/s11548-024-03316-7
Access Rights	open access
CopyRight	This article is licensed under a Creative Commons Attribution 4.0 International License.
oaire:version	Version of Record
Category	学術雑誌論文 Journal Article

Resource Type	journal article
Local Type	Journal Article
DCTERMS.bibliographicCitation	International Journal of Computer Assisted Radiology and Surgery. 2025
DC.title	Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers
DC.creator	Wataya, Tomohiro
	Miura, Azusa
	Sakisuka, Takahisa
	Fujiwara, Masahiro
	Tanaka, Hisashi
	Hiraoka, Yu
	Sato, Junya
	Tomiyama, Miyuki
	Nishigaki, Daiki
	Kita, Kosuke
	Suzuki, Yuki
	Kido, Shoji
	Tomiyama, Noriyuki
DC.publisher	Springer Science and Business Media Deutschland GmbH
DC.language' scheme='DCTERMS.RFC1766	English
DCTERMS.issued' scheme='DCTERMS.W3CDTF	2025-01-08
DC.identifier	https://doi.org/10.1007/s11548-024-03316-7
DC.subject	Active learning
	Bidirectional encoder representations from transformers
	Medical safety
	Natural language processing
	Radiology report
DC.description	The version of record of this article, first published in International Journal of Computer Assisted Radiology and Surgery, is available online at Publisher’s website: https://doi.org/10.1007/s11548-024-03316-7.
DCTERMS.abstract	Purpose: Systems equipped with natural language (NLP) processing can reduce missed radiological findings by physicians, but the annotation costs are burden in the development. This study aimed to compare the effects of active learning (AL) algorithms in NLP for estimating the significance of head computed tomography (CT) reports using bidirectional encoder representations from transformers (BERT). Methods: A total of 3728 head CT reports annotated with five categories of importance were used and UTH-BERT was adopted as the pre-trained BERT model. We assumed that 64% (2385 reports) of the data were initially in the unlabeled data pool (UDP), while the labeled data set (LD) used to train the model was empty. Twenty-five reports were repeatedly selected from the UDP and added to the LD, based on seven metrices: random sampling (RS: control), four uncertainty sampling (US) methods (least confidence (LC), margin sampling (MS), ratio of confidence (RC), and entropy sampling (ES)), and two distance-based sampling (DS) methods (cosine distance (CD) and Euclidian distance (ED)). The transition of accuracy of the model was evaluated using the test dataset. Results: The accuracy of the models with US was significantly higher than RS when reports in LD were < 1800, whereas DS methods were significantly lower than RS. Among the US methods, MS and RC were even better than the others. With the US methods, the required labeled data decreased by 15.4–40.5%, and most efficient in RC. In addition, in the US methods, data for minor categories tended to be added to LD earlier than RS and DS. Conclusions: In the classification task for the importance of head CT reports, US methods, especially RC and MS can lead to the effective fine-tuning of BERT models and reduce the imbalance of categories. AL can contribute to other studies on larger datasets by providing effective annotation.
DC.rights	This article is licensed under a Creative Commons Attribution 4.0 International License.
DC.format	100326
citation_title	Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers
citation_author	Wataya, Tomohiro
	Miura, Azusa
	Sakisuka, Takahisa
	Fujiwara, Masahiro
	Tanaka, Hisashi
	Hiraoka, Yu
	Sato, Junya
	Tomiyama, Miyuki
	Nishigaki, Daiki
	Kita, Kosuke
	Suzuki, Yuki
	Kido, Shoji
	Tomiyama, Noriyuki
citation_publisher	Springer Science and Business Media Deutschland GmbH
citation_language	English
citation_date	2025-01-08
citation_journal_title	International Journal of Computer Assisted Radiology and Surgery
citation_doi	https://doi.org/10.1007/s11548-024-03316-7
citation_public_url	https://hdl.handle.net/11094/100326

Comparison of active learning algorithms in classifying head computed tomography reports using bidirectional encoder representations from transformers

Link to primary information

Item Information

Show details