AVCLNet: Multimodal Multispeaker Tracking Network Using Audio-Visual Contrastive Learning

Li, Yihan; Li, Yidi; Xu, Zhenhuan; Guo, Hao; Liu, Mengyuan; Wan, Weiwei

doi:https://doi.org/10.1049/cit2.70092

AVCLNet: Multimodal Multispeaker Tracking Network Using Audio-Visual Contrastive Learning

Li, Yihan; Li, Yidi; Xu, Zhenhuan et al.

CAAI Transactions on Intelligence Technology, 2025

アクセス数:55件（2026-02-15 22:21 集計）

固定URL: https://hdl.handle.net/11094/103583

閲覧可能ファイル

ファイル	フォーマット	利用条件	サイズ	閲覧回数	利用開始日	説明	information
CAAITransIntellTechnol	pdf	なし	3.90 MB	22	2026.01.16

Creative Commons : 表示 - 非営利 - 改変禁止

論文情報

ファイル出力 EndNote Basic出力 Mendeley出力

タイトル	AVCLNet: Multimodal Multispeaker Tracking Network Using Audio-Visual Contrastive Learning
著者	著者名 Li, Yihan
	著者名 Li, Yidi
	著者名 Xu, Zhenhuan
	著者名 Guo, Hao
	著者名 Liu, Mengyuan
	NRID 1000020760002 1000020760002 ORCID 0000-0002-0058-2819 0000-0002-0058-2819 researchmap permalink weiweiwan weiweiwan Scopus Author ID 56238867200 56238867200 著者名 Wan, Weiwei
キーワード等	computer vision
	machine perception
	multimodal approaches
	pattern recognition
	video signal processing
内容	Li Y., Li Y., Xu Z., et al. AVCLNet: Multimodal Multispeaker Tracking Network Using Audio-Visual Contrastive Learning. CAAI Transactions on Intelligence Technology , (2025); https://doi.org/10.1049/cit2.70092.
抄録	Audio-visual speaker tracking aims to determine the locations of multiple speakers in the scene by leveraging signals captured from multisensor platforms. Multimodal fusion methods can improve both the accuracy and robustness of speaker tracking. However, in complex multispeaker tracking scenarios, critical challenges such as cross-modal feature discrepancy, weak sound source localisation ambiguity and frequent identity switch errors remain unresolved, which severely hinder the modelling of speaker identity consistency and consequently lead to degraded tracking accuracy and unstable tracking trajectories. To this end, this paper proposes a multimodal multispeaker tracking network using audio-visual contrastive learning (AVCLNet). By integrating heterogeneous modal representations into a unified space through audio-visual contrastive learning, which facilitates cross-modal feature alignment, mitigates cross-modal feature bias and enhances identity-consistent representations. In the audio-visual measurement stage, we design a vision-guided weak sound source weighted enhancement method, which leverages visual cues to establish cross-modal mappings and employs a spatiotemporal dynamic weighted mechanism to improve the detectability of weak sound sources. Furthermore, in the data association phase, a dual geometric constraint strategy is introduced by combining the 2D and 3D spatial geometric information, reducing frequent identity switch errors. Experiments on the AV16.3 and CAV3D datasets show that AVCLNet outperforms state-of-the-art methods, demonstrating superior robustness in multispeaker scenarios.
出版者	John Wiley and Sons Inc
収録物名	CAAI Transactions on Intelligence Technology
刊行年月	2025-11-29
言語	英語
ハンドルURL	https://hdl.handle.net/11094/103583
PISSN	24686557
eISSN	24682322
関連情報 (isIdenticalTo)	DOI (出版社版) https://doi.org/10.1049/cit2.70092
アクセス権	open access
権利情報	This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
出版タイプ	Version of Record
カテゴリ	学術雑誌論文 Journal Article

資源タイプ	学術雑誌論文
ローカル資源タイプ	学術雑誌論文
DCMI資源タイプ	text
DCTERMS.bibliographicCitation	CAAI Transactions on Intelligence Technology. 2025
DC.title	AVCLNet: Multimodal Multispeaker Tracking Network Using Audio-Visual Contrastive Learning
DC.creator	Li, Yihan
	Li, Yidi
	Xu, Zhenhuan
	Guo, Hao
	Liu, Mengyuan
	Wan, Weiwei
DC.publisher	John Wiley and Sons Inc
DC.language' scheme='DCTERMS.RFC1766	英語
DC.type' scheme='DCTERMS.DCMIType	text
DCTERMS.issued' scheme='DCTERMS.W3CDTF	2025-11-29
DC.identifier	https://doi.org/10.1049/cit2.70092
DC.subject	computer vision
	machine perception
	multimodal approaches
	pattern recognition
	video signal processing
DC.description	Li Y., Li Y., Xu Z., et al. AVCLNet: Multimodal Multispeaker Tracking Network Using Audio-Visual Contrastive Learning. CAAI Transactions on Intelligence Technology , (2025); https://doi.org/10.1049/cit2.70092.
DCTERMS.abstract	Audio-visual speaker tracking aims to determine the locations of multiple speakers in the scene by leveraging signals captured from multisensor platforms. Multimodal fusion methods can improve both the accuracy and robustness of speaker tracking. However, in complex multispeaker tracking scenarios, critical challenges such as cross-modal feature discrepancy, weak sound source localisation ambiguity and frequent identity switch errors remain unresolved, which severely hinder the modelling of speaker identity consistency and consequently lead to degraded tracking accuracy and unstable tracking trajectories. To this end, this paper proposes a multimodal multispeaker tracking network using audio-visual contrastive learning (AVCLNet). By integrating heterogeneous modal representations into a unified space through audio-visual contrastive learning, which facilitates cross-modal feature alignment, mitigates cross-modal feature bias and enhances identity-consistent representations. In the audio-visual measurement stage, we design a vision-guided weak sound source weighted enhancement method, which leverages visual cues to establish cross-modal mappings and employs a spatiotemporal dynamic weighted mechanism to improve the detectability of weak sound sources. Furthermore, in the data association phase, a dual geometric constraint strategy is introduced by combining the 2D and 3D spatial geometric information, reducing frequent identity switch errors. Experiments on the AV16.3 and CAV3D datasets show that AVCLNet outperforms state-of-the-art methods, demonstrating superior robustness in multispeaker scenarios.
DC.rights	This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
DC.format	103583
citation_title	AVCLNet: Multimodal Multispeaker Tracking Network Using Audio-Visual Contrastive Learning
citation_author	Li, Yihan
	Li, Yidi
	Xu, Zhenhuan
	Guo, Hao
	Liu, Mengyuan
	Wan, Weiwei
citation_publisher	John Wiley and Sons Inc
citation_language	英語
citation_date	2025-11-29
citation_journal_title	CAAI Transactions on Intelligence Technology
citation_doi	https://doi.org/10.1049/cit2.70092
citation_public_url	https://hdl.handle.net/11094/103583

AVCLNet: Multimodal Multispeaker Tracking Network Using Audio-Visual Contrastive Learning

閲覧可能ファイル

論文情報

論文詳細を表示