{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T15:31:14Z","timestamp":1782315074158,"version":"3.54.5"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T00:00:00Z","timestamp":1695772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61976107"],"award-info":[{"award-number":["61976107"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Qing Lan Project of Colleges and Universities of Jiangsu Province in 2020"},{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["FL-170100117"],"award-info":[{"award-number":["FL-170100117"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2024,2,29]]},"abstract":"<jats:p>\n            <jats:bold>Knowledge distillation (KD)<\/jats:bold>\n            is a powerful and widely applicable technique for the compression of deep learning models. The main idea of knowledge distillation is to transfer knowledge from a large teacher model to a small student model, where the attention mechanism has been intensively explored in regard to its great flexibility for managing different teacher-student architectures. However, existing attention-based methods usually transfer similar attention knowledge from the intermediate layers of deep neural networks, leaving the hierarchical structure of deep representation learning poorly investigated for knowledge distillation. In this paper, we propose a\n            <jats:bold>hierarchical multi-attention transfer framework (HMAT)<\/jats:bold>\n            , where different types of attention are utilized to transfer the knowledge at different levels of deep representation learning for knowledge distillation. Specifically, position-based and channel-based attention knowledge characterize the knowledge from low-level and high-level feature representations, respectively, and activation-based attention knowledge characterize the knowledge from both mid-level and high-level feature representations. Extensive experiments on three popular visual recognition tasks, image classification, image retrieval, and object detection, demonstrate that the proposed hierarchical multi-attention transfer or HMAT significantly outperforms recent state-of-the-art KD methods.\n          <\/jats:p>","DOI":"10.1145\/3568679","type":"journal-article","created":{"date-parts":[[2022,10,20]],"date-time":"2022-10-20T11:52:23Z","timestamp":1666266743000},"page":"1-20","update-policy":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":68,"title":["Hierarchical Multi-Attention Transfer for Knowledge Distillation"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0003-1413-0693","authenticated-orcid":false,"given":"Jianping","family":"Gou","sequence":"first","affiliation":[{"name":"College of Computer and Information Science, College of Software, Southwest University, China School of Computer Science and Communication Engineering, Jiangsu University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0002-6079-5541","authenticated-orcid":false,"given":"Liyuan","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Computer Science and Communication Engineering, Jiangsu University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0002-0761-7893","authenticated-orcid":false,"given":"Baosheng","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Computer Science, The University of Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0001-7013-9081","authenticated-orcid":false,"given":"Shaohua","family":"Wan","sequence":"additional","affiliation":[{"name":"Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0001-7225-5449","authenticated-orcid":false,"given":"Dacheng","family":"Tao","sequence":"additional","affiliation":[{"name":"School of Computer Science, The University of Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,9,27]]},"reference":[{"key":"e_1_3_1_2_2","first-page":"1","article-title":"FitNets: Hints for thin deep nets","author":"Adriana Romero","year":"2015","unstructured":"Romero Adriana, Ballas Nicolas, K. Samira Ebrahimi, Chassang Antoine, Gatta Carlo, and B. Yoshua. 2015. FitNets: Hints for thin deep nets. International Conference on Learning Representations (2015), 1\u201313.","journal-title":"International Conference on Learning Representations"},{"key":"e_1_3_1_3_2","first-page":"9163","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Ahn Sungsoo","year":"2019","unstructured":"Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. 2019. Variational information distillation for knowledge transfer. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 9163\u20139171."},{"key":"e_1_3_1_4_2","article-title":"Neural machine translation by jointly learning to align and translate","author":"Bahdanau Dzmitry","year":"2014","unstructured":"Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).","journal-title":"arXiv preprint arXiv:1409.0473"},{"key":"e_1_3_1_5_2","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Borui Zhao","year":"2022","unstructured":"Zhao Borui, Cui Quan, Song Renjie, Qiu Yiyu, and Liang Jiajun. 2022. Decoupled knowledge distillation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition."},{"key":"e_1_3_1_6_2","first-page":"7028","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"35","author":"Chen Defang","year":"2021","unstructured":"Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. 2021. Cross-layer distillation with semantic calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 7028\u20137036."},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.667"},{"key":"e_1_3_1_8_2","article-title":"End-to-end continuous speech recognition using attention-based recurrent NN: First results","author":"Chorowski Jan","year":"2014","unstructured":"Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. End-to-end continuous speech recognition using attention-based recurrent NN: First results. arXiv preprint arXiv:1412.1602 (2014).","journal-title":"arXiv preprint arXiv:1412.1602"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_1_10_2","unstructured":"Mark Everingham and John Winn. 2009. The PASCAL visual object classes challenge 2007 (VOC2007) development kit. (2009)."},{"key":"e_1_3_1_11_2","first-page":"5","article-title":"The PASCAL visual object classes challenge 2012 (VOC2012) development kit","volume":"8","author":"Everingham Mark","year":"2011","unstructured":"Mark Everingham and John Winn. 2011. The PASCAL visual object classes challenge 2012 (VOC2012) development kit. Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep 8 (2011), 5.","journal-title":"Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2020.2995049"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-021-01453-z"},{"key":"e_1_3_1_14_2","first-page":"469","volume-title":"European Conference on Computer Vision","author":"Guan Yushuo","year":"2020","unstructured":"Yushuo Guan, Pengyu Zha o, Bingxuan Wang, Yuanxing Zhang, Cong Yao, Kaigui Bian, and Jian Tang. 2020. Differentiable feature aggregation search for knowledge distillation. In European Conference on Computer Vision. 469\u2013484."},{"key":"e_1_3_1_15_2","article-title":"Learning both weights and connections for efficient neural network","volume":"28","author":"Han Song","year":"2015","unstructured":"Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems 28 (2015).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_1_17_2","first-page":"1921","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Heo Byeongho","year":"2019","unstructured":"Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. 2019. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 1921\u20131930."},{"issue":"7","key":"e_1_3_1_18_2","article-title":"Distilling the knowledge in a neural network","volume":"2","author":"Hinton Geoffrey","year":"2015","unstructured":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et\u00a0al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2, 7 (2015).","journal-title":"arXiv preprint arXiv:1503.02531"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01350"},{"key":"e_1_3_1_20_2","article-title":"Like what you like: Knowledge distill via neuron selectivity transfer","author":"Huang Zehao","year":"2017","unstructured":"Zehao Huang and Naiyan Wang. 2017. Like what you like: Knowledge distill via neuron selectivity transfer. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (2017).","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"e_1_3_1_21_2","article-title":"Binarized neural networks","volume":"29","author":"Hubara Itay","year":"2016","unstructured":"Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks. Advances in Neural Information Processing Systems 29 (2016).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_22_2","first-page":"7945","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"35","author":"Ji Mingi","year":"2021","unstructured":"Mingi Ji, Byeongho Heo, and Sungrae Park. 2021. Show, attend and distill: Knowledge distillation via attention-based feature matching. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 7945\u20137952."},{"key":"e_1_3_1_23_2","article-title":"Paraphrasing complex network: Network compression via factor transfer","volume":"31","author":"Kim Jangho","year":"2018","unstructured":"Jangho Kim, SeongUk Park, and Nojun Kwak. 2018. Paraphrasing complex network: Network compression via factor transfer. Advances in Neural Information Processing Systems 31 (2018).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_24_2","volume-title":"International Conference on Learning Representations","author":"Komodakis Nikos","year":"2017","unstructured":"Nikos Komodakis and Sergey Zagoruyko. 2017. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representations."},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2013.77"},{"key":"e_1_3_1_26_2","unstructured":"Alex Krizhevsky Geoffrey Hinton et\u00a0al. 2009. Learning multiple layers of features from tiny images. (2009)."},{"key":"e_1_3_1_27_2","doi-asserted-by":"crossref","first-page":"2225","DOI":"10.1109\/ICASSP39728.2021.9415117","volume-title":"ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing","author":"Kundu Souvik","year":"2021","unstructured":"Souvik Kundu and Sairam Sundaresan. 2021. AttentionLite: Towards efficient self-attention models for vision. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing. 2225\u20132229."},{"key":"e_1_3_1_28_2","article-title":"Pruning filters for efficient ConvNets","author":"Li Hao","year":"2016","unstructured":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient ConvNets. arXiv preprint arXiv:1608.08710 (2016).","journal-title":"arXiv preprint arXiv:1608.08710"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2019.2946102"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01264-9_8"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0911-8"},{"key":"e_1_3_1_32_2","first-page":"7573","volume-title":"International Conference on Learning Representations","author":"Mitsuno Kakeru","year":"2021","unstructured":"Kakeru Mitsuno, Yuichiro Nomura, and Takio Kurita. 2021. Channel planting for deep neural networks using knowledge distillation. In International Conference on Learning Representations. 7573\u20137579."},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.434"},{"key":"e_1_3_1_34_2","first-page":"5007","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Peng Baoyun","year":"2019","unstructured":"Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. 2019. Correlation congruence for knowledge distillation. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 5007\u20135016."},{"key":"e_1_3_1_35_2","first-page":"249","volume-title":"Chinese Conference on Pattern Recognition and Computer Vision (PRCV\u201920)","author":"Qu Yan","year":"2020","unstructured":"Yan Qu, Weihong Deng, and Jiani Hu. 2020. H-AT: Hybrid attention transfer for knowledge distillation. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV\u201920). 249\u2013260."},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46493-0_32"},{"key":"e_1_3_1_37_2","article-title":"Faster R-CNN: Towards real-time object detection with region proposal networks","volume":"28","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems 28 (2015).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00474"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.74"},{"key":"e_1_3_1_40_2","first-page":"3243","article-title":"Channel-wise distillation for semantic segmentation","author":"Shu Changyong","year":"2020","unstructured":"Changyong Shu, Yifan Liu, Jianfei Gao, Lin Xu, and Chunhua Shen. 2020. Channel-wise distillation for semantic segmentation. arXiv e-prints (2020), 3243\u20133249.","journal-title":"arXiv e-prints"},{"key":"e_1_3_1_41_2","first-page":"1","article-title":"Efficient semantic segmentation via self-attention and self-distillation","author":"Shumin An","year":"2022","unstructured":"An Shumin, Liao Qingmin, Lu Zongqing, and Xue Jing-Hao. 2022. Efficient semantic segmentation via self-attention and self-distillation. IEEE Transactions on Intelligent Transportation Systems (2022), 1\u201311.","journal-title":"IEEE Transactions on Intelligent Transportation Systems"},{"key":"e_1_3_1_42_2","article-title":"Contrastive representation distillation","author":"Tian Yonglong","year":"2019","unstructured":"Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive representation distillation. arXiv preprint arXiv:1910.10699 (2019).","journal-title":"arXiv preprint arXiv:1910.10699"},{"key":"e_1_3_1_43_2","unstructured":"Catherine Wah Steve Branson Peter Welinder Pietro Perona and Serge Belongie. 2011. The Caltech-UCSD Birds-200-2011 Dataset. (2011)."},{"key":"e_1_3_1_44_2","first-page":"5776","article-title":"MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers","volume":"33","author":"Wang Wenhui","year":"2020","unstructured":"Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems 33 (2020), 5776\u20135788.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_45_2","first-page":"588","volume-title":"European Conference on Computer Vision","author":"Xu Guodong","year":"2020","unstructured":"Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. 2020. Knowledge distillation meets self-supervision. In European Conference on Computer Vision. 588\u2013604."},{"key":"e_1_3_1_46_2","first-page":"294","volume-title":"European Conference on Computer Vision","author":"Yao Anbang","year":"2020","unstructured":"Anbang Yao and Dawei Sun. 2020. Knowledge transfer via dense cross-layer mutual-distillation. In European Conference on Computer Vision. 294\u2013311."},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.754"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098135"},{"key":"e_1_3_1_49_2","article-title":"Wide residual networks","author":"Zagoruyko Sergey","year":"2016","unstructured":"Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 (2016).","journal-title":"arXiv preprint arXiv:1605.07146"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"e_1_3_1_51_2","article-title":"Student network learning via evolutionary knowledge distillation","author":"Zhang Kangkai","year":"2021","unstructured":"Kangkai Zhang, Chunhui Zhanga, Shikun Li, Dan Zeng, and Shiming Ge. 2021. Student network learning via evolutionary knowledge distillation. IEEE Transactions on Circuits and Systems for Video Technology (2021).","journal-title":"IEEE Transactions on Circuits and Systems for Video Technology"},{"key":"e_1_3_1_52_2","volume-title":"International Conference on Learning Representations","author":"Zhang Linfeng","year":"2020","unstructured":"Linfeng Zhang and Kaisheng Ma. 2020. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations."},{"key":"e_1_3_1_53_2","article-title":"Highlight every step: Knowledge distillation via collaborative teaching","author":"Zhao Haoran","year":"2020","unstructured":"Haoran Zhao, Xin Sun, Junyu Dong, Changrui Chen, and Zihe Dong. 2020. Highlight every step: Knowledge distillation via collaborative teaching. IEEE Transactions on Cybernetics (2020).","journal-title":"IEEE Transactions on Cybernetics"},{"key":"e_1_3_1_54_2","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Zhendong Yang","year":"2022","unstructured":"Yang Zhendong, Li Zhe, Jiang Xiaohu, Gong Yuan, Yuan Zehuan, Zhao Danpei, and Chun Yuan. 2022. Focal and global knowledge distillation for detectors. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition."},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2022.3141255"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1002\/int.22819"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/3568679","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/dl.acm.org\/doi\/pdf\/10.1145\/3568679","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:08:34Z","timestamp":1750183714000},"score":1,"resource":{"primary":{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/3568679"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,27]]},"references-count":55,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,2,29]]}},"alternative-id":["10.1145\/3568679"],"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/doi.org\/10.1145\/3568679","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,27]]},"assertion":[{"value":"2022-03-28","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-10-02","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}