{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T09:45:12Z","timestamp":1753868712760,"version":"3.41.2"},"reference-count":38,"publisher":"Wiley","issue":"18","license":[{"start":{"date-parts":[[2021,7,6]],"date-time":"2021-07-06T00:00:00Z","timestamp":1625529600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/http\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["U19A2080"],"award-info":[{"award-number":["U19A2080"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Concurrency and Computation"],"published-print":{"date-parts":[[2023,8,15]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>The development of lightweight networks makes neural networks more efficient to be widely applied to various tasks. Considering the deployment of hardware like edge devices and mobile phones, we prioritize lightweight networks. However, their accuracy has always lagged far behind SOTA networks. In this article, we present a simple yet effective activation function, called WReLU, to improve the performance of lightweight networks significantly by adding a residual spatial condition. Moreover, we use a strategy to switch activation functions after determining which convolutional layer to use. We perform experiments on ImageNet 2012 classification dataset in CPU, GPU, and edge devices. Experiments demonstrate that WReLU improves the accuracy of classification significantly. Meanwhile, our strategy balances the effect of additional parameters and multiply accumulate. Our method improves the accuracy of SqueezeNet and SqueezeNext by more than 5% without increasing extensive parameters and computation. For the lightweight network with a large number of parameters, such as MobileNet and ShuffleNet, there is also a significant improvement. Additionally, the inference speed of most lightweight networks using our WReLU strategy is almost the same as the baseline model on different platforms. Our approach not only ensures the practicability of the lightweight network but also improves its performance.<\/jats:p>","DOI":"10.1002\/cpe.6350","type":"journal-article","created":{"date-parts":[[2021,7,7]],"date-time":"2021-07-07T05:26:16Z","timestamp":1625635576000},"update-policy":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Novel activation function with pixelwise modeling capacity for lightweight neural network design"],"prefix":"10.1002","volume":"35","author":[{"ORCID":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/orcid.org\/0000-0003-3056-7713","authenticated-orcid":false,"given":"Yi","family":"Liu","sequence":"first","affiliation":[{"name":"Lab of Artificial Networks, Institute of Semiconductors Chinese Academy of Sciences  Beijing China"},{"name":"University of Chinese Academy of Sciences  Beijing China"}]},{"given":"Xiaozhou","family":"Guo","sequence":"additional","affiliation":[{"name":"Lab of Artificial Networks, Institute of Semiconductors Chinese Academy of Sciences  Beijing China"},{"name":"University of Chinese Academy of Sciences  Beijing China"}]},{"given":"Kaijun","family":"Tan","sequence":"additional","affiliation":[{"name":"Lab of Artificial Networks, Institute of Semiconductors Chinese Academy of Sciences  Beijing China"},{"name":"University of Chinese Academy of Sciences  Beijing China"}]},{"given":"Guoliang","family":"Gong","sequence":"additional","affiliation":[{"name":"Lab of Artificial Networks, Institute of Semiconductors Chinese Academy of Sciences  Beijing China"},{"name":"University of Chinese Academy of Sciences  Beijing China"}]},{"given":"Huaxiang","family":"Lu","sequence":"additional","affiliation":[{"name":"Lab of Artificial Networks, Institute of Semiconductors Chinese Academy of Sciences  Beijing China"},{"name":"University of Chinese Academy of Sciences  Beijing China"},{"name":"Center for Excellence in Brain Science and Intelligence Technology Chinese Academy of Sciences  Shanghai China"},{"name":"Beijing Key Laboratory of Semiconductor Neural Network Intelligent Sensing and Computing Technology  Beijing China"}]}],"member":"311","published-online":{"date-parts":[[2021,7,6]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_2_8_2_1","DOI":"10.13328\/j.cnki.jos.005942"},{"doi-asserted-by":"crossref","unstructured":"DaiJ QiH XiongY et al. Deformable convolutional networks. Paper presented at: Proceedings of the IEEE International Conference on Computer Vision. Rio de Janeiro Brazil;2017:764\u2010773.","key":"e_1_2_8_3_1","DOI":"10.1109\/ICCV.2017.89"},{"doi-asserted-by":"publisher","key":"e_1_2_8_4_1","DOI":"10.1007\/978-3-642-75988-8_28"},{"key":"e_1_2_8_5_1","first-page":"3","article-title":"Rectifier nonlinearities improve neural network acoustic models","volume":"30","author":"Maas AL","year":"2013","journal-title":"Proceedings of the ICML"},{"doi-asserted-by":"crossref","unstructured":"HeK ZhangX RenS SunJ. Delving deep into rectifiers: surpassing human\u2010level performance on imagenet classification. Paper presented at: Proceedings of the IEEE International Conference on Computer Vision. Beijing China;2015:1026\u20101034.","key":"e_1_2_8_6_1","DOI":"10.1109\/ICCV.2015.123"},{"unstructured":"ClevertDA UnterthinerT HochreiterS. Fast and accurate deep network learning by exponential linear units (elus);2015. arXiv preprint arXiv:1511.07289 2015.","key":"e_1_2_8_7_1"},{"key":"e_1_2_8_8_1","first-page":"972","article-title":"Self\u2010normalizing neural networks","author":"Klambauer G","year":"2017","journal-title":"Proceedings of the 31st international conference on neural information processing systems. Long Beach, CA"},{"unstructured":"NairV HintonGE. Rectified linear units improve restricted Boltzmann machines. ICML;2010.","key":"e_1_2_8_9_1"},{"unstructured":"IoffeS SzegedyC. Batch normalization: Accelerating deep network training by reducing internal covariate shift;2015. arXiv preprint arXiv:1502.03167.","key":"e_1_2_8_10_1"},{"key":"e_1_2_8_11_1","first-page":"1097","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky A","year":"2012","journal-title":"Adv Neural Inf Process Syst"},{"doi-asserted-by":"crossref","unstructured":"SzegedyC LiuW JiaY et al. Going deeper with convolutions. Proceedings of the IEEE Conference on computer vision and pattern recognition;2015:1\u20109.","key":"e_1_2_8_12_1","DOI":"10.1109\/CVPR.2015.7298594"},{"unstructured":"IandolaFN HanS MoskewiczMW AshrafK DallyWJ KeutzerK. SqueezeNet: AlexNet\u2010level accuracy with 50x fewer parameters and<\u20090.5 MB model size;2016. arXiv preprint arXiv:1602.07360.","key":"e_1_2_8_13_1"},{"doi-asserted-by":"crossref","unstructured":"CholletF.Xception: deep learning with depthwise separable convolutions. Paper presented at: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu Hawaii;2017:1251\u20101258.","key":"e_1_2_8_14_1","DOI":"10.1109\/CVPR.2017.195"},{"doi-asserted-by":"crossref","unstructured":"SzegedyC VanhouckeV IoffeS ShlensJ WojnaZ. Rethinking the inception architecture for computer vision. Paper presented at: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas Nevada;2016:2818\u20102826.","key":"e_1_2_8_15_1","DOI":"10.1109\/CVPR.2016.308"},{"unstructured":"SzegedyC IoffeS VanhouckeV AlemiA. Inception\u2010ResNet and the Impact of Residual Connections on Learning;2016. arXiv preprint arXiv:1602.07261.","key":"e_1_2_8_16_1"},{"doi-asserted-by":"crossref","unstructured":"HeK ZhangX RenS SunJ. Deep residual learning for image recognition. Paper presented at: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas Nevada;2016:770\u2010778.","key":"e_1_2_8_17_1","DOI":"10.1109\/CVPR.2016.90"},{"doi-asserted-by":"crossref","unstructured":"HeK ZhangX RenS SunJ. Identity mappings in deep residual networks. Paper presented at: Proceedings of the European Conference on Computer Vision. Amsterdam The Netherlands;2016:630\u2010645.","key":"e_1_2_8_18_1","DOI":"10.1007\/978-3-319-46493-0_38"},{"doi-asserted-by":"crossref","unstructured":"GholamiA KwonK WuB et al. Squeezenext: hardware\u2010aware neural network design. Paper presented at: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City Utah;2018:1638\u20101647.","key":"e_1_2_8_19_1","DOI":"10.1109\/CVPRW.2018.00215"},{"unstructured":"HowardAG ZhuM ChenB et al. Efficient convolutional neural networks for mobile vision applications;2017. arXiv preprint arXiv:1704.04861.","key":"e_1_2_8_20_1"},{"doi-asserted-by":"crossref","unstructured":"SandlerM HowardA ZhuM ZhmoginovA ChenLC. Mobilenetv2: Inverted residuals and linear bottlenecks. Paper presented at: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City Utah;2018:4510\u20104520.","key":"e_1_2_8_21_1","DOI":"10.1109\/CVPR.2018.00474"},{"unstructured":"ZhangX ZhouX LinM SunJ. An extremely efficient convolutional neural network for mobile devices;2017. arXiv preprint arXiv:1707.01083.","key":"e_1_2_8_22_1"},{"unstructured":"HanS MaoH DallyWJ. Deep compression: Compressing deep neural networks with pruning trained quantization and Huffman coding;2015. arXiv preprint arXiv:1510.00149.","key":"e_1_2_8_23_1"},{"key":"e_1_2_8_24_1","first-page":"1135","article-title":"Learning both weights and connections for efficient neural network","volume":"28","author":"Han S","year":"2015","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_8_25_1","first-page":"2074","article-title":"Learning structured sparsity in deep neural networks","volume":"29","author":"Wen W","year":"2016","journal-title":"Adv Neural Inf Process Syst"},{"doi-asserted-by":"crossref","unstructured":"JacobB KligysS ChenB et al. Quantization and training of neural networks for efficient integer\u2010arithmetic\u2010only inference. Paper presented at: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City Utah;2018:2704\u20102713.","key":"e_1_2_8_26_1","DOI":"10.1109\/CVPR.2018.00286"},{"unstructured":"KrishnamoorthiR. Quantizing deep convolutional networks for efficient inference: a whitepaper;2018. arXiv preprint arXiv:1806.08342.","key":"e_1_2_8_27_1"},{"doi-asserted-by":"crossref","unstructured":"RastegariM OrdonezV RedmonJ FarhadiA. Xnor\u2010net: imagenet classification using binary convolutional neural networks. Paper presented at: Proceedings of the European Conference on Computer Vision. Amsterdam The Netherlands;2016:525\u2010542.","key":"e_1_2_8_28_1","DOI":"10.1007\/978-3-319-46493-0_32"},{"doi-asserted-by":"crossref","unstructured":"JaderbergM VedaldiA ZissermanA. Speeding up convolutional neural networks with low rank expansions;2014. arXiv preprint arXiv:1405.3866.","key":"e_1_2_8_29_1","DOI":"10.5244\/C.28.88"},{"doi-asserted-by":"publisher","key":"e_1_2_8_30_1","DOI":"10.1038\/323533a0"},{"unstructured":"HintonGE SrivastavaN KrizhevskyA SutskeverI SalakhutdinovRR. Improving neural networks by preventing co\u2010adaptation of feature detectors;2012. arXiv preprint arXiv:1207.0580.","key":"e_1_2_8_31_1"},{"unstructured":"MaN ZhangX SunJ. Funnel activation for visual recognition;2020. arXiv preprint arXiv:2007.11824.","key":"e_1_2_8_32_1"},{"doi-asserted-by":"crossref","unstructured":"ZhuX HuH LinS DaiJ. Deformable convnets v2: more deformable better results. Paper presented at: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach CA;2019:9308\u20109316.","key":"e_1_2_8_33_1","DOI":"10.1109\/CVPR.2019.00953"},{"key":"e_1_2_8_34_1","first-page":"2017","article-title":"Spatial transformer networks","volume":"28","author":"Jaderberg M","year":"2015","journal-title":"Advances in Neural Information Processing Systems"},{"doi-asserted-by":"crossref","unstructured":"CaoY XuJ LinS WeiF HuH. GCNet: non\u2010local networks meet squeeze\u2010excitation networks and beyond;2019. arXiv preprint arXiv:1904.11492.","key":"e_1_2_8_35_1","DOI":"10.1109\/ICCVW.2019.00246"},{"unstructured":"LoshchilovI HutterF. SGDR: stochastic gradient descent with warm restarts;2016. Arxiv.","key":"e_1_2_8_36_1"},{"doi-asserted-by":"publisher","key":"e_1_2_8_37_1","DOI":"10.1007\/s11263\u2010015\u20100816\u2010y"},{"doi-asserted-by":"crossref","unstructured":"LiuY LiZ ChenX GongG LuH. Improving the accuracy of SqueezeNet with negligible extra computational cost. Paper presented at: Proceedings of the 2020 International Conference on High Performance Big Data and Intelligent Systems (HPBD&IS). Shenzhen China;2020:1\u20106.","key":"e_1_2_8_38_1","DOI":"10.1109\/HPBDIS49115.2020.9130577"},{"doi-asserted-by":"crossref","unstructured":"RadosavovicI KosarajuRP GirshickR HeK Doll\u00e1rP. Designing network design spaces;2020. Arxiv.","key":"e_1_2_8_39_1","DOI":"10.1109\/CVPR42600.2020.01044"}],"container-title":["Concurrency and Computation: Practice and Experience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cpe.6350","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/cpe.6350","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cpe.6350","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,17]],"date-time":"2023-08-17T15:58:55Z","timestamp":1692287935000},"score":1,"resource":{"primary":{"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/onlinelibrary.wiley.com\/doi\/10.1002\/cpe.6350"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,6]]},"references-count":38,"journal-issue":{"issue":"18","published-print":{"date-parts":[[2023,8,15]]}},"alternative-id":["10.1002\/cpe.6350"],"URL":"https:\/\/summer-heart-0930.chufeiyun1688.workers.dev:443\/https\/doi.org\/10.1002\/cpe.6350","archive":["Portico"],"relation":{},"ISSN":["1532-0626","1532-0634"],"issn-type":[{"type":"print","value":"1532-0626"},{"type":"electronic","value":"1532-0634"}],"subject":[],"published":{"date-parts":[[2021,7,6]]},"assertion":[{"value":"2020-08-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-04-13","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-07-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e6350"}}