11# OpenVtuber-虚拟爱抖露共享计划
22
3- ## Kizuna-Ai MMD demo : face capture via single RGB camera
3+ Kizuna-Ai MMD demo : face capture via single RGB camera
44
55<p align =" center " ><img src =" docs/images/one.gif " /></p >
66<p align =" center " ><img src =" docs/images/two.gif " /></p >
77
88## Installation
9+
910### Requirements
1011
1112* Python 3.5+
@@ -22,56 +23,65 @@ While not required, for optimal performance(especially for the detector) it is h
2223* ` python3.7 ./PythonClient/vtuber_usb_camera.py --gpu -1 `
2324
2425## 人脸检测 (Face Detection)
25- * [ RetinaFace: Single-stage Dense Face Localisation in the Wild] ( https://arxiv.org/abs/1905.00641 )
26- * [ RetinaFace (mxnet version)] ( https://github.com/deepinsight/insightface/tree/master/RetinaFace )
27-
28- RetinaFace is a practical single-stage [ SOTA] ( http://shuoyang1213.me/WIDERFACE/WiderFace_Results.html ) face detector which is initially described in [ arXiv technical report] ( https://arxiv.org/abs/1905.00641 )
2926
30- ![ demoimg1 ] ( https://github.com/deepinsight/insightface/blob /master/resources/11513D05.jpg )
27+ [ RetinaFace: Single-stage Dense Face Localisation in the Wild ] ( https://openaccess.thecvf.com/content_CVPR_2020/html/Deng_RetinaFace_Single-Shot_Multi-Level_Face_Localisation_in_the_Wild_CVPR_2020_paper.html ) of ** CVPR 2020 ** , is a practical single-stage [ SOTA ] ( http://shuoyang1213.me/WIDERFACE/WiderFace_Results.html ) face detector. It is highly recommended to read the official repo [ RetinaFace (mxnet version) ] ( https:// github.com/deepinsight/insightface/tree /master/RetinaFace ) .
3128
32- ![ demoimg2 ] ( https://github.com/deepinsight/insightface/blob/master/resources/widerfacevaltest.png )
29+ However, since the detection target of the face capture system is in the middle-close range, there is no need for complex pyramid scaling. We designed and published [ Faster RetinaFace ] ( https://github.com/1996scarlet/faster-mobile-retinaface ) to trade off between speed and accuracy, which can reach 500 ~ 1000 fps on normal laptops.
3330
34- ## 头部姿态估计(Head Pose Estimation)
35- * [ head-pose-estimation] ( https://github.com/lincolnhard/head-pose-estimation )
31+ | Plan | Inference | Postprocess | Throughput Capacity (FPS)
32+ | --------|-----|--------|---------
33+ | 9750HQ+1660TI | 0.9ms | 1.5ms | 500~ 1000
34+ | Jetson-Nano | 4.6ms | 11.4ms | 80~ 200
3635
3736## 特征点检测(Facial Landmarks Tracking)
37+
3838The 2D pre-trained model is from the [ deep-face-alignment] ( https://github.com/deepinx/deep-face-alignment ) repository.
39+
3940* Algorithm from [ TPAMI 2019] ( https://arxiv.org/pdf/1808.04803.pdf )
4041* Training set is based on i-bug 300-W datasets. It's annotation is shown below:<br ><br >
4142![ ibug] ( https://cloud.githubusercontent.com/assets/16308037/24229391/1910e9cc-0fb4-11e7-987b-0fecce2c829e.JPG )
4243
43- ## 注视估计(Gaze Estimation)
44+ ## 头部姿态估计(Head Pose Estimation)
4445
45- - [ Laser Eye : Gaze Estimation via Deep Neural Networks ] ( https://github.com/1996scarlet/Laser-Eye )
46+ * [ head-pose-estimation ] ( https://github.com/lincolnhard/head-pose-estimation )
4647
47- ## MMD Loader
48+ ## 注视估计(Gaze Estimation)
4849
49- - [ Three.js Webgl Loader ] ( https://threejs.org/examples/?q=MMD#webgl_loader_mmd )
50+ * [ Laser Eye : Gaze Estimation via Deep Neural Networks ] ( https://github.com/1996scarlet/Laser-Eye )
5051
51- ## Live2D
52+ ## MMD Loader
5253
53- - [ 插件版本] ( https://github.com/EYHN/hexo-helper-live2d )
54- - [ 打包版本] ( https://github.com/galnetwen/Live2D )
54+ We apply [ Three.js Webgl Loader] ( https://threejs.org/examples/?q=MMD#webgl_loader_mmd ) to render MMD model on web pages.
5555
56- ## Thanks
56+ ## Special Thanks
5757
58- - [ threejs.org] ( https://threejs.org/ )
59- - [ kizunaai.com] ( http://kizunaai.com/ )
58+ * [ threejs.org] ( https://threejs.org/ )
59+ * [ kizunaai.com] ( http://kizunaai.com/ )
6060
6161## Citation
6262
63- ```
63+ ``` bibtex
64+ @misc{sun2020backbone,
65+ title={A Backbone Replaceable Fine-tuning Network for Stable Face Alignment},
66+ author={Xu Sun and Yingjie Guo and Shihong Xia},
67+ year={2020},
68+ eprint={2010.09501},
69+ archivePrefix={arXiv},
70+ primaryClass={cs.CV}
71+ }
72+
6473@article{Bulat2018Hierarchical,
65- title={Hierarchical binary CNNs for landmark localization with limited resources},
66- author={Bulat, Adrian and Tzimiropoulos, Yorgos},
67- journal={IEEE Transactions on Pattern Analysis & Machine Intelligence},
68- year={2018},
74+ title={Hierarchical binary CNNs for landmark localization with limited resources},
75+ author={Bulat, Adrian and Tzimiropoulos, Yorgos},
76+ journal={IEEE Transactions on Pattern Analysis & Machine Intelligence},
77+ year={2018},
6978}
70-
71- @inproceedings{deng2019retinaface,
72- title={RetinaFace: Single-stage Dense Face Localisation in the Wild},
73- author={Deng, Jiankang and Guo, Jia and Yuxiang, Zhou and Jinke Yu and Irene Kotsia and Zafeiriou, Stefanos},
74- booktitle={arxiv},
75- year={2019}
79+
80+ @InProceedings{Deng_2020_CVPR,
81+ author = {Deng, Jiankang and Guo, Jia and Ververas, Evangelos and Kotsia, Irene and Zafeiriou, Stefanos},
82+ title = {RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild},
83+ booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
84+ month = {June},
85+ year = {2020}
7686}
7787```
0 commit comments