最近开发一个需要从Java端将图像传至native层的功能，发现了一个奇怪现象，native层每次获取到的图像都会有一些像素值发生随机变化。原始代码类似如下
1 | cv::Mat imgbuf2mat(JNIEnv *env, jbyteArray buf, int width, int height){ |
简单来说就是先把图像内容转成cv::Mat，然后对图像做一些处理，但是即使传入相同的一张图像，处理结果每次都不一样。后来发现其实应该先处理图像，再释放引用，修改后的代码类似如下
1 | cv::Mat imgbuf2mat(JNIEnv *env, jbyte *ptr, int width, int height){ |
The statistics are from Tim Dettmers’ blog Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning.
最近尝试使用 Caffe 复现 MTCNN，感觉坑很大，记录一下训练过程，目前还没有好的结果。网上也有很多童鞋在尝试训练 MTCNN，普遍反映使用 TensorFlow 可以得到比较好的结果，但是使用 Caffe 不是很乐观。
2018.09.11：目前只训练了 12net，召回率偏低。
以blankWorld/MTCNN-Accelerate-Onet为 baseline，blankWorld 在 FDDB 上的测试性能如下图
这个效果很不错，但是我自己生成样本后训练 12net，召回率有明显下降。性能对比如下图
暂且不管 12net 的测试结果为什么会这么差，两个模型的性能差距是可以反映的。
2018.09.11
训练数据生成
参考AITTSMD/MTCNN-Tensorflow提供的 prepare_data 进行数据生成。数据集情况如下表
| Positive | Negative | Part | Landmark | |
|---|---|---|---|---|
| Training Set | 156728/189530 | 470184/975229 | 156728/547211 | 313456/357604 |
| Validation Set | 10000 | 10000 | 10000 | 10000 |
其中 Pos:Neg:Part:Landmark = 1:3:1:2，样本比例参考原作的比例。Pos、Neg、Part 来自于WiderFace，Landmark 来自于CelebA。其中正样本进行了人工的数据筛选，筛选的原因是根据 WiderFace 生成的正样本，有很多都是质量很差的图像，包含人脸大面积遮挡或十分模糊的情况。之前召回率很差的性能来自没有经过筛选的训练集，因为使用了 OHEM，只有 loss 值在前 70%的样本才参与梯度计算，感觉如果质量差的样本占比较大，网络学习到的特征是错误的，那些质量好的图像可能得不到充分的学习。
训练参数设置
初始训练参数如下
1 | type:"Adam" |
第一轮训练在 75000 次迭代(17.5 个 epoch)时停止，测试记录如下
1 | I0911 10:16:25.019253 21722 solver.cpp:347] Iteration 75000, Testing net (#0) |
注意：分类测试结果是 0.4638 是因为测试集没有打乱，1-10000 为 pos 样本，10001-20000 为 neg 样本，20001-30000 为 part 样本，30001-40000 为 landmark 样本。因此，实际分类正确率应该是 0.9276
降低学习率至 0.001，训练 135000 次迭代(31.5 个 epoch)时停止，测试记录如下
1 | I0911 13:14:36.482010 23543 solver.cpp:347] Iteration 135000, Testing net (#0) |
实际分类正确率是 0.9428。训练 260000 次迭代后停止，测试记录如下
1 | I0911 16:58:47.514267 28442 solver.cpp:347] Iteration 260000, Testing net (#0) |
实际分类正确率是 0.943752。
问题： 训练结果看似还可以，但是召回率很低，在阈值设置为 0.3 的情况下，召回率也才将将达到 90%。阈值要设置到 0.05，才能达到 97%-98%的召回率，ROC 曲线如下图。严格来说这个测试并不严谨，应该用检测器直接在图像中进行检测，但是为了方便，我直接用 val 集上的性能画出了 ROC 曲线，其中的 FDDB 曲线是将的人脸区域截取出来进行测试得到的。
2018.09.14
使用上述 12net 在 WiderFace 上提取正负样本，提取结果如下：
| Thresholed | Positive | Negative | Part |
|---|---|---|---|
| 0.05 | 85210 | 36745286 | 632861 |
| 0.5 | 66224 | 6299420 | 354350 |
2018.09.17
准备 24net 的训练样本。由于生成 12net 检测到的正样本数目有限，训练 24net 的 pos 样本包含两部分，一部分是训练 12net 的正样本，一部分是经过筛选的 12net 检测到的正样本；neg 样本和 part 样本全部来自 12net 的难例；landmark 与 12net 共用样本。经过采样后达到样本比例 1:3:1:2，样本数目如下表：
| Positive | Negative | Part | Landmark | |
|---|---|---|---|---|
| Training Set | 225172 | 675516 | 225172 | 313456 |
| Validation Set | 10000 | 10000 | 10000 | 10000 |
训练过程与 12net 类似，学习率从 0.01 下降到 0.0001，最终的训练结果如下
1 | I0917 15:19:00.631140 36330 solver.cpp:347] Iteration 70000, Testing net (#0) |
实际分类正确率是 0.963。ROC 曲线如下图，同样使用 val 集上的性能画出曲线。
2018.09.18
使用 24net 在 WiderFace 上提取正负样本，提取结果如下：
| Thresholed | Positive | Negative | Part |
|---|---|---|---|
| 0.5, 0.5 | 86396 | 83212 | 225285 |
利用以上数据生成 48net 的训练样本，由于 24net 生成的样本数量有限，结合前两次训练所用的数据，生成训练集：
| Positive | Negative | Part | Landmark | |
|---|---|---|---|---|
| Training Set | 283616 | 850848 | 283616 | 567232 |
| Validation Set | 10000 | 10000 | 10000 | 10000 |
2018.09.19
在训练 48net 的过程中，首先尝试了 Adam 算法进行优化，后来发现训练十分不稳定。转而使用 SGD 进行优化，效果好转。训练初始参数如下：
1 | type:"SGD" |
48net 的训练结果比较一般，性能如下:
1 | I0919 18:02:22.318362 3822 solver.cpp:347] Iteration 165000, Testing net (#0) |
实际的分类精度为 0.9324。整体来看基本实现了文章中参考文献[19]在验证集上的性能，性能对比如下表
| CNN | 12-net | 24-net | 48-net |
|---|---|---|---|
| [19] | 94.4% | 95.1% | 93.2% |
| MTCNN | 94.6% | 95.4% | 95.4% |
| Ours | 94.3% | 96.3% | 93.2% |
2018.09.20
整个系统连通后进行测试，发现人脸框抖动比较厉害，这应该是训练过程和样本带来的问题。
比较奇怪的问题是在 Caffe 上进行 CPU 运算时，速度极慢，尤其 12net 运行速度慢 30 倍左右。通过观察参数分布发现，有大量 kernel 都是全零分布，初步感觉是因为 Adam 和 ignore label 相互作用的结果，即 ignore label 的样本会产生 0 值 loss，这些 loss 会影响 Adam 的优化过程，具体原因还需进一步理论推导。目前的解决方案是将含有大量 0 值 kernel 的层随机初始化，使用 SGD 进行训练。至于抖动的问题，需要进一步分析。重训后的模型性能如下表：
| 12-net | 24-net | 48-net | |
|---|---|---|---|
| Accuracy | 94.59% | 96.52% | 93.94% |
2018.09.26
目前来看回归器的训练是完全失败。失败的原因可能有以下几点：
先训练一个不带 ignore label 的回归器，看 loss 是否发生变化。
2018.09.27
做了一个实验，用 20 张图像生成一个数据集，经过训练，网络是可以完全过拟合这个数据集的，说明数据和代码是没有问题的，但是加大数据量后，bounding box 的回归依旧不好。通过跟大神讨论，发现自己犯了一个很弱智的低级错误，回归问题是不可以做镜像的，回归问题是不可以做镜像的，回归问题是不可以做镜像的，重要的事情说三遍，如果图像做镜像操作，那么标注也需要进行镜像操作。
重训后的模型性能如下表：
| 12-net | 24-net | 48-net | |
|---|---|---|---|
| Accuracy | 94.35% | 97.45% | 94.67% |
2018.10.01
实验还是不够顺利，有些受挫感。目前来看 12net 和 24net 的训练是相对顺利的，至少能够去除大量虚检并给出相对准确的回归框。48net 的训练比较失败，存在虚检、定位不准以及关键点定位不准的问题。
在训练 12net 和 24net 时，因为网络较小，而且这两个网络都不负责输出关键点，所以训练时只训练了分类和框回归问题。训练 48net 时三个 loss 都很重要，感觉按照原文的 loss weight 进行配置会造成回归问题学习不充分的问题，需要提高回归问题的 loss weight。
接下来的工作包括：
2018.10.13
最近一次训练的记录如下：
12net：
12net 的样本由随机采样得来。
| Positive | Negative | Part | |
|---|---|---|---|
| Training Set | 156728 | 470184 | 156728 |
| Validation Set | 10000 | 10000 | 10000 |
验证集上的性能为：
1 | Test net output #0: cls_Acc = 0.9435 |
24net
24net 的样本全部来自 12net 的检测结果。
| Positive | Negative | Part | |
|---|---|---|---|
| Training Set | 60149 | 180447 | 120298 |
| Validation Set | 1500 | 1500 | 1500 |
验证集上的性能为：
1 | Test net output #0: cls_Acc = 0.977588 |
48net
48net 的正样本和 part 样本来自于 24net 在 widerface 上的检测结果，负样本来自于 24net 在 widerface 和 celeba 上的检测结果，landmark 样本来自于 24net 在 celeba 上的检测结果。
| Positive | Negative | Part | landmark | |
|---|---|---|---|---|
| Training Set | 242862 | 728586 | 242862 | 485724 |
| Validation Set | 5000 | 5000 | 5000 | 5000 |
在验证集上的性能为：
1 | Test net output #0: cls_Acc = 0.978155 |
使用 0.5,0.5,0.5 作为阈值，在 FDDB 上测得的 discROC 曲线如下图
TITLE: Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks
AUTHOR: Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li
ASSOCIATION: Chinese Academy of Sciences, The Chinese University of Hong Kong
FROM: IEEE Signal Processing Letters
This work is very much like a traditional slide-window based face detection method. A cascade classifier is used to classify an image patch. The first classifier is very simple so that it can efficiently remove easy negative patches. Then more complex classifiers are used to remove harder patches. The last CNN is used as a regressor to localize face landmarks. Along with the classifiers, bounding box is also localized. To handle multiple scales, an image pyramid is built.
The pipeline of this work is shown as follows. The input image is first resized to different scales to build an image pyramid, which is sent to the three-stage cascaded framework.
Stage 1: A fully convolutional network is proposed, called Proposal Network (P-Net), to obtain the candidate facial windows and their bounding box regression vectors. Then candidates are calibrated based on the estimated bounding box regression vectors. After that, non-maximum suppression (NMS) is utilized to merge highly overlapped candidates.
Stage 2: All candidates are fed to another CNN, called Refine Network (R-Net), which further rejects numerous false candidates, performs calibration with bounding box regression, and conducts NMS.
Stage 3: This stage is similar to the second stage, but this stage aims to identify face regions with more supervision. In particular, the network will output five facial landmarks’ positions.
The architecture of the three networks are shown in the following figure.
At training stage, the face classification is trained as typical two-class classification task using cross-entropy loss. In each mini-batch, the samples are sorted by their losses and only the top 70% of them are selected to compute gradients. On the other hand, Bounding box regression and facial landmark localization are trained using Euclidean loss.
Face detection performance:
Face landmark regression performance:
Jupyter Notebook 有两种键盘输入模式。编辑模式，允许你往单元中键入代码或文本；这时的单元框线是绿色的。命令模式，键盘输入运行程序命令；这时的单元框线是灰色。
Every coder will feel something seeing this :)
TITLE: DetNet: A Backbone network for Object Detection
AUTHOR: Xuepeng Shi, Shiguang Shan, Meina Kan, Shuzhe Wu, Xilin Chen
ASSOCIATION: Tsinghua University, Face++
FROM: arXiv:1804.06215
There are two problems using the classification backbone for object detection tasks. (i) Recent detectors, e.g., FPN, involve extra stages compared with the backbone network for ImageNet classification in order to detect objects with various sizes. (ii) Traditional backbone produces higher receptive field based on large downsampling factor, which is beneficial to the visual classification. However, the spatial resolution is compromised which will fail to accurately localize the large objects and recognize the small objects.
To sumarize, there are 3 main problems to use current pre-trained models, including
To address these problems, DetNet has following characteristics. (i) The number of stages is directly designed for Object Detection. (ii) Even though more stages are involved, high spatial resolution of the feature maps is mainted, while keeping large receptive field using dilated convolution.
The main architecture of DetNet is designed based on ResNet-50. The first 4 stages are kept same with ResNet-50. The main differences are illustrated as follows:
The following figure shows the dialted bottleneck with $1 \times 1$ conv projection and the architecture of DetNet.
TITLE: Real-Time Rotation-Invariant Face Detection with Progressive Calibration Networks
AUTHOR: Xuepeng Shi, Shiguang Shan, Meina Kan, Shuzhe Wu, Xilin Chen
ASSOCIATION: Chinese Academy of Sciences
FROM: arXiv:1804.06039
Given an image, all face candidates are obtained according to the sliding window and image pyramid principle, and each candidate window goes through the detector stage by stage. In each stage of PCN, the detector simultaneously rejects most candidates with low face confidences, regresses the bounding boxes of remaining face candidates, and calibrates the RIP orientations of the face candidates. After each stage, non-maximum suppression (NMS) is used to merge those highly overlapped candidates.
PCN progressively calibrates the RIP orientation of each face candidate to upright for better distinguishing faces from non-faces.
The following figure illustrates the framework.
For each input window $x$, PCN-1 has three objectives: face or non-face classification, bounding box regression, and calibration, formulated as follows:
$$[f, t, g] = F_{1}(x)$$
where $F_{1}$ is the detector in the first stage structured with a small CNN. The $f$ is face confidence score, $t$ is a vector representing the prediction of bounding box regression, and $g$ is orientation score. Overall, the objective for PCN-1 in the first stage is defined as:
$$\min L = L_{cls} +\lambda_{reg} \cdot L_{reg} + \lambda_{cal} \cdot L_{cal}$$
where $\lambda_{reg}$, $\lambda_{cal}$ are parameters to balance different loss. The first objective, which is also the primary objective, aims for distinguishing faces from non-faces. The second objective attempts to regress the fine bounding box. The third objective aims to predict the coarse orientation of the face candidate in a binary classification manner, telling the candidate is facing up or facing down.
The PCN-1 can be used to filter all windows to get a small number of face candidates. For the remaining face candidates, firstly they are updated to the new regressed bounding boxes. Then the updated face candidates are rotated according to the predicted coarse RIP angles.
Similar as the PCN-1 in the first stage, the PCN-2 in the second stage further distinguishes the faces from non-faces more accurately, regresses the bounding boxes, and calibrates face candidates. Differently, the coarse orientation prediction in this stage is a ternary classification of the RIP angle range, telling the candidate is facing left, right or front.
After the second stage, all the face candidates are calibrated to an upright quarter of RIP angle range, i.e. [$-45^{\circ}$,$45^{\circ}$]. Therefore, the PCN-3 in the third stage can easily and accurately determine whether it is a face and regress the bounding box. Since the RIP angle has been reduced to a small range in previous stages, PCN-3 attempts to directly regress the precise RIP angles of face candidates instead of coarse orientations.
The early stages only predict coarse RIP ori- entations, which is robust to the large diversity and further benefits the prediction of successive stages.
The calibration based on the coarse RIP prediction can be efficiently achieved via flipping original image three times, which brings almost no additional time cost. Rotating the original image by $-90^{\circ}$, $90^{\circ}$ and $180^{\circ}$ to get image-left, image-right, and image-down. And the windows with $0^{\circ}$,$-90^{\circ}$, $90^{\circ}$ and $180^{\circ}$ can be cropped from original image, image-left, image-right, and image-down respectively, as the following figure shows.
TITLE: Pelee: A Real-Time Object Detection System on Mobile Devicesn
AUTHOR: Robert J. Wang, Xiang Li, Shuang Ao, Charles X. Ling
ASSOCIATION: University ofWestern Ontario
FROM: arXiv:1804.06882
Two-Way Dense Layer. A 2-way dense layer is used to get different scales of receptive fields. One branch uses a small kernel size (3x3) to capture small-size objects. The other branch stacks two 3x3 convolution layers for larger objects. The structure is shown in the following figure.
Stem Block. This block is placed before the first dense layer for the sake of cost efficiency. This stem block can effectively improve the feature expression ability without adding computational cost too much. The structure is shown as follows.
Dynamic Number of Channels in Bottleneck Layer. The number of channels in the bottleneck layer varies according to the input shape to make sure the number of output channels does not exceed the number of its input channels.
Transition Layer without Compression. experiments show that the compression factor proposed by DenseNet hurts the feature expression so that the number of output channels is kept the same as the number of input channels in transition layers.
Composite Function. The post-activation (Convolution - Batch Normalization - Relu) is used for speed acceleration. For post-activation, all batch normalization layers can be merged with convolution layer at the inference stage. To compensate for the negative impact on accuracy caused by this change, a shallow and wide network structure is designed. In addition, a 1x1 convolution layer is added to the last dense block to get a stronger representational ability.
The framework of the work is illustrated in the following table.
Feature Map Selection. 5 scale feature maps (19x19, 10x10, 5x5, 3x3, and 1x1) are selected. Larger resolution features are discarded for speed acceleration.
Residual Prediction Block. For each feature map used for detection, a residual block (ResBlock) is constructed before conducting prediction, shown in the following figure.
The classification performance on ILSVRC2012 is shown in the following table.
The detection performance on VOC2007 is shown in the following table.
The detection performance on COCO2015 is shown in the following table.
From my own experince, DW convolution is not pruning friendly so that recently pruning methods, such as ThiNet and Net-Trim, works poorly on DW convolution. This work uses conventional convolutional layers, so maybe those pruning methods can play a role.
最近在给封装一个动态库，需要支持古老的windows xp系统。而我的开发系统是windows 10，使用visual studio 2013作为IDE。
一通谷歌百度之后，我采用了曝光度最高的方法。具体来说包括两个步骤：
最初并没有发现这样做有什么问题，后来写了一个接口函数，release模式下没有发现问题，但是debug模式下调用该接口的函数在出栈时一直崩溃，错误如下
因为对这一块儿实在不熟悉，就抱着死马当活马医的态度，把所有MT/MTd都改成了MD/MDd，又把所有依赖库和自己的库编译了一遍。在目标测试机上安装，居然成功了。
在网上搜了一些解释：
Debug Assertion Failed! Expression: __acrt_first_block == header
As this is a DLL, the problem might lie in different heaps used for allocation and deallocation (try to build the library statically and check if that will work).
The problem is, that DLLs and templates do not agree together very well. In general, depending on the linkage of the MSVC runtime, it might be problem if the memory is allocated in the executable and deallocated in the DLL and vice versa (because they might have different heaps). And that can happen with templates very easily, for example: you push_back() to the vector inside the removeWhiteSpaces() in the DLL, so the vector memory is allocated inside the DLL. Then you use the output vector in the executable and once it gets out of scope, it is deallocated, but inside the executable whose heap doesn’t know anything about the heap it has been allocated from. Bang, you’re dead.
This can be worked-around if both DLL and the executable use the same heap. To ensure this, both the DLL and the executable must use the dynamic MSVC runtime - so make sure, that both link to the runtime dynamically, not statically. In particular, the exe should be compiled and linked with /MD[d] and the library with /LD[d] or /MD[d] as well, neither one with /MT[d]. Note that afterwards the computer which will be running the app will need the MSVC runtime library to run (for example, by installing “Visual C++ Redistributable” for the particular MSVC version).
You could get that work even with /MT, but that is more difficult - you would need to provide some interface which will allow the objects allocated in the DLL to be deallocated there as well. For example something like:
2
3
4
5
6
>void deallocVector(std::vector<std::string> &x) {
std::vector<std::string> tmp;
v.swap(tmp);
>}(however this does not work very well in all cases, as this needs to be called explicitly so it will not be called e.g. in case of exception - to solve this properly, you would need to provide some interface from the DLL, which will cover the vector under the hood and will take care about the proper RAII)
EDIT: the final solution was actually was to have all of the projects (the exe, dll and the entire googleTest project) built in Multi-threaded Debug DLL (/MDd) (the GoogleTest projects are built in Multi-threaded debug(/MTd) by default)
说实话，对计算机原理的理解十分欠缺，遇到稍微专业一些的问题只能照着网上的一些方法试一试，如果成了也就不会再深入研究了，如果不成也不知道为什么不成，只能再去试别的方法。:-(