0%

TITLE: Understanding intermediate layers using linear classifier probes

AUTHOR: Guillaume Alain, Yoshua Bengio

ASSOCIATION: Université de Montréal

FROM: arXiv:1610.01644

CONTRIBUTIONS

The concept of the linear classifier probe (probe) is introduced to understand the roles of the intermediate layers of a neural network, to measure how much information is gained at every layer (answer : technically, none). This powerful concept can be very useful to understand the dynamics involved in a deep neural network during training and after.

Linear Classifier Probes

Probes

The probes are implemented in a very simple manner, that using a fully-connected layer and a softmax as a linear classifier. The classifier’s error takes NO part in the back-propagation process and is only used to measure the features’ ability of solving classification problems, which are extracted from different layers of different depth in the network.

Probes on untrained model

Given an untrained network, the probes are set to see whether each layer would give useful features for a classification task. The data is generated from a Gaussian distribution, a very easy task.

The probe to layer 0 corresponding to the raw data are able to classify perfectly. And the performance degrades when applying random transformations brought by the intermediate layers. The phenomenon indicates that at the beginning on training, the usefulness of layers decays as we go deeper, reaching the point where the deeper layers are utterly useless. The authors give a very strong claim: garbage forwardprop, garbage backprop

Auxiliary loss branches and skip connections

From the experiment in the paper, it seems that Auxiliary loss branches help make the untrainable model trainable.

From another experiment, if we added a bridge (a skip connection) between layer 0 and layer 64, the model completely ignores layers 1-63. The following figure illustrates the phenomenon.

Some Ideas

  1. the probes can be used to visualize the role of each layers.
  2. ResNet is really necessary? Why it works if the skip will ignore the layers it covers.
  3. Training stage by stage could be very useful when a very deep network is use.

有时候不要做无谓的挣扎,尤其是在总部跟你要东西的时候。

我的成就感来自于完成一项工作,所以我盼望着新的工作。但是,如果还没有来得及体会成就感就有新工作,多半我会讨厌这个新工作,然后充满斗志,就为了再次获得成就感。

供暖之前,又下雨,冷。窝在床上读东野的《我杀了他》,广播里放着北京交通广播的《有我陪着你》

没什么新鲜事,应该督促自己读paper了。

一切事情都不能太自以为是,不然是要被打脸的。总部同事以前就怀疑过SVM预测会占用大量CPU资源,但是我们却觉得2000*100的矩阵向量乘应该是轻量级运算,但是经过排查果然还是这个计算出了问题。

在make之前一定要make clean。各种调试代码,一个小时以后,最后却是通过make clean解决了,一切代码又恢复原状,无用功啊!

记住,玛雅:我们在二十岁有共鸣的东西到了四十岁的时候不一定能产生共鸣,反之亦然。书本如此,生活亦如此。——《岛上书店》

以前就觉得自己慢热,但是直到最近一段时间才发觉自己是慢热到极致,慢热到连成熟度都比实际年龄拖后好几年,每次测心理年龄都比实际低那么三四岁。想想这可能真是个事实,比如读《哈利波特》,本来是一部在初中特别流行的小说,但是那个时候我却更喜欢《指环王》,给自己找的一个偏爱的理由是《指环王》比《哈利波特》具有更宏大的背景,但其实现在回想起来,应该还不能理解中土世界的宏大和那个世界的来龙去脉,真正的理由应该是“好人打坏人”。《哈利波特》虽然也是“好人打坏人”,但是其中混杂着角色的成长、复杂的人格和人物之间的矛盾,读起来总不如一场酣畅淋漓的杀敌竞赛来的痛快。后来上了大学,才突然对《哈利波特》的故事着迷起来,可能直到那时才与书中的角色发生了共鸣,才能将自己带入到角色中去。相应的,对于《指环王》的喜爱也转换为开始探究中土世界的来源。除了读书,谈恋爱也一样,好像现在还处于小孩谈恋爱的状态,甚至会羡慕一些综艺节目里的桥段。

即便慢热,到现在却也开始觉得年龄大了,时不时会有一种焦躁,仔细想想估计就是怕变老吧,想永远年轻有活力,可以有无限的时间。想做的事情太多,比如做饭、画画、练字、看文献、读书、做有意思的小项目、制作模型、出去旅行……有些时候是贪多嚼不烂,这么多事情不可能都做得到。有些时候总想等以后再做,比如等有了自己的房子再开始干嘛干嘛,租着房子以后搬家好麻烦。有些时候就是因为穷,机票好贵。真羡慕《指环王》里的精灵,长生不老,厌倦了一件事之后,可以拾起另一件事。估计也不会有“当其欣于所遇,暂得于己,快然自足,曾不知老之将至”的痛苦。或者好像《阳光姐妹淘》里的sunny girls,虽然大家一度“向之所欣,俯仰之间,已为陈迹”,但是依然能够有一次最后的疯狂,只可惜我希望自己可以一直疯狂。大二大三的时候曾经有过“要是我一直都是二十一二岁该多好”的想法,估计是满足于无忧无虑的生活,而现在可是结结实实地还念了。

如果现在还是对十几岁的东西感兴趣,好像总得提醒一下自己那样是不是太幼稚了,应该学着成熟。孙燕姿《年轻无极限》里“二十五岁的我,学着大人应该有的动作”,以前觉得二十五岁还年轻,现在早过了二十五,却还是没有什么大人的动作。真是矛盾,既想要成熟的思想,又想有年轻的心态。

Recently I hang much with Ubuntu because I am trying some interesting computer vision projects which are developed in Linux environment. In addition Steam supports to play Sid Meier’s Civilization V using its SteamOS platform, which means that I do not have to log into Windows to play my favourite computer game. In the purpose of using Ubuntu well, I have learn much from some websites that I have listed following.

  1. The Linux Command Line

    Designed for the new command line user, this 540-page volume covers the same material as LinuxCommand.org but in much greater detail. In addition to the basics of command line use and shell scripting, The Linux Command Line includes chapters on many common programs used on the command line, as well as more advanced topics.

    A very useful Chinese version can be found at 快乐的 Linux 命令行.

  2. LinuxTOY

    This is a websites in Chinese that provide visitors with Linux OS related information. The website was started back in 2006. There are many useful and interesting posts, including news, softwares, games and tutorials.

  3. Ubuntu Official Website

    Ubuntu is an open source project that develops and maintains a cross-platform, open-source operating system based on Debian. It includes Unity, a consistent user interface for the smartphone, the tablet and the PC. Upgrades are released every six months and support is guaranteed by Canonical for up to five years. Canonical also provides commercial support for Ubuntu deployments across the desktop, the server and the cloud.

  4. Pro Git

    Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Though Git is not only used on Linux OS but also on Windows/Mac, it really much more convenient and ecological on Linux.

  5. A Great Vim Cheat Sheet

    I’ve compiled a list of essential vim commands that I use every day. I then give a few instructions on how to making vim as great as it should be, because it’s painful without configuration.

  6. climate

    The ultimate command line tool for Linux! climate provides a huge number of command line options for developers to automate their Linux system.

  7. Mastering Bash and Terminal

    If there is one tool that every developer uses regardless of language, platform, or framework it’s the terminal. If we are not compiling code, executing git commands, or scp-ing ssl certificates to some remote server, we are finding a new version of cowsay to entertain ourselves while we wait on one of the former. As much as we use the terminal it is important that we are efficient with it. Here are some ways I make my time in the terminal efficient and effective.

以前也尝试使用ubuntu,但是苦于连个像样的拼音输入法都没有,用了一段时间之后觉得太不友好了。现在的开源社区真是繁荣了,好多软件都有linux版本了。

一些必备工具

  1. 搜狗拼音输入法

    搜狗拼音输入法是最好用的拼音输入法之一,现在不只支持传统的Windows和Android系统,还支持Linux和Mac系统,甚至IOS都已经可以使用了,可见其在拼音输入法领域内的地位。

  2. 有道词典

    有道词典也支持几乎所有的流行平台,是看paper和科学上网的必备利器。

  3. 微信客户端

    微信客户端用的是Electronic WeChat是利用Electron开源框架开发的一款第三方微信客户端,支持Linux和MacOS X系统。

  4. 网易云音乐

    网易云音乐听音乐的利器,也支持各大操作系统。

  5. Steam

    以往linux系统的娱乐性很差,现在有了Steam,各种大型游戏也不在话下了。

  6. QQ

    腾讯对linux系统的支持比较差,上面提到的微信客户端也是爱好者自己利用网页版微信开发的。我在linux用QQ的解决方法来自这里,网上也有很多类似的教程,退而求其次地用wine qq吧。

Gstreamer

The official website of Gstreamer is here. What is Gstreamer is quoted from its website:

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.

Applications can take advantage of advances in codec and filter technology transparently. Developers can add new codecs and filters by writing a simple plugin with a clean, generic interface.

GStreamer is released under the LGPL. The 1.x series is API and ABI stable and supersedes the previous stable 0.10 series. Both can be installed in parallel.

A Simple Tutorial

I started to learn to use Gstreamer just two days ago, and I found a very useful tutorial called Gstreamer Small Tutorial authored by Arash Shafiei. One could use this tutorial as a stepping-stone to develop more complex applications. A more detailed tutorial can be found in Gstreamer’s website.

My Own Trial

Though Arash Shafiei provided an excellent sample, there is some modifications that need to be done to run the application to play video correctly.

  1. more recent API should be used by replacing gst_pad_get_caps with gst_pad_query_caps.
  2. there’s a mistake in Arash Shafiei’s sample code of static void pad_added_handler(GstElement *src, GstPad *new_pad, CustomData *data). The original code will not link video sink if the audio sink has already linked no matter whether video sink is linked or not.
  3. a demuxer is added to the pipeline to handle the input of a file source element.

my own code can be found following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
#include <gst/gst.h>
#include <glib.h>
/* Structure to contain all our information, so we can pass it to callbacks */
typedef struct _CustomData
{
GstElement *pipeline;
GstElement *source;
GstElement *demuxer;
GstElement *video_convert;
GstElement *audio_convert;
GstElement *video_sink;
GstElement *audio_sink;
} CustomData;
/* Handler for the pad-added signal */
/* This function will be called by the pad-added signal */
static void pad_added_handler(GstElement *src, GstPad *new_pad, CustomData *data)
{
GstPad *sink_pad_audio = gst_element_get_static_pad(data->audio_convert, "sink");
GstPad *sink_pad_video = gst_element_get_static_pad(data->video_convert, "sink");
GstPadLinkReturn ret;
GstCaps *new_pad_caps = NULL;
GstStructure *new_pad_struct = NULL;
const gchar *new_pad_type = NULL;
g_print("Received new pad '%s' from '%s':\n", GST_PAD_NAME(new_pad), GST_ELEMENT_NAME(src));

/* Check the new pad's type */
new_pad_caps = gst_pad_query_caps(new_pad, 0);
new_pad_struct = gst_caps_get_structure(new_pad_caps, 0);
new_pad_type = gst_structure_get_name(new_pad_struct);
if (g_str_has_prefix(new_pad_type, "audio/x-raw"))
{
/* If our audio converter is already linked, we have nothing to do here */
if (gst_pad_is_linked(sink_pad_audio))
{
g_print(" Type is '%s'.\n", new_pad_type);
g_print(" We are already linked. Ignoring.\n");
goto exit;
}
/* Attempt the link */
ret = gst_pad_link(new_pad, sink_pad_audio);
if (GST_PAD_LINK_FAILED(ret))
{
g_print(" Type is '%s' but link failed.\n", new_pad_type);
}
else
{
g_print(" Link succeeded (type '%s').\n", new_pad_type);
}
}
else if (g_str_has_prefix(new_pad_type, "video/x-raw"))
{
/* If our video converter is already linked, we have nothing to do here */
if (gst_pad_is_linked(sink_pad_video))
{
g_print(" Type is '%s'.\n", new_pad_type);
g_print(" We are already linked. Ignoring.\n");
goto exit;
}
/* Attempt the link */
ret = gst_pad_link(new_pad, sink_pad_video);
if (GST_PAD_LINK_FAILED(ret))
{
g_print(" Type is '%s' but link failed.\n", new_pad_type);
}
else
{
g_print(" Link succeeded (type '%s').\n", new_pad_type);
}
}
else
{
g_print(" It has type '%s' which is not raw audio. Ignoring.\n", new_pad_type);
goto exit;
}
exit:
/* Unreference the new pad's caps, if we got them */
if (new_pad_caps != NULL)
gst_caps_unref(new_pad_caps);
/* Unreference the sink pad */
gst_object_unref(sink_pad_audio);
gst_object_unref(sink_pad_video);
}

int main(int argc, char *argv[])
{

if(argc != 2)
{
g_printerr("usage: ./player <path_to_a_video>\n");
return 0;
}

CustomData data;
GstBus *bus;
GstMessage *msg;
GstStateChangeReturn ret;
gboolean terminate = FALSE;
/* Initialize GStreamer */
gst_init(&argc, &argv);
/* Create the elements */
data.source = gst_element_factory_make("filesrc", "source");
data.demuxer = gst_element_factory_make("decodebin", "demuxer");
data.audio_convert = gst_element_factory_make("audioconvert", "audio_convert");
data.audio_sink = gst_element_factory_make("autoaudiosink", "audio_sink");
data.video_convert = gst_element_factory_make("videoconvert", "video_convert");
data.video_sink = gst_element_factory_make("autovideosink", "video_sink");
/* Create the empty pipeline */
data.pipeline = gst_pipeline_new("test-pipeline");
if (!data.pipeline || !data.source || !data.audio_convert ||
!data.audio_sink || !data.video_convert || !data.video_sink)
{
g_printerr("Not all elements could be created.\n");
return -1;
}
/* Build the pipeline. Note that we are NOT linking the source at this point. We will do it later. */
gst_bin_add_many(GST_BIN(data.pipeline), data.source, data.demuxer,
data.audio_convert, data.audio_sink, data.video_convert, data.video_sink, NULL);
if (!gst_element_link(data.source, data.demuxer))
{
g_printerr("Elements could not be linked.\n");
gst_object_unref(data.pipeline);
return -1;
}
if (!gst_element_link(data.audio_convert, data.audio_sink))
{
g_printerr("Elements could not be linked.\n");
gst_object_unref(data.pipeline);
return -1;
}
if (!gst_element_link(data.video_convert, data.video_sink))
{
g_printerr("Elements could not be linked.\n");
gst_object_unref(data.pipeline);
return -1;
}
/* Set the file to play */
g_object_set(data.source, "location", argv[1], NULL);
/* Connect to the pad-added signal */
g_signal_connect(data.demuxer, "pad-added", G_CALLBACK(pad_added_handler), &data);
/* Start playing */
ret = gst_element_set_state(data.pipeline, GST_STATE_PLAYING);
if (ret == GST_STATE_CHANGE_FAILURE)
{
g_printerr("Unable to set the pipeline to the playing state.\n");
gst_object_unref(data.pipeline);
return -1;
}
/* Listen to the bus */
bus = gst_element_get_bus(data.pipeline);
do
{
msg = gst_bus_timed_pop_filtered(bus, GST_CLOCK_TIME_NONE,
(GstMessageType)(GST_MESSAGE_STATE_CHANGED | GST_MESSAGE_ERROR | GST_MESSAGE_EOS));
/* Parse message */
if (msg != NULL)
{
GError *err;
gchar *debug_info;
switch (GST_MESSAGE_TYPE(msg))
{
case GST_MESSAGE_ERROR:
gst_message_parse_error(msg, &err, &debug_info);
g_printerr("Error received from element %s: %s\n", GST_OBJECT_NAME(msg->src), err->message);
g_printerr("Debugging information: %s\n", debug_info ? debug_info : "none");
g_clear_error(&err);
g_free(debug_info);
terminate = TRUE;
break;
case GST_MESSAGE_EOS:
g_print("End-Of-Stream reached.\n");
terminate = TRUE;
break;
case GST_MESSAGE_STATE_CHANGED:
/* We are only interested in state-changed messages from the pipeline */
if (GST_MESSAGE_SRC(msg) == GST_OBJECT(data.pipeline))
{
GstState old_state, new_state, pending_state;
gst_message_parse_state_changed(msg, &old_state, &new_state, &pending_state);
g_print("Pipeline state changed from %s to %s:\n",
gst_element_state_get_name(old_state), gst_element_state_get_name(new_state));
}
break;
default:
/* We should not reach here */
g_printerr("Unexpected message received.\n");
break;
}
gst_message_unref(msg);
}
} while (!terminate);
/* Free resources */
gst_object_unref(bus);
gst_element_set_state(data.pipeline, GST_STATE_NULL);
gst_object_unref(data.pipeline);
return 0;
}

the compiling commond is

1
gcc player.c -o player `pkg-config --cflags --libs gstreamer-1.0`

微信很早就发布了一款在Windows下的PC客户端,在使用PC的时候可以方便与亲友聊天,而不需要频频举起手机。最近一段时间我总是在Ubuntu环境下使用电脑,很长一段时间以来只能用网页版的微信,今天发现了一个好东西——Electronic WeChat。

Electronic WeChat是利用Electron开源框架开发的一款第三方微信客户端,支持Linux和MacOS X系统。Electronic WeChat具有一些不错的特性,包括拖入图片、文件即可发送,显示贴纸消息,以及直接打开重定向的链接等等。

要在Linux下安装ElectronicWeChat,可以到这里选择适合自己平台的版本,例如我选择的是linux-x64.tar.gz版本,执行:tar zxvf linux-x64.tar.gz后直接运行electronic-wechat,然后使用手机扫描二维码即可登录。

First Post on 20161006

今天搞了一天Gstreamer,早晨九点起来,就开始弄,不知不觉就到了午饭时间,吃了饭回来继续,不知不觉又到了晚饭时间。第一次搞这种编解码的库,真是头大,搞了这么整整一天,还是没什么结果。只是大概觉得gstreamer的工作原理就是把一堆element整合到一个pipeline里,然后一个多媒体文件就通过这个pipline把音频、视频解码出来,并送到对应的输出设备里。

说起来好像不太难,而且用gst命令也可以播放视频,但是如何把它们通过C代码融合到自己的程序里呢,同时又怎么将视频全屏显示也是个问题,网上很多例子都是用GTK,这个又是个新东西。在github上搜了很多,但是也没有发现一个简单易用的例子,真是挠头。看来还是得耐下心来看文档里。

Update on 20161007

发现了一个很好的Gstreamer教程——《Gstreamer Small Tutorial》只有十页,读起来很顺畅,作者把Gstreamer的工作原理阐述得十分清晰,而且其中附有一个实例,对于理解有很大帮助。

Today I accidentally upgraded my Ubuntu 14.04 to Ubuntu 16.04. Then evidently Caffe can not be built. Several modifications are of need to bypass the issues.

  1. hack the Cuda source code. Suppress the error of unsupported GNU version! gcc versions later than 4.9 are not supported! by replacing #if __GNUC__ > 5 || (__GNUC__ == 5 && __GNUC_MINOR__ > 9) with #if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 9).
  2. config the Makefile of Caffe. Replace NVCCFLAGS += -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS) with NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

Other steps of using Caffe can be found here and here.