Modulating early visual processing by language

VQA

• More and more evidence accumulates that words set visual priors which alter how visual information is processed from the very beginning
• it is observed that P1 signals, which are related to low-level visual features, are modulated while hearing
  specific words

率先将CBN引入到VQA的问题当中，通过MLP对输入的question进行embedding，然后再基于得到的embedding来调控ResNet每个BN层的参数，进而使得模型能在训练前期实现对图像信息的调控

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Style transfer

整个网络结构主要由两个部分构成：

• Image Transformation Networks （采用了类似ResNet的Encoder-Decoder结构）
• Perceptual Loss Functions
  • Feature Reconstruction Loss（作用在relu3_3,确保网络生成的图片能与content image在high-level feature上保持一致，但是在low-level上可以存在差异）
  • Style Reconstruction Loss.(作用在每一层）

A Learned Representation For Artistic Style

• Two images are similar in content if their high-level features as extracted by a trained
  classifier are close in Euclidian distance.
• Two images are similar in style if their low-level features as extracted by a trained classifier
  share the same statistics or, more concretely, if the difference between the features’ Gram
  matrices has a small Frobenius norm

Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization

• the affine parameters in IN can completely change the style of the output image
• Motivated by these observations, we argue that instance normalization performs a form of style normalization by normalizing feature statistics, namely the mean and variance

• 通过实验验证了IN能够起到风格归一化的作用
• 每个affine parameters都从输入的style image中计算得到

Dynamic Instance Normalization for Arbitrary Style Transfer

Style Transfer

文章主要基于AdaIN的两个缺点提出

• content rncoder与style encoder的结构许保持一致
• encoder的结构得足够复杂，以确保其能够提取出更具代表性的图像特征

• content image对应的encoder由Depthwise Separable Convolutional layer构成，属于一个轻量级的特征提取网络
• DIN的输入为一张style image,其输出为convolution operator的weight与bias,然后基于学得的weight与bias对IN层输出的feature map进行调整

Positional Normalization

Style Transfer

• PONO: processes each position independently, and compute both statistics across the channels
• Moment Shortcut: 将encoder中每个PONO输出的参数通过shortcut的方式传到decoder中，以使得decoder上采样得到的deature map能更好地保留原始图片的空间结构信息
• Dynamic Moment Shortcut: 和MS不同的一点在于，decoder反卷积层的PONO参数是通过一个额外的网络（值得注意的是，该网络以相应的encoder层输出的mean和standard deviation作为输入）学习得到的

Semantic Image Synthesis with Spatially-Adaptive Normalization

Semantic Image Synthesis

cGANs with Projection Discriminator

GAN