原文链接:https://blog.csdn.net/gaohuazhao/article/details/69568267
学习caffe后跑了自带的例子是不是感觉很不过瘾,学习caffe的目的不是简单做几个练习,而是要用到自己的实际项目或者科研中,所以本文介绍如何从自己的原始图片到lmdb数据,再到训练和测试模型的整个流程。
详细代码可以去我的github上https://github.com/EddyGao/Caffe-taobao_Image-Identification参考,下载myfile4放到你的caffe/examples/下面,执行demo.sh即可。
一.准备数据
1)我们借用网上某童鞋的数据集,来自于淘宝10个商品类1000张图片,每个类100张,当然你也可以根据自己的需要搜集自己想要识别的图片集
下载地址,百度网盘:https://pan.baidu.com/s/1o77w1wI
下载后将文件解压。
2)在caffe/examples文件夹下新建文件夹myfile4,并在myfile4中新建文件夹data用于存放我们的数据集,将下载的数据集中train和val文件复制到data中即
caffe/examples/myfile4/data/train
caffe/examples/myfile4/data/val
二.转换为lmdb格式
1)在myfile4文件夹中新建create_filelist.sh
-
#!/usr/bin/env sh
-
DATA=examples/myfile4/data
-
MY=examples/myfile4/data
-
echo "Create train.txt..."
-
rm -rf $MY/train.txt
-
find $DATA/train -name 15001*.jpg | cut -d '/' -f4-5 | sed "s/$/ 0/">>$MY/train.txt
-
find $DATA/train -name 15059*.jpg | cut -d '/' -f4-5 | sed "s/$/ 1/">>$MY/train.txt
-
find $DATA/train -name 62047*.jpg | cut -d '/' -f4-5 | sed "s/$/ 2/">>$MY/train.txt
-
find $DATA/train -name 68021*.jpg | cut -d '/' -f4-5 | sed "s/$/ 3/">>$MY/train.txt
-
find $DATA/train -name 73018*.jpg | cut -d '/' -f4-5 | sed "s/$/ 4/">>$MY/train.txt
-
find $DATA/train -name 73063*.jpg | cut -d '/' -f4-5 | sed "s/$/ 5/">>$MY/train.txt
-
find $DATA/train -name 80012*.jpg | cut -d '/' -f4-5 | sed "s/$/ 6/">>$MY/train.txt
-
find $DATA/train -name 92002*.jpg | cut -d '/' -f4-5 | sed "s/$/ 7/">>$MY/train.txt
-
find $DATA/train -name 92017*.jpg | cut -d '/' -f4-5 | sed "s/$/ 8/">>$MY/train.txt
-
find $DATA/train -name 95005*.jpg | cut -d '/' -f4-5 | sed "s/$/ 9/">>$MY/train.txt
-
echo "Create test.txt..."
-
rm -rf $MY/val.txt
-
find $DATA/val -name 15001*.jpg | cut -d '/' -f4-5 | sed "s/$/ 0/">>$MY/val.txt
-
find $DATA/val -name 15059*.jpg | cut -d '/' -f4-5 | sed "s/$/ 1/">>$MY/val.txt
-
find $DATA/val -name 62047*.jpg | cut -d '/' -f4-5 | sed "s/$/ 2/">>$MY/val.txt
-
find $DATA/val -name 68021*.jpg | cut -d '/' -f4-5 | sed "s/$/ 3/">>$MY/val.txt
-
find $DATA/val -name 73018*.jpg | cut -d '/' -f4-5 | sed "s/$/ 4/">>$MY/val.txt
-
find $DATA/val -name 73063*.jpg | cut -d '/' -f4-5 | sed "s/$/ 5/">>$MY/val.txt
-
find $DATA/val -name 80012*.jpg | cut -d '/' -f4-5 | sed "s/$/ 6/">>$MY/val.txt
-
find $DATA/val -name 92002*.jpg | cut -d '/' -f4-5 | sed "s/$/ 7/">>$MY/val.txt
-
find $DATA/val -name 92017*.jpg | cut -d '/' -f4-5 | sed "s/$/ 8/">>$MY/val.txt
-
find $DATA/val -name 95005*.jpg | cut -d '/' -f4-5 | sed "s/$/ 9/">>$MY/val.txt
-
echo "All done"
在shell中运行此脚本,在caffe根目录下执行
# sh examples/myfile4/create_filelist.sh
执行完成后你会发现在examples/myfile4/data文件夹中生成train.txt和val.txt文件
2)在myfile4文件夹中新建create_lmdb.sh文件
-
#!/usr/bin/env sh
-
MY=examples/myfile4
-
TRAIN_DATA_ROOT=/home/ghz/caffe/examples/myfile4/data/
-
VAL_DATA_ROOT=/home/ghz/caffe/examples/myfile4/data/
-
echo "Create train lmdb.."
-
rm -rf $MY/img_train_lmdb
-
build/tools/convert_imageset \
-
--shuffle \
-
--resize_height=32 \
-
--resize_width=32 \
-
$TRAIN_DATA_ROOT \
-
$MY/data/train.txt \
-
$MY/img_train_lmdb
-
echo "Create test lmdb.."
-
rm -rf $MY/img_val_lmdb
-
build/tools/convert_imageset \
-
--shuffle \
-
--resize_height=32 \
-
--resize_width=32 \
-
$VAL_DATA_ROOT \
-
$MY/data/val.txt \
-
$MY/img_val_lmdb
-
echo "All Done.."
在shell中运行此脚本,在caffe根目录下执行
# sh examples/myfile4/create_lmdb.sh
执行后会在myfile4文件夹下生成img_train_lmdb文件夹和img_val_lmdb文件夹
三.计算均值并保存
myfile4中新建文件create_meanfile.sh
-
EXAMPLE=examples/myfile4
-
DATA=examples/myfile4
-
TOOLS=build/tools
-
$TOOLS/compute_image_mean $EXAMPLE/img_train_lmdb $DATA/mean.binaryproto
-
echo "Done."
生成mean.binaryproto文件
四.创建模型并编写配置文件
在myfile4中创建myfile4_train_test.prototxt文件
-
name: "myfile4"
-
layer {
-
name: "data"
-
type: "Data"
-
top: "data"
-
top: "label"
-
include {
-
phase: TRAIN
-
}
-
transform_param {
-
mean_file: "examples/myfile4/mean.binaryproto"
-
}
-
data_param {
-
source: "examples/myfile4/img_train_lmdb"
-
batch_size: 50
-
backend: LMDB
-
}
-
}
-
layer {
-
name: "cifar"
-
type: "Data"
-
top: "data"
-
top: "label"
-
include {
-
phase: TEST
-
}
-
transform_param {
-
mean_file: "examples/myfile4/mean.binaryproto"
-
}
-
data_param {
-
source: "examples/myfile4/img_val_lmdb"
-
batch_size: 50
-
backend: LMDB
-
}
-
}
-
layer {
-
name: "conv1"
-
type: "Convolution"
-
bottom: "data"
-
top: "conv1"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
convolution_param {
-
num_output: 32
-
pad: 2
-
kernel_size: 5
-
stride: 1
-
weight_filler {
-
type: "gaussian"
-
std: 0.0001
-
}
-
bias_filler {
-
type: "constant"
-
}
-
}
-
}
-
layer {
-
name: "pool1"
-
type: "Pooling"
-
bottom: "conv1"
-
top: "pool1"
-
pooling_param {
-
pool: MAX
-
kernel_size: 3
-
stride: 2
-
}
-
}
-
layer {
-
name: "relu1"
-
type: "ReLU"
-
bottom: "pool1"
-
top: "pool1"
-
}
-
layer {
-
name: "conv2"
-
type: "Convolution"
-
bottom: "pool1"
-
top: "conv2"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
convolution_param {
-
num_output: 32
-
pad: 2
-
kernel_size: 5
-
stride: 1
-
weight_filler {
-
type: "gaussian"
-
std: 0.01
-
}
-
bias_filler {
-
type: "constant"
-
}
-
}
-
}
-
layer {
-
name: "relu2"
-
type: "ReLU"
-
bottom: "conv2"
-
top: "conv2"
-
}
-
layer {
-
name: "pool2"
-
type: "Pooling"
-
bottom: "conv2"
-
top: "pool2"
-
pooling_param {
-
pool: AVE
-
kernel_size: 3
-
stride: 2
-
}
-
}
-
layer {
-
name: "conv3"
-
type: "Convolution"
-
bottom: "pool2"
-
top: "conv3"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
convolution_param {
-
num_output: 64
-
pad: 2
-
kernel_size: 5
-
stride: 1
-
weight_filler {
-
type: "gaussian"
-
std: 0.01
-
}
-
bias_filler {
-
type: "constant"
-
}
-
}
-
}
-
layer {
-
name: "relu3"
-
type: "ReLU"
-
bottom: "conv3"
-
top: "conv3"
-
}
-
layer {
-
name: "pool3"
-
type: "Pooling"
-
bottom: "conv3"
-
top: "pool3"
-
pooling_param {
-
pool: AVE
-
kernel_size: 3
-
stride: 2
-
}
-
}
-
layer {
-
name: "ip1"
-
type: "InnerProduct"
-
bottom: "pool3"
-
top: "ip1"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
inner_product_param {
-
num_output: 64
-
weight_filler {
-
type: "gaussian"
-
std: 0.1
-
}
-
bias_filler {
-
type: "constant"
-
}
-
}
-
}
-
layer {
-
name: "ip2"
-
type: "InnerProduct"
-
bottom: "ip1"
-
top: "ip2"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
inner_product_param {
-
num_output: 10
-
weight_filler {
-
type: "gaussian"
-
std: 0.1
-
}
-
bias_filler {
-
type: "constant"
-
}
-
}
-
}
-
layer {
-
name: "accuracy"
-
type: "Accuracy"
-
bottom: "ip2"
-
bottom: "label"
-
top: "accuracy"
-
include {
-
phase: TEST
-
}
-
}
-
layer {
-
name: "loss"
-
type: "SoftmaxWithLoss"
-
bottom: "ip2"
-
bottom: "label"
-
top: "loss"
-
}
创建myfile4_solver.prototxt文件
-
net: "examples/myfile4/myfile4_train_test.prototxt"
-
test_iter: 2
-
test_interval: 50
-
base_lr: 0.001
-
lr_policy: "step"
-
gamma: 0.1
-
stepsize: 400
-
momentum: 0.9
-
weight_decay: 0.004
-
display: 10
-
max_iter: 2000
-
snapshot: 2000
-
snapshot_prefix: "examples/myfile4/my"
-
solver_mode: CPU
五.训练和测试
在caffe根目录下执行
# build/tools/caffe train -solver examples/myfile4/myfile4_solver.prototxt
我们执行了2000此迭代并在迭代最后保存了模型,如果没有错误训练完成后会在文件夹下出现my_iter_2000.caffemodel文件
六.用训练好的模型进行分类
1)在myfile4中新建文件synset_words.txt
-
biao
-
fajia
-
kuzi
-
xiangzi
-
yizi
-
dianshi
-
suannai
-
xiangshui
-
hufupin
-
xiezi
2)在myfile4中新建文件deploy.prototxt
-
name: "myfile4"
-
layer {
-
name: "data"
-
type: "Input"
-
top: "data"
-
input_param{shape: {dim:1 dim:3 dim:32 dim:32}}
-
}
-
layer {
-
name: "conv1"
-
type: "Convolution"
-
bottom: "data"
-
top: "conv1"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
convolution_param {
-
num_output: 32
-
pad: 2
-
kernel_size: 5
-
stride: 1
-
}
-
}
-
layer {
-
name: "pool1"
-
type: "Pooling"
-
bottom: "conv1"
-
top: "pool1"
-
pooling_param {
-
pool: MAX
-
kernel_size: 3
-
stride: 2
-
}
-
}
-
layer {
-
name: "relu1"
-
type: "ReLU"
-
bottom: "pool1"
-
top: "pool1"
-
}
-
layer {
-
name: "conv2"
-
type: "Convolution"
-
bottom: "pool1"
-
top: "conv2"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
convolution_param {
-
num_output: 32
-
pad: 2
-
kernel_size: 5
-
stride: 1
-
}
-
}
-
layer {
-
name: "relu2"
-
type: "ReLU"
-
bottom: "conv2"
-
top: "conv2"
-
}
-
layer {
-
name: "pool2"
-
type: "Pooling"
-
bottom: "conv2"
-
top: "pool2"
-
pooling_param {
-
pool: AVE
-
kernel_size: 3
-
stride: 2
-
}
-
}
-
layer {
-
name: "conv3"
-
type: "Convolution"
-
bottom: "pool2"
-
top: "conv3"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
convolution_param {
-
num_output: 64
-
pad: 2
-
kernel_size: 5
-
stride: 1
-
}
-
}
-
layer {
-
name: "relu3"
-
type: "ReLU"
-
bottom: "conv3"
-
top: "conv3"
-
}
-
layer {
-
name: "pool3"
-
type: "Pooling"
-
bottom: "conv3"
-
top: "pool3"
-
pooling_param {
-
pool: AVE
-
kernel_size: 3
-
stride: 2
-
}
-
}
-
layer {
-
name: "ip1"
-
type: "InnerProduct"
-
bottom: "pool3"
-
top: "ip1"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
inner_product_param {
-
num_output: 64
-
}
-
}
-
layer {
-
name: "ip2"
-
type: "InnerProduct"
-
bottom: "ip1"
-
top: "ip2"
-
param {
-
lr_mult: 1
-
}
-
param {
-
lr_mult: 2
-
}
-
inner_product_param {
-
num_output: 10
-
}
-
}
-
layer {
-
name: "prob"
-
type: "Softmax"
-
bottom: "ip2"
-
top: "prob"
-
}
3)在myfile4中新建文件夹images,其中放入你想要分类的图片,比如我的是images/111.jpg,测试图片可以来自于我们下载的图像集中
4)在myfile4中新建文件demo.sh
./build/examples/cpp_classification/classification.bin examples/myfile4/deploy.prototxt examples/myfile4/my_iter_2000.caffemodel examples/myfile4/mean.binaryproto examples/myfile4/synset_words.txt examples/myfile4/images/111.jpg
在caffe根目录下执行demo.sh
# sh examples/myfile4/demo.sh
执行后会在shell显示识别的结果
至此,我们完成了Caffe训练和测试自己的数据集,如有问题欢迎交流。