引用:
* 注
* 1. 准备数据文件
* 2. 准备对应导入数据格式的`logstash`配置文件
* 3. 配置容器版`Logstash`,使用`Dockerfile`自定义一款镜像
* 4. 使用`Dockerfile`创建`logstash_image`镜像
* 5. 启动 `ES` 服务
* 6. 启动`logstash`将数据导入`ES`
注
/home/kyle/explore-es
为安装 docker
的虚拟机文件路径,所有操作都在虚拟机里执行
1. 准备数据文件
/home/kyle/explore-es/source/movies.csv
下载最MovieLens最小测试数据集:
https://grouplens.org/datasets/movielens
2. 准备对应导入数据格式的logstash
配置文件
/home/kyle/explore-es/pipeline/logstash.conf
input {
file {
path => "/user/share/source/movies.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["id","content","genre"]
}
mutate {
split => { "genre" => "|" }
remove_field => ["path", "host","@timestamp","message"]
}
mutate {
split => ["content", "("]
add_field => { "title" => "%{[content][0]}"}
add_field => { "year" => "%{[content][1]}"}
}
mutate {
convert => {
"year" => "integer"
}
strip => ["title"]
remove_field => ["path", "host","@timestamp","message","content"]
}
}
output {
elasticsearch {
hosts => "http://192.168.80.129:9200"
inde 大专栏 在docker环境中通过 logstash 给 ES 注入数据x => "movies"
document_id => "%{id}"
}
stdout {}
}
3. 配置容器版Logstash
,使用Dockerfile
自定义一款镜像
/home/kyle/explore-es/Dockerfile
FROM logstash:7.1.0 # 注: 版本必须与 `ES` 版本一致
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
ADD ./pipeline /usr/share/logstash/pipeline/
ADD ./source /user/share/source
4. 使用Dockerfile
创建logstash_image
镜像
/home/kyle/explore-es
docker build -t logstash_image .
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
logstash_image latest ba7fdb87469e 21 hours ago 848 MB
5. 启动 ES
服务
docker ps -a
9f43384e454b elasticsearch:7.1.0 "/usr/local/bin/do..." 21 hours ago Exited (143) 21 hours ago es7_01
ce011bccd327 elasticsearch:7.1.0 "/usr/local/bin/do..." 21 hours ago Exited (143) 21 hours ago es7_02
docker start es7_01 es7_02
es7_01
es7_02
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f43384e454b elasticsearch:7.1.0 "/usr/local/bin/do..." 21 hours ago Up 26 seconds 0.0.0.0:9200->9200/tcp, 9300/tcp es7_01
ce011bccd327 elasticsearch:7.1.0 "/usr/local/bin/do..." 21 hours ago Up 25 seconds 9200/tcp, 9300/tcp es7_02
6. 启动logstash
将数据导入ES
docker run --name logstash logstash_image