安装 Logstash
下载并解压缩 Logstash
笔者下载的是二进制安装包中的 MACOS 版本
Elastic 官方网站:https://www.elastic.co/cn/
Logstash介绍页面:https://www.elastic.co/cn/logstash
Logstash下载页面:https://www.elastic.co/cn/downloads/logstash
准备 logstash.conf 配置文件
运行 Logstash
bin/logstash -f logstash.conf
下载Movielens测试数据集
下载最MovieLens最小测试数据集:https://grouplens.org/datasets/movielens/
编写logstash.conf配置文件
创建 logstash.conf 配置文件并写入一下内容(注意文件路径)
input {
file {
path => "/Users/mankvis/Work/elasticsearchStudy/ml-latest-small/movies.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["id","content","genre"]
}
mutate {
split => {
"genre" => "|" }
remove_field => ["path", "host","@timestamp","message"]
}
mutate {
split => ["content", "("]
add_field => {
"title" => "%{
[content][0]}"}
add_field => {
"year" => "%{
[content][1]}"}
}
mutate {
convert => {
"year" => "integer"
}
strip => ["title"]
remove_field => ["path", "host","@timestamp","message","content"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "movies"
document_id => "%{
id}"
}
stdout {
}
}
尝试通过 Logstash 导入数据
在 logstash/bin 目录下运行一下命令
sudo ./logstash -f logstash.conf
以下为输出信息
查看已经导入的数据
打开 Kibana 的 Dev Tools 并输入 GET /movies/_search
可以看到下载的测试数据集中的数据已经全部倒入 Elasticsearch 中