一 序
本文属于极客时间 Elasticsearch核心技术与实战学习笔记系列。
二 安装Logstash
下载路径:https://artifacts.elastic.co/downloads/logstash/logstash-7.6.2.zip
我是使用下载zip,直接解压缩的格式安装的。也可使用homebrew安装。
我为什么使用解压的方式,因为最好是下载与ES相同版本号的logstash,(7.2.0),并解压到相应目录。
homebrew不指定就是最新的版本。没试过不知道兼容性如何。刚开始学还是少折腾自己。
如果使用使用Homebrew进行安装
1.1 首先需要点击Elastic Homebrew存储库
brew tap elastic/tap
1.2 点击Elastic Homebrew存储库后,可以brew install用来安装Logstash的默认发行版
brew install elastic/tap/logstash-full
2.使用Homebrew进行Logstash启动
2.1 要立即启动启动elastic / tap / logstash-full并在登录时重新启动
brew services start elastic/tap/logstash-full
2.2 要运行Logstash,请在前台运行
logstash
官网上还有其他安装方式:
Installing from Package Repositoriesedit
We also have repositories available for APT and YUM based distributions. Note that we only provide binary packages, but no source packages, as the packages are created as part of the Logstash build.
We have split the Logstash package repositories by version into separate urls to avoid accidental upgrades across major versions. For all 7.x.y releases use 7.x as version number.
We use the PGP key D88E42B4, Elastic’s Signing Key, with fingerprint
4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4to sign all our packages. It is available from https://pgp.mit.edu.
APTedit
Download and install the Public Signing Key:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -You may need to install the
apt-transport-https
package on Debian before proceeding:sudo apt-get install apt-transport-httpsSave the repository definition to
/etc/apt/sources.list.d/elastic-7.x.list
:echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.listUse the
echo
method described above to add the Logstash repository. Do not useadd-apt-repository
as it will add adeb-src
entry as well, but we do not provide a source package. If you have added thedeb-src
entry, you will see an error like the following:Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)Just delete the
deb-src
entry from the/etc/apt/sources.list
file and the installation should work as expected.Run
sudo apt-get update
and the repository is ready for use. You can install it with:sudo apt-get update && sudo apt-get install logstashSee Running Logstash for details about managing Logstash as a system service.
YUMedit
Download and install the public signing key:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearchAdd the following in your
/etc/yum.repos.d/
directory in a file with a.repo
suffix, for examplelogstash.repo
[logstash-7.x] name=Elastic repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-mdAnd your repository is ready for use. You can install it with:
sudo yum install logstashThe repositories do not work with older rpm based distributions that still use RPM v3, like CentOS5.
See the Running Logstash document for managing Logstash as a system service.
三下载测试数据
- 下载最MovieLens最小测试数据集:https://grouplens.org/datasets/movielens/
- Logstash下载:https://www.elastic.co/cn/downloads/logstash
- Logstash参考文档:https://www.elastic.co/guide/en/logstash/current/index.html
因为网速不好,全的文件250M太大了。所以就使用最小的1M哪个测试。
四 导入数据
进入到logstash解压缩文件夹。bin下。
创建配置文件如下:
input {
file {
path => "/Users/yiruan/dev/elk7/logstash-7.0.1/bin/movies.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["id","content","genre"]
}
mutate {
split => { "genre" => "|" }
remove_field => ["path", "host","@timestamp","message"]
}
mutate {
split => ["content", "("]
add_field => { "title" => "%{[content][0]}"}
add_field => { "year" => "%{[content][1]}"}
}
mutate {
convert => {
"year" => "integer"
}
strip => ["title"]
remove_field => ["path", "host","@timestamp","message","content"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "movies"
document_id => "%{id}"
}
stdout {}
}
注意,上面的配置文件里面,要把path修改为,你实际的movies.csv路径
#启动Elasticsearch实例,然后启动 logstash,并制定配置文件导入数据
bin ./logstash -f logstash.conf
效果如下:
这是为以后的使用做好数据准备。
好了,本节课就到此结束了。