如何扩展python的logging组件支持json日志输出！如何扩展python的logging组件支持json日志输出！

这两天在优化公司一个python的项目，顺便研究了一下如何将python日志转成json格式，原来在Java的项目中搞过类似的事情，知道日志转成json之后有很多便利的之处，最常见的就是可以直接对接各种日志分析系统，如开源的ELK，将数据导入之后就能快速的进行查询和分析，方便做各种统计，监控或报警等。

python里面的logging组件，其实已经是组件化了，有Logger组件，Handler组件，Fomatter组件，如下图所示：

如何扩展python的logging组件支持json日志输出！

进群“960410445” 即可获取书十套PDF！

logger=>handler=>formatter分别是一对多的关系，日志的格式其实是由formatter决定的，所以想要扩展成你想要的各种格式，就重写定制formatter组件就可以了，它实际上和Java里面Log4j的LayOut组件类似。

核心代码如下：

 REMOVE_ATTR 
=
 
[
"filename"
,
 
"module"
,
 
"exc_text"
,
 
"stack_info"
,
 
"created"
,
 
"msecs"
,
 
"relativeCreated"
,
 
"exc_info"
,
 
"msg"
]
class
 
JSONFormatter
(
logging
.
Formatter
):
 host_name
,
 host_ip 
=
 
HostIp
.
get_host_ip
()
 
def
 format
(
self
,
 record
):
 extra 
=
 self
.
build_record
(
record
)
 self
.
set_format_time
(
extra
)
 
# set time
 self
.
set_host_ip
(
extra
)
 
# set host name and host ip
 extra
[
'message'
]
 
=
 record
.
msg 
# set message
 
if
 record
.
exc_info
:
 extra
[
'exc_info'
]
 
=
 self
.
formatException
(
record
.
exc_info
)
 
if
 self
.
_fmt 
==
 
'pretty'
:
 
return
 json
.
dumps
(
extra
,
 indent
=
1
,
 ensure_ascii
=
False
)
 
else
:
 
return
 json
.
dumps
(
extra
,
 ensure_ascii
=
False
)
 
@classmethod
 
def
 build_record
(
cls
,
 record
):
 
return
 
{
 attr_name
:
 record
.
__dict__
[
attr_name
]
 
for
 attr_name 
in
 record
.
__dict__
 
if
 attr_name 
not
 
in
 REMOVE_ATTR
 
}
 
@classmethod
 
def
 set_format_time
(
cls
,
 extra
):
 now 
=
 datetime
.
datetime
.
utcnow
()
 format_time 
=
 now
.
strftime
(
"%Y-%m-%dT%H:%M:%S"
 
+
 
".%03d"
 
%
 
(
now
.
microsecond 
/
 
1000
)
 
+
 
"Z"
)
 extra
[
'@timestamp'
]
 
=
 format_time
 
return
 format_time
 
@classmethod
 
def
 set_host_ip
(
cls
,
 extra
):
 extra
[
'host_name'
]
 
=
 
JSONFormatter
.
host_name
 extra
[
'host_ip'
]
 
=
 
JSONFormatter
.
host_ip

使用的时候，可以通过简单配置即可：

 
[
loggers
]
keys
=
root
[
handlers
]
keys
=
console
,
file
[
formatters
]
keys
=
json
,
json_pretty
[
logger_root
]
level
=
DEBUG
;
handlers
=
console
,
file
,
rotate
handlers
=
console
,
file
[
handler_console
]
class
=
StreamHandler
level
=
INFO
formatter
=
json_pretty
args
=(
sys
.
stderr
,)
[
handler_file
]
class
=
FileHandler
level
=
INFO
formatter
=
json
args
=(
'log/base_conf.log'
,
'a'
,
'utf-8'
)
[
handler_rotate
]
class
=
handlers
.
TimedRotatingFileHandler
level
=
INFO
formatter
=
json
args
=(
'log/rotate.log'
,
 
'D'
,
1
,
0
,
'utf-8'
)
[
formatter_json
]
class
=
format
.
json_formatter
.
JSONFormatter
[
formatter_json_pretty
]
format
=
pretty
class
=
format
.
json_formatter
.
JSONFormatter

如下的一段异常代码：

 fileConfig
(
'log_conf.ini'
)
 log 
=
 logging
.
getLogger
(
__name__
)
 
try
:
 a 
=
 
1
 
/
 
0
 
except
 
Exception
:
 log
.
exception
(
" occurred exception "
)

输出结果如下：

{
 
"name"
:
 
"__main__"
,
 
"args"
:
 
[],
 
"levelname"
:
 
"ERROR"
,
 
"levelno"
:
 
40
,
 
"pathname"
:
 
"C:/texx.py"
,
 
"lineno"
:
 
17
,
 
"funcName"
:
 
"base_configuration"
,
 
"thread"
:
 
10608
,
 
"threadName"
:
 
"MainThread"
,
 
"processName"
:
 
"MainProcess"
,
 
"process"
:
 
11916
,
 
"@timestamp"
:
 
"2019-01-10T12:50:20.392Z"
,
 
"host_name"
:
 
"your-PC"
,
 
"host_ip"
:
 
"192.168.10.11"
,
 
"message"
:
 
" occurred exception "
,
 
"exc_info"
:
 
"Traceback (most recent call last):
 File "C:/txxx.py", line 14, in base_configuration
 a = 1 / 0
ZeroDivisionError: division by zero"
}

可以看到内容非常详细，并且组件还支持自定义字段的加入，在收集到日志系统上，可以非常的方便检索统计。

详细的解释和代码，可以fork我的github：https://github.com/qindongliang/python_log_json

如何扩展python的logging组件支持json日志输出！如何扩展python的logging组件支持json日志输出！

猜你喜欢