Protobuf是谷歌开源的协议框架,以高效传输著称并且支持多种语言。工作中也用到了,在此做个总结。
Protobuf运行原理
下面一张图可以说明:
Python中使用Protobuf
2.1安装protoc程序
protoc --version
我安装的是3.6.1版本
2.2定义协议
syntax = "proto3";
package com.union.fun;
message Metric {
string name = 1;
string type = 2;
float value = 3;
repeated string tags = 4;
}
2.3生成接口
//Python接口生成
protobuf/bin/protoc -I=/home/tiger/python --python_out=/home/tiger/python metric.proto
//另外java和C++生成方式如下
//Java接口生成
protobuf/bin/protoc -I=/home/tiger/python --java_out=/home/tiger/python metric.proto
//C++接口生成
protobuf/bin/protoc -I=/home/tiger/python --cpp_out=/home/tiger/python metric.proto
2.4读写调用
(1)单个对象
import metric_pb2
def writepb():
my_metric = metric_pb2.Metric()
my_metric.name = 'sys.cpu'
my_metric.type = 'gauge'
my_metric.value = 99.9
my_metric.tags.extend(['my_tag', 'foo:bar'])
with open('out.bin', 'wb') as f:
f.write(my_metric.SerializeToString())
def readpb():
with open('out.bin', 'rb') as f:
read_metric = metric_pb2.Metric()
read_metric.ParseFromString(f.read())
print(read_metric)
# do something with read_metric
if __name__ == "__main__":
writepb()
readpb()
运行结果如下:python3 method1.py
(2)连续多个对象
import metric_pb2
from google.protobuf.internal.encoder import _VarintBytes
from google.protobuf.internal.decoder import _DecodeVarint32
import random
def writepb():
with open('out.bin', 'wb') as f:
my_tags = ("my_tag", "foo:bar")
for i in range(128):
my_metric = metric_pb2.Metric()
my_metric.name = 'sys.cpu'
my_metric.type = 'gauge'
my_metric.value = round(random.random(), 2)
my_metric.tags.extend(my_tags)
size = my_metric.ByteSize()
f.write(_VarintBytes(size))
f.write(my_metric.SerializeToString())
def readpb():
with open('out.bin', 'rb') as f:
buf = f.read()
n = 0
while n < len(buf):
msg_len, new_pos = _DecodeVarint32(buf, n)
n = new_pos
msg_buf = buf[n:n+msg_len]
n += msg_len
read_metric = metric_pb2.Metric()
read_metric.ParseFromString(msg_buf)
print(read_metric)
# do something with read_metric
if __name__ == "__main__":
writepb()
readpb()
运行结果如下:python3 method2.py
参考文档如下:
https://www.datadoghq.com/blog/engineering/protobuf-parsing-in-python/