线上系统使用thrift rpc框架, 突然有一天某个server挂了,查看错误日志发现如下错误信息:
[05-26 16:53:51] [ERROR] [org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer:348] Read an invalid frame size of -2137128701. Are you using TFramedTransport on the client side? [05-26 16:53:57] [ERROR] [org.apache.thrift.server.TThreadedSelectorServer:553] run() exiting due to uncaught error java.lang.OutOfMemoryError at sun.misc.Unsafe.allocateMemory(Native Method) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174) at sun.nio.ch.IOUtil.read(IOUtil.java:195) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:142) at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:539) at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:388) at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203) at org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:590) at org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:545)
可以看出来,是因为服务端收到了一个非常大的数据包,thrift需要申请内存,造成服务器OOM,从而宕机。为了避免这个错误,我们需要在服务端加上如下参数: