最近上了一个项目,有些地方要抠一些细节,达到效果最优,其中有一处就是要从es流中整合出完整的帧。搜了一些资料,然后参照live555的代码做一下总结,留着以后备用。
PS: 从事流媒体开发时间不长,有哪些地方理解有误,还希望能帮我指正出来,共同进步^_^
首先讲一下es流的结构,看完标准发现h.264没有帧的概念,是以片(slice也有翻译成条带的)为单元的。
如下图:
参考文章:http://zhongcong386.blog.163.com/blog/static/1347278042012112762445528/
NAL_header(1Byte)+(slice_header)+(slice_data)+......+NAL_header(1Byte)+(slice_header)+(slice_data)
slice_data= rbsp_byte=宏块(实实在在的视频数据),当然上面nal_header之后的rbsp数据格式不同类型的NAL也不同。
比如:nal_unit_type ==5 的idr slice 格式是slice_header( )+slice_data( )+rbsp_slice_trailing_bits( )。 具体的各个nal slice的格式参看标准的7.3.2.*
(nal_unit_type <= 5 && nal_unit_type > 0)的nal单元都负载的VCL数据(视频流),每个vcl类型的nal单元存储的是一个片(slice)。
接下来就是分理出完整一帧数据。假设我们已经分离出了连续的nal_unit_type =1~5的NAL。
第一种情况 没有下一个nal则表示当前的nal是当前帧的最后一部分数据。
第二种情况 nal_unit_type !=next_nal_unit_type 则表示上一帧已经结束。
第三种情况 nal_unit_type == next_nal_unit_type的时候有可能是同一帧的数据,也有可能是不同帧的数据。(如:两个PP帧在一块就有可能出现这种情况)
出现第三种情况的时候我们需要分析slice_header里面的frame_num这一个值。如果frame_num != next_frame_num 则表示当前nal已经是当前帧的最后一部分数据。
slice_header()的句法如下: PS 太多啦,有需求直接去看标准;
slice_header( ) {
first_mb_in_slice 2 ue(v)
slice_type 2 ue(v)
pic_parameter_set_id 2 ue(v)
frame_num 2 u(v)
if( !frame_mbs_only_flag ) {
field_pic_flag 2 u(1)
if( field_pic_flag )
bottom_field_flag 2 u(1)
}
if( nal_unit_type = = 5 )
idr_pic_id 2 ue(v)
if( pic_order_cnt_type = = 0 ) {
pic_order_cnt_lsb 2 u(v)
if( pic_order_present_flag && !field_pic_flag )
delta_pic_order_cnt_bottom 2 se(v)
}
..................................
..................................
if( num_slice_groups_minus1 > 0 && slice_group_map_type >= 3 && slice_group_map_type <= 5)
slice_group_change_cycle 2 u(v)
}
具体live555分析的代码我摘录下来,贴在下面
fHaveSeenFirstByteOfNALUnit =true表示当前帧结束
unsigned H264VideoStreamParser::parse() {
..........................
..........................
// If this NAL unit is a VCL NAL unit, we also scan the start of the next NAL unit, to determine whether this NAL unit
// ends the current 'access unit'. We need this information to figure out when to increment "fPresentationTime".
// (RTP streamers also need to know this in order to figure out whether or not to set the "M" bit.)
Boolean thisNALUnitEndsAccessUnit = False; // until we learn otherwise
if (haveSeenEOF()) {//最后一个nal肯定是帧的最后一部分了
thisNALUnitEndsAccessUnit = True;
} else {
Boolean const isVCL = nal_unit_type <= 5 && nal_unit_type > 0; //Would need to include type 20 for SVC and MVC #####
if (isVCL) {//
u_int8_t firstByteOfNextNALUnit;
testBytes(&firstByteOfNextNALUnit, 1);
u_int8_t next_nal_ref_idc = (firstByteOfNextNALUnit&0x60)>>5;
u_int8_t next_nal_unit_type = firstByteOfNextNALUnit&0x1F;
if (next_nal_unit_type >= 6) {//nal_unit_type !=next_nal_unit_type的情况
// The next NAL unit is not a VCL; therefore, we assume that this NAL unit ends the current 'access unit':
thisNALUnitEndsAccessUnit = True;
} else {
// The next NAL unit is also a VCL. We need to examine it a little to figure out if it's a different 'access unit'.
// (We use many of the criteria described in section 7.4.1.2.4 of the H.264 specification.)
Boolean IdrPicFlag = nal_unit_type == 5;
Boolean next_IdrPicFlag = next_nal_unit_type == 5;
if (next_IdrPicFlag != IdrPicFlag) {
// IdrPicFlag differs in value
thisNALUnitEndsAccessUnit = True;
} else if (next_nal_ref_idc != nal_ref_idc && next_nal_ref_idc*nal_ref_idc == 0) {
//nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0
thisNALUnitEndsAccessUnit = True;
} else if ((nal_unit_type == 1 || nal_unit_type == 2 || nal_unit_type == 5)
&& (next_nal_unit_type == 1 || next_nal_unit_type == 2 || next_nal_unit_type == 5)) {//这是第三种情况主要的功能在
//analyze_slice_header中
// Both this and the next NAL units begin with a "slice_header".
// Parse this (for each), to get parameters that we can compare:
// Current NAL unit's "slice_header":
unsigned frame_num, pic_parameter_set_id, idr_pic_id;
Boolean field_pic_flag, bottom_field_flag;
analyze_slice_header(fStartOfFrame + fOutputStartCodeSize, fTo, nal_unit_type,
frame_num, pic_parameter_set_id, idr_pic_id, field_pic_flag, bottom_field_flag);
// Next NAL unit's "slice_header":
u_int8_t next_slice_header[NUM_NEXT_SLICE_HEADER_BYTES_TO_ANALYZE];
testBytes(next_slice_header, sizeof next_slice_header);
unsigned next_frame_num, next_pic_parameter_set_id, next_idr_pic_id;
Boolean next_field_pic_flag, next_bottom_field_flag;
analyze_slice_header(next_slice_header, &next_slice_header[sizeof next_slice_header], next_nal_unit_type,
next_frame_num, next_pic_parameter_set_id, next_idr_pic_id, next_field_pic_flag, next_bottom_field_flag);
if (next_frame_num != frame_num) {
// frame_num differs in value
thisNALUnitEndsAccessUnit = True;
} else if (next_pic_parameter_set_id != pic_parameter_set_id) {
// pic_parameter_set_id differs in value
thisNALUnitEndsAccessUnit = True;
} else if (next_field_pic_flag != field_pic_flag) {
// field_pic_flag differs in value
thisNALUnitEndsAccessUnit = True;
} else if (next_bottom_field_flag != bottom_field_flag) {
// bottom_field_flag differs in value
thisNALUnitEndsAccessUnit = True;
} else if (next_IdrPicFlag == 1 && next_idr_pic_id != idr_pic_id) {
// IdrPicFlag is equal to 1 for both and idr_pic_id differs in value
// Note: We already know that IdrPicFlag is the same for both.
thisNALUnitEndsAccessUnit = True;
}
}
..........................
}
先就这样,下次再补充。
这一篇博客有更好的方法和大家一块分享http://www.360doc.com/content/13/0913/15/13084517_314201133.shtml