openpyxl format cells 格式化单元格文中介绍了自动化triage report格式化问题,信息提取部分将在本文中介绍。
背景:
还是在日常triage API automation时遇到一些痛点,基于Cucumber BDD测试框架,每天smoke 加regression多达1万多的cases,怎样提高triage效率,怎么减少人力工作,做为一个爱思考的QA都必须想办法解决,哈哈!
怎么才能只获取Failed的Scenarios信息,还能将cucumber report和REST log结合起来,统统放在一个表格里就能获取所有资源信息呢,这样就不用东找西找,还能生成一个漂亮的triage report,将人力工作统统自动化。
成果:
很早就有这个想法,等待一个时机就开始行动了,并且最终实现了,也就400行Python 脚本就能解决的问题。效果如下,每个fail的Scenario details信息,对应的REST log 文件link,每个Feature的汇总信息,对应的Cucumber Report link,是不是有点酷,反正我是很有成就感了,也得到了小伙伴们的称赞,大大提高了triage效率。
Failed Scenarios Details:
Features Summary Report:
实现依据:
以上信息的提取主要结合cucumber json report, REST log信息和第三方maven-cucumber-reporting 产生的report。其实maven-cucumber-reporting产生的report也是基于cucumber json report生成的。
cucumber json report是将cucumber feature files里每个scenarios的运行结果以json的形式输出。只需在Cucumber Runner Class中配置即可产生这样一个json report:
format = {
"json:target/cucumber.json"}
下面就来解读一下这个json文件:
每个{ }就是一个feature, 有feature name, feature uri信息,下图中就运行了8个feature files
elements里就是这个feature的所有scenarios
element中每个{ }就是一个scenario执行信息, scenario line, scenaio name, scenario uri,下图这个feature有3个scenarios,
Step中的元素就是这个scnario中具体的steps执行情况,
每个{ }就是一个step,有step name, step line, step result(status, error messages)
Scenarios的运行信息都可以通过这个json file获取,至于REST log和 feature report信息就得结合feature uri, step uri找到对应的mapping关系就可以了。
代码
就只贴提取信息代码,格式化代码参考openpyxl format cells 格式化单元格
定义Scenario dataset
class Scenario_Dataset:
def __init__(self,feature_name, feature_rest_log_dir,scenario_name, scenario_line, step_name, step_line, step_error_message, step_error_type=None, step_rest_error_message=None, step_rest_error_url=None):
self.feature_name = feature_name
self.feature_rest_log_dir = feature_rest_log_dir
self.scenario_name = scenario_name
self.scenario_line = scenario_line
self.step_name = step_name
self.step_line = step_line
self.step_error_message = step_error_message
self.step_rest_error_message = step_rest_error_message
self.step_rest_error_url = step_rest_error_url
self.step_error_type = step_error_type
def set_rest_error_message(self, step_rest_error_message):
self.step_rest_error_message = step_rest_error_message
def set_rest_error_url(self, step_rest_error_url):
self.step_rest_error_url = step_rest_error_url
定义Feature dataset
class Feature_Dataset:
def __init__(self, feature_name, total_passed_scenarios, total_failed_scenarios, total_scenarios, error_type_set, feature_report_url):
self.feature_name = feature_name
self.total_passed_scenarios = total_passed_scenarios
self.total_failed_scenarios = total_failed_scenarios
self.total_scenarios = total_scenarios
self.error_type_set = error_type_set
self.feature_report_url = feature_report_url
提取Feature info
def read_cucumber_json(cucumber_file_name):
with io.open(cucumber_file_name,'rt',encoding='utf-8') as f:
data = json.load(f)
return data
def extract_feature_infor_from_cucumber_json(cucumber_file_name):
data = read_cucumber_json(cucumber_file_name)
feature_error_list = []
features_summary_list = []
for item in data:
feature_dataset, scenario_error_list = extract_scenario_info(item)
if len(scenario_error_list) > 0:
feature_error_list.append(scenario_error_list)
features_summary_list.append(feature_dataset)
return features_summary_list, feature_error_list
提取scenario info
def extract_scenario_info(item):
scenario_error_list = []
feature_dataset = None
error_type_set = dict()
elements = item["elements"]
if elements[0]["type"] == 'background':
total_scenarios = len(elements) - 1
else:
total_scenarios = len(elements)
total_failed_scenarios = 0
feature_name = item['name']
feature_rest_log_dir = get_feature_log_dir(item['uri'])
feature_report_url = ''
for element in elements:
scenario_dataset = extract_steps_info(feature_name, feature_rest_log_dir, element)
if scenario_dataset is not None:
total_failed_scenarios += 1
if scenario_dataset.step_error_type in error_type_set:
error_type_set[scenario_dataset.step_error_type] += 1
else:
error_type_set[scenario_dataset.step_error_type] = 1
scenario_error_list.append(scenario_dataset)
if total_failed_scenarios > 0:
feature_report_url = get_report_file_path(item['uri'].replace('/', '-').replace('.','-'))
feature_dataset = Feature_Dataset(feature_name, total_scenarios-total_failed_scenarios, total_failed_scenarios, total_scenarios, str(error_type_set), feature_report_url)
return feature_dataset, scenario_error_list
提取step info
def extract_steps_info(feature_name, feature_rest_log_dir,element):
scenario_name = element['name']
scenario_line = element['line']
steps = element['steps']
step_error_type = None
scenario_dataset = None
for step in steps:
step_result = step['result']
step_status = step_result['status']
if step_status == 'failed':
step_error_message = step_result['error_message']
step_line = step['line']
step_name = step['name']
end_index = step_error_message.find('at ')
step_error_message = step_error_message[:end_index].strip()
if step_error_message.find('Expected status code') > 0:
status_code_end_index = step_error_message.find('>.')
step_error_type = int(step_error_message[status_code_end_index - 3:status_code_end_index])
elif step_error_message.find('UnknownHostException') >= 0:
step_error_type = 'Name or service not known'
else:
step_error_type = 'the response does not match the expected'
scenario_dataset = Scenario_Dataset(feature_name, feature_rest_log_dir, scenario_name, scenario_line, step_name, step_line, step_error_message, step_error_type=step_error_type)
break
return scenario_dataset
如果要运行在jenkins上,还得结合这些变量找到对应的文件关系。
WORKSPACE = os.getenv('WORKSPACE')
BUILD_URL = os.getenv('BUILD_URL')
ARTIFACT_URL = join(BUILD_URL, 'artifact')
JOB_NAME = os.getenv('JOB_BASE_NAME')