Video Retrieval 中的评价指标

几个指标 Recall MedR MnR

We evaluate the performance of our model with standard retrieval metrics: recall at rank N (R@N, higher is better), median rank (MdR, lower is better) and mean rank (MnR, lower is better).The MedR and MnR measures the median and average rank of correct items in the retrieved ranking list respectively

· Recall:

在这里插入图片描述

·MedR

衡量搜索正确的选项在序列中位置的中位数。

·MnR

衡量搜索正确的选项在序列中位置的平均数。

代码

mmt的代码有点复杂，所有选择写的比较清楚的hgr的代码，原理不变
代码地址：https://github.com/cshizhe/hgr_v2t/blob/master/t2vretrieval/models/evaluation.py


import numpy as np


def eval_q2m(scores, q2m_gts):
    '''
  Image -> Text / Text -> Image
  Args:
    scores: (n_query, n_memory) matrix of similarity scores
    q2m_gts: list, each item is the positive memory ids of the query id
  Returns:
    scores: (recall@1, 5, 10, median rank, mean rank)
    gt_ranks: the best ranking of ground-truth memories
  '''
    n_q, n_m = scores.shape
    gt_ranks = np.zeros((n_q,), np.int32)

    for i in range(n_q):
        # 以每一个询问为单位，记录所有的得分
        s = scores[i]
        # 找到置信度最大的下标，例如找到最匹配的视频的序号，按从大到小排序
        sorted_idxs = np.argsort(-s)
        # 初值设置为所有查询的长度
        rank = n_m
        # q2m_gts[i]：第i个问询的模型计算出的最匹配的id list
        for k in q2m_gts[i]:
            # 在计算中找到一个序号最小（置信度最大），因为可能会计算出多个
            tmp = np.where(sorted_idxs == k)[0][0]
            if tmp < rank:
                rank = tmp
        # 每个问询最优的排序
        gt_ranks[i] = rank

    # compute metrics
    # gt_ranks 是每个问询的正确选项排名，np.sum(cols == X) 则是指正确在X之前的个数，这个个数为正样本个数
    r1 = 100 * len(np.where(gt_ranks < 1)[0]) / n_q
    r5 = 100 * len(np.where(gt_ranks < 5)[0]) / n_q
    r10 = 100 * len(np.where(gt_ranks < 10)[0]) / n_q
    # 取每一个查询正确的中位数
    medr = np.median(gt_ranks) + 1
    # 平均每一个查询正确的位置
    meanr = gt_ranks.mean() + 1
    return (r1, r5, r10, medr, meanr)

参考资料

Recall : https://zhuanlan.zhihu.com/p/146711298

https://blog.csdn.net/weixin_37821353/article/details/88367211?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522162669915516780264042929%2522%252C%2522scm%2522%253A%252220140713.130102334…%2522%257D&request_id=162669915516780264042929&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2_allsobaiduend~default-3-88367211.first_rank_v2_pc_rank_v29&utm_term=axis+%3D+1&spm=1018.2226.3001.4187

https://blog.csdn.net/Laox1ao/article/details/73289320?utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-2.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-2.control

https://blog.csdn.net/weixin_42866962/article/details/82811082?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522162669471316780274176209%2522%252C%2522scm%2522%253A%252220140713.130102334…%2522%257D&request_id=162669471316780274176209&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2_alltop_positive~default-1-82811082.first_rank_v2_pc_rank_v29&utm_term=np.newaxis&spm=1018.2226.3001.4187