3.1决策树的构造
3.1.1信息增益
面对机器学习boss二代目——决策树,大多人应该不陌生,决策树的一些基本概念就不表了,下面直接贴上计算数据集的香农熵的代码:
from math import log def calcShannonEnt(dataSet): numEntries = len(dataSet) labelCounts = {} for featVec in dataSet: currentLabel = featVec[-1] if currentLabel not in labelCounts.keys(): labelCounts[currentLabel] = 0 labelCounts[currentLabel] += 1 shannonEnt = 0.0 for key in labelCounts: prob = float(labelCounts[key])/numEntries shannonEnt -= prob * log(prob,2) return shannonEnt贴心的作者还给我们提供测试的代码,这个依然是加入到trees.py模块中,代码为:
def createDataSet(): dataSet = [[1,1,'yes'], [1,1,'yes'], [1,0,'no'], [0,1,'no'], [0,1,'no']] labels = ['no surfacing','flippers'] return dataSet,labels这次依然选择jupyter作为测试平台,代码:
import sys sys.path.append('D:\xx\xxx')
import trees myDat,labels = trees.createDataSet()测试结果为:
在数据集中添加分类,观察熵的变化:
3.1.2划分数据集
程序代码如下:
def splitDataSet(dataSet,axis,value): retDataSet = [] for featVec in dataSet: if featVec[axis] == value: reducedFeatVec = featVec[:axis] reducedFeatVec.extend(featVec[axis + 1:]) retDataSet.append(reducedFeatVec) return retDataSet
测试结果如下:
这个splitDataSet函数的作用就是根据axis位置上的value来划分数据集。
接下来将splitDataSet和calcShannonEnt函数结合起来,以此确定哪种数据集划分方式最好,代码如下:
def chooseBestFeatureToSplit(dataSet): numFeatures = len(dataSet[0]) - 1 baseEntropy = calcShannonEnt(dataSet) bestInfoGain = 0.0; bestFeature = -1 for i in range(numFeatures): featList = [example[i] for example in dataSet] uniqueVals = set(featList) newEntropy = 0.0 for value in uniqueVals: subDataSet = splitDataSet(dataSet,i,value) prob = len(subDataSet)/float(len(dataSet)) newEntropy += prob * calcShannonEnt(subDataSet) infoGain = baseEntropy - newEntropy if (infoGain > bestInfoGain): bestInfoGain = infoGain bestFeature = i return bestFeature接下来进行测试,结果为:
可见,将myDat的第0个特征作为划分数据集的特征最好。这段代码看起来很简单,可叹我绿豆大的脑容量,足足思考了两天才将这段代码和之前的代码结合起来,理解最终用途,真是惭愧。
3.1.3 递归构建决策树
准备工作做完,现在开始最核心的部分,构建决策树。这里书上有一个划分数据集时的数据路径,初次看的时候没注意,回头再看这个图对理解后面代码很有帮助,整个代码的流程就是按照这个图来走的,所以还是要好好看图:
(看到这张图,第一感觉是自己就是鱼,被决策树这章炸的头昏脑胀,想到后面......不说了,这片池塘的水,我先干为敬,大家随意。)
接下来作者给到了一个majorityCnt的函数,如果你第一次读到这,你一定会像我一样懵,完全没有get到作者的意图,这个majorityCnt函数的作用得结合后面创建树的代码理解,先给出代码:
def majorityCnt(classList): classCount = {} for vote in classList: if vote not in classCount.keys():classCount[vote] = 0 classCount[vote] += 1 sortedClassCount = sorted(classCount.items(),\ key = operator.itemgetter(1),reverse = True) return sortedClassCount[0][0]根据后面“种树”代码,我们可知这的classList对象是:
classList = [example[-1] for example in dataSet]看看这个函数的处理结果:
至于为什么这么处理,是因为”种树“过程中,作者说了:
if len(dataSet[0]) == 1即如果数据集只有一个特征,那么你还费这么大劲干什么,直接看看这一个特征中哪个特征值最多就完了。好吧,这个函数就这样了,接下来开始”种树“。(对了,这段函数要执行,需要在trees.py中导入:
import operator
最重要的”种树“代码:
def createTree(dataSet,labels): classList = [example[-1] for example in dataSet] if classList.count(classList[0]) == len(classList): return classList[0] if len(dataSet[0]) == 1: return majorityCnt(classList) bestFeat = chooseBestFeatureToSplit(dataSet) bestFeatLabel = labels[bestFeat] myTree = {bestFeatLabel:{}} subLabels = labels[:] del(subLabels[bestFeat]) featValues = [example[bestFeat] for example in dataSet] uniqueVals = set(featValues) for value in uniqueVals: myTree[bestFeatLabel][value] = createTree(splitDataSet(dataSet, bestFeat, value),subLabels) return myTree这段代码基本上按照原书上照敲的,但是做了一些改动,原书中有个极为潇洒的操作:
del(labels[bestFeat])这个操作太过潇洒,最终的结果是你的labels列表改变了,可能之后的操作会受到很大影响,所以像我这样的菜鸡选手,还是不用的为好,要不出错了找都找不到,所以综合网上大佬们的意见,这里处理成:
subLabels = labels[:] del(subLabels[bestFeat])具体代码解释我就不写了,网上不少详细的解释,这里贴一个 点我参观代码解释。
测试下代码:
和原书上一样。
3.2 绘制树形图
3.2.1 Matplotlib 注解
直接贴代码:
import matplotlib.pyplot as plt decisionNode = dict(boxstyle = 'sawtooth',fc = '0.8') leafNode = dict(boxstyle = 'round4',fc = '0.8') arrow_args = dict(arrowstyle = '<-') def plotNode(nodeTxt,centerPt,parentPt,nodeType): createPlot.axl.annotate(nodeTxt,xy = parentPt,xycoords = 'axes fraction',\ xytext = centerPt,textcoords = 'axes fraction',\ va = 'center',ha = 'center',bbox = nodeType,arrowprops = arrow_args) def createPlot(): fig = plt.figure(1,facecolor = 'white') fig.clf() createPlot.axl = plt.subplot(111, frameon=False) plotNode('决策节点',(0.5,0.1),(0.1,0.5),decisionNode) plotNode('叶节点',(0.8,0.1),(0.3,0.8),leafNode) plt.show()输出结果:
说实话,这段代码看不懂,照着敲的。可以看到图中的中文显示出错,这是由于默认字体中没有中文字体,措施是在treePlotter
中添加以下代码:
from pylab import * mpl.rcParams['font.sans-serif'] = ['SimHei']修改之后结果为:
3.2.2 构造注解树
获取叶节点的数目和树的层数,代码:
def getNumLeafs(myTree): numLeafs = 0 firstStr = myTree.keys()[0] secondDict = myTree[firstStr] for key in secondDict.keys(): if type(secondDict[key]).__name__ == 'dict': numLeafs += getNumLeafs(secondDict[key]) else: numLeafs += 1 return numLeafs def getTreeDepth(myTree): maxDepth = 0 firstStr = myTree.keys()[0] secondDict = myTree[firstStr] for key in secondDict.keys(): if type(secondDict[key]).__name__ == 'dict': thisDepth = 1 + getTreeDepth(secondDict[key]) else: thisDepth = 1 if thisDepth > maxDepth: maxDepth = thisDepth return maxDepth具体解释可以看@ SnailTyan大佬的代码解释, http://blog.csdn.net/quincuntial/article/details/50477508
原书作者为了节省大家的时间,在treePlotter.py中定义一个retrieveTree的函数来输出预先存储的额树信息,代码如下:
def retrieveTree(i): listOfTrees = [{'no surfacing': {0:'no',1:{'flippers': \ {0:'no',1:'yes'}}}}, {'no surfacing': {0:'no',1:{'flippers':\ {0:{'head':{0:'no',1:'yes'}},\ 1:'no'}}}} ] return listOfTrees[i]接下来到了测试时间,在你按照原书敲下测试代码后
myTree = treePlotter.retrieveTree(0) treePlotter.getNumLeafs(myTree) treePlotter.getTreeDepth(myTree)你会发现程序报错了 .......错误原因是:'dict_keys' object does not support indexing ,好吧好吧,多亏了@ chienchia 的普及: 在python2.x中,dict.keys()返回一个列表,在python3.x中,dict.keys()返回一个dict_keys对象,比起列表,这个对象的行为更像是set,所以不支持索引的。
解决方案:list(dict.keys())[index]
so,将获取叶节点数目和树的层数的代码更改为:
def getNumLeafs(myTree): numLeafs = 0 firstStr = list(myTree.keys())[0]#修改行 secondDict = myTree[firstStr] for key in secondDict.keys(): if type(secondDict[key]).__name__ == 'dict': numLeafs += getNumLeafs(secondDict[key]) else: numLeafs += 1 return numLeafs def getTreeDepth(myTree): maxDepth = 0 firstStr = list(myTree.keys())[0] #修改行 secondDict = myTree[firstStr] for key in secondDict.keys(): if type(secondDict[key]).__name__ == 'dict': thisDepth = 1 + getTreeDepth(secondDict[key]) else: thisDepth = 1 if thisDepth > maxDepth: maxDepth = thisDepth return maxDepth好了,这次测试结果为:
画图的的准备工作做完了,下面更新画图这部分的代码:
def plotMidText(cntrPt, parentPt, txtString): xMid = (parentPt[0] - cntrPt[0]) / 2.0 + cntrPt[0] yMid = (parentPt[1] - cntrPt[1]) / 2.0 + cntrPt[1] createPlot.axl.text(xMid,yMid,txtString) def plotTree(myTree,parentPt,nodeTxt): numLeafs = getNumLeafs(myTree) depth = getTreeDepth(myTree) firstStr = list(myTree.keys())[0] cntrPt = (plotTree.xOff + (1.0 + float(numLeafs))/2.0/plotTree.totalW,\ plotTree.yOff) plotMidText(cntrPt,parentPt,nodeTxt) plotNode(firstStr, cntrPt,parentPt,decisionNode) secondDict = myTree[firstStr] plotTree.yOff = plotTree.yOff - 1.0/plotTree.totalD for key in secondDict.keys(): if type(secondDict[key]).__name__ == 'dict': plotTree(secondDict[key],cntrPt,str(key)) else: plotTree.xOff = plotTree.xOff + 1.0/plotTree.totalW plotNode(secondDict[key],(plotTree.xOff,plotTree.yOff),\ cntrPt,leafNode) plotMidText((plotTree.xOff,plotTree.yOff),cntrPt,str(key)) plotTree.yOff = plotTree.yOff + 1.0/plotTree.totalD def createPlot(inTree): fig = plt.figure(1,facecolor='white') fig.clf() axprops = dict(xticks = [],yticks = []) createPlot.axl = plt.subplot(111,frameon = False, **axprops) plotTree.totalW = float(getNumLeafs(inTree)) plotTree.totalD = float(getTreeDepth(inTree)) plotTree.xOff = -0.5/plotTree.totalW plotTree.yOff = 1.0 plotTree(inTree,(0.5,1.0),' ') plt.show()最终测试结果为:
3.3测试和存储分类器
3.3.1测试算法:使用决策树执行分类(这块的东西没看懂,不知道这块的处理目的是什么)
代码:
def classify(inpuTree,featLabels,testVec): firstStr = list(inputTree.keys())[0] secondDict = inpuTree[firstStr] featIndex = featLabels.index(firstStr) for key in secondDict.keys(): if testVec[featIndex] == key: if type(secondDict[key]).__name__ == 'dict': classLabel = classify(secondDict[key],featLabels,testVec) else: classLabel = secondDict[key] return classLabel测试结果:
3.3.2使用算法:决策树的存储
这段代码基本上与原书一样,但是个别位置需要修改,否则程序会报错
代码如下:
def storeTree(inputTree,filename): import pickle fw = open(filename,'wb') pickle.dump(inputTree,fw) fw.close() def grabTree(filename): import pickle fr = open(filename,'rb') return pickle.load(fr)验证一下:
3.4 示例:使用决策树预测隐形眼镜类型
照旧敲代码:
import os os.chdir('D:\xx\xx\MLiA_SourceCode') os.listdir() fr = open('lenses.txt') lenses = [inst.strip().split('\t') for inst in fr.readlines()] lensesLabels = ['age','prescript','astigmatic','tearRate'] lensesTree = trees.createTree(lenses,lensesLabels) lensesTree treePlotter.createPlot(lensesTree)结果如图:
结果看起来挺像回事的。
我提一些自己的想法:
首先来看看lenses.txt文件的样子:
再回过头看看作者代码,代码都是针对具体数据结构写的,所以在具体现实决策树应用中,难点之一就是准备数据这一步骤。如何针对具体数据对代码进行优化,实际上需要丰富的经验。