版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/BabyFish13/article/details/81027074
本次的解析，只是将日志中的所需要的字段给析取出来。有些请求参数可能没有提取，相对而言，下一篇所述的那种方式更好更合适一些。
1、示例日志文件
Mac下的文件
/Users/nisj/Downloads/localhost_access_log.min.txt
/Users/nisj/PycharmProjects/BiDataProc/love/accessLogPhase1.py
本方法，之前有做过类似的。 数据解析出最终的样式后，根据需要装载进行hive库中供数据分析后续使用。
附：最终的数据样式
1、示例日志文件
Mac下的文件
/Users/nisj/Downloads/localhost_access_log.min.txt
112.123.103.141 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appSource=212&_appVersion=5.0.3&_appkey=CSAndroid&_identifier=127793878&_imei=868981032402390&_loc=117.492657%2C30.653728&_locRadius=178.0&_locType=bd09ll&_sign=a7a3d52e7efe0256c9c67c9de78d8ae6&_t=1529690399659&_wsign=a360e9863060bcd77167ab096619115799316d00&codes=10006%2C10007%2C10009%2C10008&device_api_int=27&device_board=k71v1_64_bsp&device_brand=vivo&device_codename=REL&device_cpuabi=arm64-v8a&device_cpuabi2=armeabi-v7a&device_cs=ff656d930ed6b0f859003ff011b13a20&device_devicename=PD1801&device_display=O11019+release-keys&device_ds=ff656d930ed6b0f859003ff011b13a20&device_es=38b8caf769c58d77a6c260ad52a892baa&device_finger=vivo%2FPD1801%2FPD1801%3A8.1.0%2FO11019%2F1527863420%3Auser%2Frelease-keys&device_fsa=bbc8151c9533667d4d61749470137a4b223&device_hardware=mt6771&device_host=compiler17&device_id=3d6b396903002f35&device_incremental=1527863420&device_mac=020000000000&device_manufacturer=vivo&device_model=vivo+X21i+A&device_product=PD1801&device_ram=6008250368&device_release=8.1.0&device_serial=55FQKFGAWS65SKNJ&device_tags=release-keys&device_type=user&device_user=compiler&device_versionid=O11019&os_version=27&screenHeight=2208&screenWidth=1080&token=1180bcd9a93cf9e8g2bd99ce8 HTTP/1.1 200 48 0.002 - - Android ChushouTV/5.0.3
111.32.72.4 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appSource=212&_appVersion=5.0.4&_appkey=CSAndroid&_identifier=8068017&_imei=868512031137750&_loc=117.27557%2C39.064607&_locRadius=40.0&_locType=bd09ll&_sign=1eb945b3563e3efb5efab476a879e8a5&_t=1529690399786&_wsign=88db3c161755e3abb7947661b127cb95d1d46813&_xappVersion=10601&codes=10006%2C10007%2C10009%2C10008&device_api_int=25&device_board=msm8953&device_brand=vivo&device_codename=REL&device_cpuabi=arm64-v8a&device_cpuabi2=armeabi-v7a&device_cs=c8f51209714a3d702b07959b3f0515f1&device_devicename=PD1708&device_display=N2G47H+release-keys&device_ds=c8f51209714a3d702b07959b3f0515f1&device_es=34f41d329562eb6dafce10632af7871ea&device_finger=vivo%2FPD1708%2FPD1708%3A7.1.2%2FN2G47H%2Fcompil05152054%3Auser%2Frelease-keys&device_fsa=a696445f2b9a4129ad4d5fcd3f497a1b286&device_hardware=qcom&device_host=compiler028&device_id=276865e6a5b67878&device_incremental=eng.compil.20180515.205412&device_mac=020000000000&device_manufacturer=vivo&device_model=vivo+Y79A&device_product=PD1708&device_ram=3763171328&device_release=7.1.2&device_serial=4415da1e&device_tags=release-keys&device_type=user&device_user=compiler&device_versionid=N2G47H&os_version=25&screenHeight=1356&screenWidth=720&token=5c5ad6639c0aca6ag51e8336a HTTP/1.1 200 48 0.001 - - Android ChushouTV/5.0.4
182.138.212.243 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/feedback.htm?advertTime=1529678125541&advertPrimaryKey=1529678125541326&opType=1&advertId=1028&positionId=22&scheduleId=118&marketId=1&_appEnv=1&_appSource=811&_appVersion=5.0.2.26526&_appkey=CSIos&_identifier=0F564805-6266-4D15-B175-499706037464&_idfa=1F907805-77EE-4E2D-8669-A5A8C8A08A86&_loc=104.057351%2C30.710412&_locRadius=10.000000&_locType=bd09ll&_sign=5c0e290cf5c1f33fb6a053dbaa9376eb&_t=1529690399970&_wsign=09980838df4f5563fde1cf72f3732df2bf5e01db&advertId=1028&advertPrimaryKey=1529678125541326&advertTime=1529678125541&device_cs=6354a91b4a0d0579b5ab6b3a85e245cb&device_ds=0F564805-6266-4D15-B175-499706037464&device_es=hb3079bc8ba8e08328fd4f10bd36cbeb3&device_model=iPhone10%2C3&device_release=11.4&device_user=%E7%8E%8B%E5%8B%87%E7%9A%84%20iPhone&marketId=1&opType=1&positionId=22&scheduleId=118&token=71f9bea869199147g5236aa47 HTTP/1.1 200 33 0.000 - - Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15F79 iOS ChushouTv/5.0.2.26526
182.138.212.243 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/feedback.htm?advertTime=1529690399951&advertPrimaryKey=1529690399951487&opType=1&advertId=-1&positionId=4&scheduleId=79&marketId=2&_appEnv=1&_appSource=811&_appVersion=5.0.2.26526&_appkey=CSIos&_identifier=0F564805-6266-4D15-B175-499706037464&_idfa=1F907805-77EE-4E2D-8669-A5A8C8A08A86&_loc=104.057351%2C30.710412&_locRadius=10.000000&_locType=bd09ll&_sign=4530f1951a3b71b663172e43dee6100e&_t=1529690400089&_wsign=5be3669bf8dba6b131f970dcef69f8910aee21bd&advertId=-1&advertPrimaryKey=1529690399951487&advertTime=1529690399951&device_cs=6354a91b4a0d0579b5ab6b3a85e245cb&device_ds=0F564805-6266-4D15-B175-499706037464&device_es=hb3079bc8ba8e08328fd4f10bd36cbeb3&device_model=iPhone10%2C3&device_release=11.4&device_user=%E7%8E%8B%E5%8B%87%E7%9A%84%20iPhone&marketId=2&opType=1&positionId=4&scheduleId=79&token=71f9bea869199147g5236aa47 HTTP/1.1 200 33 0.001 - - Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15F79 iOS ChushouTv/5.0.2.26526
27.197.216.94 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appEnv=1&_appSource=811&_appVersion=5.0.2.26526&_appkey=CSIos&_identifier=7D14547D-25BD-4FE4-8219-F9402CD96D7C&_idfa=968AF15C-9FBB-45CC-94FC-A7F6C7AE74D1&_locRadius=10.000000&_locType=bd09ll&_sign=e127a079c980b8877c4c2ac4806aabe3&_t=1529690399894&_wsign=8e200b73dacd2d340af4333bd9e6f4c3ba08b746&codes=20011&device_cs=e427a59ba4f043cc7b1f0ac8ab83d84e&device_ds=7D14547D-25BD-4FE4-8219-F9402CD96D7C&device_es=h531740cc6561647845c64f68921c1a42&device_model=iPhone9%2C1&device_release=11.2.6&device_user=iPhone%20%282%29&screenHeight=667&screenWidth=375&token=71a0b317df9d5861g46cd6161 HTTP/1.1 200 48 0.001 - - Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_6 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Mobile/15D100 iOS ChushouTv/5.0.2.26526
121.207.60.108 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/feedback.htm?_appSource=568&_appVersion=5.0.2.20594&_appkey=CSAndroid&_identifier=103726604&_imei=864034032232252&_loc=4.9E-324%2C4.9E-324&_locRadius=0.0&_locType=bd09ll&_sign=8a78a3f580ef638c5347ea597bc329f0&_t=1529690398293&_wsign=5b30ce86f84974e5c2be5d31fc3f927a8d51bbd3&advertId=-1&advertPrimaryKey=1529690395509908&advertTime=1529690395509&device_api_int=24&device_board=STF&device_brand=HONOR&device_codename=REL&device_cpuabi=arm64-v8a&device_cpuabi2=armeabi-v7a&device_cs=23c75271ec687bebf5e3a622275c3d52&device_devicename=HWSTF&device_display=STF-TL10C01B155&device_ds=23c75271ec687bebf5e3a622275c3d52&device_es=381f3fc7874745e9b5ff990eee2984988&device_finger=HONOR%2FSTF-TL10%2FHWSTF%3A7.0%2FHUAWEISTF-TL10%2FC01B155%3Auser%2Frelease-keys&device_fsa=706bd931f1766b5c11f5a749dd08a3d42d4&device_hardware=hi3660&device_host=LFG1000749995&device_id=d97b1fcabc2ec44b&device_incremental=C01B155&device_mac=020000000000&device_manufacturer=HUAWEI&device_model=STF-TL10&device_product=STF-TL10&device_ram=6009204736&device_release=7.0&device_serial=ERMDU17719004449&device_tags=release-keys&device_type=user&device_user=test&device_versionid=HUAWEISTF-TL10&marketId=2&opType=6&os_version=24&positionId=3&scheduleId=78 HTTP/1.1 200 33 0.001 - - Android ChushouTV/5.0.2.20594
119.85.20.4 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appSource=209&_appVersion=5.0.4&_appkey=CSAndroid&_identifier=2011225951&_imei=A1000049D9D9C8&_loc=108.308172%2C31.350436&_locRadius=47.356655&_locType=bd09ll&_sign=7f8c139fe129bdd22ecef2c62a0aea9a&_t=1529690399607&_wsign=4c6f92bce852cc29d359672cb8cf514d072a00a6&_xappVersion=10601&codes=10003&device_api_int=21&device_board=msm8916&device_brand=vivo&device_codename=REL&device_cpuabi=armeabi-v7a&device_cpuabi2=armeabi&device_cs=0e71fbf721118360ddcc9dd370fdea26&device_devicename=PD1421V&device_display=LRX22G+release-keys&device_ds=0e71fbf721118360ddcc9dd370fdea26&device_es=310f678fed9bfe961768042053abf89bd&device_finger=vivo%2FPD1421V%2FPD1421V%3A5.0.2%2FLRX22G%2Fcompiler08021524%3Auser%2Frelease-keys&device_fsa=c2710e37b759813a7e4608914c59d7b62e3&device_hardware=qcom&device_host=compiler033&device_id=91fc862c745108ae&device_incremental=eng.compiler.20160802&device_mac=3ca348953c68&device_manufacturer=vivo&device_model=vivo+X5Pro+V&device_product=PD1421V&device_ram=2003296256&device_release=5.0.2&device_serial=440bdedf&device_tags=release-keys&device_type=user&device_user=compiler&device_versionid=LRX22G&os_version=21&screenHeight=1920&screenWidth=1080&token=8d48df74439a6e63g522eeb9d HTTP/1.1 200 48 0.001 - - Android ChushouTV/5.0.4
117.136.79.90 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appSource=212&_appVersion=5.0.3&_appkey=CSAndroid&_identifier=897957993&_imei=867366030842857&_loc=113.319278%2C22.732977&_locRadius=124.0&_locType=bd09ll&_sign=2c4e728b5def29c7c9f8b6ac174dd7a1&_t=1529690399851&_wsign=acfe6651777e0cab2f56f518da0d4b0f78ade373&codes=10006%2C10007%2C10009%2C10008&device_api_int=25&device_board=sdm660&device_brand=vivo&device_codename=REL&device_cpuabi=arm64-v8a&device_cpuabi2=armeabi-v7a&device_cs=cc34e938005b5e24815594d717049556&device_devicename=PD1709&device_display=NMF26X+release-keys&device_ds=cc34e938005b5e24815594d717049556&device_es=3823b17c729b058f9f0874c8b5e000bbc&device_finger=vivo%2FPD1709%2FPD1709%3A7.1.1%2FNMF26X%2Fcompil05151836%3Auser%2Frelease-keys&device_fsa=284399fe8a67b15f8fb48b4d4fb9b3ec288&device_hardware=qcom&device_host=compiler031&device_id=d985e31284298e19&device_incremental=eng.compil.20180515.183659&device_mac=020000000000&device_manufacturer=vivo&device_model=vivo+X20&device_product=PD1709&device_ram=3915128832&device_release=7.1.1&device_serial=9ba6c868&device_tags=release-keys&device_type=user&device_user=compiler&device_versionid=NMF26X&os_version=25&screenHeight=2034&screenWidth=1080&token=b5256f7df7f14d93g490cd76d HTTP/1.1 200 48 0.002 - - Android ChushouTV/5.0.3
111.3.34.46 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appEnv=1&_appSource=811&_appVersion=5.0.2.26526&_appkey=CSIos&_identifier=8A383EC6-BAF0-49A0-B71F-6DD9A478B462&_idfa=89BA8D0B-99A9-4B02-8DEF-685957C2CFAC&_loc=120.584361%2C30.645005&_locRadius=10.000000&_locType=bd09ll&_sign=317dc2e17fb338b55109547d0a617ffe&_t=1529690399965&_wsign=1cb470802b4abee1172c1a498ac7e658a3ac6f6a&codes=20006%2C20007&device_cs=6b9ed8e460a3a427ef3df1489bcd6b4f&device_ds=8A383EC6-BAF0-49A0-B71F-6DD9A478B462&device_es=h26261c09d067d22a3aac13df1bda725b&device_model=iPad6%2C11&device_release=11.4&device_user=iPad&screenHeight=480&screenWidth=320&token=641381dbfca5d674g49280a74 HTTP/1.1 200 48 0.001 - - Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15F79 iOS ChushouTv/5.0.2.26526
113.247.80.43 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appSource=313&_appVersion=5.0.3&_appkey=CSAndroid&_identifier=291347367&_imei=867306030507209&_loc=112.409116%2C28.114568&_locRadius=1000.0&_locType=bd09ll&_sign=4ef9665b3ccb7265077052fa82c67591&_t=1529690399538&_wsign=069c0ee36a7c93425273a093b372c0c0c43d0066&codes=10003&device_api_int=25&device_board=msm8953&device_brand=xiaomi&device_codename=REL&device_cpuabi=arm64-v8a&device_cpuabi2=armeabi-v7a&device_cs=f72a7db9bc3572a901e009f269b053f7&device_devicename=tiffany&device_display=QL1515-tiffany-build-20180525212811&device_ds=f72a7db9bc3572a901e009f269b053f7&device_es=3dd3e4e4b6c8fe90fa02f71481ad4a302&device_finger=xiaomi%2Ftiffany%2Ftiffany%3A7.1.2%2FN2G47H%2FV9.6.1.0.NDBCNFD%3Auser%2Frelease-keys&device_fsa=11ba46a58786fa35de03186112250a452b1&device_hardware=qcom&device_host=c3-miui-ota-bd67.bj&device_id=e1f871c0308c621e&device_incremental=V9.6.1.0.NDBCNFD&device_mac=020000000000&device_manufacturer=Xiaomi&device_model=MI+5X&device_product=tiffany&device_ram=3772723200&device_release=7.1.2&device_serial=5eaa9af90604&device_tags=release-keys&device_type=user&device_user=builder&device_versionid=N2G47H&os_version=25&screenHeight=1920&screenWidth=1080&token=2720d4f66ff05501g50461d01 HTTP/1.1 200 48 0.001 - - Android ChushouTV/5.0.3
116.25.40.54 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appEnv=1&_appSource=811&_appVersion=5.0.2.26526&_appkey=CSIos&_identifier=8F092F29-508A-47D8-A0A6-CACF49DC0020&_idfa=E4B17751-B2F5-438C-8666-68375D95A02F&_locRadius=10.000000&_locType=bd09ll&_sign=26bd34cc7ade33bd4b81d8d3cfaf8196&_t=1529690397989&_wsign=0fcecfc7576eb5b1cfd00839aef4f55d7e79d917&codes=20009%2C20008&device_cs=37aa9bbb6251aef5dc09896aacc3c575&device_ds=8F092F29-508A-47D8-A0A6-CACF49DC0020&device_es=h54a988e7a71fdfd6f16f508d46be4bd3&device_model=iPad2%2C2&device_release=9.3.5&device_user=iPad&screenHeight=480&screenWidth=320&token=e02a43b80262bb9fg5046b361 HTTP/1.1 200 48 0.001 - - Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_5 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13G36 iOS ChushouTv/5.0.2.26526
182.138.212.243 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/feedback.htm?advertTime=1529690399769&advertPrimaryKey=1529690399769488&opType=1&advertId=1061&positionId=22&scheduleId=114&marketId=1&_appEnv=1&_appSource=811&_appVersion=5.0.2.26526&_appkey=CSIos&_identifier=0F564805-6266-4D15-B175-499706037464&_idfa=1F907805-77EE-4E2D-8669-A5A8C8A08A86&_loc=104.057351%2C30.710412&_locRadius=10.000000&_locType=bd09ll&_sign=e7cc8e2fb8bc6cc5ee79de1689cb1d01&_t=1529690400141&_wsign=ff9d5d7cba5678e593f81f43e18d76dc9f8c2afd&advertId=1061&advertPrimaryKey=1529690399769488&advertTime=1529690399769&device_cs=6354a91b4a0d0579b5ab6b3a85e245cb&device_ds=0F564805-6266-4D15-B175-499706037464&device_es=hb3079bc8ba8e08328fd4f10bd36cbeb3&device_model=iPhone10%2C3&device_release=11.4&device_user=%E7%8E%8B%E5%8B%87%E7%9A%84%20iPhone&marketId=1&opType=1&positionId=22&scheduleId=114&token=71f9bea869199147g5236aa47 HTTP/1.1 200 33 0.001 - - Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15F79 iOS ChushouTv/5.0.2.26526
106.5.27.196 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appEnv=1&_appSource=811&_appVersion=5.0.2.26526&_appkey=CSIos&_identifier=FAFBF3E0-3FAD-475C-A05D-FC2E5FCC3C3B&_idfa=950C0768-CF12-49D9-9288-8EBB3493AB3A&_loc=115.015795%2C26.844929&_locRadius=10.000000&_locType=bd09ll&_sign=4dc73501e3bfb8625ceeffa967fb552e&_t=1529690399393&_wsign=fb436d3166b65c9e7afee13c3c27a0373b52ec60&codes=20006%2C20007&device_cs=e9347fc7e65121d0dc4efd7d397e740a&device_ds=FAFBF3E0-3FAD-475C-A05D-FC2E5FCC3C3B&device_es=h57f5e8bbed3e3f57f974b123c5fbc0f6&device_model=iPhone9%2C2&device_release=11.3&device_user=iPhone%207%20Plus&screenHeight=736&screenWidth=414&token=e458c3ee44aa903fg9cec1 HTTP/1.1 200 48 0.001 - - Mozilla/5.0 (iPhone; CPU iPhone OS 11_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E216 iOS ChushouTv/5.0.2.26526
118.212.200.198 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appSource=311&_appVersion=5.0.3&_appkey=CSAndroid&_identifier=660838218&_imei=864320036032859&_loc=115.809154%2C28.696803&_locRadius=353.0&_locType=bd09ll&_sign=f0e9f2be33428d97ce8a7bea1d64a66e&_t=1529690399577&_wsign=1e1ee6b89eecb272e79870bc6c1567e4bfda299f&codes=10006%2C10007%2C10009%2C10008&device_api_int=23&device_board=msm8953&device_brand=OPPO&device_codename=REL&device_cpuabi=arm64-v8a&device_cpuabi2=armeabi-v7a&device_cs=65f7826bc3e1635d0a6c7a9e19a84cf7&device_devicename=R9sk&device_display=R9sk_11_A.25_180418&device_ds=65f7826bc3e1635d0a6c7a9e19a84cf7&device_es=3b1ebe33f43ea5d253a7803dabad63b59&device_finger=OPPO%2FR9sk%2FR9sk%3A6.0.1%2FMMB29M%2F1522231161%3Auser%2Frelease-keys&device_fsa=21f28384a617e0945a8cdf4abf4794602b4&device_hardware=qcom&device_host=ubuntu-121-139&device_id=f6762dec1ae6601b&device_incremental=eng.root.20180418.162155&device_mac=020000000000&device_manufacturer=OPPO&device_model=OPPO+R9sk&device_product=R9sk&device_ram=3769978880&device_release=6.0.1&device_serial=bb30aa3d&device_tags=dev-keys&device_type=user&device_user=root&device_versionid=MMB29M&os_version=23&screenHeight=1920&screenWidth=1080&token=57c77ca2e66f0dbg51f49adb HTTP/1.1 200 48 0.001 - - Android ChushouTV/5.0.3
117.136.79.127 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/feedback.htm?_appSource=306&_appVersion=5.0.4&_appkey=CSAndroid&_identifier=477494778&_imei=862266034478358&_loc=113.803454%2C22.746902&_locRadius=102.23267&_locType=bd09ll&_sign=e8692b993c7fe4f078961472ee4038a7&_t=1529690399349&_wsign=f9b56b7b5c7f31624bc6b075c35b01ece9ae88a6&_xappVersion=10601&advertId=-1&advertPrimaryKey=1529690398001848&advertTime=1529690398001&device_api_int=24&device_board=EVA-TL00&device_brand=HUAWEI&device_codename=REL&device_cpuabi=arm64-v8a&device_cpuabi2=armeabi-v7a&device_cs=839b394e24599fcc7cc8c0dcf4cfbf83&device_devicename=HWEVA&device_display=EVA-TL00C01B399SP17&device_ds=61ecb3763843971d54c8ab67b84eae48&device_es=3150dcf4fe3d8af1ac900b4acf3ebce6f&device_finger=HUAWEI%2FEVA-TL00%2FHWEVA%3A7.0%2FHUAWEIEVA-TL00%2FC01B399SP17%3Auser%2Frelease-keys&device_fsa=8621a630cad8a9646dd01e23bfb81446244&device_hardware=hi3650&device_host=huawei-105-38&device_id=bbba118f552c9f9c&device_incremental=C01B399SP17&device_mac=020000000000&device_manufacturer=HUAWEI&device_model=EVA-TL00&device_product=EVA-TL00&device_ram=2915397632&device_release=7.0&device_serial=B2T0216820010970&device_tags=release-keys&device_type=user&device_user=test&device_versionid=HUAWEIEVA-TL00&marketId=2&opType=6&os_version=24&positionId=3&scheduleId=78&token=49cf8aeee36b58bfg5044d3bf HTTP/1.1 200 33 0.001 - - Android ChushouTV/5.0.4
27.197.216.94 172.16.1.72 - - [23/Jun/2018:02:00:00 +0800] GET /api/advert/getList.htm?_appEnv=1&_appSource=811&_appVersion=5.0.2.26526&_appkey=CSIos&_identifier=7D14547D-25BD-4FE4-8219-F9402CD96D7C&_idfa=968AF15C-9FBB-45CC-94FC-A7F6C7AE74D1&_locRadius=10.000000&_locType=bd09ll&_sign=4d4eec8156a40ea1162111f6b72602c2&_t=1529690400323&_wsign=03e4805d9b0e516911121c23f387171544e22654&codes=20011&device_cs=e427a59ba4f043cc7b1f0ac8ab83d84e&device_ds=7D14547D-25BD-4FE4-8219-F9402CD96D7C&device_es=h531740cc6561647845c64f68921c1a42&device_model=iPhone9%2C1&device_release=11.2.6&device_user=iPhone%20%282%29&screenHeight=667&screenWidth=375&token=71a0b317df9d5861g46cd6161 HTTP/1.1 200 48 0.001 - - Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_6 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Mobile/15D100 iOS ChushouTv/5.0.2.26526/Users/nisj/PycharmProjects/BiDataProc/love/accessLogPhase1.py
# -*- coding=utf-8 -*-
import os, re, time
def readFileFromOs(filePath):
with open('{filePath}'.format(filePath=filePath), 'r') as f:
fileDatas = f.read().split('\n')
PendingData_list = []
for pending_list in fileDatas:
pending = re.split(' ',
pending_list.replace('\n', '').replace('?', '?&').replace('&&', '&').replace(' HTTP/1.1',
''))
PendingData_list.append(pending)
for PendingDataL in PendingData_list:
for pdl in PendingDataL:
if 'GET /' in pdl:
PendingDataD = eval(
'{\'' + pdl.replace('&', '\',\'').replace('=', '\':\'').replace('GET ', 'request\':\'') + '\'}')
# print PendingDataD
token = ""
_appSource = ""
_appkey = ""
_identifier = ""
request = ""
for key, value in PendingDataD.iteritems():
if key == 'token':
token = value
token = token[token.find('g') + 1:]
if key == '_appSource':
_appSource = value
if key == '_appkey':
_appkey = value
if key == '_identifier':
_identifier = value
if key == 'request':
request = value
print token, _appSource, _appkey, _identifier, request
readFileFromOs(filePath='/Users/nisj/Downloads/localhost_access_log.min.txt')本方法，之前有做过类似的。 数据解析出最终的样式后，根据需要装载进行hive库中供数据分析后续使用。
附：最终的数据样式
/usr/bin/python /Users/nisj/PycharmProjects/BiDataProc/love/accessLogPhase1.py
2bd99ce8 212 CSAndroid 127793878 /api/advert/getList.htm?
51e8336a 212 CSAndroid 8068017 /api/advert/getList.htm?
5236aa47 811 CSIos 0F564805-6266-4D15-B175-499706037464 /api/advert/feedback.htm?
5236aa47 811 CSIos 0F564805-6266-4D15-B175-499706037464 /api/advert/feedback.htm?
46cd6161 811 CSIos 7D14547D-25BD-4FE4-8219-F9402CD96D7C /api/advert/getList.htm?
568 CSAndroid 103726604 /api/advert/feedback.htm?
522eeb9d 209 CSAndroid 2011225951 /api/advert/getList.htm?
490cd76d 212 CSAndroid 897957993 /api/advert/getList.htm?
49280a74 811 CSIos 8A383EC6-BAF0-49A0-B71F-6DD9A478B462 /api/advert/getList.htm?
50461d01 313 CSAndroid 291347367 /api/advert/getList.htm?
5046b361 811 CSIos 8F092F29-508A-47D8-A0A6-CACF49DC0020 /api/advert/getList.htm?
5236aa47 811 CSIos 0F564805-6266-4D15-B175-499706037464 /api/advert/feedback.htm?
9cec1 811 CSIos FAFBF3E0-3FAD-475C-A05D-FC2E5FCC3C3B /api/advert/getList.htm?
51f49adb 311 CSAndroid 660838218 /api/advert/getList.htm?
5044d3bf 306 CSAndroid 477494778 /api/advert/feedback.htm?
46cd6161 811 CSIos 7D14547D-25BD-4FE4-8219-F9402CD96D7C /api/advert/getList.htm?
Process finished with exit code 0