背景
工欲善其事,必先利其器。最近笔者的产品又对我提出了很多的非分的需求,数据库没有的字段也非要我导到Excel里。几千条数据,让我一个一个去查,笔者每天加班到死,手里一堆活,于是乎跟产品开始撕X,产品气得说要自己查。虽然表面上产品像我的死党一样,但是毕竟几千条数据,也不好查,况且产品还是个妹纸。表面很冷,但豆腐心肠的我,准备把导出的数据用java导入内存,然后用Httpclient爬取数据,拼装后再导出。
传统的请求方式
发现最新的Httpclient变动还是挺大,于是参考了这篇博客:HttpClient用法–这一篇全了解,该博客的一个示例如下:
public class HttpClientUtils {
private static PoolingHttpClientConnectionManager connectionManager = null;
private static HttpClientBuilder httpBuilder = null;
private static RequestConfig requestConfig = null;
private static int MAXCONNECTION = 10;
private static int DEFAULTMAXCONNECTION = 5;
private static String IP = "cnivi.com.cn";
private static int PORT = 80;
static {
//设置http的状态参数
requestConfig = RequestConfig.custom()
.setSocketTimeout(5000)
.setConnectTimeout(5000)
.setConnectionRequestTimeout(5000)
.build();
HttpHost target = new HttpHost(IP, PORT);
connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(MAXCONNECTION);//客户端总并行链接最大数
connectionManager.setDefaultMaxPerRoute(DEFAULTMAXCONNECTION);//每个主机的最大并行链接数
connectionManager.setMaxPerRoute(new HttpRoute(target), 20);
httpBuilder = HttpClients.custom();
httpBuilder.setConnectionManager(connectionManager);
}
public static CloseableHttpClient getConnection() {
CloseableHttpClient httpClient = httpBuilder.build();
return httpClient;
}
public static HttpUriRequest getRequestMethod(Map<String, String> map, String url, String method) {
List<NameValuePair> params = new ArrayList<NameValuePair>();
Set<Map.Entry<String, String>> entrySet = map.entrySet();
for (Map.Entry<String, String> e : entrySet) {
String name = e.getKey();
String value = e.getValue();
NameValuePair pair = new BasicNameValuePair(name, value);
params.add(pair);
}
HttpUriRequest reqMethod = null;
if ("post".equals(method)) {
reqMethod = RequestBuilder.post().setUri(url)
.addParameters(params.toArray(new BasicNameValuePair[params.size()]))
.setConfig(requestConfig).build();
} else if ("get".equals(method)) {
reqMethod = RequestBuilder.get().setUri(url)
.addParameters(params.toArray(new BasicNameValuePair[params.size()]))
.setConfig(requestConfig).build();
}
return reqMethod;
}
public static void main(String args[]) throws IOException {
Map<String, String> map = new HashMap<String, String>();
map.put("account", "");
map.put("password", "");
HttpClient client = getConnection();
HttpUriRequest post = getRequestMethod(map, "http://cnivi.com.cn/login", "post");
HttpResponse response = client.execute(post);
if (response.getStatusLine().getStatusCode() == 200) {
HttpEntity entity = response.getEntity();
String message = EntityUtils.toString(entity, "utf-8");
System.out.println(message);
} else {
System.out.println("请求失败");
}
}
}
第一感觉是,前端的axios都可以那么简单,我就发送一个Http请求,Java怎么那么麻烦。于是对HttpClient进行了封装。
笔者的请求方式
这里笔者以必应搜索首页抓到的请求为例:https://cn.bing.com/HPImageArchive.aspx?format=js&idx=0&n=1&nc=1563378874917&pid=hp
@Test
public void testGet() {
try {
String url = "https://cn.bing.com/HPImageArchive.aspx?format=js&idx=0&n=1&nc=1563378874917&pid=hp";
R<Map<String,Object>> result = Request.url(url).get();
System.out.println(result.getData());
} catch (Exception e) {
e.printStackTrace();
}
}
打印结果:
{
"images": [{
"startdate": "20190716",
"fullstartdate": "201907161600",
"enddate": "20190717",
"url": "/th?id=OHR.Narrenmuehle_ZH-CN5582540867_1920x1080.jpg&rf=LaDigue_1920x1080.jpg&pid=hp",
"urlbase": "/th?id=OHR.Narrenmuehle_ZH-CN5582540867",
"copyright": "Narrenmuehle(愚人磨坊风车),德国Dülken (© dpa picture alliance/Alamy)",
"copyrightlink": "/search?q=Narrenmuehle&form=hpcapt&mkt=zh-cn",
"title": "",
"quiz": "/search?q=Bing+homepage+quiz&filters=WQOskey:%22HPQuiz_20190716_Narrenmuehle%22&FORM=HPQUIZ",
"wp": true,
"hsh": "d9b06221f1008b2d9208638a18b9b55c",
"drk": 1,
"top": 1,
"bot": 1,
"hs": []
}],
"tooltips": {
"loading": "正在加载...",
"previous": "上一个图像",
"next": "下一个图像",
"walle": "此图片不能下载用作壁纸。",
"walls": "下载今日美图。仅限用作桌面壁纸。"
}
}
笔者还有几个带body, header, 以及返回结果泛型自动转化的例子等,这里再列举几个:
@Test
public void testJsonGet() {
try {
String url = "http://local.home:9200/_search";
String json = "{\n"
+ " \"query\": {\n"
+ " \"match_all\": {}\n"
+ " }\n"
+ "}";
R result = Request.url(url).body(json).get();
System.out.println(result.getData());
} catch (Exception e) {
e.printStackTrace();
}
}
@Test
public void testJsonGet2() {
try {
String url = "http://local.home:9200/_search";
HashMap<String, Object> f2 = new HashMap<>();
f2.put("match_all", new Object());
HashMap<String, Object> json = new HashMap<>();
json.put("query", f2);
R result = Request.url(url).body(json).get();
System.out.println(result.getData());
} catch (Exception e) {
e.printStackTrace();
}
}
@Test
public void testJsonGet3() {
try {
String url = "http://local.home:9200/_search";
HashMap<String, Object> f2 = new HashMap<>();
f2.put("match_all", new Object());
HashMap<String, Object> json = new HashMap<>();
json.put("query", f2);
R<Map<String, Object>> result = Request.url(url).body(json).dataType(new TypeReference<Map<String, Object>>() {
}).get();
System.out.println(result.getData());
} catch (Exception e) {
e.printStackTrace();
}
}
封装之后的方法,发送一个请求是不是简单很多了。
使用
maven引用
<dependency>
<groupId>com.github.rxyor</groupId>
<artifactId>carp-common-util</artifactId>
<version>1.0.4</version>
</dependency>