首先请大牛们见谅菜鸟重复造轮子的学习方式,本文适合新手看~
下面使用的同步http是HttpClient 3.X 的版本,不过早已 不在维护 ,如果刚开始使用http,建议大家都换成 4.X 版本,别看下面的有关同步http的部分了,4.x效率有质地提高,总结3.X只是因为无奈旧项目还在使用。后面再更新一篇有关4.x的,最新的HttpClient 4.X官方地址: http://hc.apache.org/httpcomponents-client-4.5.x/index.html
但鉴于可能有些旧的系统还是采用3.X版本的HttpClient,所以本文还是先记录下使用方法。
相反下面的异步http是Async Http Client 的 1.9.8 版本,这个版本还是挺好的。API请见: http://asynchttpclient.github.io/async-http-client/apidocs/com/ning/http/client/AsyncHttpClient.html
http使用场景很多,据以往经验,对于客户端来说,我们使用http一般会发出以下几种常见的场景:
- 以get方式请求服务器
- 不带任何参数
- 带上key-value对
- 以post方式请求服务器
- 不带任何参数
- 带上key-value对
- 带上字节数组
- 带上文件
- 带上文件+key-value对
以上的场景一般可以满足一般的需求,然后,我们可以在这基础上扩展一点点:假如遇到一个类似于报表的子系统,主系统要在关键的逻辑链路中“打点”,通过http调用报表子系统记录一些相关的信息时,那么如果我们使用同步http来请求报表子系统的话,一旦报表子系统挂了,那么肯定会影响到主系统的运行。
为了不影响到主系统的运行,我们可以采用“ 异步 ” 的方式通过http(AsyncHttpClient )请求报表子系统,那么即使子系统挂了,对主系统的关键链路的执行也不会产生多大的影响。所以,封装一个http组件,自然而然少不了封装异步http请求。而异步http所能够做的事情,也应该覆盖上面提到的几种场景。
再者,考虑到效率问题,除非有足够的理由,否则每次调用http接口,都创建立一个新的连接,是相当没效率的,所以MultiThreadedHttpConnectionManager 诞生了,HttpClient在内部维护一个 连接池 ,通过MultiThreadedHttpConnectionManager 我们可以设置“默认连接数”、“最大连接数”、“连接超时”、“读取数据超时”等等配置,从而来提高效率。
废话完了,怎么实现以上需求呢。
包的引用:
同步的http我使用的是org.apache.commons.httpclient的HttpClient的3.1版本。
maven配置为:
1
2
3
4
5
6
|
<!-- httpClient -->
<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>
3.1
</version>
</dependency>
|
异步的http我使用的是com.ning.http.client的AsyncHttpClien的1.9.8版本。 注意 ,其需要依赖几个日志相关的组件、分别为log4j、slf4j、slf4j-log4j
maven配置为:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
<!-- slf4j-log4j -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>
1.7
.
7
</version>
</dependency>
<!-- slf4j -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>
1.7
.
5
</version>
</dependency>
<!-- log4j -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>
1.2
.
16
</version>
</dependency>
<!-- 异步IO -->
<dependency>
<groupId>com.ning</groupId>
<artifactId>async-http-client</artifactId>
<version>
1.9
.
8
</version>
</dependency>
|
为了实现连接池,我们通过一个工厂类来生成httpClient,为了上一层方便调用,我们定义了一个接口,规范了同步、异步http应该实现的方法。包结构如下:
一、同步的HttpClient 3.X
从工厂入手,工厂负责初始化httpClient的配置,包括“默认连接数”、“最大连接数”、“连接超时”、“读取数据超时”等等,不同的服务我们应该创建不同的manager,因为不可能我们调服务A和调服务B使用同一套配置是吧,比如超时时间,应该考虑会有所差异。初始化完配置后,把 manager传到实现类,在实现类中new HttpClient。
工厂代码如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
// 专门针对xx服务器的连接管理对象
// 因为不同服务可能超时等参数不用,所以针对不同服务,把连接管理对象区分开来,这只是其中一个
private
static
MultiThreadedHttpConnectionManager xxconnectionManager =
new
MultiThreadedHttpConnectionManager();
static
{
// 专门针对xx服务器的连接参数
xxconnectionManager =
new
MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams paramsSearch =
new
HttpConnectionManagerParams();
paramsSearch.setDefaultMaxConnectionsPerHost(
1000
);
// 默认连接数
paramsSearch.setMaxTotalConnections(
1000
);
// 最大连接数
paramsSearch.setConnectionTimeout(
30000
);
// 连接超时
paramsSearch.setSoTimeout(
20000
);
// 读数据超时
xxconnectionManager.setParams(paramsSearch);
}
/*
* 返回针对XX服务的httpClient包装类
*/
public
static
SyncHttpClientWapperImpl getXXSearchHttpClient() {
return
new
SyncHttpClientWapperImpl(xxconnectionManager);
}
|
注意 一点,这些连接数,超时等的配置,要做要调查工作之后再定夺,是根据访问服务的不同,我们自己的机器能有多少剩余的可用空间的不同而不同的,而不是随随便便就设置一个参数。
实现类的构造方法如下:
1
2
3
4
5
6
7
8
|
private
HttpClient client;
// httpClient
private
final
static
String CHARACTER =
"UTF-8"
;
// 构造器,由工厂调用
public
SyncHttpClientWapperImpl(MultiThreadedHttpConnectionManager connectionManager) {
client =
new
HttpClient(connectionManager);
// 字符集
client.getParams().setParameter(HttpMethodParams.HTTP_CONTENT_CHARSET, CHARACTER);
}
|
这里有一个 挺困惑 的点:HttpClient有必要弄成静态的吗?即直接在工厂里面为每种服务生成一个静态的HttpClient,然后传到实现类?经测试,改成静态的效率并没有提高,在文件传输的测试中,甚至下降了,这个有点困惑,大家可以试一试一起讨论一下。
然后,在实现类中实现各种方法。
第一种,通过URL,以get方式请求服务器,返回字节数组。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
public
byte
[] getWithQueryURL(String queryURL)
throws
HttpClientException {
if
(queryURL ==
null
) {
throw
new
HttpClientException(
"queryURL is null."
);
}
byte
[] newbuf = executeByGet(queryURL);
if
((newbuf ==
null
) || (newbuf.length ==
0
)) {
throw
new
HttpClientException(
"Server response is null: "
+ queryURL);
}
return
newbuf;
}
private
byte
[] executeByGet(String url)
throws
HttpClientException {
HttpMethod method =
new
GetMethod(url);
// RequestHeader
method.setRequestHeader(
"Content-type"
,
"text/html; charset=UTF-8"
);
// 提交请求
try
{
client.executeMethod(method);
}
catch
(Exception e) {
method.releaseConnection();
throw
new
HttpClientException(url, e);
}
// 返回字节流
byte
[] responseBody =
null
;
try
{
responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
}
catch
(IOException e) {
throw
new
HttpClientException(e);
}
finally
{
method.releaseConnection();
}
return
responseBody;
}
|
接着,写一个通用的流解析方法,负责把返回的流解析成字节数组。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
private
byte
[] getBytesFromInpuStream(InputStream instream)
throws
IOException {
ByteArrayOutputStream outstream =
new
ByteArrayOutputStream();
try
{
int
length;
byte
[] tmp =
new
byte
[
8096
];
while
((length = instream.read(tmp)) != -
1
) {
outstream.write(tmp,
0
, length);
}
return
outstream.toByteArray();
}
finally
{
instream.close();
outstream.close();
}
}
|
这样就完成了最简单的get请求的调用了。
第二种:通过URL和paramsMap参数,以post方式请求服务器,返回字节数组。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
public
byte
[] postWithParamsMap( String queryURL, Map<String,String> paramsMap)
throws
HttpClientException{
if
(queryURL ==
null
) {
throw
new
HttpClientException(
"queryURL is null."
);
}
byte
[] newbuf = executeByPostWithParamsMap(queryURL,paramsMap);
if
((newbuf ==
null
) || (newbuf.length ==
0
)) {
throw
new
HttpClientException(
"Server response is null: "
+ queryURL);
}
return
newbuf;
}
private
byte
[] executeByPostWithParamsMap(String URL, Map<String,String> paramsMap)
throws
HttpClientException {
PostMethod method =
new
PostMethod(URL);
// 构造参数
if
(paramsMap !=
null
) {
Set<Entry<String, String>> entrySet = paramsMap.entrySet();
Iterator<Entry<String, String>> iterator = entrySet.iterator();
NameValuePair[] nvps =
new
NameValuePair[paramsMap.size()];
int
i =
0
;
while
(iterator.hasNext()) {
Entry<String, String> entry = iterator.next();
if
(entry.getKey() !=
null
) {
NameValuePair nvp =
new
NameValuePair(entry.getKey(),entry.getValue());
nvps[i++] = nvp;
}
}
method.setRequestBody(nvps);
}
// RequestHeader,key-value对的话,httpClient自动带上application/x-www-form-urlencoded
method.setRequestHeader(
"Content-type"
,
"application/x-www-form-urlencoded; charset=UTF-8"
);
// 提交请求
try
{
client.executeMethod(method);
}
catch
(Exception e) {
method.releaseConnection();
throw
new
HttpClientException(URL, e);
}
// 返回字节流
byte
[] responseBody =
null
;
try
{
responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
}
catch
(IOException e) {
throw
new
HttpClientException(e);
}
finally
{
method.releaseConnection();
}
return
responseBody;
}
|
第三种:通过URL和bytes参数,以post方式请求服务器,返回字节数组。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
|
public
byte
[] postWithBytes(String queryURL ,
byte
[] bytes)
throws
HttpClientException{
if
(queryURL ==
null
) {
throw
new
HttpClientException(
"queryURL is null."
);
}
byte
[] newbuf = executeByPostWithBytes(queryURL,bytes);
if
((newbuf ==
null
) || (newbuf.length ==
0
)) {
throw
new
HttpClientException(
"Server response is null: "
+ queryURL);
}
return
newbuf;
}
private
byte
[] executeByPostWithBytes(String queryURL,
byte
[] bytes)
throws
HttpClientException {
PostMethod method =
new
PostMethod(queryURL);
RequestEntity requestEntity =
new
ByteArrayRequestEntity(bytes);
method.setRequestEntity(requestEntity);
// RequestHeader
method.setRequestHeader(
"Content-type"
,
"text/plain; charset=UTF-8"
);
// 提交请求
try
{
client.executeMethod(method);
}
catch
(Exception e) {
method.releaseConnection();
throw
new
HttpClientException(queryURL, e);
}
// 返回字节流
byte
[] responseBody =
null
;
try
{
responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
}
catch
(IOException e) {
throw
new
HttpClientException(e);
}
finally
{
method.releaseConnection();
}
return
responseBody;
}
|
第四种:通过URL、fileList、paramMap参数,以post方式请求服务器,返回字节数组。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
|
public
byte
[] postWithFileListAndParamMap(String queryURL,List<File> fileList,Map<String,String> paramMap)
throws
HttpClientException, HttpException, IOException {
if
(queryURL ==
null
) {
throw
new
HttpClientException(
"queryURL is null."
);
}
if
(fileList ==
null
) {
throw
new
HttpClientException(
"file is null."
);
}
if
(paramMap ==
null
){
throw
new
HttpClientException(
"paramMap is null."
);
}
return
executeByPostWithFileListAndParamMap(queryURL, fileList, paramMap);
}
private
byte
[] executeByPostWithFileListAndParamMap (String queryURL,List<File> fileList,Map<String,String> paramMap)
throws
HttpException, IOException, HttpClientException {
if
(queryURL !=
null
&& fileList !=
null
&& fileList.size() >
0
) {
// post方法
PostMethod method =
new
PostMethod(queryURL);
// Part[]
Part[] parts =
null
;
if
(paramMap !=
null
) {
parts =
new
Part[fileList.size()+paramMap.size()];
}
else
{
parts =
new
Part[fileList.size()];
}
int
i =
0
;
// FilePart
for
(File file : fileList){
Part filePart =
new
FilePart(file.getName(),file);
parts[i++] = filePart;
}
// StringPart
if
(paramMap !=
null
) {
Set<Entry<String, String>> entrySet = paramMap.entrySet();
Iterator<Entry<String, String>> it = entrySet.iterator();
while
(it.hasNext()) {
Entry<String, String> entry = it.next();
Part stringPart =
new
StringPart(entry.getKey(),entry.getValue());
parts[i++] = stringPart;
}
}
// Entity
RequestEntity requestEntity =
new
MultipartRequestEntity(parts, method.getParams());
method.setRequestEntity(requestEntity);
// RequestHeader,文件的话,HttpClient自动加上multipart/form-data
// method.setRequestHeader("Content-type" , "multipart/form-data; charset=UTF-8");
// excute
try
{
client.executeMethod(method);
}
catch
(Exception e) {
method.releaseConnection();
throw
new
HttpClientException(queryURL, e);
}
// return
byte
[] responseBody =
null
;
try
{
responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
}
catch
(IOException e) {
throw
new
HttpClientException(e);
}
finally
{
method.releaseConnection();
}
return
responseBody;
}
return
null
;
}
|
二、异步的AsyncHttpClient
同样的,按照这种思路,异步的AsyncHttpClient也有类似的实现,不过写法不同而已,在工厂中,AsyncHttpClient使用的是AsyncHttpClientConfig.Builder作为管理配置的类,也有类似连接超时,最大连接数等配置。
工厂类:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
// 专门针对xx服务器的连接管理对象
// 因为不同服务可能超时等参数不用,所以针对不同服务,把连接管理对象区分开来,这只是其中一个
private
static
AsyncHttpClientConfig.Builder xxbuilder =
new
AsyncHttpClientConfig.Builder();
static
{
xxbuilder.setConnectTimeout(
3000
);
// 连接超时
xxbuilder.setReadTimeout(
2000
);
// 读取数据超时
xxbuilder.setMaxConnections(
1000
);
// 最大连接数
}
/*
* 返回针对XX服务的httpClient包装类
*/
public
static
AsyncHttpClientWapperImpl getXXSearchHttpClient() {
return
new
AsyncHttpClientWapperImpl(xxbuilder);
}
|
其使用了builder 的设计模式,活生生的一个例子,值得学习。
实现类的构造方法:
1
2
3
4
5
|
private
AsyncHttpClient client;
public
AsyncHttpClientWapperImpl(Builder xxbuilder) {
client =
new
AsyncHttpClient(xxbuilder.build());
}
|
这样,AsyncHttpClient对象就创建完毕了。接下来是各种场景的实现,感觉异步的AsyncHttpClient封装得比HttpClient 3.X更加容易使用,设计得更好。
第一种:通过URL,以get方式请求服务器,返回字节数组。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
public
byte
[] getWithQueryURL(String queryURL)
throws
HttpClientException, HttpClientException {
if
(queryURL ==
null
) {
throw
new
HttpClientException(
"queryURL为空"
);
}
byte
[] newbuf = executeByGet(queryURL);
if
((newbuf ==
null
) || (newbuf.length ==
0
)) {
throw
new
HttpClientException(
"Server response is null: "
+ queryURL);
}
return
newbuf;
}
private
byte
[] executeByGet(String queryURL)
throws
HttpClientException {
byte
[] responseBody =
null
;
try
{
Future<Response> f = client.prepareGet(queryURL).execute();
responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
}
catch
(Exception e) {
throw
new
HttpClientException(e);
}
return
responseBody;
}
|
同样的,我们写了一个getBytesFromInputStream()方法解析服务端返回的流,我们发现,两个实现类里面都有一些共同的方法,这里可以考虑写一个父类,把这些方法提取出来。
第二种:通过URL和paramsMap参数,以post方式请求服务器,返回字节数组。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
|
public
byte
[] postWithParamsMap(String queryURL,
Map<String, String> paramsMap)
throws
HttpClientException {
if
(queryURL ==
null
) {
throw
new
HttpClientException(
"queryURL为空"
);
}
byte
[] newbuf = executeByPostByParamMap(queryURL,paramsMap);
if
((newbuf ==
null
) || (newbuf.length ==
0
)) {
throw
new
HttpClientException(
"Server response is null: "
+ queryURL);
}
return
newbuf;
}
private
byte
[] executeByPostByParamMap(String queryURL,Map<String,String> paramsMap)
throws
HttpClientException {
byte
[] responseBody =
null
;
try
{
RequestBuilder requestBuilder =
new
RequestBuilder();
// 添加 key-value参数
if
(paramsMap !=
null
&& paramsMap.size() >
0
) {
Set<Entry<String, String>> entrySet = paramsMap.entrySet();
Iterator<Entry<String, String>> iterator = entrySet.iterator();
while
(iterator.hasNext()) {
Entry<String, String> entry = iterator.next();
if
(entry.getKey() !=
null
) {
requestBuilder.addFormParam(entry.getKey(), entry.getValue());
}
}
}
// 添加RequestHeader,key
requestBuilder.addHeader(
"Content-type"
,
"application/x-www-form-urlencoded; charset=UTF-8"
);
requestBuilder.setMethod(
"POST"
);
// 添加URL
requestBuilder.setUrl(queryURL);
// request
Request request = requestBuilder.build();
// 提交
ListenableFuture<Response> f = client.executeRequest(request);
responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
}
catch
(Exception e) {
throw
new
HttpClientException(e);
}
return
responseBody;
}
|
第三种:通过URL和bytes参数,以post方式请求服务器,返回字节数组。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
public
byte
[] postWithBytes(String queryURL,
byte
[] bytes)
throws
HttpClientException {
if
(queryURL ==
null
) {
throw
new
HttpClientException(
"queryURL is null."
);
}
byte
[] newbuf = executeByPostWithBytes(queryURL,bytes);
if
((newbuf ==
null
) || (newbuf.length ==
0
)) {
throw
new
HttpClientException(
"Server response is null: "
+ queryURL);
}
return
newbuf;
}
private
byte
[] executeByPostWithBytes(String queryURL,
byte
[] bytes)
throws
HttpClientException {
byte
[] responseBody =
null
;
try
{
RequestBuilder requestBuilder =
new
RequestBuilder();
// 添加 bytes参数
requestBuilder.setBody(bytes);
// 添加RequestHeader,key
requestBuilder.addHeader(
"Content-type"
,
"text/plain; charset=UTF-8"
);
requestBuilder.setMethod(
"POST"
);
// 添加URL
requestBuilder.setUrl(queryURL);
// request
Request request = requestBuilder.build();
// 提交
ListenableFuture<Response> f = client.executeRequest(request);
responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
}
catch
(Exception e) {
throw
new
HttpClientException(e);
}
return
responseBody;
}
|
第四种:通过URL、fileList、paramMap参数,以post方式请求服务器,返回字节数组。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
|
public
byte
[] postWithFileListAndParamMap(String queryURL,
List<File> fileList, Map<String, String> paramMap)
throws
HttpClientException, HttpException, IOException {
if
(queryURL ==
null
) {
throw
new
HttpClientException(
"queryURL is null."
);
}
if
(fileList ==
null
|| fileList.size() ==
0
) {
throw
new
HttpClientException(
"fileList is null."
);
}
if
(paramMap ==
null
|| paramMap.size() ==
0
) {
throw
new
HttpClientException(
"paramMap is null."
);
}
return
executeByPostWithFileListAndParamMap(queryURL, fileList, paramMap);
}
private
byte
[] executeByPostWithFileListAndParamMap (String queryURL,List<File> fileList,Map<String,String> paramsMap)
throws
HttpException, IOException, HttpClientException {
if
(queryURL !=
null
&& fileList !=
null
&& fileList.size() >
0
) {
byte
[] responseBody =
null
;
try
{
RequestBuilder requestBuilder =
new
RequestBuilder();
// FilePart
for
(File file : fileList){
Part filePart =
new
FilePart(file.getName(),file);
requestBuilder.addBodyPart(filePart);
}
// StringPart
if
(paramsMap !=
null
) {
Set<Entry<String, String>> entrySet = paramsMap.entrySet();
Iterator<Entry<String, String>> it = entrySet.iterator();
while
(it.hasNext()) {
Entry<String, String> entry = it.next();
Part stringPart =
new
StringPart(entry.getKey(),entry.getValue());
requestBuilder.addBodyPart(stringPart);
}
}
// 添加RequestHeader,key
requestBuilder.addHeader(
"Content-type"
,
"multipart/form-data; charset=UTF-8"
);
requestBuilder.setMethod(
"POST"
);
// 添加URL
requestBuilder.setUrl(queryURL);
// request
Request request = requestBuilder.build();
// 提交
ListenableFuture<Response> f = client.executeRequest(request);
responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
}
catch
(Exception e) {
throw
new
HttpClientException(e);
}
return
responseBody;
}
return
null
;
}
|
OK,入了个门后,更多的用法可以自己去看文档了,请不要局限以上几种常用的场景。