Fastdfs批量下载问题 Connection reset,Broken pipe (Write failed),recv package size -1 != 10
1、由于对接第三方用的fastdfs上传的,在刚开始我这边不清楚的时候用http请求下载的,导致了文件超过一定的大小就会中断下载,
2、后面使用fastdfs下载,再网上找的下载例子,在测试下载中总是出现连接重置之类的错误,再排除了防火墙问题后,后面才搞明白是由于把fastdfs的new实例写在for循环内的原因。
最初写成了一个下载方法在使用的时候循环调用downloadFile:
以下代码 内联代码片
。
public static boolean downloadFile(String url, String dir, String filename) {
FileOutputStream fileOutputStream = null;
try {
File path = new File(dir);
if (!path.exists()) {
path.mkdirs();
}
//加载fastdfs-client.properties配置文件
ClientGlobal.initByProperties("bhc.properties");
//ClientGlobal.init("bhc.properties");
//定义TrackerClient,用于请求TrackerServer
TrackerClient trackerClient = new TrackerClient();
//连接tracker
TrackerServer trackerServer = trackerClient.getConnection();
//获取storage
StorageServer storeStorage = trackerClient.getStoreStorage(trackerServer);
//创建storageClient
StorageClient1 storageClient1 = new StorageClient1(trackerServer, storeStorage);
//下载文件
//文件id
String fileId=getFileGroup(url);
logger.info("下载的文件id:"+fileId);
byte[] bytes = storageClient1.download_file1(fileId);
//使用输出流保存文件
fileOutputStream = new FileOutputStream(new File(dir + filename));
fileOutputStream.write(bytes);
return true;
} catch (Exception e) {
e.printStackTrace();
return false;
} finally {
try {
if(null!=fileOutputStream) {
fileOutputStream.close();
logger.info("关闭成功!");
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
后面在网上查了下面的资料后发现fastdfs最大了高并发连接数是256,这时就联想到了循环调用的原因:
调整后这部分代码放到循环外:
FileOutputStream fileOutputStream = null;
TrackerServer trackerServer = null;
StorageServer storeStorage = null;
StorageClient1 storageClient1 = null;
try {
//加载fastdfs-client.properties配置文件
ClientGlobal.initByProperties("bhc.properties");
//ClientGlobal.init("bhc.properties");
//定义TrackerClient,用于请求TrackerServer
TrackerClient trackerClient = new TrackerClient();
//连接tracker
trackerServer = trackerClient.getConnection();
//获取storage
storeStorage = trackerClient.getStoreStorage(trackerServer);
//创建storageClient
storageClient1 = new StorageClient1(trackerServer, storeStorage);
这部分代码放到循环内:
File path = new File(filepath + "/" + file_path + "/");
if (!path.exists()) {
path.mkdirs();
}
//下载文件
//文件id
String fileId = getFileGroup(url);
logger.info("下载的文件id:" + fileId);
byte[] bytes = storageClient1.download_file1(fileId);
//使用输出流保存文件
fileOutputStream = new FileOutputStream(new File(filepath + "/" + file_path + "/" + filename));
fileOutputStream.write(bytes);
最后一定要关闭连接:
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (null != fileOutputStream) {
fileOutputStream.close();
trackerServer.close();
storeStorage.close();
logger.info("关闭成功!");
}
} catch (IOException e) {
e.printStackTrace();
}
}