dolphinscheduler执行datax报错:text is busy

报错

[INFO] 2021-11-04 01:00:00.846  - [taskAppId=TASK-113-8535-33164]:[127] - task dir : /tmp/dolphinscheduler/exec/process/9/113/8535/33164
[INFO] 2021-11-04 01:00:00.846  - [taskAppId=TASK-113-8535-33164]:[141] - datax task params {"targetTable":"ti_resource_rate_hist","postStatements":[],"jobSpeedRecord":1000,"de":"MYSQL","jobSpeedByte":0,"dataSource":12,"dataTarget":13,"sql":"select ResourceId   \t\tas ResourceId   \t\t\n       ,RateId       \t\tas RateId       \t\t\n       ,EffectieDate\t\n  from resource_rate_hist;\t","preStatements":["truncate table ti_resource_rate_hist;"]}
[INFO] 2021-11-04 01:00:01.180  - [taskAppId=TASK-113-8535-33164]:[105] - tenantCode user:combexec, task dir:113_8535_33164
[INFO] 2021-11-04 01:00:01.185  - [taskAppId=TASK-113-8535-33164]:[109] - create command file:/tmp/dolphinscheduler/exec/process/9/113/8535/33164/113_8535_33164.command
[INFO] 2021-11-04 01:00:01.185  - [taskAppId=TASK-113-8535-33164]:[128] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /home/dolphinscheduler/application/dolphinscheduler/conf/env/dolphinscheduler_env.sh
/tmp/dolphinscheduler/exec/process/9/113/8535/33164/113_8535_33164_node.sh
[INFO] 2021-11-04 01:00:01.187  - [taskAppId=TASK-113-8535-33164]:[375] - task run command:
sudo -u combexec sh /tmp/dolphinscheduler/exec/process/9/113/8535/33164/113_8535_33164.command
[INFO] 2021-11-04 01:00:01.194  - [taskAppId=TASK-113-8535-33164]:[160] - process start, process id is: 40748
[INFO] 2021-11-04 01:00:01.221  - [taskAppId=TASK-113-8535-33164]:[170] - process has exited, work dir:/tmp/dolphinscheduler/exec/process/9/113/8535/33164, pid:40748 ,exitStatusC
[INFO] 2021-11-04 01:00:01.220  - [taskAppId=TASK-113-8535-33164]:[109] -  -> /tmp/dolphinscheduler/exec/process/9/113/8535/33164/113_8535_33164.command:行5: /tmp/dolphinscheduler/e5/33164/113_8535_33164_node.sh: 文本文件忙

初步定位,原因可能为代码流未释放,Files.write()会自动释放流,假如还没释放流,就去执行该文件,一边写入,一边执行,就会报文件正忙的错,项目上线一年了,第一次遇到该情况,很难复现:

 Files.write(path, dataxCommand.getBytes(), StandardOpenOption.APPEND);

修改,换成有close方法的文件类,出现问题之后,再没出现过,有待验证:

  FileUtils.writeStringToFile(path.toFile(), dataxCommand, StandardCharsets.UTF_8);

一边写文件,一边执行文件,倒是可以复现文件正忙的错,但是无法复现生产中的错

在这里插入图片描述
在这里插入图片描述
测试代码:

package org.apache.dolphinscheduler.server.worker;


import java.io.File;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;

import org.apache.commons.io.FileUtils;
import org.apache.dolphinscheduler.common.utils.OSUtils;

public class FileTest2 {
	public static void main(String[] args) throws Exception {
		testFilesWrite();
	}

    private static String testFilesWrite()
        throws Exception {
    	StringBuilder sbr = new StringBuilder();
    	String fileName = "F:\\test.bat";
    	Path path = new File(fileName).toPath();
    	for(int i=0;i<100;i++) {
    		Thread.sleep(1000);
    		sbr.append("cd /");
    		String dataxCommand = sbr.toString();
    		if (OSUtils.isWindows()) {
    			if(!path.toFile().exists()) {
    				Files.createFile(path);
    			}
    		}
    		//Files.write(path, dataxCommand.getBytes(), StandardOpenOption.APPEND);
    		FileUtils.writeStringToFile(path.toFile(), dataxCommand, StandardCharsets.UTF_8);
        }
        
        
        return fileName;
    }

}

package org.apache.dolphinscheduler.server.worker;

public class FileTest {
	public static void main(String[] args)  {
		test();
	}
	private static void test() {
		Runtime run = Runtime.getRuntime();
    	for(int i=0;i<100000;i++) {
    		try {
    			Thread.sleep(100);
				run.exec("F:\\test.bat");
				System.out.println("执行结束"+i);
			} catch (Exception e) {
				System.out.println(e.getMessage());
				test();
			}
    		
    	}
	}
}

猜你喜欢

转载自blog.csdn.net/qq_36434219/article/details/121624217