EFS and SOLR Cloud Backup(3)Cron Backup and Archive
Try to clean the directories older than X days
>find /home/ec2-user/users/carl/ -mtime +3 -exec sudo rm -fr {} \;
Try to clean the directory under directory, only keep X
>sudo rm -fr `ls -t /Users/carl/company/code/solr-alljobs/crontask/test | awk ‘NR>3'`
Try to make a Docker and Task Definition in ECS and Start a scheduled Task there.
I first create a Docker Image and Task Definition
Dockerfile
#Start a Clean Machine to Execute Shell Script
#Prepre the OS
FROM centos:7
MAINTAINER Carl Luo <[email protected]>
#Start the Application
RUN mkdir -p /app/
ADD start.sh /app/
WORKDIR /app
CMD [ "./start.sh" ]
Makefile
IMAGE=odt/cron-task
TAG=1.6
NAME=cron-task
REPOSITORY=xxxxxx.dkr.ecr.us-east-1.amazonaws.com
push-local:
docker push $(REPOSITORY)/$(IMAGE):$(TAG)
docker-context:
build: docker-context
docker build -t $(REPOSITORY)/$(IMAGE):$(TAG) .
run:
docker run -d --name $(NAME) $(REPOSITORY)/$(IMAGE):$(TAG)
clean:
docker stop ${NAME}
docker rm ${NAME}
logs:
docker logs ${NAME}
publish:
docker push ${IMAGE}
The start.sh will be the key part, it will call backup and archive the too old backup files.
#!/bin/sh -ex
DATE=`date +%y-%m-%d-%H-%M-%S`
COLLECTION=${COLLECTION:-"allJobs"}
CLUSTER=${CLUSTER:-"alljobstest"}
echo "Start to check the backup and archive ${DATE}"
echo "list the backup directory"
ls /efs
echo "start the backup process"
curl "http://${CLUSTER}.us-east-1.elasticbeanstalk.com/solr/admin/collections?action=BACKUP&name=allJobsBackup${DATE}&collection=${COLLECTION}&location=/efs&async=${DATE}"
echo "backup request is sent"
echo "start to archive data 3 days ago"
#find /efs/ -mtime +3 -exec sudo rm -fr {} \;
sudo rm -fr `ls -t /efs/ | awk 'NR>3'`
I build and push this Docker image to the ECS Repositories
From the Task Definition, some settings are as follow:
Task Definition Name: Cron-Task
Task Role: None
Network Mode: Bridget
Compatibilities: EC2
Volumes: efs = /efs
Container Name:
CronTask: Image: xxxxx.dkr.ecr.us-east-1.amazonaws.com/odt/cron-task:1.6
Mount Points:
Container Path: /efs
Source Volume: efs
Log Configuration
Log driver: awslogs
awslogs-group: Feed2GLogging
awslogs-region: us-east-1
awslogs-stream-prefix: crontask
Creat a Cron Service
Go to the cluster page, choose [Scheduled Tasks]—> [Create]
You can choose, Run at Fixed interval, I put 2 hours there for testing.
You can choose Cron expression as well, for example
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
0 23 * * ? * (every 11pm)
Put the task Definition string : Cron-Task:10
That is it.
References:
https://stackoverflow.com/questions/13489398/delete-files-older-than-10-days-using-shell-script-in-unix
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html
https://lucene.apache.org/solr/guide/7_2/making-and-restoring-backups.html#delete-snapshot-api
EFS and SOLR Cloud Backup(3)Cron Backup and Archive
猜你喜欢
转载自sillycat.iteye.com/blog/2407637
今日推荐
周排行