shell & bash
shell指允许用户通过文本操作计算机的程序。
interactive shell:从是否通过标准输入输出与用户进行交互的角度分为交互式shell(interactive)和非交互式shell(non-interactive)。
login shell:从是否以一个指定用户及其环境进入shell角度分为登录式shell和非登录式shell,登录式shell会额外source /etc/profile
,/etc/profile.d
,~/.profile
,~/.bash_login
等profile相关脚本。
变量$-
包含了当前shell的选项参数,一个字母代表一种选项。部分如下:
H
histexpandm
monitorh
hashallB
braceexpandi
interactive
判断是否是交互式shell:根据$-
输出是否包含字母i来判断。
case "$-" in
*i*) echo This shell is interactive ;;
*) echo This shell is not interactive ;;
esac
if [[ "$-" == *i* ]]; then
echo This shell is interactive
fi
当前使用的shell类型(bash, cshell...):echo $0;注意$SHELL是默认shell,并非当前shell,也就说如果在echo $SHELL不是当前shell的类型。
uniq只会删除连续重复的行,也就说uniq不在输入的所有行上做distinct,只是在剪掉连续出现的行。想要对在输入的所有行上做distinct,可以考虑先sort,然后uniq,但需要注意这样会改变输入中行字符串出现的顺序。
iostat
查看系统io,iftop
查看网络流的情况。
ssh连接在很短的不活跃时间后自动断开,如需延长这个闲置时间,通过在ssh配置文件中修改ServerAliveInterval ServerAliveCountMax两个量实现,前者指给远程发送心跳包的时间间隔,后者指连续发送心跳包的最大次数。
~/.ssh/config或/etc/ssh/ssh_config中添加两行:
# 每60秒发送一个心跳包
ServerAliveInterval 60
# 最多连续发送3次
ServerAliveCountMax 3
ssh命令行参数提供密码:
sshpass -p PASSWORD ssh -p PORT USER@HOST
sudo -E
sudo -i 登入root用户shell
查看当前使用的shell:
echo $SHELL
查看系统可用的shel:
cat /etc/shells
更改用户默认的shell:
修改/etc/passwd文件对应用户最后的shell配置。
bash相关配置文件
登录bash shell相关(ubuntu):/etc/profile -> /etc/bash.bashrc(在/etc/profile中被source) -> /etc/profile.d/ -> ~/.bash_profile -> ~/.bash_login -> .bashrc
交互式bash shell: /etc/bash.bashrc -> .bashrc
如果shell不是bash,/etc/profile中不会source /etc/bash.bashrc,如启动intellij idea IDE时。
shell script脚本
shell脚本(shell script)是指由一系列shell命令组成的脚本程序文件。shell script不是interative shell,也不是login shell。
小心ls、grep等带有颜色的输出,输出字符串看起来一样的,在做uni时被视为不一样,则小心其中有个输出可能带有颜色字符,用cat -A确认。
for, while中计数
counter=0
for/while...
counter=$((counter+1))
定义环境变量:
export name=value
取消定义:
交换文件中以逗号分割的前两列的纯bash脚本实现:
while IFS=, read x1 x2 remainning
do
echo ${x2},${x1},$remainning
done < FILE
$PS1中的特殊符号:
\u:用户名
\h:短主机名
\W:working directory(basename)
\s: shell名字(bash、sh)
\v: shell version
\d: 英文日期
\D{fmt}:日期格式(man strftime)
\n: 换行(多行提示符相关)
\w: full path of working directory
\H: full hostname
以及颜色标签[\exxx]
如果不存在则创建目录
[ -d /path/to/dir ] || mkdir -p /path/to/dir
io重定向 io redirect:
stdout stderr输出到同一个文件:
ls -l /usr > out 2>&1
或者 ls -l /usr &> out
。
stdout stderr输出到单独文件 ls -l /usr > stdout 2> stderr
。
忽略stderr输出 ls -l /usr > stdout 2>/dev/null
date
将epoc seconds格式化输出 date -d @1305741420。注意,需带@
前缀。 若想将millis(毫秒)格式化输出则先去掉数值后3位:echo 1305741420 | cut -b1-10 | xargs -I {} date -d @{}
xargs
将xargs的输入(通常是前面管道的输出)作为待执行命令最后一个参数,如果需要拼接固定(或其他形式格式化),可以指定替换串形式,如:echo 1305741420 | xargs -I {} | date -d @{}
,则指定{}
为替换串,会将命令形式date -d @{}
中的{}替换为xargs接收到的输入。
awk
行处理,根据列是否满足条件输出、或做进一步处理,这样的需求完全可以考虑使用awk。
忽略#开头的行 awk '$1 ~ /^[^#]/'
忽略#开头的行,输出第三列(默认以空格分隔)以<开头的行awk '$1 ~ /^[^#]/ && $3 ~ /^</'
设置分隔符为逗号,:awk -F,
awk 'BEGIN{FS=,}'
sed
comm
找出已排序的两个文件中的相同行、不同行,差集、交集、并集皆可实现。
输出3列,分别是:FILE1中独有行、FILE2中独有行、共有行。
comm
接受-
comm -12
仅输出两文件共有行
comm -23
仅输出FILE1独有行
comm -13
仅输出FILE2独有行
join
合并已排序的两个文件中的行。
top
- -b 输出到文件
- -p 指定pid, -pN1,N2,N3
- -O 打印可以输出的字段,相当于对-o的帮助信息
- -n 迭代次数 -n num
- -O 排序字段 [+|-]fieldName
lsblk
分辨ssd/hdd:lsblk -d -o name,rota
输出0代表ssd,1代表hdd。
lsblk -o name
可展示出物理盘的从属虚拟硬盘(slave disk),如物理盘虚拟出的lvm盘。
迭代文件中的行
cat
对稍大一点(上百M)的文件,速度相当慢。
IFS=$'\n'
for line in `cat file`
do
...
done
控制流程:
if ... ; then
...
elif ... ; then
...
else
...
fi
function
#定义函数,关键字function是可不写
function fun(){
#获取参数
echo $1 , $2;
#返回值,可选
return $(($1 + $2))
}
#调用函数,需要先定义
fun 5 7;
#获取函数返回值
s = $?
输出错误信息到stderr: echo .... >&2
字符串
判断字符串是否包含给定字符串:
if [[ <string> == *"sub-string"* ]]; then
...
fi
下载
wget
根据url下载小文件 wget -nd URL。选项-nd
指明不创建目录,否则程序将根据url创建层级目录;选项-b
指明后台下载;选项-d
打印处理过程;-v
指明更详细输出。
定时任务
编写crontab文件
crontab <crontab-file>
crontabl -l # 列出任务
linux文件权限rwx
文件:
目录:
连接其他电脑
ssh ...
#在ssh中
export DISPLAY=:0
[sudo] notify-send <msg> #右上角显示提示信息
[sudo] xmessage <msg> [-timeout <n>(单位:sec)] [-buttons OK,Cancle] #以xmessage丑陋的窗口弹窗显示信息
#某些可以从命令行打开GUI的程序,如redis-desktop-manager
...
命令及功能:
[ -d "~" ] is false
[ -d ~ ] is true
[ -d "$HOME" ] is true
apt-cache show app # check whether app is installed
apt-get update # update apt repository
telnet <CTRL+]> back to telnet then quit or <CTRL+D> to quit telnet
tail -n +NUM FILE # show line NUM to end
export NAME=xxx
unset NAME
# detect encoding of tile content
enca FILE
# convert encoding of file content
enconv FILE
# convert encoding of text
iconv -f from-encoding -t to-encoding
# convert encoding of file name
convmv -f from-encoding -t to-encoding --notest
# convert file content between character sets
recode FROM-ENC..TO-ENC FILE (e.g. recode gbk..utf8 /a.txt)
# list the supported character sets of recode
recode -l
lsusb
lscpu
# specify an encoding before extracting
unzip -O encoding
unzip -O cp936 -l chap04.zip | head -n -2 | sed -n '4,$p' | awk '{print $4}' | grep
# read the contents of files in a zip file; extract file into stdout
unzip -c ZIPFILE ENTRY
locate:
locate -b(basename) xx ; locate -r regex
# manual for command
man xx; man [1-7] xx; man 7 man
# help for shell-built command
help <CMD> # help cd; help pwd
# document
info xx
# search man page names and descriptions
apropos xx
# block device, uuid, &...
(sudo) blkid /device... ; blkid -s UUID /device ; blkid -s LABEL /device
# To cut out the third field of text from file or stdin, which is delimited by a #:
cut -d # -f3
# first char of column instead of whole column, cut twice
cut -d # -f3 | cut -c 1 # cannot cut -f3 -c 1
# sort file name numerically
ls /dev/sda* | sort -k 1.9n
# list files sorted by size(desending order)
ls -S
# list files sorted by time(desending order)
ls -t
# list files, dereference symbolic link
ls -L ...
# short desciption of ascii characters
man 7 ascii
# directory or file usage
du -hs FILE
du -a <directory> # including files not only directory
du -d <depth> FILE
# disk usage
df -lh /mount/point /# or mounted /dev/xxx, if /dev/xx not mounted, output is not what we expected
# disk ssd or hdd
lsblk -d -o name,rota # 0 for ssd, 1 for hdd
cat /sys/block/sdx/quene/rotational
# create an iso image file from a cd/dvd/bd
dd if=/dev/cdrom of=/path/to/image.iso
# eject a disc
eject eject cdrom eject /mount/point eject /dev/cdrom
# mount Windows shared folder, mount samba share (cifs <- sambfs)
mount -t cifs -o username="WinUser",password="secret",uid="localuser",gid="localgroup",dir_mode=0777 //192.168.x.x /mnt/win
# maybe need run the following command before mount
apt install cifs-utils # if no cifs filesystem supporting lib in system (by checking if /sbin/mount.cifs exists)
# mount an iso image file
mount -o loop /path/to/image.iso /path/to/mount/point
# mount partition by non-superuser
gvfs-mount -d /dev/sda1 /mnt/point/ # gvfs-* for gnome user only
# unmount
gvfs-mount -u <location>
# list
gvfs-mount -l
无密码访问远程主机R
# generate key pair on local host
ssh-keygen -t rsa
#copy public key id_rsa.pub into remote host R
scp x.pub user@host:/home/u/.ssh/
# on remote host
cat x.pub >> .ssh/authrized_keys
# cgroup: control group
# sudo apt-get install cgroup-bin
# share a read-only/readable-writable directory over lan
?????
# find devices on lan
????
# find a device on lan then mount
?????
# list open files for a process
ls /proc/PID/fd
lsof -p PID
# determine / identify processes using files or sockets
fuser name...
# merge two folders
rsync -abviuzP src/ dest/
-i turns on the itemized format, which shows more information than the default format
-b makes rsync backup files that exist in both folders, appending ~ to the old file. You can control this suffix with --suffix .suf
-u makes rsync transfer skip files which are newer in dest than in src
-z turns on compression, which is useful when transferring easily-compressible files over slow links
-P turns on --partial and --progress
--partial makes rsync keep partially transferred files if the transfer is interrupted
--progress shows a progress bar for each transfer, useful if you transfer big files
# merge two folders with cp, tip: hard link: cp -r --link dir1/* dir2/* merged/ && rm -r dir1/ dir2/
# IMPOSSIBLE: create necessary destination directories when copying file, e.g. cp foo.txt /path/not/existing/foo.txt
# telnet to a ssl port(e.g. 443)
openssl s_client -connect host:port
# shared library dependencies
ldd BINARY
# list symbols from object files
nm OBJ(?.o) LIB(?.so ?.a)
# set shared library search paths(directories), add paths to /etc/ld.so.conf, then run ldconfig
vim /etc/ld.so.conf
ldconfig
# report missing objects
ldd -r /path/to/ELF_binary
ldd /path/to/executable | grep 'not found'
#
readelf -d <binary> | grep RPATH
# restore trash that deleted as root
su
cd /path/to/parent/of/file/trashed
restore-trash
# change label of partitions
# mlabel or fatlabel for fat32
# ntfslabel for ntfs
# exfatlabel for exFAT
# e2label for ext2/ext3/ext4
# btrfs for BTRFS
# change label of a fat32 filesystem
sudo mlabel -i /dev/sdb1 ::"new_label"
#if fails & print '... add mtools_skip_check=1 to .mtoolsrc', do
#echo "mtools_skip_check=1" >> ~/.mtoolsrc # as the program says
# get or set label of ms-dos filesystem
sudo fatlabel /dev/sdc1 [new_label]
# change label of a ntfs filesystem
sudo ntfslabel /dev/sda7 "new_label"
# change label of a exfat filesystem
sudo exfatlabel /dev/sdc1 "new_label"
# change label of a ext2/3/4 filesystem
sudo e2label /dev/sdc1 "new_label"
# list background jobs
jobs
bg
# foreground jobs
fg
# get gpu information
lspci -v | grep VGA -A 12
# grep
grep PATTERN {FILE | stdin}
-r # deep in FILE recursively
-i # case-insensitive
-v # not matching
-A NUM, --after-context=NUM # print NUM lines after matching lines
-B NUM, --before-context=NUM # print NUM lines before matching lines
-C NUM, --context=NUM
-H, --with-filename
-h, --no-filename
sudo ssldump -i lo port 443
# delete entries from a zip file
zip -d foo.zip foo/entry/to/delete
# delete a directory from a zip file
zip -d foo.zip "foo/dir/*"
# command information
type <command>
# locate a command
which <command>
# make naming fifo
mkfifo
# make special files
mknod <name> <type>
# change password of current user
passwd
# change another user
sudo passwd <username>
# add group
sudo addgroup <group-name>
# add user
sudo adduser -ingroup <group-name> <user-name>
# login as / become another user, default as root
su - <user-name>
# tell login shell or not
shopt login_shell
# add user in sudoers
# just put the user into 'sudo' group
usermod -a -G sudo <USER-NAME>
su - # login as root
visudo
# then move to the last line, write <user-name> ALL=(ALL:ALL) ALL then ctrl-X to exit, Alt-B to backup previous version file
# screen resolution
xdpyinfo | grep dimensions
# groups of user
groups <user-name>...
# list groups of current user
groups
# add user into group
usermod -a -G <group-name> <user-name> #e.g usermod -a -G sudo jack
# top usage
memory usage summary value scale: E (circle in KB<-->PB)
memory usage of process: swap between size/percentage: ?
sort by memory usage:M
sort by cpu usage:P
toggle command/command-line: c
# memory usage
free -m (by M unit)
free -g (by G unit)
ps aux -p <PID>
screen [-S session-name] # create a screen sesion
# run some long-time-consuming script
screen -d [session-name] # <Ctrl-A d> detach from session
screen -ls # list sessions
screen -r <session-name> # attach to session
screen -d -r <session-name> # attach , detach first if necessary
# which sesseion i am in
echo $STY
# exit: close session then exit
# to leave the process running after closing the terminal
<ctrl+z> <- suspend the process and back to terminal
bg <- show background jobs, find out the job id of corresponding process
disown -h %<jobib> <- let the job run in background
pgrep -f <pattern> # search pattern in command line (not only in command)
pgrep -a <pattern> # print full command
# find files whose name ends with .c in (and desending into) current directory
find . -name "*.c" # quotes are must. or: find . -name \*.c or: find .-name '*.c'
# 在大范围内查找(如根目录/)是很鲁莽的,find在大范围内查找的开销较大
#!!! * in a shell command line will be expanded as file names(hidden files excluded), if it is a wildcard in an argument, should be enclosed by qutoes or escaped
# find . -name *.c => means find . -name x.c y.c z.c where x.c y.c z.c are only files ending with .c in current directory
# find . -name \*.c or "*.c" '*.c' => means find files in .(and desending), which name ending with .c
# find files on which no read-permission
find <FILE> ! -perm -u+r # user-non-readable
find <FILE> ! -perm -g+r # no permision on group
# find files by size
find . -size +100M # larger than 100M
find . -size -100M # smaller than 100M
# find files by owner
find . -user <USER> # username or user id all allowed
# find files by time in day unit, -ctime, -atime, -mtime; c: change(attributes been changed), a: access, m: modify
# by time in minute unit, -cmin, -amin, -mmin
# time argument, +<NUM> over NUM day/min, -<NUM> within NUM day/min
find . -ctime +2 -and -mmin 60 #
# striping leading './' of output of find . ; find * will not search hidden file/dir(file of name starts with '.')
find -printf '%P\n'
# md5 checksum
md5sum FILE...
# gui sudo
gksudo <cmd line>
# print http headers
curl -v <URL>
# startup services management
sysv-rc-conf
# change owner of symbolic file, put -h (no dereference) option
chown -h user:group FILE
# mount remote directory conneting by ssh
sshfs user@host:/path /local/path
# umount remote directory
fusermount -u /local/path
# determine linux distribution info
ls /etc/*-release
# or
lsb_release -a
# print kernal version
uname -a
# print listening, established connections with specific port
(sudo) lsof -i:<PORT> # lsof -i:8080
(sudo) netstat -tunlp|grep <PORT> # -tunlp: tcp, udp
# list symbols in a shared library
readelf -s /path/to/libx.s
# encode a image or a file in base64
base64 <FILE>
# system time(date)
date
# hardware clock time
hwclock # sudo hwclock
# dmesg, messages from kernel
dmesg
# awk
# swap columns
awk '{print $2"\t"$1}' <FILE>
# accumulate numbers, one per line (sum)
awk '{s+=$1} END {print $s}' <FILE>
# using shell variable in awk, you cannot use one directly
awk -v home=$HOME 'print(home)' # careful: it's home, not $home
# output columns satisfy criteria, e.g not begin with #, and 3rd column starts with <
awk '$1 ~ /^[^#]/ && $3 ~ /^</'
# create directory if not exist
[ -d /dir ] || mkdir -p /dir
ss -tlnp
# rename network interface temporarily, 'eth0' -> 'eth1'
ifconfig eth0 down
ip link set eth0 name eth1
ifconfig eth1 up
# rename network interface permanently, edit /etc/udev/rules.d/some-file
# io statistics
iostat
iostat -x 2 5 # extented info, update every 2sec, total 5 updates
# top io wait process
iotop
# list open files
lsof -p <pid>
# copy into gui clipboard
xsel -b
# compare unique file line by line, make lines *sorted* & *unique* in each file
comm FILE1 FILE2 # a hyphen - means stdin, only one hyphen allowed
-1 supress column 1 - unique lines in FILE1
-2 supress column 2 - unique lines in FILE2
-3 supress column 3 - unique lines in FILE1 & FILE2
-12 tells intersect, -23 tells {FILE1} - {FILE2}, -13 tells {FILE2} - {FILE1}
# file status
stat FILE
# system uptime
uptime
# network monitor
iftop
# change user name
usermod -l new_name user_name
# move home dir, -m -d
usermod -m(move) -d <NEW_DIR> <USER>
# change uid
usermod -u <NEW_UID> <USER>
# append user to group, add group to user, -a -G
usermod -a -G <GROUP,GROUP2...> <USER>
# list user and group id
id
# list id and belonging groups
id <USER>
#change group name
groupmod -n <NEW_GP> <GROUP>