UNIX/LINUX

1. Use Ctrl+D for specifying EOF for stdin.

2. Removing the files returned by svn status command.

svn status | awk '{cmd = "rm " $2; system(cmd)}'


3. To find the Linux distribution on the machine

more /etc/*-release

4.  top, htop, iotop,sysstat  for seeing the system usage.

5.  jstack   for seeing the threads inside a java process

6. In LDAP access log , to print all the lines which has etimes greater than 10 ms and also print next line if it has "SRCH" into it, use following command

awk -F= '/etime/ && $7>0.02000 {print FNR; print;getline; print FNR;if (match($0,"SRCH")) {print}}'  access

6.1  To list all the lines which have etime > 1000 ms 

 awk -F"etime=" '$2>1000 {print }' access





7.  In LDAP access log, print the previous line and the line where etimes greater than 10 ms.

awk -F= '/etime/ && $7>0.01000 { print line;print;} {line=$0}'  access


8. Sudo to root without changing current directory.


sudo su -c "cd $(pwd); bash"

9.  .pem is typically the private key file . So if you have .pem , you can always .ssh into another box as far as  your public key is part of authorized_keys in that server.  assuming you are using OpenSSH protocol.    Remember you use your private key and provide your public key to remote systems.

ssh -i  <private key path>  <remote host>

To be able to ssh into another box without specifying the key on the command line , you need to have your private key into one of standard  paths which systems looks for.  like  .ssh/identity  or .ssh/id_rsa  etc.

ssh  <remote host> 

To debug ssh , use following:

ssh -v <remote host>

At times , some systems look for private key on .ssh/id_rsa and not .ssh/identity and vice versa. So you have be aware of that.

9.1 "No supported authentication methods available(Server sent ....) . 
Many times this issue is simply because we have not copies the public key in the authorized_key on  remote host properly. The key is normally a single line. When you copy and paste it , at times it ends up adding new line characters into it.

10.  use  "cut"  command to get the nth column from a line.


11. To quickly identify which directories are occupying most space


du | awk '$1>10000 {print;}'


You can narrow down the directories taking the most space  by changing $1>10000   condition.

12. Using WinSCP, if you copy a file and if target system is out of disk space, it does not give that messages but rather one like  General Failure "permission denied"  .

13.   running command over a list of values

#!/bin/bash
while read -r line || [[ -n "$line" ]]; do
        echo "Text read from file: $line"
        ID=$line
        echo ${ID}
        # in case you need to create a command dynamically use below format
        cmd="ls -l ${ID}" 
        eval "$cmd"
done < "$1"




14.   when you run commands over ssh on remote server   the echo $?
 still returns the status of the command executed on remote server. 

15.   Redirecting the stdout and stderr

cmd  & > /dev/null  will redirect all of the out to  /dev/null or any other place you want 

cmd > out.log 2>&1 => will redirect std out to out.log and stderr to stdout, which then goes to out.log.

cmd 2>&1 | tee out.log  ...will redirect both stderr to stdout and stdout to out.log and screen both. 

16.  Be careful invoking ssh or some other command inside while loop above.  As those will eat the stdin from $1 also and your while loop wont work.

while read -r line || [[ -n "$line" ]]; do
   IP=$line
   ssh ${IP}  sudo service jboss-as restart
done < "$1"

won't work, change it to below format

for IP in $(cat $1); do
    ssh ${IP}  sudo service jboss-as restart
done;

17. testing a cluster of nodes. It takes the a file which has list of  node ips in the cluster and there are two testcommands which you want to run on each node to make sure it is healthy.
 ----------------------------------------------------
 #!/bin/bash

for IP in $(cat $1); do
        #echo $IP;
       
        ssh centos@${IP} testcommand1  &> /dev/null
        if [ "$?" -eq 0 ]
        then
                ssh centos@${IP} testcommand2 > /dev/null

                if [ "$?" -eq 0 ]
                then
                        echo "Node ${IP} is good."
                else
                        echo "Node ${IP} is bad."
                fi
        else
                echo "Node ${IP} is bad."
        fi

done
--------------------------------------------

 18.  Listing the files modified before 30 days 


find  <path>  -mtime +30 -exec ls -l {} \;


19.  listing the 10 largest files under a dirctory


du -a / | sort -n -r | head -n 10    

above command includes directories also. If you want only files, use below command.

find . -type f -exec du -a {} + | sort -n -r  | head -n 10 


To delete largest 10 files in /var/log/ directory use following command 

find /var/log/ -type f -exec du -a {} + | sort -n -r | head -n 10 |cut  -f 2 | xargs rm 

20. To execute command on remote systems


#!/bin/bash

for IP in $(cat $1); do
        echo ${IP}
        echo $2
        cmd="sudo $2"
        echo ${cmd}
        ssh  -tt centos@${IP} ${cmd}
done


where $1 is list of host IPs and $2 is command to be executed. If command has multiple words , enclose in double quotes.

21. To see the established connections on a particular port 

netstat -anp

22. To get the timestamp with each history entry  set the format for HISTTIMEFORMAT

for example on linux:

 export HISTTIMEFORMAT="%d/%m/%y %T "

23.  AWK ...To parse tabular data and calculate running total of a column. For example in hdfs , you want to see the total space consumed by certain pattern of files. 

hdfs dfs -du -s <path_pattern> | awk '{sum +=$1; printf("File:%s, size:%d, Total:%d\n",$3,$1,sum)}'
 
24.  printing quote inside awk 

Let us say you want to print file name inside single quotes.  Every place where you need quote , use '\'' . 

hdfs dfs -du -s <path_pattern> | awk '{sum +=$1; printf("File:'\''%s'\'', size:%d, Total:%d\n",$3,$1,sum)}'

25.  for i  in {1..10..2}   syntax you can not use command line parameters for start (1), end(10) , step(2) values as the braces are evaluated before evaluating the any values inside. 

26. Also if you need to increment the value of integer variable inside a while loop  use following syntax

i=$(($i+$step))

27. Formatted date 

date +%y%m%d

27. To extract / list first few lies from a file , you can use  any of the following commands
head/awk/sed

This is especially important if you are dealing with very large  files which are hard to open using regular editors.

28.  be careful while using sed . Unless you full understand the meaning of characters in pattern. It can have unintended consequences.  The . matches with any character.  I tried something similar as I wanted to replace "refined/" with "../refined/"  and I ended up clearing my whole scripts.

sed -i 's/ refined/ ..\/refined/g' *.sh

see space before refined ..makes sure it only picks up the relative paths    (  \ is used to escape  /) .

This command worked fine.

replaces ../../user/logs  by /zero/first/sec/third/fourth/fifth/sixth/seventh/eighth/ninth/tenth/elevnth

sed -ibak "s/\.\.\/\.\.\/user\/logs/\/zero\/first\/sec\/third\/fourth\/fifth\/sixth\/seventh\/eighth\/ninth\/tenth\/elevnth/g"  file.sh


29.  This command makes no differnce to the input file having "minusMonths(1)"  string in it.

sed -i.bak "s/minusMonths\(1\)/minusDays\(29\)/g" abc.txt

what is wrong ? it should replace. it appears there is no need to escape ()  characters here . So following command worked fine.

ed -i.bak "s/minusMonths(1)/minusDays(29)/g" abc.txt


30. "jobs -l"   display the status of the job along with some information ..like for a job stopped on output  it will display "Stopped (tty output).....".

31. To make the apache web server start every time server is restarted.

chkconfig httpd on