SED and AWK command in Linux

-------------------AWK--------------------
1) Pattern scanning and text processing language


awk one-liner Tips

Print column1, column5 and column7 of a data file or output of any columns list

$awk ‘{print $1, $5, $7}’ data_file

$cat file_name |awk ‘{print $1 $5 $7}’

$ls –al |awk ‘{print $1, $5, $7}’ -- Prints file_permissions,size and date

Syntax of running an awk program

Awk ‘program’ input file(s)


List all files names whose file size greater than zero.

$ls –al |awk ‘$5 > 0 {print $9}’


List all files whose file size equal to 512bytes.

$ls –al |awk ‘$5 == 0 {print $9}’

print all lines

$awk ‘{print }’ file_name

$awk ‘{print 0}’ file_name

Number of lines in a file

$awk ‘ END {print NR}’ file_name

Number of columns in each row of a file

$awk ‘ {print NF’} file_name

Sort the output of file and eliminate duplicate rows

$awk ‘{print $1, $5, $7}’ |sort –u

List all file names whose file size is greater than 512bytes and owner is "oracle"

$ls –al |awk ‘$3 == "oracle" && $5 > 512 {print $9}’

List all file names whose owner could be either "oracle" or "root"

$ls –al |awk ‘$3 == "oracle" || $3 == "root" {print $9}’

list all the files whose owner is not "oracle

$ls –al |awk ‘$3 != "oracle" {print $9}’

List all lines which has atlease one or more characters

$awk ‘NF > 0 {print }’ file_name

List all lines longer that 50 characters

$awk ‘length($0) > 50 ‘{print }’ file_name

List first two columns

$awk ‘{print $1, $2}’ file_name

Swap first two columns of a file and print

$awk ‘{temp = $1; $1 = $2; $2 = temp; print }’ file_name

Replace first column as "ORACLE" in a data file

$awk ‘{$1 = "ORACLE"; print }’ data_file

Remove first column values in a data file

$awk ‘{$1 =""; print }’ data_file

Calculate total size of a directory in Mb

$ls –al |awk ‘{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}’

Calculate total size of a directory including sub directories in Mb

$ls –lR |awk ‘{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}’

Find largest file in a directory including sub directories

$ls –lR |awk ‘{print $5 "\t" $9}’ |sort –n |tail -1

$ awk -F":" '{ print $1 $3 }' /etc/passwd

$ awk -f myscript.awk myfile.in


------------------SED---------------------
1)  Stream editor for filtering and transforming text
2) Sed works as filter, which makes it particularly useful for scripts, sed works line oriented. The most simple commands are a pattern and an action . if no pattern is given, then action is applied to all lines, otherwise it is applied only to lines matching the pattern.

EXAMPLE::
1) A sample application of sed would be to delete the first 10 lines of stdin and echo the rest to stdout:

$cat /etc/passwd| sed -e '1,10d'

2) If we wanted to print only the first ten lines, we would have deleted all the lines starting with 11:

$cat /etc/passwd| sed -e '11,$d'

3)Another way to get only the first 10 lines is to use the -n option:

$cat /etc/passwd | sed -n -e '1,10p'

4)Commands in sed programs are separated by new lines. So if we wanted to delete the lines 1 to 4 and 6 to 9, we could use:

$cat /etc/passwd | sed -e '1,4d 
6,9d'

5)If we want to put command in the script files instead of using the command line.

$sed -f program.sed