Variable Substitution

# !/ bin / bash # example showing the use of default variable values echo echo read echo echo

$ { STR : - " Hello World ! " } $ { STR - " Hello World ! " } STR $ { STR : - " Hello World ! " } $ { STR - " Hello World ! " }

" ( Default if Null ) " " ( Default if not set ) " " ( Default if Null ) " " ( Default if not set ) "

${STR:-abc} Use default abc if null ${STR-abc} Use default abc if not set ${STR=abc} or ${STR:=abc} set STR to abc ${STR?} or ${STR:?} print error

# !/ bin / bash : $ {1? " Usage : $ 0 ARGUMENT " }

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 32

Variable Substitution - String Chopping

# !/ bin / bash # example showing the use of string chopping STR = " This is a String " echo $ { STR :0:10} chopped $ { STR :10} ${STR:x} All characters starting at character x

${STR:x:y} All characters starting at x up to (not including) y ${STR# pattern } delete pattern (smallest match from start)

${STR% pattern } delete pattern (smallest match from end)

${STR## pattern } delete pattern (largest match from start)

${STR%% pattern } delete pattern (largest match from end)

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 33

Partial List of Commands/Variables/. . . so far

man $STR #! let $3 $a -le $b mv $$ script sufx chmod esac ${STR:-x} apropos read mkdir continue shell builtin emacs touch chgrp a=$((3+5)) ${STR%.txt} cd $* $@ echo $STR rm - - help bla rmdir ${STR:3:8} ls echo $STR if expr cp echo $PS3 u+x info vi a=$[3+5] $? clear break 2 $# ${#a} a=3 test pwd for $RANDOM

shift

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 34

Part III Shell Cong

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 35

Cong of PS1-PS4 (Prompt Statement Variables)

\d \h \H \l \n \s \t \u \w

...

current date hostname fully qualied domain name name of the terminal newline name of the shell time 24-hour-format (try \T, \@) username current working directory (try \W)

PS1 = " \u ,\ d ,\ t ,\ W /: "

PS1 - Default interaction prompt PS2 - Continuation interactive prompt PS3 - Prompt used by select inside shell script PS4 - Used by set -x to prex tracing output
36

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013

Special Directories
. current directory .. parent directory ~ own home directory ~user home directory of user ~- previous directory cd / usr / local / bin pwd cd ~ pwd cd ~ pwd cd .. pwd

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 37

Aliases and Variables

alias - short form for any command
alias alias ll = ' ls -l '

environment variables Variables are not only within scripts, but also in the shell By setting a variable, it is present in the current script/shell By exporting it (export VARIABLE=value), it is also present at all child processes (NOT at the parent) Many environment variables are already set by default
PATH =/ home / neckel / bin /: $ PATH ; echo $ PATH env

local variables set shows all local and global variables and functions unset deletes a variable
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 38

Bash cong
prole and rc
especially aliases are used in every session should not be dened each time e.g. clean up (kill jobs) on logout Prole is used for complete session bashrc is used for a single terminal You might even run arbitrary code on login or logout

Files
/etc/profile /etc/bash.bashrc $HOME/.bash profile $HOME/.bashrc $HOME/.bash logout
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 39

Pipes
stdin/stdout: Standard input/output for programs Most Linux programs read from stdin and write to stdout Pipes are used to redirect input and output cmd1 | cmd2 connect the output of cmd1 with the input of cmd2 cmd > file redirect the output of cmd to le cmd 2> file redirect stderr of cmd to le cmd > file 2>&1 redirect stdout to le and stderr to stdout cmd &> file redirect all output to le cmd >> file redirect the output of cmd and append it to le cmd < file use the content of le as stdin for cmd cmd <<Endmark Read from stdin until Endmark is inserted in a

seperate line

; command separator
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 40

Wildcards

Wildcards are expanded by the shell * zero or more characters ? exactly one character [abcd] one of the characters a-d [a-d] same [!a-d] any other character {first,second} either first or second Similar pattern exist in other contexts as well (compare regular

expressions), but always a bit different. . . :-(

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 41

Part IV Bash Advanced: Regular Expressions and More

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 42

Regular Expressions???
The set of regular languages over an alphabet and the corresponding regular expressions are dened recursively as follows: The empty language is a regular language, and the corresponding regular expression is . The empty string {} is a regular language, and the corresponding regular expression is . For each a in , the singleton language { a } is a regular language, and the corresponding regular expression is a. If A and B are regular languages, and r1 and r2 are the corresponding regular expressions, Then A U B (union) is a regular language, and the corresponding regular expression is (r1+r2) AB (concatenation) is a regular language, and the corresponding regular expression is (r1r2) A* (Kleene star) is a regular language, and the corresponding regular expression is (r1*)
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 43

Regular Expressions!!!
What does all this mean to you, as a user? Absolutely nothing. As a user, you dont care if its regular, nonregular, unregular, irregular, or incontinent. So long as you know what you can expect from it, you know all you need to care about. Jeffrey Friedl, author of Mastering Regular Expressions Did you ever. . .
. . . search for a character or string in a text le? . . . use tab for auto-completion? . . . use the * in a terminal for selecting a group of les? ...

Then youve already somehow used regular expressions.

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 44

Regular Expressions
When searching for a string, exactly the given character

sequence is searched

regular expressions get powerful as a tool to nd patterns Additionally to ordinary characters, which stand for themselves

special characters are used which are interpreted in a special way. The exact syntax for special characters differs between different implementations Example to identify an email address: [^@]\+@.\+\.[^.]\+ regular expressions usually look very cryptic! But imagine you would have to write a normal program to identify an email address! Regular expressions can be used to nd/replace groups of strings which can be described by a pattern
45

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013

Special Characters for Regular Expressions

char A character maching itself * matches zero or more occurences (greedy!) of the previous

expression

\+ matches one or more occurences (GNU extension) \? matches zero or one occurence (GNU extension) \{i\} matches i occurences \{i,\} matches i or more occurences \{i,j\} matches i to j occurences . matches an arbitrary character matches the beginning of a string $ matches the end of a string [list] matches a single character from the list ([list] any character

not in the list)

\n matches newline \(\) denes a group, reuse via \1 (1st group), \2 (2nd group) \| allows for a logical OR
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 46

Character Classes
[:digit:] 0 to 9 (alternative: [0-9]). [:alnum:] alphanumeric character 0-9 or A-Z or a-z. [:alpha:] character A-Z or a-z. [:xdigit:] Hexadecimal notation 0-9, A-F, a-f. [:punct:] Punctuation symbols, e.g. . , ? ! ; : # $ % & ( ) [:print:] Any printable character. [:space:] whitespace (space, tab, ...). [:upper:] uppercase character A-Z (alternative: [A-Z]). [:lower:] lowercase character a-z (alternative: [a-z]).

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 47

programs using regular expressions

grep print lines matching a pattern grep -i case insensitive grep -v invert match grep -n additionally print line number tr translate: echo "lower/UPPER"| tr "A-Z""a-z" sed stream editor awk pattern scanning and processing language find search for les in a directory hierarchy ... most editors can handle regular expressions

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 48

sed - Stream Editor

works (as most bash programs) on a stream of data data is processed linewise no way of going back a line ( efcient) apply some action on selected lines (using addresses to select) Addresses address: line number or regular expression zero, one or two (comma-separated) addresses can be used zero addresses: all lines are processed one address: all lines matching the address are processed two addresses: match from rst to second address n match only line number n n~step starting from line n, match every stepth line /pattern/ lines matching the given regular expression i,j all lines from i to j (including both) $ match the last line

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 49

sed - Editing Commands

parameters
-n supress output (per default, all lines are printed) -e editing command follows in command line -f script to be read from le

commands
p print line = print line number i\text and a\text insert text before/after matched line c\text replace matched line by text s replace pattern, e.g. sed -e 's/search/replace/g'

(note: g at end optional, but replaces globally!)

w write line into given le, sed -n '/patt/ w out.txt' demo.txt

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 50

Operating on Files
less is more than more
more displays the content of text les pagewise only downward-scrolling is possible less is an extended and more exible more

outputting les
cat outputs the content of a le to stdout (try tac) cat concatenates several les head outputs the rst part of les tail outputs the last part of les for i in ` seq 30 ` ; do echo $ i >> temp . txt ; sleep 1; \ done & tail -f temp . txt

Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 51

Operating on Files (2)

file nd out more about le type cmp Compare les byte by byte diff Compare les line by line (graphical: kdiff3) patch apply patch (created by diff) to a le tar pack and compress les (try tar -xzf and tar -czf) sort sort lines of text les (and write to stdout) uniq report or omit repeated lines wc print newline, word, and byte counts for each le cut remove sections from each line of les echo " 1;2;3;4 " | cut -b 4 -5 echo " 1;2;3;4 " | cut -d " ; " -f 3 cut -d ' ' -f1 ,2 / etc / mtab
Tobias Neckel: Scripting with Bash and Python Compact Course @ Max-Planck, October 07 - 18, 2013 52