Basic System Administration

Your daily commands as root

 Becoming root
Avoid login as root over network (denied by default)

Use "/bin/su -" command from your regular account

- - runs user LOGIN scripts
- root can su to any userid without password.
- Note utility for NOLOGIN IDs.

$HOME is sometimes not /root but instead /; so watch what you delete!

Remove the current working directory (".") from your PATH

Never execute any regular user's program as root (possible Trojan Horse)

Use SSH, not TELNET over network to avoid sniffers

 root access - sudo
visudo (as root) creates /etc/sudoers file in the following format:
usernames/group servername = (usernames command can be run as) command

To implement: sudo <command>

Groups are the same as user groups and are differentiated from regular users by a % at
the beginning. The Linux user group "users" would be represented by %users.
You can have multiple usernames per line separated by commas.
Multiple commands also can be separated by commas. Spaces are considered part of the
command.
The keyword ALL can mean all usernames, groups, commands and servers.
If you run out of space on a line, you can end it with a back slash (\) and continue on the
next line.
sudo assumes that the sudoers file will be used network wide, and therefore offers the
option to specify the names of servers which will be using it in the servername position
in Table 9-1. In most cases, the file is used by only one server and the keyword ALL
suffices for the server name.
The NOPASSWD keyword provides access without prompting for your password.
Same concept as the SUID bit in permissions (4000).
 sudo examples

sudo command to run the command as USER

Examples:
- user1 localhost=/sbin/halt user1 can halt local system
[user1@student1]$ sudo /sbin/halt
password:
[user1@student1]$ System going down now!

- user2 ALL= NOPASSWD: /sbin/halt user2 can halt any system w/o password
- user3 instructor = /usr/sbin/* user 3 can run any command in /usr/sbin
 System Administration tools
man: Sections - 1 commands, 2 system calls, 3 C library routines, 4 devices
and networks, 5 file formats, 6 games and demos, 7 miscellaneous, 8 system
administration
info textinfo man page
vi editor (front-end to a lot of utilities)
su, sudo
df/du, mount
dump/restore, dd, cpio, tar, rmt, find, rsync
ps, at, batch, crontab, anacron, watch, kill, nice, nohup, killall,
useradd, usermod, userdel. passwd
groupadd, groupmod, groupdel
who, whoami, w, id
syslog
system configuration files /etc
 System information
hostname
uname a
dmesg
who, whoami, w, id
last (reboot)
which, where/whereis, apropos
hwclock
date
ulimit (user limits)
sysctl (system limits/settings)
cgroups
/etc/sysconfig
/etc/security
/proc
ps, pstree,
 System monitoring
sar
pmap
vmstat,
mpstat
iostat
nstat (network),
pidstat
dstat
free
lsof
top, ntop, iftop, latencytop
ulimit a (view), ulimit n (set) ulimits Hn, ulimits -Sn per user limits,
/etc/security/limits.*
See /etc/security/limits.conf: sysctl q (view), sysctl w (set) system
limits, /etc/sysctl.conf
strace (debugging)
 /etc/sysconfig
The /etc/sysconfig directory is where many of the files that control the system
configuration are stored for daemon processes or system services like networking.

Contents vary depending on products installed.

/etc/sysconfig is usually sourced (. /etc/sysconfig) in SYSV startup scripts.

Files in the /etc/sysconfig/ Directory

amd , apmd authconfig , clock , desktop , devlabel, dhcpd, firstboot, gpm, harddisks,
hwconf, i18n, init, ip6tables-config , iptables-config, irda, keyboard , kudzu, mouse,
named, netdump, network, network-scripts, ifup-xxxx, ntpd, pcmcia, radvd, rawdevices,
selinux, logrotate, samba , sendmail, spamassassin, squid , tux , vncservers, xinetd
 /proc
/proc is a virtual filesystem. It's sometimes referred to as a process information pseudo-
file system. It doesn't contain 'real' files but runtime system information (e.g. system
memory, devices mounted, hardware configuration, etc) for all processes started by init
including PID and startup commands. /proc was developed as a LINUX extension to
keep track of all the complex processes started in the system

For this reason it can be regarded as a control and information centre for the kernel. In
fact, quite a lot of system utilities are simply calls to files in this directory. For example,
'lsmod' is the same as 'cat /proc/modules' while 'lspci' is a synonym for 'cat /proc/pci'. By
altering files located in this directory you can even read/change kernel parameters
(sysctl) while the system is running.

The most distinctive thing about files in this directory is the fact that all of them have a
file size of 0, with the exception of kcore, mtrr and self.
 /etc/security
Central directory for system defaults

The limits.conf file defines process resource limits for users. (see ulimit)

opasswd - Store old passwords.

access.conf used to allow or restrict access to the system.

chroot.conf used to restrict users to there home directories

console.apps contains files which are same as service names.

console.perms and console.perms.d directory determine the permissions that

will be given to

The rest are PAM (Program Authentication Module) related.

 sysctl system limits
sysctl q, sysctl w, sysctl p file, sysctl -A
/etc/sysctl.conf

For network:
# Enable IP spoofing protection
net.ipv4.conf.all.rp_filter=1
# Disable IP source routing
net.ipv4.conf.all.accept_source_route=0
# Ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.icmp_ignore_bogus_error_messages=1
# Make sure spoofed packets get logged
net.ipv4.conf.all.log_martians = 1
# disable IPv6
net.ipv6.conf.all.disable_ipv6=1

Kernel isolation (test carefully on test system):

Turn on execshield
kernel.exec-shield=1
kernel.randomize_va_space=1
 ulimit user limits
ulimit - set user limits
-c maximum core file size (in 512-byte blocks)
-d maximum size of data segment or heap (in kbytes)
-f maximum file size (in 512-byte blocks)
-n maximum file descriptor plus 1
-s maximum size of stack segment (in kbytes)
-t maximum CPU time (in seconds)
-v maximum size of virtual memory (in kbytes)
-S soft limit
-H hard limit
/etc/security/limits.conf
 c(ontrol)groups
Cgroups allow you to allocate resourcessuch as CPU time, system memory, network bandwidth, or
combinations of these resourcesamong user-defined groups of tasks (processes) running on a
system.

A *cgroup* associates a set of tasks with a set of parameters for one or more subsystems. A
*subsystem* is a module that makes use of the task grouping facilities provided by cgroups to treat
groups of tasks in particular ways. A subsystem is typically a "resource controller in a hierarchy of
processes.

A cgroup is mounted as a virtual filesystem and can be modified to re-alllocate kernel resources.
Each cgroup is represented by a directory in the cgroup file system containing the following files
describing that cgroup:
- tasks: list of tasks (by pid) attached to that cgroup
- releasable flag: cgroup currently removeable?
- notify_on_release flag: run the release agent on exit?
- release_agent: the path to use for release notifications (this file exists in the top cgroup only) Other
subsystems such as cpusets may add additional files in each cgroup dir.
 PAM
Pluggable Authentication Module
Centralized authentication mechanism
Plug in different authentication methods
Different services can have different
authentication policies
Highly secure systems can require multiple
passwords to authenticate
 PAM Framework

ftp login ssh Applications

libpam PAM Library

conf

Modules

pam_unix pam_ldap pam_securetty

 PAM Stack
Modules are stacked (order is important)
Sample PAM configuration in /etc/pam.d:
interface control flag module name
auth required pam_nologin.so
auth required pam_securetty.so
auth sufficient pam_unix.so
auth required pam_ldap.so
 Security Enhanced LINUX
Kernel level security included since 2.60 kernel
Not an application interface. But sends access return codes to applications.
Can be combined with ACLs
/etc/selinux directory
Old gui system-config-selinux, new gui: policycureutils-gui
BE CAREFUL with changes. Especially deleting files.
See enforcing=0 or selinux=0 on Grub edit menu
Protects, files, processes, applications
Based on security context
An SE Linux security context is comprised of three parts: an "identity", a
"role", and a "type" for users and files or "domain" for processes. Default
context for root, the role is sysadm_r, and the domain is sysadm_t.
Configuration directory: /etc/selinux directory
 SELINUX Modes/Types
Enforcing: enable and enforce the SELinux
security policy on the system, denying access
and logging actions in /var/log/audit/audit.log
Permissive: enabled but will not enforce the
security policy, only warn and log actions.
Used for troubleshooting SELinux issues
Disabled: SELinux is turned off
Targeted: Specific processes
MLS/STRICT: VERY secure - systemwide
 SELINUX Policy
Policy: a set of rules the SELinux security engine that
defines types for file objects and domains for processes,
and user defined (identities) roles to limit the domains that
can be entered.
Strict - minimum access
Targeted specific processes
Unconfined not under SELINUX control, outside
SELINUX context
 SELINUX Access Control
Type Enforcement (TE): Type Enforcement is the
primary mechanism of access control used in
the targeted policy
Role-Based Access Control (RBAC): Based around
SELinux users (not necessarily the same as the Linux
user), but not used in the default targeted policy
Multi-Level Security (MLS): Not commonly used and
often hidden in the default targeted policy.
Shown with Z option (ls Z, ps Z etc)
 SELINUX Commands
sestatus: show SELINUX status
getenforce: show SELINIX status
setenforce: set SELINUX status
semanage: command line policy management
chcon: change SELINUX context
restorecon: restore default SELINIX context
audit2allow: Generate SELINUX policy from
/var/log/audit/audit.log
sealert: troubleshooting tool
ls Z, ps Z: show SELINUX context for files, processes
id: show the current user id context.
 Process info: ps -aux(BSD)
Common options:
-a print all processes involving terminals
-e print environment and arguments
-l long listing
-u print user information
-xi nclude processes with no terminals

Meaning of user information columns:

%CPU percentage use of CPU
SZ total size (in 1024 byte pages) of the process
RSS total resident size (in pages) of the process
STAT state of the process
TIME time, including both user and system time
 Process info: ps ef (System V)
Common options:
-e print all processes
-f print full listing
-l long listing (more info than -f)

pstree

Meaning of full listing columns:

S state
PRI priority
SZ total size (in 4096 byte pages) of the process
RSS total resident size (in pages) of the process
TIME starting time
TIME cumulative execution time
 Process Management
at schedule onetime batch job (scripts or commands).
Example: at now +1 minutes f somecommandfile.txt
batch interactive at command on some systems.
atq, atrm etc

anacron (Linux) workstation scheduler. See /etc/anacrontab

watch - execute a program periodically, display results fullscreen

cron scheduler routine must be a started process. See /etc/crontab/, /etc/cron.d,

/etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly
- crontab [-e] [-l] [-r] [u user] [filename]. Creates cron table -euses vi syntax.
- Each line contains:
mm(0-59) hh(0-23) dd(1-31) mm(1-12) day (0-6, 0=Sunday**) command
* - Is treated as a wild card. Meaning any possible value.
*/5 - Is treated as ever 5 minutes, hours, days, or months.
2,4,6 - Treated as an OR, if placed in the hours, this could mean at 2, 4, or 6 o-clock.
9-17 - Treats for any value between 9 and 17. If placed in day of month, days 9 -17.
** Note 1-7 on some UNIX systems
Example */5 * * * * echo hi there > /dev/tty2 2>&1
 Process Management
kill pid stop a process. -9 kills absolutely. killall

nice pid or command set process priority

nohup command run a process after logging off. Nohup.out contains job
output.

sighup option on some commands for forcing process to reinitialize.

zombie processes killed or abended processes with no parent.

Usually requires a reboot to reclaim resources. Can cause system instability.

See also: Ctrl Z, Ctrl C, fg, bg, &

 Runlevels

Run Level Generic Fedora Core Slackware Debian

0 Halt Halt Halt Halt

1 Single-user mode Single-user mode Single-user mode Single-user mode

Basic multi-user User definable -

User definable
2 mode (without configured the same Multi-user mode
(Unused)
networking) as runlevel 3

Multi-user mode -
Full (text based)
3 Multi-user mode default Slackware
multi-user mode
runlevel

X11 with
4 Not used Not used KDM/GDM/XDM Multi-user mode
(session managers)

Full multi-user mode

User definable -
Full (GUI based) (with an X-based
5 configured the same Multi-user mode
multi-user mode login screen) -
as runlevel 3
default runlevel

6 Reboot Reboot Reboot Reboot

 service/chkconfig
See systemd (RHEL/Fedora/Centos)
See upstart (Debian/Ubuntu)

service <service name> start/stop/restart/status

chkconfig --list [name]

chkconfig --add name
chkconfig --del name
chkconfig [--level levels] name <on|off|reset>
chkconfig [--level levels] name
 Backup/Restore
dump/restore - backs up file systems, has interactive mode, can do
incremental backups, maintains "sparse files", is most commonly used utility

cpio - can back up individual files/directories, handles special files, packs data
tighter than tar, skips bad spots on media on restore, use with find (some
versions of find have -cpio option for this purpose)

tar - backs up directory trees, does not back up special files, poor error
handling with media errors, does not pack blocks (GNU tar solves some of
these problems). Some LINUX/UNIX systems have built-in compress with z
flag.

dd - copies/converts files, can go from one medium to another, processes

whole entity or select blocks, can swap bytes and do ASCII/EBCDIC
conversions. Performs physical backup of raw devices.

rmt - used for remote tape operations. Varies by OS.

rsync used for directory synchronization, e.g. hot folders

 Backup Strategy
Physical (dd, cpio) usually devices (as root)
dd if=devicefile if=outputfile bs=blocksize count=#blocks

Logical (rmt, tar, dump/restore). Backup marker.

create: tar cvf tarfilename.tar [directory list] | compress
list: tar tvf tarfilename.tar
extract: tar xvf tarfilename.tar

Can use logical backups in conjunction with find command exec option (next panel)
for differential or incremental backups

Backup types: Full (everything)

Incremental (Difference since last backup)
Differential (Difference since last full backup)
Full + Incremental or Differential = Backup set
 find
Syntax: find starting-dir(s) matching-criteria-and-actions Matching criteria
-atime n file was accessed n days ago
-mtime n file was modified n days ago
-size n file is exactly n 512-byte blocks
-type c file type (e.g., f=plain, d=dir)
-name nam file name (e.g., `*.c')
-user usr file's owner is usr
-perm p file's access mode is p
-print display pathname
-exec cmd execute command ({} expands to file)

find examples
find . -name \*.c -print
find / -size +1000 -mtime +30 \ -exec ls -l {} \;
find / \( -name a.out -o -name core \ -o -name' \) -type f -atime +14 \ -exec rm -f {} \;
find / \( -perm 2000 -o -perm 4000 \) \ -print | diff - files.secure
 Disk management
df

mount / umount

du | sort rn | more

find / -name core -exec rm -f {} \;

Filesystems: /home, /var, /tmp (noexec), / (never full!)

mkdev, mkfs, fdisk

 User management
Set system account parameters (e.g., password aging, account expiration,
quotas, login scripts - /etc/profile, /etc/bashrc etc)

Determine login name, user ID (UID), group ID (GID)

Assign password (passwd)

/etc/passwd - logname:passwd:uid:gid:user info:home:shell

Passwords stored in /etc/shadow (pwconv)

Commands: useradd, usermod, userdel, chage, passwd

/etc/group: group:passwd:gid:members

Commands: groupadd, groupmod, groupdel

 User Security
ALWAYS use /etc/shadow (pwconv command)

Password aging:
get: chage -l userid
set: chage -M 60 -m 7 -W 7 userid

Lock/unlock and account

passwd l userid
passwd u userid

Limit password reuse:

vi /etc/pam.d/system-auth (RHEL/Fedora)
vi /etc/pam.d/common-password (Ubuntu)
Add: password sufficient pam_unix.so use_authtok md5 shadow remember=10

Verify root IDs: awk -F: '($3 == "0") {print}' /etc/passwd

Verify no password IDs: awk -F: '($2 == "") {print $1 }' /etc/shadow | grep /etc/passwd
Make sure they are /bin/nologin

No ROOT LOGIN, use su or sudo.

Disable ROOT login under FTP., SSH, GUI (procedures vary)

Configure password policy (LINUX: see pam_cracklib.so, other vary).

See defaults in /etc/login.defs for /etc/shadow values.

 SYSLOG
(r)syslog is a utility for tracking and logging all manner of system
messages from the merely informational to the extremely critical.
Sysogs stored in

In LINUX, system logs are stored in /var/log. System messages are

recorded in /var/log/messages. Other OSes may use different files in
different directories (e.g. /var/adm).

Each system message sent to the syslog server has two descriptive
labels associated with it that makes the message easier to handle.

- The first describes the function (facility) of the application that

generated it. For example, applications such as mail and cron generate
messages with easily identifiable facilities named mail and cron.
- The second describes the degree of severity of the message.
 SYSLOG
Severity: Level: Keyword: Description

0 emergencies System unusable

1 alerts Immediate action required
2 critical Critical condition
3 errors Error conditions
4 warnings Warning conditions
5 notifications Normal but significant conditions
6 informational Informational messages
7 debugging Debugging messages
 SYSLOG
configuration file /etc/rsyslog.conf or/etc/syslog.conf.

File consists of two columns.

- First lists the facilities and severities of messages to expect
- Second lists the files to which they should be logged.
- LINUX default directory is /var/log

Example:
*.info;mail.none;authpriv.none;cron.none /var/log/messages

Note other services may record messages in other files (e.g. sendmail)

Syslog is also a network service. A common implementation is to forward

system info to a common syslog server. TCP or UDP can be used.

Logs compressed, ,stored and optionally e-mailed by the logrotate function.

Definitions stored in /etc/logrotate.conf and /etc/logrotate.d
 System shutdown
Shutdown will run SysV K* scripts.

shutdown {h|-r} {time in minutes|now}

See also wall command

Other commands: halt, reboot, Ctrl-Alt-Del may bypass some processing. Not
recommended for production systems.

Reboots recorded in /var/log/wtmp or utmp

last (reboot) displays info