http://www.s0ftpj.org/docs/lkm.

htm

Detecting Loadable Kernel Modules (LKM) Purpose The purpose of this paper is cover LKM basics, detecting "trojaned" LKM's and figuring out which LKM is installed on your machine. LKM What is a LKM? Loadable Kernel Modules (LKM) are files that contain dynamically loadable kernel components. LKM's are normally used to load device drivers and other hardware drivers. LKM's can be found on Linux, Solaris and BSD (Open, Free and Net). This paper will focus on Linux. Linux comes with various tools to assist the system administrator in loading, listing and unloading of the kernel modules. Before we cover the tools, lets look at some important files and directories associated with Kernel Modules. The first directory we want to look at is /lib/modules/"kernel_version". Lets look and see what files we can find under /lib/modules."Kernel_version": block build cdrom fs ipv4 misc modules.dep modules.isapnpmap modules.pcimap modules.usbmap net pcmcia scsi usb video Table 1. /lib/modules listing Table 1 shows us both directories and files related to LKM's. The only ones listed that are not directories are modules.dep, modules.isapnpmap, modules.pcimap and mosules.usbmap. Those are actual files that list modules within the system (combined). Lets take a quick look at the net directory (This is not a complete listing, do to space): 3c59x.o 3c90x.o 82596.o 8390.o ac3200.o acenic.o arlan-proc.o Table 2. net directory There are a couple of important bits of information we can gather from this table. The first bit is, modules are listed as .o. Why? Because they are object files

1 of 8

10/30/2012 05:18 PM

http://www.s0ftpj.org/docs/lkm.htm that contain the actual module itself. The second bit is, the listing is not complete. I did not have room to put all of the modules. The second important directory/file we want to look at it /etc/conf.modules. Conf.modules is the configuration file that allows the system administrator to specify a variety of parameters that control the loading of the modules. The conf.modules file typically looks like this: alias parport_lowlevel parport_pc

Table 3. conf.modules file The conf.modules file allows the administrator to assign alias to commonly used modules. Note: This file is not required. A system administrator can create his/her own configuration file. A system administrator can do that by using the modprobe C command. Since I mentioned modprobe earlier, lets briefly take a look at modprobe and the other tools available in Linux that load and unload modules. The first tool is modprobe. Modprobe can load single and multiple modules. It uses the modules.dep file to look up dependencies. Depmod creates a Makefile like dependency file. This file is called modules.dep. The last three commands I will cover are lsmod, insmod and rmmod. lsmod provides the administrator a listing of all modules currently loaded in the kernel. This list can also be found at /proc/modules. This is helpful if you want to figure out what modules are currently running. You will see a little later that lsmod is not always a good tool to use for the detection of rootkit LKMs. insmod loads modules(seems simple). Some of the trojaned LKM code I have seen uses insmod -f to load the trojaned module. rmmod is normally used to remove modules from the kernel. Normally rmmod will NOT remove any trojaned LKM. We will discuss details of that a little bit later. Hopefully, this section provided everybody the basics of Loadable Kernel Modules. If this was not detailed enough see http://www.kernel.org/LDP/. LKM rootkits and kstat Recently, there has been a lot of press about adore. Well, the worm adore not the LKM adore. So I decided to look into the LKM adore(everyone else has looked at the worm). BTW, if want to download adore go to http://packetstorm.securify.com/filedesc/adore-0.34.html. Adore is a Linux LKM rootkit. It is easy to install and only requires a few minor adjustments when configuring. Adore can be installed with a default configuration or the user can make changes to some of the code. The readme file recommends that the user change the settings for ELITE_CMD and for HIDDEN_PORT. When you run ./configure, adore asks you for a password. This is for the backdoor port (hence, change the HIDDEN_PORT) If you want more information on the install you will have to download it. I ran the default installation, which consisted of running./configure and make. After running make, you will see two files (ava, startadore). In order for adore too execute you have to run startadore. Once you run startadore the user can then run ava. Table 4 is the output from ./ava. Usage: ./ava {h,u,r,R,i,v,U} [file, PID or dummy (for U)] h hide file u unhide file r execute as root 2 of 8 10/30/2012 05:18 PM

http://www.s0ftpj.org/docs/lkm.htm

R U i v

remove PID forever uninstall adore make PID invisible make PID visible

Table 4. Ava output Table 4 shows us the options that adore provides to us. I will not cover each switch and what it does only because it does not help us detect this rootkit. Now that we have covered the basics of adore (LKM) lets look at how we detect adore and other rootkits. Many rootkits hide processes, directories, files and even connections. But many of them do so by modifying the source code of binaries such as ps, df, netstat, top and lsof. There are a couple of ways to detect these types of rootkits (i.e. t0rn): 1) md5 checksums 2) Compiling these binaries from a known good source (i.e. cd-rom, disk). 3) Some rootkits have a default port that an administrator can focus in on. 4) Running a program such as chkrootkit Many of the techniques used to detect rootkits like t0rn are not effective against LKM rootkits (chkrootkit can detect some of them). Since LKM rootkits access the kernel, they can hide processes, connections, directories and files without modifying the binaries. MD5 checksums become useless because there are no files being modified. This means the checksums will not change. Checking ports MIGHT be an option BUT that only tells you that you have been rooted. Well, how can I detect these monsters? Easy. There is a program I highly recommend to everyone concerned with LKM rootkits. This program is called KSTAT. You can find it at http://s0ftpj.org/en/site.html. KSTAT works by checking the memory (/dev/kmem) for information about the host(including LKMs). Want to see what kstat looks like? Good. Here is the output for kstat: Usage: kstat [-i iff] [-P] [-p pid] [-M] [-m addr] [-s] -i -P -p -M -m iff may be specified as 'all' or as name (e.g. eth0) displays info about the queried interface displays all processes pid is the process id of the queried task displays the kernel's LKMs' linked list addr is the hex address of the queried module displays info about the module to be found at addr

3 of 8

10/30/2012 05:18 PM

http://www.s0ftpj.org/docs/lkm.htm

-s

displays info about the system calls' table

As you can see, kstat provides a person with many options for detecting these rootkits. Lets go through some of the options and from these options we will learn how to detect the following LKM rootkits: 1) knark 2) adore 3) rkit BTW, if theres any rootkits I have omitted please let me know and I will update the paper. : ) Kstat s seems to be the best way to detect LKM rootkits. The other options are really great, but s works all the time. Here is an output from kstat s: SysCall Address sys_exit 0xc0117ce4 sys_fork 0xc0108ebc sys_read 0xc012604c sys_write 0xc0126110 sys_open 0xc0125c10 sys_close 0xc0125d60 sys_waitpid 0xc0117ff8 sys_creat 0xc0125ca4 sys_link 0xc012de60 sys_unlink 0xc012dc90 sys_execve 0xc0108f18 sys_chdir 0xc01254a0 sys_time 0xc01184b4 sys_mknod 0xc012d77c sys_chmod 0xc01256e4

Table 5. kstat s Table 5 is not a complete output from kstat s, but it does give us an idea as too what the output looks like. Remember s provides us with a picture of the sys_call_table. Keep this information fresh as we will revisit this again later. Kstat P is another switch that is very effective. Kstat -P shows us all of the processes running at that time. This includes the processes hidden by an LKM rootkit. Table 6 is an example of kstat P: PID 1 2 3 4 5 6 241 256 PPID 0 1 1 1 1 1 1 1 UID 0 0 0 0 0 0 1 0 GID 0 0 0 0 0 0 0 0 COMMAND init kflushd kupdate kpiod kswapd mdrecoveryd portmap lockd

4 of 8

10/30/2012 05:18 PM

http://www.s0ftpj.org/docs/lkm.htm

257 266 280 331

256 1 1 1

0 0 0 0

0 0 0 0

rpciod rpc.statd apmd syslogd

Table 6. kstat P When I first ran this program I wasnt sure kstat could do what it said butit proved my doubts WRONG. Here is what I did to verify this switch. I used ava -i 241(portmap). I then ran kstat P. As you can see from table 6, portmap(bold) shows up. I then ran ps-ef(Table 7)and could not find it. Lsof also did not show it. Table 7 is the output from ps ef: UID root root root root root root root root root root root root nobody nobody nobody nobody nobody daemon root root root root root root xfs root root root root PID 1 2 3 4 5 6 256 257 266 280 331 340 354 357 359 360 361 372 386 404 418 462 477 491 531 571 572 573 574 PPID 0 1 1 1 1 1 1 256 1 1 1 1 1 354 357 357 357 1 1 1 1 1 1 1 1 1 1 1 1 C 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 STIME Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 Mar30 TTY ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? tty1 tty2 tty3 tty4 TIME 00:00:06 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:01 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 CMD init [3] [kflushd] [kupdate] [kpiod] [kswapd] [mdrecoveryd] [lockd] [rpciod] rpc.statd /usr/sbin/apmd -p 10 -w 5 -W -s syslogd -m 0 klogd identd -e -o identd -e -o identd -e -o identd -e -o identd -e -o /usr/sbin/atd crond inetd lpd sendmail: accepting connections gpm -t ps/2 httpd xfs -droppriv -daemon -port -1 login -- root /sbin/mingetty tty2 /sbin/mingetty tty3 /sbin/mingetty tty4

5 of 8

10/30/2012 05:18 PM

http://www.s0ftpj.org/docs/lkm.htm root root nobody nobody nobody nobody nobody nobody nobody nobody root root 575 1 0 Mar30 tty5 00:00:00 /sbin/mingetty tty5 576 1 0 Mar30 tty6 00:00:00 /sbin/mingetty tty6 4290 491 0 Apr01 ? 00:00:00 httpd 4291 491 0 Apr01 ? 00:00:00 httpd 4292 491 0 Apr01 ? 00:00:00 httpd 4293 491 0 Apr01 ? 00:00:00 httpd 4294 491 0 Apr01 ? 00:00:00 httpd 4295 491 0 Apr01 ? 00:00:00 httpd 4298 491 0 Apr01 ? 00:00:00 httpd 4299 491 0 Apr01 ? 00:00:00 httpd 8073 571 0 Apr02 tty1 00:00:00 -bash 10659 8073 0 11:24 tty1 00:00:00 ps -ef Table 7. ps ef output There are two other switches we will go over, they are kstat p and kstat M. First, lets look at kstat p. kstat p gives us more information about a process. In order to run kstat p you will need to provide a process id. For example: kstat p 241. This checks process 241 (portmap) and provide an output. Sometimes it can provide us with more information about a LKM rootkit. TABLE 8 is the output from kstat p 241: Name: portmap State: S (sleeping) Pid: 241 Ppid: 1 (init) Uid: 1 1 1 1 Gid: 0 0 0 0 Flags: PF_FORKNOEXEC PF_SUPERPRIV Crucial Capabilities Check Open Files 0 CHAR /dev/null 1 CHAR /dev/null 2 CHAR /dev/null 3 0.0.0.0:111 0.0.0.0:0 4 0.0.0.0:111 0.0.0.0:0 7 FIFO /// 8 FIFO /// 21 CHAR /dev/null

Table 8. kstat p output Finally, lets look at kstat M. Under normal conditions an administrator can perform lsmod or more /proc/modules and find out what modules he/she are running. When a machine has been rooted you cant trust lsmod for accurate information. Kstat M will catch many of the basic LKM rootkits that I have tested. Kstat M list all of the modules loaded. I have seen it where kstat M did list anything for knark, butnothings perfect. The output from kstat M is similar to lsmod. Detecting knark, adore and other LKM rootkits So far we have covered a lot of material about LKMs, rootkits and kstat. Now we are going to put all of that together and learn how to detect some of the

6 of 8

10/30/2012 05:18 PM

http://www.s0ftpj.org/docs/lkm.htm LKM rootkits available today. The first LKM rootkit we want to look at is knark. Knark is probably the best-known LKM rootkit and one of the best written as well. Detecting it with kstat is fairly straight up as well. Remember Table 5? Well in order to detect a knark LKM rootkit you will need to run kstat s. Once ran you need to look for the following:

sys_fork 0xc284652c WARNING! Should be at 0xc0108c88 sys_read 0xc2846868 WARNING! Should be at 0xc012699c sys_execve 0xc2846bb8 WARNING! Should be at 0xc0108ce4 sys_kill 0xc28465d4 WARNING! Should be at 0xc01106b4 sys_ioctl 0xc2846640 WARNING! Should be at 0xc012ff78 sys_settimeofday 0xc2846a8c WARNING! Should be at 0xc0118364 sys_clone 0xc2846580 WARNING! Should be at 0xc0108ca4

Table 9. knark detection Lets take a closer look at this table and see what all of this mess means to us. First, we see that there are seven (7) sys_call_table entries that have warnings. Lets look at sys_settimeofday, here we see that currently it is making its home in memory at 0xc2846a8c. kstat s tells us that this is wrong and it should be at 0xc0118364. When knark is installed it changes the sys_call_table and the memory locations where you can find: sys_fork, sys_read, sys_execve, sys_kill, sys_ioctl, sys_settimeofday and sys_clone. This is how you could detect knark. Knowing which sys_call_table entries have been changed could help you identify the actual rootkit itself. Knark changes seven(7) total. After running kstat s, the next step would be to run ps-ef along with kstat P and compare the two. If the are any differences in the two you can then take the correct action. Keep in mind that memory locations can change from box to box. As I said earlier, detecting LKM rootkits with kstat is quite simple. Lets look at adore. Table 10 will show us what adore changes:

sys_fork sys_write sys_close sys_kill sys_mkdir sys_clone sys_getdents

0xc4051428 0xc4051590 0xc405163c 0xc40514d0 0xc405172c 0xc405147c 0xc40512a4

WARNING! WARNING! WARNING! WARNING! WARNING! WARNING! WARNING!

Should Should Should Should Should Should Should

be be be be be be be

at at at at at at at

0xc0108c88 0xc01269b8 0xc01264a4 0xc011060c 0xc012e540 0xc0108ca4 0xc013022c

Table 10. Adore detection Adore changes seven (7) entries as well (just like knark). Knark and adore only change three (3) of the same sys_call_table entries. They are sys_fork, sys_kill, sys_clone. This is a good thing because it allows us the ability to detect each one separately. Again, once this has been done, the administrator can then run kstat P and ps-ef and compare the two to figure out what processes are running and hiding.

7 of 8

10/30/2012 05:18 PM

http://www.s0ftpj.org/docs/lkm.htm The last rootkit we will look at is rkit. This rootkit does not hide itself as well as the other two do. As a matter of fact it only changes sys_setuid and can be found using kstat M. Conclusion LKM rootkits can make a system administrators life a nightmare. They are hard to detect, but using tools like kstat and understanding what the rootkit changes can make our life easier. Since tools like kstat are available, it would help systems administrators if they took a picture of the sys_call_table after a fresh install and any upgrades. This will help in identifying the good from the bad. Although I have not had the time, one could write a shell | perl script to automate this process and check weekly, daily, every 10 minutes or whatever. Resources Kstat http://s0ftpj.org/en/site.html Knark http://members.prestige.net/tmiller12/papers/KNARK.htm LKM rootkits http://packetstorm.securify.com/groups/thc/LKM_HACKING.html Author Toby Miller is a GIAC Certified Analyst and MCP. He is currently working towards his CISSP and RHCE. Toby has contributed to 2 books, written papers for SANS, Securityfocus, performs Risk Assessments and runs a lab for a living. For entertainment Toby likes to analyze exploit signatures on his home network. Toby can be reached at tmiller@va.prestige.net.

8 of 8

10/30/2012 05:18 PM