jmglov's Linux Kernel Hacking FAQ

Please note that this FAQ is based on my own kernel hacking, which has been exclusively on the Intel i386 (also know as 80x86) architecture. Thus, the FAQ may be less helpful on other architectures. In any case, as the Linux kernel evolves quickly, following my instructions verbatim is probably a bad idea. Use this FAQ as a guide to understanding how to hack on the kernel. Do *not* copy my code, paste it into the kernel, and expect good results.

Just in case, this FAQ is provided with no warranty whatsoever, and no guarentees that the code and instructions contained herein will not cause your hardware to spontaneously combust. Proceed at your own risk.

0. Table of Contents

1. Doesn't kernel hacking take a huge brain? How do I get started?
2. What tools do I need to hack on the kernel?
3. How do I declare and use global variables?
4. How do I create a new system call?
5. Given a filename, how do I grab the inode to which it refers?
6. Can you explain the ext2 filesystem to me?
7. Who is responsible for this FAQ, and how do I shower them with thanks?

The short answer is no, anyone with some decent knowledge of C should be able to write code that actually *works* in the kernel. You do not need a degree in Computer Science / Computer Engineering / Rocket Science, or a membership in MENSA.

To get started, grab yourself a kernel and read the section on necessary tools.

Creating a new system call requires three steps. First, you must add a #define to include/asm-i386/unistd.h (if you are hacking on a kernel that will need to run on a different architecture, or want your system call to work on non-Intel architectures, you will need to check out the other include/asm-* directories). Open this file in your favourite text editor, then locate the long list of

#define __NR_foo     25

#define __NR_<syscall>  <num>

The second step is to add a symbol definition to the syscall table in arch/i386/kernel/entry.S (again, if you care about architectures other than the i386, you will need to deal with them as well). Open this file in a text editor and locate the lines:

.data
ENTRY(sys_call_table)

.rept NR_syscalls-190
        .long SYMBOL_NAME(sys_ni_syscall)
.endr

.long SYMBOL_NAME(sys_<syscall>)

The third and final step to implementing a new system call is to actually write the function. The easiest thing to do is to add your function to a source file that is already included in a standard Linux build. It makes sense to put the function in a source file that contains similar functions. For example, if you are implementing a syscall that reads the entire contents of a file at once, stick it in fs/read_write.c, where similar syscalls reside.

Now that you have chosen a file for your syscall implementation, you must resist the urge to slap an ordinary C function into it. Why? It will not work. Syscalls are magical, as they must run in kernel mode. On most architectures, the only way to get from user mode to kernel || supervisor || privileged mode is to toss an interrupt. On the i386 architecture, Linux handles syscalls like this:

Warning! The following expects you to know a bit about the Intel i386 architecture and assembler. If you do not, you might want to remain in the dark as to how syscalls really work and skip this explanation.

1. When the user program invokes a syscall, the system call software interrupt is thrown. Thanks to an i386 trap gate sitting at address 0x80 in the Interrupt Descriptor Table (which is set up *very* early in the boot process), the interrupt causes the kernel function system_call() to be called, with the syscall number sitting in register eax.
2. The system_call() function reads the syscall number and uses it as an offset to read the syscall table (defined in arch/i386/kernel/entry.S, remember?). It saves all the registers associated with user mode by invoking the SAVEALL macro, then jumps to the address sitting at that offset, which should correspond to a function.
3. In order for this voodoo to succeed, the function that system_call() tries to jump to must be declared with the asmlinkage keyword. What asmlinkage tells the compiler is this: "Instead of handling functional arguments in the normal C fashion (i.e. looking for the first several in registers), they will all be on the stack." If you leave out the asmlinkage in your function definition, the function is unlikely to work the way you expect it to, as you have no idea what will be lurking in the registers that C normally uses for function arguments. (A syscall with no arguments *might* work, but I would not get into the habit of ignoring the asmlinkage requirement.)
4. Now that the actual function that implements the syscall that was called by the user program is executing in kernel mode, it behaves exactly as a normal C function would be expected to behave. When the function ends (probably with some return value--few syscalls are void), the system_call() function quickly puts the return value into the location where register eax was saved previously. Then the kernel executes the ret_from_sys_call() function (also assembler, also in arch/i386/kernel/entry.S).
5. ret_from_sys_call() mainly restores the user context by invoking the RESTORE_ALL macro. After this, it returns to userland (most likely rescheduling the user process that made the syscall, but we can ignore this and skip ahead to the next time that the user process runs). At this point, the return value of the function that was executed by system_call() is where the user program expects to find it (C convention: return value is in the eax register).
6. There you have it, that is how a syscall works. However, there are two important caveats left to consider:
   • Return value: the Linux convention is for most syscalls to return a long. Given that the i386 architecture (on which most Linux kernels run) is 32-bit, the biggest thing that will fit in the eax register is a long. Hence, long is the largest valid return value for a function. "Now wait just a blessed second!" you might be saying, "C lets you return things that will not fit in eax by sticking a pointer to the return value in eax!" Yes, this is correct, but the reason that Linux does not allow this is this: if you try to return a pointer to something from a syscall, which is running in kernel mode, said pointer will point to an address in memory that is simply *not accessible* from user mode. Which would end up doing very little good. Therefore, if you feel the need to have your syscall return something larger than a long, you will have to be a bit creative. Read on.
   • Arguments: If you are a reasonably seasoned C programmer, you are probably in the habit of passing damn near everything by reference or as a pointer. Pass by value is not too useful in most cases. Well, you cannot do this with a syscall. Why not? For the same reason that you cannot return a pointer from a syscall (see above): the kernel uses a different chunk of memory than user mode, and thus code that runs in kernel mode uses a different addressing scheme (or rather, virtual addresses in the kernel address space, as opposed to virtual addresses in the address space of whatever user process is running). Therefore, when you are implementing your syscall, you need some way of translating back and forth between userland addresses and kernel memory addresses. Luckily, Linux provides you with just such a method: the copy_to_user() and copy_from_user() macros. Basically, all you need to do inside your syscall function to deal with a pointer argument is:

     copy_from_user( &target, source, sizeof( source ) );

     where &target is the address of your struct that you declared somewhere in kernel memory (you have to define the struct or do some malloc() stuff, because the kernel is not going to do it for you!), and source is a pointer to a struct in userland (the argument in question). If you need to copy a normal data type instead of a struct, you should use put_user() and get_user() instead. (Of course, I would just pass by value if I wanted an int argument...) Conversely, when you are ready to return from the syscall function, and you expect the user to have access to the structure to which you were given a pointer, you need to do this:

     copy_to_user( target, &source, sizeof( source ) );

     But this time, things are backwards: target is the value of the pointer argument, and &source is the address in kernel memory where your struct lives.

Right, so now that you know how a syscall works, write the function and put it in the appropriate source file. If you skipped the above description of syscall voodoo (presumably because you wanted to avoid the specifics of Intel i386 architectural organisation and the unfortunate side-effect: a splitting headache), here is what you need to know:

• Stick an asmlinkage in front of your function definition or expect the function to mysteriously fail to handle arguments.
• You are advised to make your function return a long. If you do not want to return a long, you really should read the bit about return values, above.
• If you want to make some of the arguments to your function pointers, read my caveat on arguments, above.

retval = syscall( num, arg0, arg1, ... argN );

Example: We want to add a system call named foobar() to the kernel, which is supported on the Intel i386 architecture. We open include/asm-i386/unistd.h in our favourite editor (XEmacs) and find the last syscall definition:

#define __NR_vfork              190

#define __NR_foobar             191

Now, we open the arch/i386/kernel/entry.S file and, right before

.rept NR_syscalls-190
        .long SYMBOL_NAME(sys_ni_syscall)
	.endr

.long SYMBOL_NAME(sys_foobar)

Since our foobar() syscall is going to fill out a trivial struct that we claim has some relevance to read()ing and write()ing, we will chose fs/read_write.c as the target for our function. We open the file and slap this code into it, right at the bottom:

/** trivial_t{}
 *
 * A pretty trivial data structure.
 */
 
struct trivial_t
{
  int  number;   /* some trivial number */
  long bignum;   /* some big number */
} trivial; /* trivial_t{} */


/** sys_foobar()
 *
 * Implements the foobar() syscall: copies the kernel's trival struct into
 * userland.
 *
 * Returns:
 *  retval  0 on success, -1 on failure
 */
 
asmlinkage long sys_foobar( struct trivial_t *buf /** pointer to a trivial_t
                                                   *  struct that lives in
						   *  userland */
			    )
{
  /* Copy our struct to the userland one */
  if( (copy_to_user( buf, &trivial, sizeof( trivial ) )) != 0 )
    return -1;
  
  /* Victory! */
  return 0;
  
} /* sys_foobar()

Here ends our example.

Many syscalls in Linux take filenames that could be relative, and could contain ".." or ".", and could be hard or soft (symbolic) links. Given this mess, it would be nice to have a way to take a filename and resolve it to the underlying inode.

Luckily, there is just such a procedure. The function that you are after is path_walk(), which is defined (in Linux 2.4.5) at line 422 of fs/namei.c. What it does is takes a valid pathname, and returns the actual dentry (wrapped in a nameidata struct, but who cares) to which the pathname refers.

Use LXR to do an identifier search on path_walk and dentry. What you probably want to do is something like this:

asmlinkage long sys_foo(const char * pathname)
{
  int error 0;
  char * name;
  char fqfn[1024]; /* this will hold your fully qualified filename */
  struct dentry *dentry;
  struct nameidata nd;

  name getname(pathname);

  ...

  if (path_init(name, LOOKUP_PARENT, &nd))
    error = path_walk(name, &nd);

  ...

  strcpy(fqfn, nd.dentry->dname->name);

fqfn will now hold your full filename. Your inode will be nd.dentry->d_inode.

Maybe. Take a look at the slides (PDF) of a lecture I gave on the subject as part of my "Linux Kernel Internals" course at The College of William and Mary. If you feel like updating the slides for the 2.6 kernel (please?) or otherwise want the LaTeX sources, help yourself to a tarball.

The principle parties are:

• Josh Glover -- Wrote this FAQ
• Dr. Phil Kearns -- Taught the "Linux Kernel Internals" course at The College of William and Mary that allowed me to learn how to hack
• Godwin Stewart -- Provided me with some helpful feedback / corrections

I am actively maintaining this FAQ, so please do send questions, comments, and corrections to me. My email address is: "jmglov{SYMBOL_0}wmalumni.com{SYMBOL_1}{TLD}", where {SYMBOL_0} == '@', {SYMBOL_1} == '.', and {TLD} == "com". There, I hope *that* stops the bloody harvester bots.