kernel authority and noted software
Interview by Stephen
Stephen Ibaraki, I.S.P., has an exclusive
interview with the internationally known
Linux expert, speaker, software engineer,
and author, Robert Love.
contributing editor for Linux Journal and a
software engineer for MontaVista Software.
His kernel projects include the process
scheduler, the preemptive kernel, the VM,
and multiprocessing enhancements. He
maintains schedutils and procps.
Amongst his most
recent writings is the book, Linux Kernel
Development, a practical guide to the
design and implementation of the Linux
Q: Robert, your
unique expertise will be of great value to
our audience. Thank you for doing this
A: It is a
Q: How did you
get into computing and into writing?
A: My interest in
computers goes back nearly as far as I can
remember. But while computers have always
been an interest, it was not until I started
programming that I could really call
computing a passion. Interestingly, I found
myself intrigued by operating systems very
early on, even before I began programming.
The concept of the inner internals
fascinated me more than the outer
applications. Growing involved with Linux,
and then ultimately the kernel itself, has
been extremely gratifying.
Similarly, I have
always enjoyed writing. Awhile back, I had
the opportunity to write an article for
Linux Journal, which I really enjoyed.
The entire writing process was very fun and
it is quite rewarding to write for an
audience similar to you and help teach them
something new. Thankfully, Linux Journal
has continued to provide an outlet for my
Q: How have you
resolved the roadblocks in your career? What
assets and processes proved to be the most
A: I feel the most important approach to
anything you do in life is to genuinely
enjoy it. If you sincerely revere something,
and it challenges you, a natural reaction is
to challenge it back and be the best you can
be at it.
Q: Describe the
major challenges you face in your job and
how you overcome them?
A: Some aspects
of kernel development can be rather
frustrating. I talk about some of them - and
their solutions - in my book.
The kernel is not
afforded many of the luxuries of user-space
programs and, consequently, some facets of a
kernel make working inside the kernel harder
than working outside of it. Wrapping your
head around some of the nastier bugs, for
example, is sometimes particularly painful.
Nonetheless, through eternal vigilance, all
problems are surmountable.
Q: Describe your
latest book? Who should read this book and
what background should they have to get the
greatest value from it? Why would someone
want to carefully study this book—what makes
this book particularly unique and special?
Kernel Development is a book on the
design and implementation of the latest
Linux kernel. I approach the material in a
hopefully fun and practical manner, with an
eye toward helping developers either
understand the kernel for the first time or
gain a stronger grasp on kernel internals.
The book is of interest to developers
wishing to (better) understand the Linux
kernel, either simply for fun or so they can
work on kernel projects. The book is also of
interest to anyone simply curious about how
the kernel works.
To really get the
most out of the book, the reader should be
familiar with Unix and Linux systems, have
at least a basic understanding of the
concepts behind operating system design, and
know C. Really none of these are hard
requirements, however, as the book tries to
explain as much as possible from the ground
up. Where help is needed, I included a
bibliography of some of my favorite books.
A few things make
the book unique. First, its approach is
rather special in that it tackles the
subject in a pragmatic and fun manner. I
hope that the reader can relate to the book
and find the information accessible. Second,
as a kernel developer both by hobby and by
vocation, the book is ripe with insider
views and practical tips.
Q: Which parts of
the kernel appeal to you most and why?
A: My personal
operating system interests are scheduling,
process management, virtual memory
management, and locking.
Q: What specific
tips can you provide on the kernel that
would be useful to kernel developers?
A: Play around.
Study the source. Never give up. Have fun.
Q: What new
features are in the current kernel, and what
features do you see in future kernels and
when do you expect them to appear?
A: The new 2.6
kernel is rich with neat new features. A lot
of these are discussed in the book. Some of
the most notable are a new process
scheduler, two new I/O schedulers, a new
block I/O layer, a device model, a reworked
VM, improved scalability, a preemptive
kernel, and improved threading support.
Anything is up
for grabs in future kernels. We should see
development start on the next kernel, 2.7,
in the coming months.
Q: Provide tips
from your work for Linux Journal?
A: When technical
writing, the most important thing to keep in
mind is your audience. You need to relate to
them, understand them, and convey the
information to them in terms they can
understand. It does not hurt to have a sense
of humor, either!
Q: Describe good
kernel coding practices.
A: There is a lot
behind good kernel coding practices. Sane,
clean, KISS designs are important. The core
kernel developers, and consequently the
kernel itself, have a well-defined style. A
lot of it is just an extension of the Unix
style and approach taken to normal Unix
programming and Unix kernels in particular.
But there is certainly a bit of Linux flavor
in there. Understanding that style requires
understanding the source code, watching the
kernel development process, and so on.
Following the kernel style is definitely
Q: Share your
more vexing problems working with the
A: Even though I
am not a big user of debuggers, oftentimes
the lack of an official or standard kernel
debugger is a nuisance. Thankfully, in 2.5
and 2.6, we have had kgdb (a kernel
debugger) patches kept very up to date and
Another gripe is
our release behavior. I would like it if we
could switch to a release structure like
GNOME's. They do time-based releases. For at
least one release, I would love to try that
Q: What are the
major problems and successes with Open
A: Open Source
has many positive attributes. Obviously, I
believe in it. It has definitely fostered
the creation of best-of-breed projects: the
Apache web server, the Linux kernel, and the
GNU C compiler, for example. Open source
creates a meritocracy, free from political
or marketing agenda, with which projects are
designed and built entirely in the open.
This provides the "many eyes" phenomenon,
which is often described as "with enough
eyes, all bugs are shallow."
But not all is
perfect. Sometimes innovation needs to be
pushed from somewhere other than within, and
open source might not provide that. It is
often argued that open source projects,
which are usually run entirely by
developers, lack accessibility required by
or features for non-developers. While I
think today that is largely an issue of the
past - just look at the resounding success
of GNOME, for example - it is still a cited
are largely issues of the past now that open
source has received corporate backing and
funding, which can fund work in the
Q: Can you make
future predictions about specific products
and services coming from the Open Source
A: I think we are
going to see more and more markets outside
of personal computing and serving moving to
open source based solutions. When I say
this, I think of markets such as traditional
embedded, scientific computing, and consumer
electronics. It is not that these markets
are going to engineer their solutions in
open source, although that undoubtedly would
be great, but that they are going to find
open source based systems the way to go due
to cost, technical superiority, wide
support, and vendor neutrality.
Q: What are some
common problems and their solutions facing
A: These days,
systems are growing more and more complex
and the barrier to entry can only be greater
than it ever was before. I think this
happens in all aspects of society, stuff
just gets more complicated and involved, but
in computing the complexity is not augmented
by better tools or better methodologies. The
disparity between the current generation,
who understand the systems because they
built them, and the next generation, who
face this high barrier of entry, is growing
wider and wider. To counter this, it is very
important that documentation and education
improve to match.
Q: Do you have
some stories about very challenging
situations and their resolution? Any
A: I once spent a
couple days debugging a kernel problem,
trying new approaches, recompiling,
rebooting my target system, and growing more
and more frustrated. It turned out, for
reasons still unknown, that my NFS (a
networked file system) mount, which was how
I was accessing the target, was not
synchronizing before I rebooted the target,
so I never actually tried the new code. I do
not even remember what the bug I was trying
to fix was or why NFS was misbehaving. But,
needless to say, when I figured out the
problem I put a brown paper bag over my head
for a week or two. The moral of this story
is to always check your build versions to
make sure you are actually running what you
think you are. And do not have freaky NFS
Q: What trends do
you see in program development? Please make
some predictions about the future, and
future technologies that businesses and IT
professionals should be considering.
professionals should be considering the
potential of open source technology, of
Q: With regards
to security, please provide your detailed
recommendations in this area?
A: It is
important that security be an initial design
feature and built into the system from the
very beginning. Security cannot be an
afterthought. It cannot be tacked on to an
existing system. The initial design needs to
ask the critical questions and design the
requisite answers into the system. The
implementation then needs to carry through
with the design and correctly implement the
security through obscurity (the practice of
"securing" systems by hiding how they work)
is nonsense. A system that is provably
correct and provably secure can have both
its design and implementation handed to the
attacker. It should not matter.
matter what you do, if the implementation
does not match the design (e.g., you have a
bug), then you have problems. If your system
is filled with buffer overflows and other
errors, then no secure design in the world
will help you. Many people blame these sorts
of errors on the programming language or
maintaining old systems. To some end, I
certainly agree with them. But, at the end
of the day, the errors are still occurring
due to incorrect programming. Many systems
are programmed in C and C++. Nearly all of
Linux is. Nearly all of Windows is. It is
inexcusable if either system were
continually plagued with security flaws,
despite not using "modern" languages.
Thankfully, Linux has proven very secure
over the years.
Q: Which ten
resources do you find the most useful?
A: Ten? Hmm. My
friends, family, and coworkers. My personal
workstation and my Apple ThinkPad. I love my
Apple. Three great books: Knuth's The Art
of Computer Programming, Stevens'
Advanced UNIX Programming, and K&R's
The C Programming Language. I probably
never look at these any more nowadays,
except maybe the first, but they are still
incredible books. LWN [http://www.lwn.net/]
to keep me up-to-date with Linux news and
CNN [http://www.cnn.com/] to keep me
up-to-date with the rest of the world. That
ought to be ten.
If you were doing this interview, what three
questions would you ask of someone in your
position and what would be your answers?
1) What operating system do you most
respect, aside from Linux?
Definitely Mac OS X. I am very impressed by
Linux excels in the server, workstation, and
embedded markets. What is next?
Definitely the desktop, as in the home or
office desktop. I feel the GNOME desktop is
incredibly mature and well-featured. GNOME
is beyond the "catching up" phase; it is now
an innovator in the desktop space.
Nonetheless, I think there are still
political and technical issues that remain.
Politically, Linux needs to continue to win
deployments in large corporations.
Management needs to be aware of Linux as an
option on the desktop. They need to realize
it works and realize it is a great option.
Companies such as Ximian, Red Hat, and SuSE
are making good inroads here. Technically,
Linux still lacks application support,
although applications such as Open Office,
Mozilla, and Evolution are certainly
excellent, especially as of late. We also
need better hardware management, with the
kernel reporting hardware events (such as
the user plugging in a digital camera) and
the desktop environment handling the event.
But we are getting there. I love my Linux
Vi or Emacs? Boxers or briefs?
Q: Robert, you
provide some deep insights and an
interesting perspective. It was a pleasure
interviewing you. Thank you for sharing your
wealth of knowledge with our audience.
A: Thank you for
the opportunity. Take care.