Date: Mon, 23 Jun 2008 21:55:15 -0400
From: "David A. Harding" <dave@dtrt.org>
To: Mailing list for the Ubuntu New Jersey LoCo Team <ubuntu-us-nj@lists.ubuntu.com>
Subject: Re: ubuntu-us-nj Question
Message-ID: <20080624015515.GA24986@localhost.localdomain>
In-Reply-To: <485F94D3.5030900@gmail.com>

On Mon, Jun 23, 2008 at 08:19:31AM -0400, Rob wrote:
> You wrote to the entire Ubuntu NJ LoCo mailing list, but it appears
> you were trying to reach Joe directly. 

I doubt Joe has a problem with that: I think he runs the team so he
doesn't have to help everyone himself.

> Linux does not need to be defragged, 

No operating system I know of needs its file system defragmented, but
defragmentation can increase file access speed and free disk space --
this applies to both Microsoft and Unix-style (GNU+Linux) file systems.
Ubuntu's filefrag command lists the number of fragments in a file:

    $ sudo filefrag star_wreck_in_the_pirkinning_subtitled_xvid.avi 
    star_wreck_in_the_pirkinning_subtitled_xvid.avi: 54 extents found, perfection would be 5 extents

Extents and fragments are almost exactly the same thing. The fewer
extents, the less the spindle of your hard drive will need to move
unnecessary. In the example above, the spindle will need to move 49 more
times than necessary to read this entire file once.

The upshot is that this is a very large file -- 542 megabytes -- and
it's a 90 minute movie, so 49 unnecessary movements isn't a problem: my
movie player will keep enough of the file in memory so that I'll never
notice the minor unnecessary movement that occurs only about once every
two minutes during playback.

Ubuntu almost always stores small files in a single extent. For example,
the file filefragged below contains your email to this mailing list as I
received it:

    $ sudo filefrag Maildir/.lists.njloco/cur/1214223769.11560_2.callisto\:2\,S 
    Maildir/.lists.njloco/cur/1214223769.11560_2.callisto:2,S: 1 extent found

On Ubuntu, the fsck command will tell us how many files use more than
one extent.  Look at the final line in the output below:

    harda@callisto:~$ sudo fsck.ext3 -n -f /dev/md0
    e2fsck 1.40.11 (17-June-2008)
    Warning!  /dev/md0 is mounted.
    Warning: skipping journal recovery because doing a read-only filesystem check.
    /dev/md0: 32397/122624 files (16.2% non-contiguous), 214947/244960 blocks

The percentage of non-contiguous files is the percentage of files with
more than one extent (more than one fragment). I've never seen a Unix
filesystem more non-contiguous as this one, which stores my /home
directory. Why is it so fragmented? Because I keep it nearly full all
the time, and Ubuntu has a hard time finding enough free space to fit an
entire file in one fragment on a full filesystem.

A good rule-of-thumb I've heard for both Windows and Ubuntu is to not
worry about a filesystem that is less than 20% non-contiguous. On
Windows, you'll probably reach the 20% mark after you use it for a few
weeks; on Ubuntu, you'll probably never reach he 20% mark. The
difference comes from the design of the Windows and Ubuntu file systems.

On Ubuntu, and Unix in general, the operating system usually gets the
opportunity to re-arrange a file each time you save it. This lets Ubuntu
defragment your files as you work. On the downside, Ubuntu sometimes
takes longer to save small changes to your file while it defragments
them -- so if you use Ubuntu for long enough, you'll notice that file
save times are terribly inconsistent. I admit, that annoys me.

Moreover, Windows tries to save each new file starting at the end of the
previous file. This means that when you add something to an old file,
Windows needs to split it into additional pieces. Ubuntu instead tries
to space each file as far apart as possible; so, for example, if you
have only two files on a filesystem, the first file will start at the
beginning of the drive and the second file will start at the middle of a
drive. This means that when you add something to an old file, Ubuntu
doesn't usually need to split it into additional pieces, and it won't
unless you add a lot of stuff or your filesystem is almost full.

There are several ways to defragment your Unix filesystem. Some bored
Computer Science students have written Unix defragmentation tools and
you can also simply copy files to another filesystem and back, but both
of these methods can be dangerous and, again, probably won't provide you
any noticeable benefit unless your filesystem is more than 20% non-
contiguous or you're working with large, high-performance databases.

-Dave

P.S.  Earlier I said that a Windows filesystem will often get more than
  20% non-contiguous after only a few weeks. So how long did it take
  me to get to 16.2$ non-contiguous on a Unix file system? Three
  years -- and that's on a filesystem which is currently 93% full.

  So I suggest you enjoy the benefit of almost never needing to
  defragment your filesystems on Ubuntu.
-- 
David A. Harding            Website:  http://dtrt.org/
1 (609) 997-0765              Email:  dave@dtrt.org
            Jabber/XMPP:  dharding@jabber.org