FAQ   Search   Memberlist  
Profile    Log in to check your private messages    Register    Log in
filesystem performance tweaks

 
Post new topic   Reply to topic    BLAG Forum Index -> development
View previous topic :: View next topic  
Author Message
jebba
PostPosted: Fri Jul 30, 2004 11:11 pm    Post subject: filesystem performance tweaks Reply with quote

This was posted on the audacity-devel mailing list about how to speed up audacity with huge files. It is true outside of audacity as well. Understand what you're doing before jacking around with your filesystem, of course... Here's the email from monty:

==================begin==================
BTW, to folks running Linux: There are several things you can do to
tune your filesystem to give *much* better performance in Audacity
with medium and large projects:

1) Turn on the 'dir_index' option to ext2fs and ext3fs, using tune2fs
and -O dir_index. This causes ext2fs and ext3fs to use btrees instead
of linear search for metadata in any new directories, drastically
reducing CPU overhead accessing files in large directories.
Drawbacks: This option is unsupported in older kernels, and as such,
some rescue disks will not be able to mount a filesystem using this
option. For example, the stock Debian/Woody install/rescue can't even
mount an FS using dir_index, but the bf2.4 rescue/install disks can.
Other drawbacks: None; this is safe and recommended on an existing
filesystem, and will improve all access times on *new* directories
made since changing the setting. It's not more widespread only
because the feature is new.

2) If using ext3, set the journal to 'data_ordered' mode, using
tune2fs and -o journal_data_ordered on an *unmounted* filesystem.
This tells the journal to keep the filesystem (metadata) consistent at
all times, but be slightly less paranoid about flushing new file data
out to the disk; in a crash, an open, actively-modified file will be
in-tact, but the end of the file is not guaranteed to be exactly
correct up to the moment of the crash (unlike full journalling mode,
where the end of the file is truncated at exactly the last write to
finish). The result is a filesystem more than 4x faster when
writing.

3) If Audacity is a primary use of your workstation, consider using a
filesystem like XFS, which has about 60-100% more write throughput
than even a well-tuned ext3fs. This is not a knock against ext3fs
which is phenomenally simple and stable, but when it comes to raw
numbers, XFS is just about always faster by a large margin, mostly in
writes. Again, the XFS filesystem is new to Linux (Available in stock
2.4.26 and 2.6.somethingorother), but there are no other serious
drawbacks to using it aside from potential rescue floppy/ISO
compatability. I also find XFS gets along a bit better with hardware
and software RAID although, again, a well tuned ext3fs is much better
on RAID than a default-settings ext3fs. XFS simply deals
automatically where the ext3fs must be hand-tuned for a RAID.

4) The Linux buffercache is tuned to be interactive rather than to
have high continuous disk bandwidth; most notably it tries to buffer
far too much disk output in-cache and when it runs out of machine
memory to do so, it can cause the machine to go *completely
unresponsive* for minutes to hours (depending on the amount of RAM
being used to cache) while catching up. Chances are it won't even
ping; it's not crashed, it's simply way behind on disk writes and has
punched the panic button to catch up. The problem is worst in 2.6
exactly because the buffercache is so much faster (but the disk isn't
;-) Mark Eichin of Metacarta refers to this as the 'wings stay on /
wings fall off' switch.

The 'fix' is to tell the buffercache that getting behind on writes is
unacceptable; your writes will feel like they're slower, but in fact
you're eliminating all those occasions where the machine freezes while
trying to catch up during big I/O. Actual disk throughput will be
higher.

Eichin recommends a line like:

echo "0 5000 0 0 100 3000 0 1 0" > /proc/sys/vm/bdflush

This directs bdflush to always maximize bandwidth-to-disk during
writes. Note that if you use hardware RAID or a caching controller
(like a 3ware Escalade, which is what I have), your vendor may have
their own recommended /proc/sys/vm/bdflush settings; if they do, it's
probably best to use theirs instead.

Similarly, you want the buffercache to be *more* aggressive when
reading. Something like:

echo "512" > /proc/sys/vm/max-readahead
echo "512" > /proc/sys/vm/min-readahead

...will greatly improve sequential read throughput as well as mildly
improving writes (which usually involve some reading as well).
Audacity does alot of sequential reading and writing. This setting is
perfectly safe for other software, you're simply telling the kernel to
trade off an amount random disk access performance for greater linear
throughput.

BTW, changes made using 'tune2fs' are persistent. The /proc/sys/vm
settings are not; they will need to be set after each reboot. To make
sure this happens aty boot time, put the lines into
/etc/init.d/bootmisc.sh (for Debian, or the equivalent init file on
your distribution).

Monty
===================end===========

Cool :)

Display posts from previous:   
Post new topic   Reply to topic    BLAG Forum Index -> development
Page 1 of 1

Protected by Anti-Spam ACP