Linux swapiness - Adjusting Kernel VM settings

Before you read this, please note that I understand the benefits of caching. I'm familiar with the dogma that unused ram is wasted ram.

This question is one that I've adapted from a previous question:

deleting linux cached ram

In that question I was curious about adjusting how my server uses and caches ram. The system is fairly dynamic so I believe that the cached files doesn't really afford me much gain. Additionally, we have code on the server that has to quickly access large amounts of ram in short periods of time to process video files and I believe that I'll see a performance benefit from directly handing of ram rather then clearing it from cache and then handing it off.

I'd like to find out if any of you have experience with adjusting the default value of 60 in the following file (this happens to be on an Ubuntu server):

/proc/sys/vm/swappiness  

And if so, what affects did you see. If I replace the default value of 60 with 30 will I see less swapping and a lot more reuse of stale cache? Do I approach 0 or 100 to decrease swapiness and increase reuse of cache?

Finally, anyone know why the default is set to 60?

NOTE: If it's close to 0, Linux will prefer to keep applications in RAM and not grow the caches. If it's close to 100, Linux will prefer to swap applications out, and enlarge the caches as much as possible. The default is a healthy 60. - Thanks for the link below, 3dInfluence.

Answer

Edit: Rewrote the answer so that it's shorter and clearer I hope :)

You really need to understand how the VM subsystem works as a whole to start tweaking the tunables or you may find that you're not getting the results that you expect. This article is a pretty good starting point on how these settings work together with a desktop slant.

So more to your question. Swappiness controls when the VM subsystem reclaims process table pages by unmapping and paging them out, aka swapping. This tunable works by telling the VM subsystem to look for pages to swap when the % of memory mapped to process page tables + swappiness value is > 100. So a setting of 60 will cause the system to start paging out stale pages from the process page table when it is using more than 40% of your system's memory. If you want to allow your programs to use more memory at the expense of cache you'll want to lower the swappiness value. You'll also want to have a look at /proc/sys/vm/min_free_kbytes and /proc/sys/vm/vfs_cache_pressure. As this will also control how much memory is kept in reserve and how aggressive the caching is. See that article I linked to for more information on the latter of those.