memory - How to find what is using linux swap or what is in the swap?

I have virtual linux (Fedora 17) server with 28GB RAM and 2GB swap. The server is running a MySQL DB that is set up to use most of the RAM.

After some time running the server starts to use swap to swap out unsued pages. That is fine as my swappiness is at default 60 and it is the expected behaviour.

The strange thing is that the number in top/meminfo does not correspond with info from processes. I.e. the server is reporting these numbers:

/proc/meminfo:
SwapCached:        24588 kB
SwapTotal:       2097148 kB
SwapFree:         865912 kB

top:
Mem:  28189800k total, 27583776k used,   606024k free,   163452k buffers

Swap:  2097148k total,  1231512k used,   865636k free,  6554356k cached

If I use the script from https://serverfault.com/a/423603/98204 it reports reasonable numbers (few MBs swapped by bash'es, systemd, etc) and one big allocation from MySQL (I omitted a lot of output lines):

892        [2442] qmgr -l -t fifo -u
896        [2412] /usr/libexec/postfix/master
904        [28382] mysql -u root
976        [27559] -bash
984        [27637] -bash

992        [27931] SCREEN
1000       [27932] /bin/bash
1192       [27558] sshd: admin@pts/0
1196       [27556] sshd: admin [priv]
1244       [1] /usr/lib/systemd/systemd
9444       [26626] /usr/bin/perl /bin/innotop
413852     [31039] /usr/libexec/mysqld --basedir=/usr --datadir=/data/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/data/mysql/err --open-files-limit=8192 --pid-file=/data/mysql/pid --socket=/data/mysql/mysql.sock --port=3306
449264   Total Swap Used

So if I get the script output right the total swap usage should be 449264K = ca. 440MB with mysql using ca. 90% of the swap.

The question is why this differs so much from the top and meminfo numbers? Is there any way how to "dump" swap info to see what is actually in it instead of summing the swap usages from all processes?

When analyzing the issue I came up with different ideas but they all seems to be wrong:

1. The script output is not in KB. Even if it would be in 512 or 4KB units it won't match. Actually the ratio (1200:440) is about 3:1 which is "strange" number.


2. There are some pages in swap that are somehow shared between processes as mentioned in https://serverfault.com/a/477664/98204 . If this is true how can I find the actual number of memory used like this? I mean it would need to make cca 800MB difference. And that does not sound right in this scenario.


3. There are some "old" pages in swap used by processes that already finished. I would not mind that if I were able to find out how much is this "freeable" swap.



4. There are pages in swap that has been swapped back to memory and are in swap just in case they did not change in RAM and need to be swapped out again as mentioned in https://serverfault.com/a/100636/98204 . But the SwapCached value is only 24MB.

The strange thing is that the swap usage is slowly increasing while the sum output from the script is roughly the same. In last 3 days the swap used increased from 1100MB to current 1230MB while the sum increased from 430MB to current 449MB (ca.).

The server has enough free(able) RAM so I could just turn off the swap and turn it back on. Or I could probably set swappiness to 0 so the swap would get used only if the is no other way. But I would like to solve the issue or at least find out what is the cause of this.