The Unix utilities ps and top report memory differently with HugePages than without.
Without HugePages ps -eF seems to include the SGA memory under the SZ column:
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD ... oracle 1822 1 0 846155 16232 0 07:19 ? 00:00:00 ora_d000_orcl oracle 1824 1 0 846155 16228 0 07:19 ? 00:00:00 ora_d001_orcl oracle 1826 1 0 846155 16236 0 07:19 ? 00:00:00 ora_d002_orcl oracle 1828 1 0 846155 16232 0 07:19 ? 00:00:00 ora_d003_orcl oracle 1830 1 0 846155 16224 0 07:19 ? 00:00:00 ora_d004_orcl oracle 1832 1 0 846155 16232 0 07:19 ? 00:00:00 ora_d005_orcl oracle 1834 1 0 846155 16236 0 07:19 ? 00:00:00 ora_d006_orcl oracle 1836 1 0 846155 16228 0 07:19 ? 00:00:00 ora_d007_orcl oracle 1838 1 0 846155 16224 0 07:19 ? 00:00:00 ora_d008_orcl oracle 1840 1 0 846155 16232 0 07:19 ? 00:00:00 ora_d009_orcl oracle 1842 1 0 846155 16240 0 07:19 ? 00:00:00 ora_d010_orcl oracle 1844 1 0 846155 16228 0 07:19 ? 00:00:00 ora_d011_orcl ...
Here SZ = 846155 kilobytes = 826 megabytes. If you add up all the SZ values it comes to 81 gigabytes which wont fit in my 4 gig memory and 4 gig swap. It seems to include the amount of the SGA actually used, not the full 3 gigabyte max sga size, otherwise the total would have been hundreds of gigabytes.
Doing the same exercise with 3 gigabytes of huge pages ps looks like this:
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD ... oracle 1809 1 0 59211 15552 0 07:52 ? 00:00:00 ora_d000_orcl oracle 1811 1 0 59211 15544 0 07:52 ? 00:00:00 ora_d001_orcl oracle 1813 1 0 59211 15548 0 07:52 ? 00:00:00 ora_d002_orcl oracle 1815 1 0 59211 15544 0 07:52 ? 00:00:00 ora_d003_orcl oracle 1817 1 0 59211 15544 0 07:52 ? 00:00:00 ora_d004_orcl oracle 1819 1 0 59211 15548 0 07:52 ? 00:00:00 ora_d005_orcl oracle 1821 1 0 59211 15544 0 07:52 ? 00:00:00 ora_d006_orcl oracle 1823 1 0 59211 15544 0 07:52 ? 00:00:00 ora_d007_orcl oracle 1825 1 0 59211 15544 0 07:52 ? 00:00:00 ora_d008_orcl oracle 1827 1 0 59211 15544 0 07:52 ? 00:00:00 ora_d009_orcl oracle 1829 1 0 59211 15544 0 07:52 ? 00:00:00 ora_d010_orcl oracle 1831 1 0 59211 15544 0 07:52 ? 00:00:00 ora_d011_orcl ...
SZ = 59211 k= 57 meg. Total SZ = 5.89 gigabytes. Still this is bigger than total memory but closer to the 4 gig memory available. It’s just a guess, but I’m pretty sure that with HugePages this total doesn’t include the amount of memory in use in the SGA in the SZ for each process as it did without HugePages.
The other weird thing is how different top looks with HugePages. Here is top with the database having just come up without HugePages:
top - 07:20:16 up 3 min, 2 users, load average: 1.06, 0.33, 0.13 Tasks: 187 total, 1 running, 186 sleeping, 0 stopped, 0 zombie Cpu(s): 1.6%us, 6.3%sy, 0.0%ni, 77.8%id, 14.2%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 4050836k total, 984444k used, 3066392k free, 14460k buffers Swap: 4095996k total, 0k used, 4095996k free, 450128k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2010 oracle 20 0 3310m 51m 44m D 7.6 1.3 0:00.21 oracle 1988 oracle 20 0 3307m 50m 45m D 3.8 1.3 0:00.21 oracle 1794 oracle -2 0 3303m 15m 13m S 1.9 0.4 0:01.07 oracle
Notice that we have about 3 gigabytes free – 3066392k and nothing in swap.
Here is the same system with 3 gig of HugePages:
top - 07:53:21 up 2 min, 2 users, load average: 0.81, 0.29, 0.11 Tasks: 179 total, 1 running, 178 sleeping, 0 stopped, 0 zombie Cpu(s): 2.0%us, 8.6%sy, 0.0%ni, 69.2%id, 20.1%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 4050836k total, 3860100k used, 190736k free, 14332k buffers Swap: 4095996k total, 0k used, 4095996k free, 239104k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1781 oracle -2 0 3303m 15m 13m S 3.5 0.4 0:01.02 oracle 1 root 20 0 19400 1520 1220 S 0.0 0.0 0:01.43 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
Now only 190736k is free. But, note that in both cases top lists the oracle processes with 3300 meg of virtual memory, which is consistent with the 3 gig max sga.
I’ve still got a lot to learn about HugePages but I thought I would pass along these observations. This article on Oracle’s support site helped me learn about HugePages:
HugePages on Oracle Linux 64-bit (Doc ID 361468.1)
I ended up sizing the HugePages down to 2 gig on my 4 gig test system and reducing sga max size to 2 gig as well. My system was sluggish with so little free memory when I was using a 3 gig SGA and HugePages. It was much snappier with only 2 gig tied up in HugePages and dedicated to the SGA, leaving 2 gig for everything else.
This was all done with Oracle’s 64 bit version of Linux and 11.2.0.3 database.
– Bobby
