Demand Technology Software
Frequently Asked Questions about Memory
1. How can I tell when I am out or RAM and need to add more memory?
4. How do I find a memory leak?
Frequently Asked Questions about Memory
1. System Monitor reports a very large number of Page Faults/sec on my system, we just added RAM, and it does not seem to help. How can I tell when I am out memory and need to add more?
The best single indicator of a real memory (RAM) shortage is Available Bytes. This Counter reports the current number of pages (in bytes – there are 4096 bytes in a page) in the Zero, Free, and Standby Lists that the Memory Manager maintains. This pool of available memory is used to resolve page faults quickly. So long as there is a cushion of Available Bytes that is at least 5-10% of the size of RAM, you probably have sufficient memory to run your workload.
It is also important to monitor Available Bytes to see if it remains at or near a threshold value of 4 MB. When Available Bytes (the sum of Zero, Free, and Standby Lists) falls below approximately 4 MB, it triggers a change in the operating system’s memory management policy. The working set of a process is defined as the number of virtual memory pages that it is allowed to keep resident in RAM. Normally, the working sets of processes that are running at their maximum working set size are adjusted upwards periodically by a component of the Windows 2000 Memory Manager. This allows processes that need more RAM to grow their working sets over time. However, when Available Bytes are in short supply, the working sets of processes already running at their maximum working set size are no longer adjusted upwards. Since this change in policy often leads to increased paging activity, it is a good idea to monitor Available Bytes and set off an alarm whenever you find it hovering consistently at about 4 MB.
Figure 1 below is a graph showing the utilization of RAM on a Windows 2000 server that shows Available Bytes dropping sharply to 4 MB shortly after 20:15 and stabilizing at that point afterwards. The other Counters charted represent real memory usage in various system-managed pools: as usual, the non-pageable pool, the pageable pool, and resident bytes in the file cache are the most significant consumers of real memory. In this instance, something causes resident cache bytes to increase (evidently due to some file access activity), which serves to deplete the supply of Available Bytes. Notice that once Available Bytes drops to 4 MB, it stabilizes at that value, even as the file cache expands (probably at the expense of RAM-resident process Working set bytes (not shown). The change in memory management policy allows the Windows 2000 Memory Manager to maintain the pool of Available Bytes at a relatively constant level. But, because RAM is in short supply, something has to give. The result is a predictable increase in demand paging.
Figure 1. RAM usage on 256 MB Windows 2000 Server.
Figure 2 covers the same interval, showing the demand paging statistics from the Memory Object. These Counters report a corresponding rise in the page fault rate once Available Bytes drops to 4 MB. Cache Faults/sec represent read file operations that miss the cache and require accessing data from disk. This file cache activity apparently accounts for the expansion of the System Cache Resident Bytes depicted in Figure 1. Because there is contention for RAM among executing processes, the hard page fault rate also increases. The hard page fault rate charted here corresponds to the Page reads/sec Counter in the Memory Object. Together, Available Bytes hovering at 4 MB and a corresponding increase in the rate of hard page faults strongly suggests a situation where it would be useful to add more RAM. This example also illustrates the fact that the system file cache, which is part of the System address space working set, competes for memory on equal terms to other process working sets.
Figure 2.
Demand paging rates can
spike when the pool of Available Bytes is depleted.
Unfortunately, the relation between a memory shortage and the onset of demand paging illustrated here is usually not so clear-cut in many of the systems you will get a chance to observe. You also need to be aware that applications like MS SQL Server and Exchange running on Windows 2000 will grow their working sets dynamically to absorb any excess RAM that is available. It is not unusual to see systems running MS SQL Server or Exchange running at or near the 4 MB line without excessive paging. These applications also bypass the standard system file cache and buffer recently used database objects within their own memory-resident working sets. To diagnose shortage-of-memory conditions for these apps, it is necessary to look beyond the system level Memory statistics and examine their internal cache statistics.
2. I see a lot page faults on my system even though there appear to be plenty of Available Bytes. What is going on?
You are probably looking at the Page Faults/sec Counter and interpreting it as the rate of demand paging. Not entirely. In Windows NT and 2000, the Page Faults/sec counter includes both hard and soft page faults. (It also appears to include Cache faults/sec, which are application-related file cache read misses.) Instead of using the Page Faults/sec counter as an indicator of demand paging, you should rely on the Page Reads/sec Counter instead in both Windows NT and 2000. Page Reads/sec corresponds to the rate of hard page faults that specifically require a disk access to resolve.
There are two types of so-called “soft” faults in Windows 2000 that are included in the Page Faults/sec metric: Transition Faults/sec and Demand Zero Faults/sec. High rates of both types of soft faults are common even where there is an ample supply of RAM.
Transition faults are a by-product of the Window 2000 page replacement policy and cannot easily be avoided. To determine which pages of a process address space are not currently in use, the Windows 2000 Memory Manager trims pages aggressively from each process working set. These pages are only stolen by the operating system provisionally, however. Recently trimmed pages remain in memory on the Standby List. (The Standby List is one of the components of the pool of Available Bytes.) The idea is that the next time that threads from the process execute, they will reference the pages that identify the application’s current working set, which the Memory Manager will restore swiftly. Recently trimmed paged that are re-referenced transition fault back into the process working set quickly because they normally remain in memory for some period of time after being trimmed. Because these transition faults are resolved by the OS without ever having to retrieve a page from the paging file on disk, they are not nearly so costly as a hard page fault that must be resolved from disk.
The mechanism for handling transition faults is straightforward. Pages on the Standby List remain in memory. Their corresponding Page Table Entries (PTEs) are marked invalid, but are also flagged in transition. A process thread accessing a memory location on an invalid page triggers a hardware interrupt. During interrupt processing, the operating system quickly determines that a page in transition was referenced. The PTE entry for page is then quickly restored, the page is returned to the process working set, and the instruction that failed due to an invalid address reference is re-executed. Trimmed pages that are not re-referenced in a timely manner remain on the Standby List until they are eventually moved to the Free List and, finally, the Zero List as they age.
Demand Zero faults are requests from executing processes for new pages. For security reasons, the operating system always returns an empty, zero-filled page when a process requests a new page of virtual memory. Demand zero faults are satisfied with empty pages from the Zero List, or from the Free List if the Zero List is empty. The Demand Zero Faults/sec Counter corresponds to the rate of requests by processes for new zero-filled pages. Since requests for new pages are satisfied directly from the pool of Available Bytes, Demand Zero Faults/sec are another category of soft faults that do not require disk I/O to resolve.
Consider a server process that allocates an area of virtual memory as a work area when requests for service arrive and later frees the memory when it has finished processing the request. Since freeing memory replenishes the pool of Available Bytes, this process might generate a high rate of Demand Zero Faults/sec without ever needing to access the paging file on disk.
Because soft faults do not generate I/O to disk, it is important to ignore the rate of soft faults when you are trying to figure out if you have enough RAM.
3. How can I tell how much RAM is being used by application processes and various operating system functions?
It is not possible to get a complete accounting of RAM usage on a Windows 2000, but you can get reasonably close. RAM usage by various OS functions is measured by the following five Memory Object counters:
 | Pool Nonpaged Bytes: these represent allocations directed to the nonpaged pool, which is a set virtual memory pages that always remain resident in RAM. (These are nonpageable bytes.) Device drivers and the OS use the nonpaged pool to store data structures that must stay in physical memory and can never be paged out to disk. (For example, the TCP/IP driver must allocate some amount of nonpaged memory for every TCP/IP connection that is active on the computer for data structures that are required during processing of network adaptor interrupts when page faults cannot be tolerated.) |
| Pool Paged Resident Bytes: Most virtual memory pages that are acquired in the Operating System range of virtual addresses can be paged out. The Pool Paged Resident Bytes represent memory locations from the pageable pool that currently reside in RAM. |
| System Cache Resident Bytes: the system’s file cache occupies a reserved range of virtual memory addresses, some of which may currently reside in RAM. (Cached file segments can also be non-resident, in which case they must be fetched from disk when they are referenced by executing processes.) System Cache Resident Bytes represents segments of the file cache that are currently resident in RAM. |
| System Code Resident Bytes: memory locations associated with system code that is currently resident in RAM. |
| System Driver Resident Bytes: memory locations associated with device driver code that is currently resident in RAM. |
These five Counters account for RAM usage of virtual memory associated with operating system (and device driver) functions. As discussed above, Available Bytes represents free RAM that is not allocated to any OS function or to any executing process.
Once you know how much OS function are currently using, it ought to be a simple matter to account for RAM usage completely by factoring in the Working Set Bytes of various executing processes, as follows:
Process(_Total) Working Set Bytes = Sizeof(RAM) – (Available Bytes + Pool Nonpaged Bytes + Pool Paged Resident Bytes + System Cache Resident Bytes + System Code Resident Bytes + System Driver Resident Bytes)
However, resident pages associated with shared DLLs (e.g., DLLs like comsvcs.dll, mfc42.dll, msvbvm60.dll, etc.) are counted in each and every process Working Set that references a shared DLL. Because these resident pages are counted multiple times, Process(_Total) Working Set bytes usually reports a higher number of resident pages than are actually in use. In other words,
Sizeof(RAM) < Process(_Total) Working Set Bytes + Available Bytes + Pool Nonpaged Bytes + Pool Paged Resident Bytes + System Cache Resident Bytes + System Code Resident Bytes + System Driver Resident Bytes
Because the numbers do not add up as they seemingly ought to, we prefer creating a report like Figure 1 above that charts the six system Memory counters (including Available Bytes) and compares their sum to the amount of installed RAM. Process working sets fill the amount of RAM left over after you have accounted for these six counters, even though you cannot pin down precisely which how much memory each process occupies.
4. How do I find a memory leak?
A memory leak refers to a programming bug where an application program repeatedly allocates virtual memory, but never deletes it. Eventually, a program with a memory leak will cause something bad to happen. For example, the system or some of its applications might lock up because all the available virtual memory is allocated.
Several aspects of typically memory leaks make them especially insidious programming bugs. A program with a memory leak is not obviously incorrect, and may even produce the correct output or calculate the proper results. Memory leaks are often not evident until a program has been executing successfully for hours, days, or weeks. It is not always obvious which program is causing the memory leak. The memory leak may not manifest itself in the same way all the time. A program with a memory leak will eventually exhaust the supply of virtual memory pages available so that simple calls to allocate a new segment of virtual memory fail. When simple application calls to allocate a new segment of virtual memory fail, the results are often unpredictable. Finally, applications frequently make calls to allocate and free virtual memory. These calls are typically sprinkled in various spots in the program. Because system functions and DLL library routines also make frequent calls to allocate and free virtual memory, it is not easy to isolate what is causing the bug in the program. Fortunately, there are excellent automated tools for Windows 2000 application developers that can help programmers find many types of memory bugs. There is no excuse for not using some of these development and testing tools on your Windows 2000 applications in order to catch many types of memory leaks before they manifest themselves in critical product environments.
The procedure for finding a memory leak depends on what type of virtual memory is being allocated. The bug may be local to an application so that its only effect is on the process virtual address space where the leak is occurring. The bug may be in a shared DLL or other common module so that the bug will be manifest first in this application and then that one. Finally, the problem could be a leak affecting shared system memory, which can make it very difficult to pinpoint the specific application or module causing the leak. To diagnose a memory leak, be prepared to look at any or all of the following:
% Committed Bytes in Use. The Commit Limit is an upper limit on the amount of virtual memory that can be allocated on your system. The Commit Limit is tied to the size of RAM and the amount space defined for paging files (and remember that paging files can and do expand the system looks like it may run short of virtual memory). The % Committed Bytes in Use Counter reports Committed Bytes as a percentage of the Commit Limit. Whenever % Committed Bytes in Use exceeds 80-90%, application requests to allocate
per process Virtual Bytes. Virtual Bytes records the amount of virtual memory allocated by individual processes. If the program with the memory leak is allocating virtual memory in its own address space, the memory leak should be evident by tracking the per process Virtual Bytes Counter. If the amount of Virtual Bytes allocated for a process increases steadily over the life of a process, there is good reason to suspect a leak.
Pool Paged Bytes. Virtual memory for various system functions, including shared memory files (like DLLs), is allocated from the Paged Pool, which is an area of the system's virtual memory that is limited in size. A program with a memory leak that is allocating, but never freeing memory from the Paged Pool will eventually exhaust the Paged Pool. Subsequent allocations that request the Paged Pool will then fail, with unpredictable (but predictably bad) results. The Pool Paged Bytes Counter in the Memory Object reports the current number of bytes allocated from the Paged Pool. The upper limit on the size of the Paged Pool is calculated by the system at start-up, using to the formula documented here. This calculation can be overridden by setting the Paged Pool Limit explicitly using the Registry parameter PagedPoolSize at HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management. In either case, the upper limit on the size of the Paged Pool, according to Microsoft documentation in Windows 2000 Resource Kit is 192 MB in NT 4.0 and up to 470 MB in Windows 2000.
Pool Nonpaged Bytes. Some kernel functions and device drivers in particular require real memory buffers that can never be paged out of the system. These programs allocate memory from the nonpaged pool, which also has an upper limit. (The upper limit on the size of the nonpaged pool in Windows NT is 128 MB and 256 MB in Windows 2000.) A device driver with a memory leak will eventually exhaust the supply of Nonpaged Pool Bytes, which will cause subsequent allocations that request the nonpaged pool to fail. Running out of space in the nonpaged pool often results in a Blue Screen.