Big Heaps and Intimate Shared Memory (ISM)

Introduction

The performance of some applications is limited by the amount of memory that the JVM can address. In particular, multithreaded programs that allocate heavily often bottleneck in garbage collection, and the total time spent in garbage collection can generally be reduced by using a bigger heap.

This page describes options useful for improving performance on systems with large physical memory. These options are available starting with J2SE 1.3.1, and are only intended for large server systems. Large heaps should be avoided if there is insufficient physical memory to avoid paging.Because the options described have specific system requirements for correct operation and may require privileged access to system configuration parameters, they are not recommended for casual use.

These options may disappear in a future release, like all other options that begin with -X . For example, future JVMs supporting a full 64 bit address space will likely have different sizing needs.

Using a Large Heap

Heaps larger than 2GB are available starting with J2SE 1.3.1. These are options that can be useed to tune the use of memory by the JVM. The values given should not be taken as suggestions; best values for your application will vary and should be set by measurement.

-Xms3g -Xmx3g

This example sets the minimum and maximum total heap size to 3GB. The default maximum size is 64MB, but for many server applications it makes sense to be much larger. The g suffix may be replaced by m to measure in megabytes.
Ordinarily the JVM consumes as little memory as possible, starting with a heap of the minimum size specified by the -Xms flag, and adaptively growing as required up to the maximum specified by -Xmx. Setting these to the same value avoids being conservative, and will often improve startup time.

-XX:+AggressiveHeap

This option instructs the JVM to push memory use to the limit: the overall heap is more than 3850MB, the allocation area of each thread is 256K, the memory management policy defers collection as long as possible, and (beginning with J2SE 1.3.1_02) some GC activity is done in parallel.
Because this option sets heap size, do not use the -Xms or -Xmx options in conjunction with -XX:+AggressiveHeap. Doing so will cause the options to override each other's settings for heap size.
This option should be used with caution. Because it makes the JVM greedy with memory resources, it is not appropriate for many programs and makes it easier to run out of memory. For example, by using most of the 4GB address space for heap, some programs will run out of space for stack memory. In such cases -Xss may sometimes help by reducing stack requirements.

ISM

A large heap places harsher demands on the hardware memory system. Intimate Shared Memory (ISM) is a Solaris facility that can help by:

Enabling larger pages - 4MB instead of the default 8KB
Locking pages in memory - no paging to disk
Allowing shared virtual to physical address data structures

The 1.3.1 JVM exploits the first two. By reducing the overhead of virtual to physical address translation, a significant performance boost can often be obtained (typically 5-10%) . However, not all applications improve with ISM (it is possible to get worse), so it is important to monitor performance and resource use.

Requirements

J2SE 1.3.1 or greater
Sparc V9 (ultrasparc I or greater)
Solaris 2.6 or greater
Ability to edit /etc/system
Sufficient physical memory to dedicate to a single application

Using ISM

To use ISM, add the following to the /etc/system file:

set shmsys:shminfo_shmmax=0xffffffff
set shmsys:shminfo_shmseg=32

If your system is 2.6, you may also need to add

set enable_grp_ism=1

After those modifications, reboot and run the VM with the flag

-XX:+UseISM

Problems and Dangers

ISM locks memory and makes it unavailable to other processes or the OS. Since the number of locked pages is based on the total heap specified, one could potential hold all of the system's memory. This option is designed for systems dedicated to specific tasks, such as a web server; ISM should not be enabled on systems with no control over the applications that arbitrary users can run.

ISM does not guarantee that the entire heap uses 4MB pages. At boot time most of physical memory is contiguous, but due to fragmentation it is possible that 4MB blocks of physical memory are no longer available. ISM will use as many 4MB pages as possible and then silently use 8KB pages for the rest, reducing performance. A performance loss may be observed over time as more applications use and return fragmented pages to the OS; the only way to guarantee that the entire region is using 4MB pages is to first reboot the system.

ISM uses the system calls shmget to allocate a region of memory, and shmat with the SHM_SHARE_MMU flag to attach to this region. Under abnormal termination it is possible to have the shared memory segment stay resident in the system. Although normal exits and common signals prompt the VM to remove the shared segment, a SIGKILL (kill -9) of a process is not catchable so no cleanup will occur. This leads to a portion of physical memory permanently unavailable to any applications. To remove such a discarded segment manually, use the command ipcs to determine the shared memory ID, and ipcrm -m to remove it.