Stress testing and JVM tuning

Certify and Increase Opportunity.
Be
Govt. Certified Apache Cassandra Professional

Stress testing and JVM tuning

Stress Testing Cassandra

Stress testing (sometimes called torture testing) is a form of deliberately intense or thorough testing used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results. Reasons can include:

• to determine breaking points or safe usage limits
• to confirm intended specifications are being met
• to determine modes of failure (how exactly a system fails)
• to test stable operation of a part or system outside standard usage

The cassandra-stress tool is a Java-based stress testing utility for benchmarking and load testing a Cassandra cluster. The binary installation of the tool also includes a daemon, which in larger-scale testing can prevent potential skews in the test results by keeping the JVM warm.

There are different modes of operation:

• Inserting: Loads test data.
• Reading: Reads test data.
• Indexed range slicing: Works with RandomParititioner on indexed column families.

You can use these modes with or without the cassandra-stressd daemon running (binary installs only).

JVM Tuning in Cassandra

Useful JVM options
-XX:+UseCompressedOops # enables compressed references, reducing memory overhead on 64bit JVMs

When using compressed references, the JVM stores all references to objects, classes, threads, and monitors as 32-bit values. Use the -Xcompressedrefs and -Xnocompressedrefs command-line options to enable or disable compressed references in a 64-bit JVM. Only 64-bit JVMs recognize these options.

The use of compressed references improves the performance of many applications because objects are smaller, resulting in less frequent garbage collection and improved memory cache utilization. Certain applications might not benefit from compressed references. Test the performance of your application with and without compressed references to determine if they are appropriate. For default option settings.

Using compressed references runs a different version of the JVM. You need to enable compressed references when using the dump extractor to analyze dumps produced by the JVM.
When you are using compressed references, the following structures are allocated in the lowest 4 GB of the address space:

•     Classes
•     Threads
•     Monitors

Additionally, the operating system and native libraries use some of this address space. Small Java™ heaps are also allocated in the lowest 4 GB of the address space. Larger Java heaps are allocated higher in the address space.

Native memory OutOfMemoryError exceptions might occur when using compressed references if the lowest 4 GB of address space becomes full, particularly when loading classes, starting threads, or using monitors. You can often resolve these errors with a larger -Xmx option to put the Java heap higher in the address space.

On Windows, the operating system allocates memory in the lowest 4 GB of address space by default until that area is full. A large -Xmx value might be insufficient to avoid OutOfMemoryError exceptions. Advanced users can change the default Windows allocation options by setting the registry key named HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\AllocationPreference to (REG_DWORD) 0x100000.

From Java 6 SR7, a command-line option can be used with -Xcompressedrefs to allocate the heap you specify with the -Xmx option, in a memory range of your choice. This option is -Xgc:preferredHeapBase=<address>, where <address> is the base memory address for the heap. In the following example, the heap is located at the 4GB mark, leaving the lowest 4GB of address space for use by other processes.

-Xgc:preferredHeapBase=0x100000000

If the heap cannot be allocated in a contiguous block at the preferredHeapBase address you specified, an error occurs detailing a Garbage Collection (GC) allocation failure startup. When the -Xgc:preferredHeapBase option is used with the -Xlp option, the preferredHeapBase address must be a multiple of the large page size. If you specify an inaccurate heap base address, the heap is allocated with the default page size.
From Java 6 SR 5, 64-bit JVMs recognize the following Oracle JVM options:

-XX:+UseCompressedOops
This enables compressed references in 64-bit JVMs. It is identical to specifying the -Xcompressedrefs option.
-XX:-UseCompressedOops
This prevents use of compressed references in 64-bit JVMs.

Note: These options are provided to help when porting applications from the Oracle JVM to the IBM JVM, for 64-bit platforms. The options might not be supported in subsequent releases.

-Datastax