Stress testing and JVM tuning

Certify and Increase Opportunity.
Be
Govt. Certified Apache Cassandra Professional

Stress Testing Cassandra

Stress testing (sometimes called torture testing) is a form of deliberately intense or thorough testing used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results. Reasons can include:

to determine breaking points or safe usage limits
to confirm intended specifications are being met
to determine modes of failure (how exactly a system fails)
to test stable operation of a part or system outside standard usage

The cassandra-stress tool is a Java-based stress testing utility for benchmarking and load testing a Cassandra cluster. The binary installation of the tool also includes a daemon, which in larger-scale testing can prevent potential skews in the test results by keeping the JVM warm.

There are different modes of operation:

Inserting: Loads test data.
Reading: Reads test data.
Indexed range slicing: Works with RandomParititioner on indexed column families.

You can use these modes with or without the cassandra-stressd daemon running (binary installs only).

Usage

Packaged installs: cassandra-stress [options]
Binary installs: <install_location>/tools/bin/cassandra-stress [options]

The available options are:

Long OptionShort Option	Description
–average-size-values-V	Generate column values of average rather than specific size.
–cardinality <CARDINALITY>-C <CARDINALITY>	Number of unique values stored in columns. Default is 50.
–columns <COLUMNS>-c <COLUMNS>	Number of columns per key. Default is 5.
–column-size <COLUMN-SIZE>-S <COLUMN-SIZE>	Size of column values in bytes. Default is 34.
–compaction-strategy <COMPACTION-STRATEGY>-Z <COMPACTION-STRATEGY>	Specifies which compaction strategy to use.
–comparator <COMPARATOR>-U <COMPARATOR>	Specifies which column comparator to use. Supported types are: TimeUUIDType, AsciiType, and UTF8Type.
–compression <COMPRESSION>-I <COMPRESSION>	Specifies the compression to use for SSTables. Default is no compression.
–consistency-level <CONSISTENCY-LEVEL>-e <CONSISTENCY-LEVEL>	Consistency level to use (ONE, QUORUM, LOCAL_QUORUM, EACH_QUORUM, ALL, ANY). Default is ONE.
–create-index <CREATE-INDEX>-x <CREATE-INDEX>	Type of index to create on column families (KEYS).
–enable-cql-L	Perform queries using CQL (Cassandra Query Language).
–family-type <TYPE> -y <TYPE>	Sets the column family type.
–file <FILE>-f <FILE>	Write output to a given file.
–help-h	Show help.
–keep-going-k	Ignore errors when inserting or reading. When set, –keep-trying has no effect. Default is false.
–keep-trying <KEEP-TRYING>-K <KEEP-TRYING>	Retry on-going operation N times (in case of failure). Use a positive integer. The default is 10.
–keys-per-call <KEYS-PER-CALL>-g <KEYS-PER-CALL>	Number of keys to per call. Default is 1000.
–nodes <NODES>-d <NODES>	Nodes to perform the test against. Must be comma separated with no spaces. Default is localhost.
–nodesfile <NODESFILE>-D <NODESFILE>	File containing host nodes (one per line).
–no-replicate-on-write-W	Set replicate_on_write to false for counters. Only for counters with a consistency level of ONE (CL=ONE).
–num-different-keys <NUM-DIFFERENT-KEYS>-F <NUM-DIFFERENT-KEYS>	Number of different keys. If less than NUM-KEYS, the same key is re-used multiple times. Default is NUM-KEYS.
–num-keys <NUMKEYS>-n <NUMKEYS>	Number of keys to write or read. Default is 1,000,000.
–operation <OPERATION>-o <OPERATION>	Operation to perform: INSERT, READ, INDEXED_RANGE_SLICE, MULTI_GET, COUNTER_ADD, COUNTER_GET. Default is INSERT.
–port <PORT>-p <PORT>	Thrift port. Default is 9160.
–progress-interval <PROGRESS-INTERVAL>-i <PROGRESS-INTERVAL>	The interval, in seconds, at which progress is output. Default is 10 seconds.
–query-names <QUERY-NAMES>-Q <QUERY-NAMES>	Comma-separated list of column names to retrieve from each row.
–random-r	Use random key generator. When used –stdev has no effect. Default is false.
–replication-factor <REPLICATION-FACTOR>>-l <REPLICATION-FACTOR>	Replication Factor to use when creating column families. Default is 1.
–replication-strategy <REPLICATION-STRATEGY>-R <REPLICATION-STRATEGY>	Replication strategy to use (only on insert and when a keyspace does not exist.) Default is: SimpleStrategy.
–send-to <SEND-TO>-T <SEND-TO>	Sends the command as a request to the cassandra-stressd daemon at the specified IP address. The daemon must already be running at that address.
–skip-keys <SKIP-KEYS>-N <SKIP-KEYS>	Fraction of keys to skip initially. Default is 0.
–stdev <STDEV>-s <STDEV>	Standard deviation. Default is 0.1.
–strategy-properties <STRATEGY-PROPERTIES>–O <STRATEGY-PROPERTIES>	Replication strategy properties in the following format: <dc_name>:<num>,<dc_name>:<num>,… For use with NetworkTopologyStrategy.
–threads <THREADS>-t <THREADS>	Number of threads to use. Default is 50.
–unframed-m	Use unframed transport. Default is false.
–use-prepared-statements-P	(CQL only) Perform queries using prepared statements.

JVM Tuning in Cassandra

Useful JVM options
-XX:+UseCompressedOops # enables compressed references, reducing memory overhead on 64bit JVMs

When using compressed references, the JVM stores all references to objects, classes, threads, and monitors as 32-bit values. Use the -Xcompressedrefs and -Xnocompressedrefs command-line options to enable or disable compressed references in a 64-bit JVM. Only 64-bit JVMs recognize these options.

The use of compressed references improves the performance of many applications because objects are smaller, resulting in less frequent garbage collection and improved memory cache utilization. Certain applications might not benefit from compressed references. Test the performance of your application with and without compressed references to determine if they are appropriate. For default option settings.

Using compressed references runs a different version of the JVM. You need to enable compressed references when using the dump extractor to analyze dumps produced by the JVM.
When you are using compressed references, the following structures are allocated in the lowest 4 GB of the address space:

Classes
Threads
Monitors

Additionally, the operating system and native libraries use some of this address space. Small Java™ heaps are also allocated in the lowest 4 GB of the address space. Larger Java heaps are allocated higher in the address space.

Native memory OutOfMemoryError exceptions might occur when using compressed references if the lowest 4 GB of address space becomes full, particularly when loading classes, starting threads, or using monitors. You can often resolve these errors with a larger -Xmx option to put the Java heap higher in the address space.

On Windows, the operating system allocates memory in the lowest 4 GB of address space by default until that area is full. A large -Xmx value might be insufficient to avoid OutOfMemoryError exceptions. Advanced users can change the default Windows allocation options by setting the registry key named HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\AllocationPreference to (REG_DWORD) 0x100000.

From Java 6 SR7, a command-line option can be used with -Xcompressedrefs to allocate the heap you specify with the -Xmx option, in a memory range of your choice. This option is -Xgc:preferredHeapBase=<address>, where <address> is the base memory address for the heap. In the following example, the heap is located at the 4GB mark, leaving the lowest 4GB of address space for use by other processes.

-Xgc:preferredHeapBase=0x100000000

If the heap cannot be allocated in a contiguous block at the preferredHeapBase address you specified, an error occurs detailing a Garbage Collection (GC) allocation failure startup. When the -Xgc:preferredHeapBase option is used with the -Xlp option, the preferredHeapBase address must be a multiple of the large page size. If you specify an inaccurate heap base address, the heap is allocated with the default page size.
From Java 6 SR 5, 64-bit JVMs recognize the following Oracle JVM options:

-XX:+UseCompressedOops
This enables compressed references in 64-bit JVMs. It is identical to specifying the -Xcompressedrefs option.
-XX:-UseCompressedOops
This prevents use of compressed references in 64-bit JVMs.

Note: These options are provided to help when porting applications from the Oracle JVM to the IBM JVM, for 64-bit platforms. The options might not be supported in subsequent releases.