/
Connexion Enterprise Message Processing Throughput

Connexion Enterprise Message Processing Throughput

Owned by Jonathan Reis (Unlicensed)
Sept 02, 2014

We have made 6 fairly major changes to the Connexion Enterprise (v14) that are geared to improve the over-all throughput of message processing

  1. Message Table redesign - Message attributes that are indexed are stored in a separate table than the message data. This creates less locking, faster deletes, and improved branching performance.
  2. Multiple Message Storage Databases - Messages for different channels can be stored in different databases allowing the IO to be spread across multiple servers, and or disks.
  3. Multiple Application Servers - CPU load can be spread across multiple physical servers
  4. Message serialization/deserialization improvements - we now skip JSON serialization if the message type has a simple string representation, for example HL7 and XML.
  5. Processing history improvements - we now use a text differencing algorithm when storing the changes made to a message. This results in less data being written to the database.
  6. Message Compression - we now support a choice of compression algorithms at the Queue Device level.

Of these (1), (4), (5), and (6) will benefit standalone, single box, Connexion installs as well as the multi-computer enterprise install.

In this note I will describe Message Compression changes that have been made in v14.

A substantial amount of time, in the Connexion processing pipe-line, is spent compressing/decompressing messages. Originally trading some throughput for less storage made sense. However; in the data center, throughput is more of a concern than is space, and spending too many CPU cycles compressing and decompressing data will limit the overall scalability of the system.

 After running some tests on various compression libraries, and algorithms (see https://clientservices.mmodal.com/display/CXN/Compression+and+Throughput) we discovered that by using LZ4 compression we can can increase the throughput by up to 36% (213 messages/second GZIP vs 332 messages/second LZ4) and at the same time decrease the CPU . The trade-off for the increased speed is an increase in storage space requirements by between 5-18% depending on the size of the original message (LZ4 compression is 18% larger than GZIP for small messages, 5% larger for big messages).

Rather than switch solely to LZ4 we decided to provide an option at the Queue Device level to control which compression algorithm is used, either maximum throughput (LZ4), or maximum compression (GZIP). The default setting will be maximum through-put.

Question: The LZ4 library is licensed as BSD (see here: http://lz4net.codeplex.com/license). Do we need to pass this by M*Modal legal?

Notes:

  1. Test were done on my development machine i7 processor, 12GB ram, RAID 0 hard disks.
  2. v13 vs v14 queuing throughput for large messages is 206 m/sec vs 332 m/sec
  3. v13 vs v14 dequeuing for large messages is 366 m/sec vs 1629 m/sec. 
  4. v13 vs v14 queuing throughput for small messages is 1111 m/sec vs 2020 m/sec
  5. v13 vs v14 dequeuing throughput for small messages is 1282 m/sec vs 3076 m/sec
  6. The new .NET 4.5 GZIP in System.Compression provides better compression and throughput that the IONIC libraries used in v13.
  7. A NO compression option was tested, but the overall throughput went down. This is probably due to the increased time to store the larger sized data in the database. 
  8. Kudos go to Nick for discovering and pushing to make this change.