New Features & Updates in v16
- Nick Judson
- Jonathan Reis
Version 16.0 builds upon the 15.1 branch and includes many improvements, fixes, and new features. If you are upgrading from a pre-15.1 environment, please have a look at the changes made in 15.1.
In the past few years (2019 onwards), we have seen a push towards cloud computing and automation. Many of the updates in this branch address issues relating to the management of very large instances/clusters. The Integrator framework has evolved from hosting a small number (<10) channels to replacing on-premise Connexion instances with hundreds of channels (and has a significantly re-worked UI).
There are also many improvements which will also benefit smaller Connexion/Integrator deployments. As always, we welcome your feedback on these features and updates.
New: Connexion & Gateway API
We have built a preliminary API for both Connexion and Gateway environments. Currently, these APIs are geared towards the querying and creation of the main components of each environment (Groups, Tabs, Channels, Devices, Remote Agents). We will be evolving these APIs based on feedback.
Since we are actively evolving these APIs, the easiest way to discover what is currently available is to download the sample projects (at the end of the above linked pages) and browse the methods. There are 3 assemblies (with associated nuget packages):
Connexion.Gateway.Api.dll - Core APIs for the Integrator platform.
Connexion.Api.dll - Core APIs for the Connexion platform.
Connexion.Api.Orchestration.dll - Helpers for common tasks on both platforms, as well as the ability to create routes between Connexion and Remote Agents.
New: Batch Processing
The default queue modes process messages one-at-a-time and in the order of receipt. In 15.0, the ‘Multi-threaded’ mode was added to allow parallel processing of messages. This mode exposed the ability to group message pulled from the queue and then process those groups of messages in parallel. For example, X HL7v2 messages could be pulled from the queue, grouped by patient, and then those groups processed in parallel.
While useful, ‘multi-threaded’ mode required a number of conditions to be met and was somewhat cumbersome to implement. We no longer recommend multi-threaded mode for new implementations.
Batch processing uses a new interface and function to allow the processing of a batch of messages. A collection of messages is passed to the device, and the implementer can decide on how best to deal with them. In some cases you may want to pass the entire collection of messages in a single call (for example, a web service that takes an array of messages), or, you may want to process the messages individually in parallel.
Please note that batch mode (threading / asynchronous programming) is an advanced feature that requires care! Read more about batch processing.
Example
A device which has support for Batch processing is the Remote Agent Sender device. This device brokers messages between Connexion and Remote Agents. Initial implementations of this device sent one message-at-a-time and therefore had a maximum processing speed capped by the latency of the connection. Even on fast connections, this capped throughput at around 40 messages per second.
To overcome this limit, a custom batching mechanism was implemented which cached the messages into a local file and then periodically sent that batch of messages. This resulted in a 10x throughput improvement but required a fairly complex caching mechanism.
When operating in batch mode, this device receives a collection of messages off the queue and can instantly send this batch without any custom caching mechanism. This mode is slightly faster (than custom caching) with much lower complexity and latency.
New: Message Tracing
In large/complex deployments of Connexion (often including Remote Agents), it can be challenging to determine the path a message followed from origin to destination. ‘Message Tracing’ has been added to 16.0 to simplify this task.
Changed: Gateway User Interface Hierarchy
Remote agents are now hosting many more channels and performing more functions that the original UI was designed for. This has surfaced some manageability and performance issues we have decided to change the object hierarchy from Group > Tab > Remote Agent > Channel (15.1) to Group > Remote Agent > Tab > Channel (16.0). This hierarchy provides a more manageable and performant experience for remote agents with many channels, although it does take some getting-used-to.
Changed: Bandwidth Usage Improvements
The continued feature enhancements to both Connexion and Remote Integrator have necessitated larger and larger data payloads to be transferred between the UIs and servers. This was highlighted at the beginning of the pandemic as people moved to work-from-home over (often) slow vpns. 16.0 brings a significant improvement in load times and overall speed when running over slower network connections. We now page-in data on-demand (where possible) to get you up-and-running more quickly.
New: Gateway Map Dashboard
A new dashboard showing the approximate geographic location of each remote agent.
Updated: User Dashboard
Both Connexion and Gateway have had minor updates to the user dashboards. Both now expose the ability to export the dashboard contents (to csv). The Connexion dashboard displays several new columns (backported to newer 15.1 builds) and the Gateway dashboard will now display users connected to all Gateways within the cluster (and a new 'Host' column).
Updated: Integrator Push Updates
There have been significant updates and enhancements to the ‘Deploy Installer Package’ dialog (and related functionality). More information is now displayed within the dialog and a new ‘Remote Agent Filtering’ feature lets you exclude remote agents from deployments (for various reasons).
New ‘advanced deployment’ features let you perform multiple install operations (including installing multiple packages) within a single deployment operation.
Read about the new deployment features
New: Centralized Server Connections
Several new features to make distributing server connections to many users less painful. Improvements to the display and quick switching of connected servers.
Read about centralized server connections
New: Delay Queuing
16.0 introduces the new queue type (D)elay. Delay processing is a workflow where specific messages being dequeued cannot yet be processed, and should be retried later. However, other messages within the queue can be processed and should not be blocked.
New: Release Message(s) For Processing / Manual Message Processing
A new feature has been added to the queue device called ‘Force Process Message(s)’. It allows users to select message(s) from a paused queue within a running channel and process them. This can be useful in scenarios where you wish to process a very specific subset of messages within a queue. For example, during testing, it may be useful to select specific types of message within a queue a process them one-at-a-time. This lets you verify that each type of message is being handled correctly.
Read about Manual Message Processing
New: Kafka Reader/Writer Devices
New Kafka devices: Reader, Writer
New: Globals bulk import/export
16.0 introduces a new bulk global import/export feature designed to help provision empty Connexion systems. This tool can also be used to compare global values across systems. This feature is targeted at power-users and admins and is not designed for day-to-day operations.
Read about Bulk Global Operations
New: Gateway Nightly Backup
The nightly channel backup feature has been ported from Connexion to the Gateway. This feature allows you to backup remote agent channels via a file writer device (which allows the copying of these channel files to many different targets - ftp, AWS, Azure, etc.).
Functionally equivalent to Connexion, this feature will create a channel file per remote agent, allowing you to snapshot all the channels within each remote agent.
Updated: Connexion Plugin UI
A minor update to the device manager dialog within Connexion to indicate plugins which originate from the same assembly. When selecting a plugin, any other plugins from the same assembly will be displayed with a grey bar down the left side of the dialog.
When importing new versions of a plugin assembly, all plugins indicated by the grey bar will be affected.
Updated: Queue Query Timezone Logic
A minor update to the time translation when connecting UIs to servers in different time zones. Specifically, when selecting a Queue Date/Time filter on the queue device (Today, Last 2 Days, Last 7 Days…), the filter will be adjusted to match the time zone of the server (previous versions used the time zone of the client).
This means that when selecting a predefined filter (like ‘today’), the query is for the server’s version of today. Running a ‘today’ query directly from the server now produces the same query results as a ‘today’ query run from a different time zone (remote UI).
New: Remote Agent MongoDb Error State
The gateway Ui has been enhanced to differentiate between remote agent connectivity issues and database (MongoDb) issues. This should make error diagnosis easier.
Read about the new database error state
Revisit: Channel Settings Device & Message Verification
While included in 15.1, 16.0 utilizes the Channel Settings device for message verification. Watermarking has been removed from 16.0.
Read about Message Verification using the Channel Settings Device
New: Remote Agent Decommissioning
Remote agents components (remote agent, updaters) will automatically be uninstalled from the remote computer when fully deleting a remote agent record from the Gateway. Previously, it was up to the customer to uninstall the software from their machine(s).
Updated: Alert/Event Purging
The purge events UI has been updated in both Connexion and Gateway UIs. In previous versions, three options were exposed: Purge Today, Purge All, and Custom Purge.
It was common for users to simply click the ‘purge all’ button and flush out all events (which made retroactive troubleshooting difficult). The old options have been reduced to a single purge button:
This button shows a new unified purge UI:
The new UI defaults to clearing all events for a specific channel or device (depending on which control was clicked to open the events dialog). Clearing all events can be done by users with the required ‘clear all’ permission in the main events screen.
Updated: Database Polling Device Set Seed
The database polling device has a UI which lets a user manually set the initial seed value. However, when deploying channel templates, there was no way to preset a seed value….until now. There is a new ‘Seed' tab which lets you define a query for setting the seed value. This query can be used to query a table and return a Max or Top 1, or, simply return a calculated value.
Updated: Queue Device ‘Copy to Queue’
The ‘under-the-covers’ implementation of copying messages from one queue to another has been updated to perform the copy in batches instead of a single operation. Previous implementations could negatively impact the database performance when doing a very large (millions) message copy.
Similar updates to equivalent operations (such as a bulk re-process) will most likely be rolled out in future versions of 16.0.
New: Low Latency Mode(s)
Get near real-time message transfer when spanning Remote Agent and Connexion boundaries via this new feature. Remote Agent Sender and Secure Sender can now trigger the target queue to immediately wake from sleep and start processing.
Read about the new Low Latency Mode