In our latest Mirth Connect Tutorial Series video we do an overview and demonstration of how to set up and effectively utilize the Data Pruner in Mirth Connect. We walk you through how to set it up at both the global and channel level and by the end of the tutorial you will be able to inform your company’s infrastructure and compliance teams on how the Data Pruner can be beneficial to your company by increasing channel speed and performance.
Why use the Data Pruner?
Even with the declining prices of hard disks and flash memory, not all organizations can afford to let their databases grow indefinitely. Having limited resources is one of the main factors as to why IT teams are reaching out to their interface engineers, in the attempt to track the trending usage in order to allocate the right amount of resources.
When database tables are allowed to grow the indices that are originally created to make lookups quick. End up not being as performant in the long run. Depending on the type of database used, if the table is not analyzed frequently enough, queries will not behave as expected – as those indices were originally optimized for a smaller set of data.
This can affect application performance in the form of slow database reads and make the message browser perform slower to query for messages. This can also be observed when using a database poller on a large table, causing a channel to appear to be running slow, when in fact, the table it is querying from is no longer optimized to poll from repeatedly in succession.
When data needs to be recovered during unexpected system outages, it is easier to migrate smaller datasets during disaster recovery. Along with having limited resources, migrating only the relevant data helps keep backups and replicas slim as well.
Finally, for regulatory compliance, organizations need to adhere to their federal, state, or local laws on how long all the specific data should be held onto.
How to take full advantage of the Data Pruner
The most important thing to know about the data pruner is that it must be enabled at the settings level and on a per-channel basis.
Plenty of interface engineers have assumed that because they set the data pruner in a channels summary tab, they were all set and their data would not be growing. This can become a problem quickly when a typical workflow is copying an existing interface without making changes to the previous channel’s settings. Because of this, infrastructure or DevOps engineers should make sure that the settings level data pruner is enabled and set based on their system requirements.
The data pruner is disabled by default. Enabling it gives you some sensible settings to work from, but for our purposes in this example, we will go with default settings of 1-hour intervals and a block size of 1000.
You can set the schedule to also be based on time and a cron interval, as well as using the advanced settings to enable the pruner on specific months and days of the week and hours of the day.
We won’t be diving into the archive settings in this demo, but if you’re interested to learn more about them, let us know in the comments and we may cover it in a future video.
Reclaiming Disk Space
A few more caveats to know when running the data pruner and working with your infrastructure team. The data pruner does not reclaim physical space on a hard drive. This is dependent on the database used, but generally, the data pruner is marking the database tuples as obsolete. In an ideal world, the number of messages received equals the number of messages being pruned in order for the database size to remain stable. You will have to check in with your infrastructure team repeatedly, as the amount of interfaces and message ingestion increases.
To clarify, we are telling the data pruner that every hour on the hour, to go through the list of channels with enabled data pruning to prune any metadata and content that has been set at the channel level. Data is not immediately pruned until the data pruner has kicked off.
We’ll go ahead and click and save and head back to our channels view.
As you can see from the names of the channels, I have already set each channel to its own combination of pruning settings. Let’s dive into interface A first, to take a look at what a new channel’s pruning settings would be set to.
Pruning settings are found in the summary tab, under message pruning, and by default, we can see that Metadata is set to store indefinitely, and content is set to prune when message metadata is removed.
With this combination, the metadata, seen in the box below, the source and type, which is the default metadata for all channels for HL7 type messages, will be deleted at the same time as the message being deleted.
Heading over to interface B, here is our first step in setting a channel to prune. We have the Metadata set to prune after 1 day, with the content deleting, once the metadata has been removed.
Lastly, moving to interface C, we actually have the Metadata set to be stored indefinitely and have the content be pruned after 1 day. Changing the metadata to store indefinitely or a different day, allows your team to create custom reporting based on the metadata which is a lot smaller in comparison to the content.
The added benefit of having the metadata and content separate is that having the metadata for a longer period still allows users to visually see the amount of messages in the message browser, so metadata columns can be leveraged to store message ids and the like, to track down messages up or downstream, although we have already cleared out the contents in Mirth Connect.
We hope that this has provided you with a better understanding of the Data Pruner and how it can leverage it to reduce cost, increase performance, and meet regulatory requirements.
Be sure to watch the entire video to see a detailed demonstration of how to set up the data pruner using some common use cases.
If you need help with your Mirth Connect environment or have questions about any of the information covered here – use the link below to schedule a meeting with our integration experts.
