Introduction
HPE NonStop users are becoming more familiar with Kafka, as Kafka in turn becomes more and more prevalent inside large enterprise environments. Kafka is now used by thousands of companies, including 60% of the Fortune 100, and many NonStop shops. Well known users in the finance space include Goldman Sachs, Rabobank, Barclays, Jack Henry and PayPal, and household names that use Kafka include Netflix, Oracle, LinkedIn and AirBnB. A full list of Kafka users is available here.
These organizations use Kafka to manage “streams” of data, which have become prevalent as internet usage massively boosts the amount of data being generated and needing to be processed. Kafka allows these huge volumes of data to be processed in real-time, via a combination of “producers” and “consumers”, which work with a Kafka “cluster” – the main data repository.
NonStop Servers have valuable data on them that users may want to stream to Kafka, and NonStop applications may also have a need to send their data directly to Kafka. In these situations, NonStop users need a reliable, high-performing way to get that data streamed to Kafka. This is where uLinga for Kafka comes in.
uLinga for Kafka – Overview
uLinga for Kafka takes a unique approach to Kafka integration: it runs as a natively compiled Guardian process pair, and supports the Kafka communications protocols directly over TCP/IP. This removes the need for Java libraries or intermediate databases, providing the best possible performance on HPE NonStop. It also allows uLinga for Kafka to directly communicate with the Kafka cluster, getting streamed data across as quickly and reliably as possible.
With the latest release of uLinga for Kafka now available, a number of new key features have been implemented. The product has once again been benchmarked and proven to perform at massive volumes – over 25,000 TPS running on a single core of a NonStop X3 processor. This article provides a refresher on uLinga for Kafka and summarises those new features and performance figures.
uLinga for Kafka – Application Integration
uLinga for Kafka (uLinga) supports a range of options to communicate with HPE NonStop applications and read/process NonStop data. Applications can use standard Guardian IPC messages, or Pathsend requests, to send data to uLinga, and HTTP clients can send data via uLinga’s inbuilt HTTP interface. uLinga can process disk records directly from Enscribe files, and as of the most recent product release, also supports TMF Audit Trails, meaning updates to all TMF-protected files can be streamed by uLinga to Kafka.
User Exit Support
The most recent release of uLinga for Kafka includes support for customer-written user exit processing. Custom user exit code can manipulate/transform data before it is sent to Kafka, or can conditionally determine whether a particular record should be sent to Kafka or omitted. The functionality implemented by the custom code is entirely up to the customer – virtually anything is possible.
Custom user exit functionality can be invoked from a number of configuration points within uLinga. Consider the following implementation, where uLinga is reading/streaming data from an Enscribe file using its FILEREADER capability:
In this configuration, a uLinga USEREXIT could be invoked from the IPCSERVER, from the KAFKAPRODUCER, or both. This gives significant flexibility in terms of where customer-written code is executed.
If uLinga was configured to monitor two different sets of Enscribe files, it would have two different FILEREADER resources. In this case you could configure a different USEREXIT for each FILEREADER, providing different functionality for each file being processed.
If you wanted all data being streamed to a single topic to have the same custom code logic applied, you could configure a USEREXIT at the KAFKAPRODUCER. This would ensure that all data, regardless of the origin, would have the same custom code invoked.
Customer-written user exit code can be managed by uLinga for Kafka, using uLinga’s Worker Farm framework. Customer-written code can also be run as a Pathway server class, and can even be run on a remote HTTP server, accessed by uLinga’s built-in HTTP support.
High Performance
Because uLinga for Kafka is written in C, and does not
require any additional libraries or interim servers, it is able to achieve
impressive performance figures, with extremely low latency.
Infrasoft testing has shown that the product is able to
exceed 25,000 TPS with sub-millisecond latency, running in a single core of a
NonStop X3 processor. Perhaps more
importantly given most NonStops are used in OLTP environments, when running at
a steady state of 1000TPS, uLinga for Kafka has a very low CPU
requirement. This will allow uLinga for
Kafka to process data from the busiest of NonStop OLTP applications with
minimal overhead.
uLinga for Kafka is now available. Please contact productinfo@infrasoft.com.au for
more information or to arrange a trial.