Powerful new feature provides extensive data manipulation options
uLinga for Kafka, widely recognised as the best solution for integrating NonStop applications with Kafka, and uLinga Gateway, the newest product in the uLinga suite, have recently been enhanced to add powerful data transformation capabilities. This article gives an introductory overview to these capabilities, and then takes a deep dive into one possible use case – transforming the venerable ISO8583 standard into JSON.
uLinga Gateway – Introduction
uLinga Gateway, recently added to the uLinga suite, provides a wide range of network protocol handling and message manipulation services that can be used in a number of applications and use cases. uLinga Gateway can handle any combination of TCP/IP, TLS, HTTP, Pathway, Guardian IPC and more, allowing you to connect almost any NonStop application with any endpoint via simple configuration.
Now, with Data Transformation, uLinga product capabilities are extended to allow easily configured data transformation and message manipulation. uLinga’s Data Transformation can handle any combination of binary and JSON messages for any input or output messages, with additional message formats in development.
Use Case – ISO8583 Data Transformation
The primary focus of this article is a very common use case in NonStop circles, involving messages using the ISO8583 standard. ISO8583 has been in very wide usage in the NonStop financial transaction processing space for decades, courtesy of its broad adoption between financial institutions, switches and card brands such as Visa and Mastercard. While ISO8583 is undoubtedly powerful and very capable in this space, it is not a simple protocol, and developing new applications to support this standard can be a challenge.
uLinga’s Data Transformation now supports ISO8583 messages, and can transform them into a format more easily supported by external entities, such as JSON, allowing REST endpoints to access traditional transaction processing mechanisms.
ISO8583 messages have a number of aspects that can take some time to learn and understand, including bitmaps and variable length fields. A standard ISO8583 message looks something like this:
A common use case for this type of data streaming is to send application logs (such as a BASE24 Transaction Log File, or TLF) to Kafka. BASE24 usually has 2 or 3 TLFs active at any point in time; uLinga for Kafka supports this via FILEREADER configuration.
The Message Type Indicator (MTI) indicates what type of message this is, and bitmaps are used to indicated which fields are present in the message. The fields themselves can be fixed format, or variable with a two- or three-digit length (LLVAR and LLLVAR). The entire message can be in ASCII or in binary/binary-coded decimal. There is a good primer on ISO8583 message formats available in Wikipedia.
An example use case would be where a NonStop application processes incoming ISO8583 requests to allow access to the financial transaction processing services it provides (for instance, balance enquiries). Traditional clients would access this service using ISO8583 requests sent directly into the NonStop, as depicted below.
The JSON request does not need bitmaps, and variable length fields are handled simply through the JSON formatting – no need for LLVAR and LLLVAR fields. uLinga Gateway adds these elements as part of building its ISO8583 request.
A similar transformation takes place with the response, allowing the client to once again deal with simple JSON.
uLinga Gateway’s Data Transformation GUI (built into the WebCon UI) allows you to easily build your data transformation, mapping fields from input to output via drag and drop.
Message Queuing
Kafka can be used for message queuing, and can provide a very high performance alternative to traditional message queuing solutions. On the NonStop, uLinga for Kafka can facilitate message queueing implementations, by virtue of the range of APIs it supports, as well as its “produce” and “consume” functionality.
One way message queuing, from NonStop to Kafka (and from their to other platforms and applications) can be achieved in much the same way as already outlined in “Data Streaming” above. One way message queuing from other applications to Kafka (and from there to NonStop) via uLinga for Kafka requires ULK’s consumer functionality. uLinga for Kafka can monitor one or more topics in a Kafka cluster, consuming (reading) new messages as they are posted to the topic. Once consumed, they can be presented to a NonStop application via a range of mechanisms – via an interim file, or directly presented via Guardian IPC, TCP/IP or HTTP communications. ULK’s consumer functionality looks like this:
uLinga Gateway Data Transformation can also transform other types of binary messages into JSON, and even convert between formats – binary to binary or JSON to JSON for instance. When combined with uLinga Gateway’s protocol transformation capabilities you have an extremely powerful tool to address virtually any application connectivity requirement.
uLinga for Kafka and Data Transformation
uLinga for Kafka, which is now in use at a number of HPE NonStop installations worldwide, has a broad range of options for integrating NonStop applications and data with Kafka. Some of the key features of uLinga for Kafka include:
- FILEREADER to automatically monitor and read new records as they are written to entry-sequenced Enscribe files, and stream (produce) the data read straight to Kafka.
- AUDITREADER to automatically monitor and read new records as they are written to a TMF audit trail, and stream (produce) that data straight to Kafka.
- IPCSERVER support, allowing applications to directly send data to uLinga for Kafka, and from there be streamed (produced) straight to Kafka.
- FILEWRITER support, allowing for data to be streamed (consumed) from a Kafka topic and written to an Enscribe entry-sequenced file.
- IPCCLIENT support, allowing for data to be streamed (consumed) from a Kafka topic and sent to a NonStop application process.
With the addition of the new Data Transformation feature, this allows you to configure uLinga for Kafka to (for instance) read records from an entry-sequenced file, and convert those records into JSON before streaming them to Kafka, allowing consumers to much more easily understand the streamed data.
Do you have a data transformation need in your environment? Chances are the uLinga’s new Data Transformation feature can help. Get in touch with us at productinfo@infrasoft.com.au to arrange a demo so you can see for yourself.