TPFUG: Enhance z/DFDL to process complex schemas more quickly and efficiently using choice keys

Reference IBM Case: TS010552059 zDFDL CTL 10

Z/TPF Lab's Bradd Kadlecik's analysis of the CTL-010 revealed that the problem was not with the size of the source event message being zDFDL parsed, but the time needed to parse a complex zDFDL schema through each of the source message's 14,300 LRECs. In this case the zDFDL had many conditional parsing statements to be processed against each of the message's LRECs and associated fields.

Key Finding: The zDFDL parsing of very large messages with many LRECS using a complex zDFDL schema is very CPU intensive consuming large amounts of CPU time, MIPS and increasing latency.

Short-term Fixes:

1. Z/TPF Lab is providing a patch to the DFDL program to always use Time-Slice (tmslc) when using the IBM z/TPF Standard formatter.

   1. The patch will automatically tmslc when zDFDL is parsing thus eliminating CTL-010s. It will have an upper limit so that it can never process more than 10 or so seconds before issuing a CTL-2010 error.

1. Z/TPF Lab is providing a short term zDFDL fix that will "cast" zDFDL schema compare values into binary rather than converting them to binary each time they are compare executed. The fix will reduce MIPs

Long-Term Fixes

1. The most desired long-term solution is to enhance the zDFDL parse process to be more efficient using the lab recommended choiceDispatchKey and choiceBranchKey DFDL annotations.

1. Per the Lab's root cause analysis, parsing is troublesome when there are very large number of zDFDL parsing choice branches with discriminators with over 1000 lrecs in the data and each LRec key has a large number of TID layouts.

2. The current zDFDL parsing approach uses a discriminator and causes each branch to be evaluated in order until an expression evaluates to true.

3. The choiceDispatchKey would cause this evaluation to happen only once for all choice branches and the choiceBranchKey would provide the a simple and quick comparison match for each branch.

4. Additionally IBM z/TPF Lab will type cast the zDFDL Schema compare operands in the DFDL expression similar to what was done for PJ45813