ISSUE:
Travelport uses Moveable VIPA (MVIPA) in MQ to provide a floating IP to the TPF Queue Manager on any CPU (a single Redundant connection point for the Remote End). This serves as TPF's equivalent of High Availability (HA) which is available to MQ on other platforms.
In an 8 CPU complex, Travelport has an MQ Sender Channel and an MQ Receiver Channel definition with the same name on each LPAR. In addition, multiple sets of MQ Sender Channels and MQ Receiver Channels are associated with the same MVIPA. And finally, Travelport has one MQ MVIPA associated with each CPU.
With the current design, Travelport suffers 2 Message Sequence Number Error Conditions; and thereby, 2 Customer Service Impacts during every planned CPU activity as the MVIPA moves from "its Home" CPU to another CPU and then back to "its Home CPU" when the activity is completed.
When Queue Managers on other platforms failover they maintain Message Sequence Numbers for both MQ Sender Channels and MQ Receiver Channels. This is not happening on TPF, which puts TPF at a distinct disadvantage.
JUSTIFICATION:
Travelport cannot rely on manual procedures or the requirement to call the remote end's outsourced vendor personnel (whose contact information is not reliable) to recover Travelport's MQ Receiver Channels especially when the Message Sequence Number Error Condition (for both MQ Sender Channels and MQ Receiver Channels) is self-imposed by TPF.
As more companies outsource their MQ Support (including Travelport), Travelport finds it impossible to avoid Customer Service Impacts when a MVIPA moves from one CPU to another CPU on the TPF Queue Manager. Manual recovery options are no longer acceptable and not quick enough to avoid Customer Service Impact.
Travelport hosts a major Departure Control System (DCS) which serves over 500 airports worldwide and boards 60 million passengers annually.
Travelport has been forced to take a “Delay in MQ Message Delivery” instead of “Slower MQ Recovery Time” for planned CPU activities. Travelport manually deletes the MQ Receiver channel definitions on the “Next CPU” prior to IPL of the “Active CPU” to avoid 2 Sequence Number Error Conditions and 2 Customer Service Impacts for each planned CPU activity. The remote end receives "MQRC_UNKNOWN_CHANNEL_NAME” errors during the planned CPU Activity preventing the MQ Message Sequence Number from advancing and avoid a Message Sequence Number Error Condition while the MVIPA is active on the "Next CPU" and another Message Sequence Number Error Condition when returning to the “its Home CPU”. Travelport must manually re-add MQ Receiver channel definitions on the "Next CPU" following each CPU activity. This effectively makes MVIPA for MQ only an option for long term CPU outages e.g. CEC Upgrades; CPU taken out of the complex due to long term troubleshooting.
Travelport’s DCS customer has complained that TPF's current MVIPA MQ Sender Channels and MQ Receiver Channels are not providing the Customer Service Experience which they demand.
REQUEST:
Travelport needs an automated mechanism for preserving the Message Sequence Numbers of the TPF MQ Sender Channels and TPF Receiver Channels as the MVIPA moves across CPUs.
Travelport needs the same seamless functionality of maintaining MQ Message Sequence Numbers for both MQ Sender Channels and Receiver channels during a TPF "HA Failover" as MQ provides on other platforms.
We have an MQ Auto Channel Recovery program which CRETC to itself every 30 seconds.
It has a list of all MQ Channel definitions and uses the following IBM Internal Functions:
isChannelIndoubt(MQSCD_ENTRY *pChannel)
isChannelReady(MQSCD_ENTRY *pChannel)
isChannelStopped(MQSCD_ENTRY *pChannel)
isChannelSender(MQSCD_ENTRY *pChannel)
For all MQ Channel Types isChannelReady is TRUE then we do nothing
Else (FALSE);
For MQ Sender Channel (isChannelSender) which we control the Sequence Number and don't have to parse the Sequence Number from the Console Log we do the following:
isChannelIndoubt is TRUE
We CRETC CVAA each of the following
ZMQSC STOP CHL-channelname
ZMQSC RESOLVE CHL-channelname ACTION-BACKOUT
dlayc();
ZMQSC RESET CHL-channelname
ZMQSC START CHL-channelname
Else (FALSE);
We CRETC CVAA each of the following:
ZMQSC STOP CHL-channelname
ZMQSC RESET CHL-channelname
ZMQSC START CHL-channelname
As I stated earlier Travelport's pain point is MQ Receiver Channels for which we don't control the "Transmission Sequence Number".
At this point we would be willing to accept a ZMQSC RESET CHL-channelname SEQNUM-0 command for MQ Receiver Channels that would set a flag to automatically accept the "MESSAGE SEQUENCE NUMBER RECEIVED" as the "EXPECTED SEQUENCE NUMBER" without having to parse the Sequence Number.
IBM already has code in the MCA that already does this for Newly Defined MQ Receiver Channels: Search for: MQSC0378 in MCA code
"QUOTE
MQSC0378W CHANNEL channelname WAS RESET. THE EXPECTED SEQUENCE expseq WAS SET TO THE RECEIVED SEQUENCE seqrcvd.
Last Updated: 2025-01-24
Where
channelname
z/TPF IBM MQ receiver channel name.
expseq
The next sequence number that the receiver channel expected to receive.
seqrcvd
The sequence number that the receiver channel received from the remote sender channel.
Last updated
Added in 2019.
Explanation
A z/TPF IBM MQ receiver channel that is newly defined issues this message when a sequence number is received other than the expected sequence number.
System action
The channel automatically resets the receiver channel to the sequence number that is received.
"UNQUOTE
This would allow for simplified programmatic solution following a MQSC0347E condition.
MQ Receiver Channels Not Running:
isChannelReady is FALSE
isChannelSender is FALSE
ZMQSC STOP CHL-channelname
ZMQSC RESET CHL-channelname SEQNUM-0
ZMQSC START CHL-channelname
e.g.
CSMP0097I 02.55.24 CPU-B SS-BSS SSU-1P1P IS-01
MQSC0347E 02.55.24 CHANNEL - RADTYB1P.A1PAPE00.M7 STOPPED
BECAUSE OF MESSAGE SEQUENCE ERRORS
EXPECTED SEQUENCE NUMBER - 165800
MESSAGE SEQUENCE NUMBER RECEIVED - 165801+
We have tried to automate MQSC0347E.
Our experience has been mixed, frustrating, and inconsistent.
First automation is faced with reading a z/TPF Message Number that spans multiple lines.
For us, the message is physically 5 lines which automation must parse. The first CSMP line tells us which CPU the commands must be entered from.
Second, we have encountered situations where of the 5 lines are not consecutive. The attached console file (COM1228.LTC) shows examples (Line 1673, Line 1680).
Thirdly, when we enter the single ZMQSC RESET CHL-OTPAQG02.A1PATE01.01 SEQNUM-10158 command, we have encountered MQSC0324E Channel is in Use error conditions.
<--End //SMPAZMQSC RESET CHL-OTPAQG02.A1PATE01.01 SEQNUM-10158--><0.00>
<--Start //SMPAZMQSC RESET CHL-A1PATA02.A1PATE01.01 SEQNUM-77213-->
CSMP0097I 18.18.17 CPU-A SS-BSS SSU-1P1P IS-01
MQSC0324E 18.18.17 CHANNEL - A1PATA02.A1PATE01.01 IS IN USE
As a result, the automation mushrooms into even more complications. Now we must enter 3 separate commands, wait on the response for each, and parse and validate each response for success.
ZMQSC STOP CHL-
ZMQSC RESET CHL-
ZMQSC START CHL-
This is why we yearn for a simple single MQ Receiver Channel command that says accept the next message sequence number received.
