AETNA CASUALTY & SURETY CO Update12/12/12 7:34 PM *** Electronic submission by customer via SR tool, version 2.5 *** Preferred contact method: IBM Service Request (SR) notification. *** Customer contact full name: Steve Kerins *** Telephone: 860-273-5366 *** Alt. telephone: 860-273-5366 *** Email: kerinss@aetna.com . Problem Details . Product or Service: CICS Transaction Server for z/OS 4.2.0 Component ID: 5655S9700 . Operating System: z/OS . Problem title PM44954 - bad workload trace entries. . Problem description We have the bad workload condition as described in PM44954. Our intention is to fix the condition by merging our multiple CICS Workload specifications into a single workload specification, to eliminate the condition. We wanted to remove this condition, not only because of the trace entry records being written, but to define a better awareness for the workload. We understood that if you have multple workload specifications they don't know about each other; therefore, may have a less than optimum workload distribution among the overall set of regions involved in the multi-workload topology. So, that being said, we want to prove that our Workload Specification changes fixed the BADWRKLD condition. To prove this we wanted to gather trace records before the change and after the change. We tried to get the before change trace records by using CED0 turning on trace flag 31 in the CMAS; the router MAS(s) and Targets ; and we could not capture the trace entries. Then also tried turning on the trace flags as discussed in URL: http: //www-01.ibm.com/support/docview.wss?uid=swg21213746#WLMtrace Still no trace records, Question, how do we go about getting the BADWRKLD trace entries, to validate that our Workload Specification changes were effective fixing the BADWRKLD condition? Trace flags to set, etc, Are there any other messages or indication that the BADWRKLD condition existed and subsequently fixed? Thanks for your assistance, Steve. . Business impact ( BusImpact ) Prove to fix BADWRKLD condition to improve the overall workload awareness and optimization. . *** --- FOR SR USE ONLY --- *** XRQXSRprNode65521355339835890 *** CAG1602354 ENG Y *** MTS *** 5655S97:420/5655S9700:700 (PREF) SHALES, DAVE G 12/12/12 9:02 PM Customer rep: Steve Kerins --- Aetna - Hello there Steve, - In APAR PM44954 it says that before the APAR the trace entries were written for trace point 29. You said that you were trying to capture the trace before the change by using trace point 31. It was PM44954 that changed the trace to trace point 31. - You said that you know you have the problem. How did you determine that you have the problem? I expect that whatever indicated that you have the problem would no longer indicate that you have the problem after the PTF is applied. - Steve, please try using the WLM trace point 29 for the before trace. If you have any follow up questions on this topic please let me know. If not, please close this question. I will follow up with you in a week if I have not heard from you before then. Thanks for using System z SoftwareXcel from IBM Global Technology Services. - Friendly regards, Dave Shales - Action taken: Recommended using the WLM trace point 29 for the before trace. Action plan: Wait for follow up question or question closing. Follow up in a week. AETNA CASUALTY & SURETY CO Update12/13/12 12:24 PM *** Electronic submission by customer via SR tool, version 2.5 *** Preferred contact method: IBM Service Request (SR) notification. *** Customer contact full name: Steve Kerins *** Telephone: 860-273-5366 *** Alt. telephone: 860-273-5366 *** Email: kerinss@aetna.com *** Updated by: Steve Kerins *** Email: kerinss@aetna.com . Additional comments Before PM44954, the trace entries would occur automatically. After PM44954 they don't appear. We knew we had the problem, because our trace was flooded with the BADWRKLD trace entries. We noticed this while trying to get doc for a different CPSM issue. Was difficult to get the doc, due to the extensive quantity of BADWRKLD trace entries. You are correct; we no longer get the trace entries, indicating the problem. We would like to fix the problem. We want to prove thru some means i. e. trace entries that the problem still exists and after adjustments the entries go away. We don't get the trace entries, regardless of the COD0 TRACE FLAGS set. For workload, I set all the flags to 1 to see if could get a trace record; none appeared, We want to have doc (BADWRKLD trace entries) before our change, and doc to prove that the change worked, Otherwise, how do we prove that the change worked? Steve. Steve. . LEE, KURT K A12/13/12 4:27 PM Action Taken : ~ Hi Steve, Are you looking for a way to produce BADWRKLD trace entries? I found you have a old PMR 85410,L6Q,000, in there it said, ------------------------------------------------------------------------ The BadWRKLD trace means you have defined AORs that are the targets of the non-terminal STARTS as CPSM TORs. You have defined them in a different workload than the workload the routing TOR is part of. As a result each transaction delivered, though processed, results in a BadWRKLD trace in the AOR, which gets delivered also to the CMAS to get written there also. This can add much overhead unnecessarily. ------------------------------------------------------------------------ Can you use this way to produce BADWRKLD trace entries? Best regards.. Kurt Lee for Dave who is on vacation ~ Action Plan : Customer to review, and close or ask additional questions Followup set for 12/20 to check status. AETNA CASUALTY & SURETY CO Update12/13/12 6:42 PM *** Electronic submission by customer via SR tool, version 2.5 *** Preferred contact method: IBM Service Request (SR) notification. *** Customer contact full name: Steve Kerins *** Telephone: 860-273-5366 *** Alt. telephone: 860-273-5366 *** Email: kerinss@aetna.com *** Updated by: Steve Kerins *** Email: kerinss@aetna.com . Additional comments Yep; already know about the Non Terminal Starts; this is how we originally found out about it being invalid to have multiple workload specs and doing non-terminal START. . For example setup scenerio: Router1 ==> Target/Router2 ==>Target3, . The Router1 ==> Target/router2 is in one workload spec, The Target/Router2 ==> Target3 is in different workload spec, Therefore bad workload trace entries on non-terminal starts, Our objective is to consolidate Router1 ==> Target/Router2 ==> Target3 into a single workload spec, removing the bad workload condition and improve the workload balancing distribution as previously stated. So, how do we prove that the reorganization of our workloads specs fix the bad workload condition, if we can no longer get the trace records, under the bad workload configuration? Tests to run: . Test 1) before change of WS. Test 2) after change WS. Steve. . LEE, KURT K A12/13/12 8:22 PM Action Taken : ~ Hi Steve, Before PM44954, trace entries were written for WLM trace level 29. After PM44954, trace entries were written for WLM trace level 31. Here are the steps to set WLM trace level. - Sign on to the CMAS - Enter 'COD0' - Enter 'TRACE FLAGS' - Set WLM column 29 or 31 to 1 I think this should works. Best regards.. Kurt Lee for Dave who is on vacation ~ Action Plan : Customer to review, and close or ask additional questions Followup set for 12/20 to check status. AETNA CASUALTY & SURETY CO Update12/14/12 1:17 AM *** Electronic submission by customer via SR tool, version 2.5 *** Preferred contact method: IBM Service Request (SR) notification. *** Customer contact full name: Steve Kerins *** Telephone: 860-273-5366 *** Alt. telephone: 860-273-5366 *** Email: kerinss@aetna.com *** Updated by: Steve Kerins *** Email: kerinss@aetna.com . Additional comments Yep; that's what's documented, Sign on to the CMAS - Enter 'COD0' - Enter 'TRACE FLAGS' - Set WLM column 29 or 31 to 1 Also need to do a TRACE ON, then after test TRACE OFF. No trace entries, also had co-worker try; same results; no trace entries. Steve. . LEE, KURT K 12/14/12 1:13 PM Action Taken : ~ Hello c/t, Before PM44954, customer did see BadWRKLD trace entries when a distributed route is shipped for a workload to an AOR which is also marked as a TOR, but for a different workload. After PM44954, with the same configuration. Customer no longer sees any BadWRKLD trace entry. Where does BadWRKLD trace entries go now? CMAS or the routing AOR? Do we need to do - Sign on to the AOR. - Enter 'COD0' - Enter 'TRACE FLAGS' - Set WLM column 31 to 1 as well? Best regards.. Kurt Lee for Dave who is on vacation ~ Action Plan : Consulting c/t. Followup set for 12/21 to check status. IBM Update12/14/12 1:18 PM Requeued to CPSML6,L244 by Hursley gatekeeper (CPSML3-CPSML6) IBM Update12/14/12 3:33 PM Hi guys, . First, just a minor clarification . Prior to the APAR, the BADWRKLD traces were written as exception traces with a trace point ID of 29, not as a level 29 trace. The difference is that exception traces are always written, while level traces have to be turned on. So prior to the APAR, the traces were always written, which led to the flooding of the auxtrace data sets problem . The key here, and the reason why what you are doing to turn on the level 31 WLM tracing is not working, is that you are turning on the tracing in the CMAS, but the CMAS does not generate these trace entries, the MASes do. The BADWRKLD traces are issued by EYU0WDIN in the MASes, which will then ship them up to the CMAS for inclusion in the CMAS's auxtrace dataset also. We do that with all traces issued by a MAS, except for COMs traces (we have an issue with recursion with COMs traces). . So, use COD0 in the MASes to turn on WLM level 31, or use the WLMTRACE(31) EYUPARM for the MASes. . Hope this helps. Please let me know if there are any further questions regarding this. . Regards, Tony LEE, KURT K A12/14/12 4:12 PM Action Taken : ~ Hi Steve, Please review preceding c/t's update and let me know if you have further questions. Best regards.. Kurt Lee for Dave who is on vacation ~ Action Plan : Customer to review, and close or ask additional questions Followup set for 12/21 to check status. AETNA CASUALTY & SURETY CO Update12/14/12 4:52 PM *** Electronic submission by customer via SR tool, version 2.5 *** Preferred contact method: IBM Service Request (SR) notification. *** Customer contact full name: Steve Kerins *** Telephone: 860-273-5366 *** Alt. telephone: 860-273-5366 *** Email: kerinss@aetna.com *** Updated by: Steve Kerins *** Email: kerinss@aetna.com . Additional comments Success! Got the bad workload entries; turning on COD0 WLM 31 tracing. Also had to turn on the AUX traces as well for all MAS and CMAS, BadWRKLD - .p..WDIN....WDTRBadWRKLD.........U...7CPPAE31QAB1CICG...... K FC5A0 RET-80083A66 TIME-11:49:50.0420080129 INTERVAL- 00.0054652812 *USER E6C4E3D9 C28184E6 D9D2D3C4 CA9E31AA 23B8802E *.p..WDIN.... WDTRBadWRKLD F1D8C1C2 F1C3C9C3 C7CA9E31 AA23B7D2 2E00EF6A *.U... 7CPPAE31QAB1CICG... . LEE, KURT K A12/14/12 5:12 PM Action Taken : ~ Hi Steve, Great news!! Great success!! Is there anything else I can help? Or we can close this PMR off. Best regards.. Kurt Lee for Dave who is on vacation ~ Action Plan : Customer to review, and close or ask additional questions Followup set for 12/21 to check status. AETNA CASUALTY & SURETY CO Update12/14/12 6:23 PM *** Electronic submission by customer via SR tool, version 2.5 *** Preferred contact method: IBM Service Request (SR) notification. *** Customer contact full name: Steve Kerins *** Telephone: 860-273-5366 *** Alt. telephone: 860-273-5366 *** Email: kerinss@aetna.com *** Updated by: Steve Kerins *** Email: kerinss@aetna.com . Additional comments Almost ready to close, but 2 additional items: 1) Could the PTF PMR be enhanced so that if others need to get the suppressed exception trace entries, it will be described how to do based upon our findings in this SR as suggested by IBM C/T? 2) Also for IBM C/T, how would anyone know that they have a bad workload topology, if no error or trace entries produced; we only discovered this when researching something else and the BADWRKLD exception trace records were flooding the trace dataset. . Would it be appropriate to produce a single informational / error message, something like . DFHnnnnnnI BADWRKLD Encountered; Review your CPSM Workload Specification Topology Or something to that effect; then people could look up the DFHnnnnnnI and would find out how to get the documentation to fix? COD0 WKLD Byte 31; internal trace on; AUX trace on in CMAS.... etc, Thanks, Steve. . LEE, KURT K A12/14/12 6:53 PM Action Taken : ~ Hi Steve, I like your DFHnnnnnnI idea. Could you please to submit a RFE ( Request Function Enhancement ) to developer? Please reference URL http://www.ibm.com/developerworks/rfe/ Best regards.. Kurt Lee for Dave who is on vacation ~ Action Plan : Customer to review, and close or ask additional questions Followup set for 12/21 to check status. TROPEA, TINO A12/21/12 3:25 PM Hello Steve, Tino Tropea here now for Dave & Kurt who are both now on vacation. . I read the SR and I'm just following up here on this ticket. . #1 Your request #1 was not addressed by Kurt, so let me say yes we can do that and I have taken the steps necessary to turn the contents of this SR into a technote. That way others who do not have access to your SR ticket can get the information. . #2 Re your request #2, yes submitting a RFE is the way to go about this. . If you would like to see such a message in some future release of CPSM, you can submit a Request For Enhancement to our developers letting them know. . If you want to submit a new RFE or vote on a existing RFE please goto url: http://www.ibm.com/developerworks/rfe/ you need to create a developerWorks profile, in order to use the RFE tool. . To search for or submit CICS related entries, you can specify the following for the following three pull downs: . Brand: WebSphere Product family: Transaction Processing Product: CICS Transaction Server . When you get a minute, can you provide a update. . Happy holidays. Tino action takrn: status check action plan: await feedback / closure DCF@IBMUS QACICS AETNA CASUALTY & SURETY CO Update12/24/12 2:29 PM *** Electronic submission by customer via SR tool, version 2.5 *** Preferred contact method: IBM Service Request (SR) notification. *** Customer contact full name: Steve Kerins *** Telephone: 860-273-5366 *** Alt. telephone: 860-273-5366 *** Email: kerinss@aetna.com *** Updated by: Steve Kerins *** Email: kerinss@aetna.com . Additional comments HI, I been on vacation last week, For the #2, I'll submit an RFE, Is it ok if I copy / paste this entire text into the RFE, so they understand what I"m talking about, or just supply a re-explination? Thanks, Steve .