Generate Java-/Heap-Dump from a running JVMServer

This RFE is satisfied by CICS Transaction Server for z/OS V5.6 which is Generally Available today 12th June 2020.
For more information see the CICS TS V5.6 Knowledge Center https://www.ibm.com/support/knowledgecenter/SSGMCP_5.6.0/whats-new/intro.html#intro__gather_spi

This RFE is satisfied by CICS Transaction Server for z/OS V5.6 which is announced today 7th April 2020 and which has a planned availability date of 12th June 2020.
For more information see the CICS TS V5.6 announcement letter
https://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/7/897/ENUS220-077/index.html&request_locale=en

This is something we would like to address. The RFE is being re-opened and moved into 'Planned for Future release' status. Please note:
IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Whilst this requirement is valid, based on our current plans and priorities, it is not likely that this could be implemented in the next 12 months, or in the next CICS TS release. Correspondingly this requirement is being declined at this point. The requirement will be kept in the RFE system and might be reassessed in the future. You also have an opportunity to resubmit in twelve months time if you wish it to be reconsidered then.

Due to processing by IBM, this request was reassigned to have the following updated attributes:
Brand - Servers and Systems Software
Product family - Transaction Processing
Product - CICS Transaction Server

For recording keeping, the previous attributes were:
Brand - WebSphere
Product family - Transaction Processing
Product - CICS Transaction Server

Hi Juergen, "context 6" should do it.

Hello,

regarding jdmpview, I didn't understand, how can you select the context of the JVMServer. If I start jdmpview I get the following view:
jdmpview -core DUMP.E001.D150415.T103307.PRB88786
DTFJView version 4.26.67, using DTFJ version 1.11.26070
Loading image from DTFJ...

For a list of commands, type "help"; for how to use "help", type "help help"
Available contexts (* = currently selected context) :

Source : file:///usr/lpp/java/J7.0_64/bin/DUMP.E001.D150415.T103307.PRB88786
0 : ASID: 0xffffffffc9c1d9c8 : No JRE : No JRE
1 : ASID: 0x1 : No JRE : No JRE
2 : ASID: 0x3 : No JRE : No JRE
3 : ASID: 0x4 : No JRE : No JRE
4 : ASID: 0x7 : No JRE : No JRE
*5 : ASID: 0x349 EDB: 0x48f1e055e8 : JRE 1.7.0 z/OS s390x-64 build (pmz6470sr8-20141026_01(SR8) )
6 : ASID: 0x349 EDB: 0x4880b055e8 : JRE 1.7.0 z/OS s390x-64 build (pmz6470sr8-20141026_01(SR8) )
7 : ASID: 0x349 EDB: 0x49709055e8 : JRE 1.7.0 z/OS s390x-64 build (pmz6470sr8-20141026_01(SR8) )
8 : ASID: 0x349 EDB: 0x59a8efb0 : No JRE
9 : ASID: 0xffff : No JRE : No JRE


I've three JVMs in this CICS-Region. Context 5 is selected, but I need the heapdump from context 6. How can I switch context?

Best regards,
Juergen

Hi Juergen,

You can select the current JVM on which to operate, i.e. the 'context'...when you start jdmpview e.g.
jdmpview -core 01670.082.000.JVM.DUMP1
DTFJView version 4.29.1, using DTFJ version 1.11.28006
Loading image from DTFJ...

For a list of commands, type "help"; for how to use "help", type "help help"
Available contexts (* = currently selected context) :

Source : file:///C:/01670.082.000.JVM.DUMP1
0 : ASID: 0x1 : No JRE : No JRE
1 : ASID: 0x4 : No JRE : No JRE
2 : ASID: 0x7 : No JRE : No JRE
*3 : ASID: 0x2a9 EDB: 0x35e30878 : JRE 1.7.0 z/OS s390-31 build (pmz3170_27sr1-20140411_01(SR1) )
4 : ASID: 0xffff : No JRE : No JRE
5 : ASID: 0x40404040 : No JRE : No JRE

Although you cannot immediately tell which JVMSERVER you have selected, you can subsequently check the PID to match the JVMSERVER (from CEMT INQ JVM) thus pinpoint the correct JVM:

> info proc
Process ID:
50332370

At this time jdmpview cannot produce a javacore, but that can be done by sending kill -quit to the JVM. z/OS customers aren't usually as interested in a javacore, it is not as useful a system dump when multiple languages or JCICS calls are involved.

However, I appreciate the point about pausing 'all' of CICS with a snap dump, rather than just the target JVMSERVER, and also accept the ease of getting specific diagnostics with a JVMSERVER PERFORM command - so I'll explore this option further and consider how it might fit into release plans. In the meantime I hope I've provided enough information to allow you to achieve what you want in the short term, albeit not as smooth as you would like.

Hi Ivan,

thanks for the tip with the "jdmpview" and that this tool can also work with a CICS-System-Dump. That was new for me.
I tried it out and can see, that it is possible to create a heapdump in a separate file. But I unfortunately have some trouble:
- If my cics-region contains more than one JVMServer, I'm not able to create a heapdump for a specific JVM. I only get informations about the "current" JVM. But what is the "current" JVM and how can I switch between the JVMs?
- Same problem, if I want to inspect a Java-Thread. I'm not able to switch the JVM.
- If I want to use a tool like "IBM Thread and Monitor Dump Analyzer for Java" for analyzing a core-dump, I need a javacore-File. But "jdmpview" is not able to create a javacore-file.
- The creation of a CICS-System-Dump stops all processing in the whole CICS-Region for some seconds. If I need only a heapdump or a javacore-dump from one specific JVM, it's a very big effort and not really the best way in a productive environment.

You mentioned to use the programmatic approach. That is possible, but not useful if I want to inspect a current problem in a productive environment. We sometimes see problems in a jvm, like slowly increasing memory-leaks. That problems are mostly not reproducable in our test-environment. In that case it is really helpful, to have an easy-usable and well-known possibility to get that dumps, without changing the code.

Because of that facts, I think the easiest and comfortablest way to support the CICS-Systemprogrammer is, to extend the JVMSERVER SPI and furthermore support that commands in CICS-Explorer.

Many thanks for your support and best regards,
Juergen

Hi Juergen,

Regarding the 'irregular' state of the JVMSERVER after issuing a UNIX kill command...if you issue a "kill -3 " it would not usually change the state of the JVMSERVER or impact its stability. However, some unix consoles can interpret the -3 as SIGKILL rather than QUIT, and this will upset the JVMSERVER! To ensure that doesn't happen, it is always safer to issue "kill -quit ".

Regarding the diagnosis, are you aware that you can get a Java system dump using CEMT PERFORM SNAP? The Java heap, thread, and system diagnostics can be obtained running the "jdmpview" utility against the resulting dump. Here is a link for more information: http://www-01.ibm.com/support/knowledgecenter/SSYKE2_7.0.0/com.ibm.java.lnx.71.doc/diag/tools/dump_viewer_dtfjview/dump_viewer.html

Since the creation of the JVM server model I have taken the design stance that we (CICS) should not duplicate or add CICS 'veneer' to existing standard Java capabilities or tools - it leads to a CICS maintenance burden when there are Java changes.

However, this is not an absolute rule - and if you don't believe CEMT PERFORM SNAP and JDMPVIEW can't satisfy your request, then please let us know. In that event we could consider extending the JVMSERVER SPI to take the dumps... for example, CEMT PERFORM JVMSERVER HEAP|THREAD|SYSTEM ?

You could also consider the programmatic approach, where your application can take Java dumps using the com.ibm.jvm.Dump API .

many thanks,
Ivan

If we have some trouble in a "productive" Java-Environment the only chance to inspect the problem is to work with a Thread-, Heap- or JVM-System-Dump. Of course not every CICS-Sysprog is able to deal with a Java-problem, but that's not necessary, because it's the job for the java-people. The job of CICS-Sysprog is, to collect all necessary informations around the problem.
The integration of IBM-Health-Center in a productive environment is not practicable, it needs to much configuration until we can use it. Also using of "late binding" is not so easy, as it should be to help in a fast and comfortable way.
CICS-Explorer, CPSM or CEMT already exists. There's nothing to configure, it's an easy way and well-known to all CICS-Guys.

Can you clarify what the underlying need for obtaining Java diagnostics is, as we wouldn't expect CICS customer to be regularly requesting Java diagnostics. The IBM Health Center is an excellent tool that can help with many Java problem determination issues such as lock or memory analysis and also has the ability to request dumps if required.