(TPFUG/Visa) Tooling required to keep online and offline image unique file system files in sync and cleanup obsolete/superseded content

The list of files loaded online on TPF in /sys/tpf_pbfiles the list of files included in an OLD or TLD deck using the current content of tpf.loadfile can be different. We need a way to be able to determine what those differences are, and a way to cleanup (remove or obsolete) file system files from linux and TPF that are no longer required. This is important because old versioned files may contain vulnerabilities, and loading files to TPF that are no longer required is confusing as well as a waste of space.

• Anomalies between what is loaded to TPF and what is included in a deck that used tpf.loadfile (e.g. a full TLD deck where PROGCLEAR=YES is specified) may result from many sources including:

  • Old versioned files in TPF location /sys/tpf_pbfiles/apps/lib/ or /sys/tpf_pbfiles/apps/xxxx/lib (e.g. xxxx = tpfccv) were @@DELETEd or zfile rm’d on TPF, but the linux source management system still contains them.

    • A deck created using tpf.loadfile would reload the old versioned files on TPF which is undesirable

  • Files that were once loaded due to use of a wildcard in an IBM loadfile are no longer generated or part of current Java support or applications (e.g. /sys/tpf_pbfiles/opt/ibm/ for Java 11). The unused files may still be loaded to TPF.

    • E.g. if a lower TPF_ROOT has files in a wildcarded subdirectory location that was used at one time to load files to TPF, but the files no longer exist in the higher TPF_ROOT used to build the deck.

    • In this case, dropping the files from TPF is the right action but we are still left with old files in lower TPF_ROOT locations. This could be problematic if these files are subject to vulnerability findings.

  • Files that had a specific definition in a loadfile (i.e. were not included due to use of a wildcard) were subsequently made obsolete (loadfile definition removed), but not @@DELETEd or zfile rm’d on TPF. The obsolete files may still be loaded to TPF.

    • Dropping these files from TPF is the right action. Ideally these files should also be made obsolete in (or removed from) the source management system

  • In addition to anomalies, unnecessary files may be loaded to TPF and continue to be loaded in a deck using tpf.loadfile if:

    • Customers do not remove obsolete file definitions from tpf_app_usr.loadfile – that problem is on the customer to resolve

    • IBM do not remove obsolete specifically named file definitions from the IBM loadfiles – this doesn’t usually happen, because the obsolete specifically named file is usually removed from the appropriate IBM loadfile as part of the APEDIT that makes the file obsolete.

    • Files included by wildcard are no longer required. This is most likely to occur for versioned online Maven dependencies, but could occur for any file system file loaded as a result of residing in a wildcarded source subdirectory.

The requested tooling should provide for one-off rationalization of online/offline file system files but would ideally be able to be extended to a continuous integration/cleanup model. We see the following functional support components as essential:

1. An online utility (possibly similar to ZDECK IPAT or ZDECK RIAT) that creates a list of all files (including the full /sys/tpf_pbfiles/..path) that reside in /sys/tpf_pbfiles and writes that list to a user defined file system file which we can then FTP (or similar) offline.

   • The utility needs to understand the conventions used in /sys/tpf_pbfiles e.g. hidden directories and suffixed file names that are part of e-loader support, and only produce a list of discrete file name loaded to TPF that can be compared to a load deck of file system files created using tpf.loadfile.

2. A linux utility that takes the file produced by the online utility above, compares it to the list of files in a deck created using tpf.loadfile and produces a list of differences.

3. A means of auditing/cleaning up old files from wildcard’d locations.

   • Need to be able to identify and remove unneeded files from TPF

   • Need to be able to identify and remove unneeded file from linux for any given TPF_ROOT path

One possible idea for requirement 3) is

• Provide a way to identify which files in a deck created from tpf.loadfile were added as a result of a wildcard loadfile entry

• Provide a way to create a current list of files in wildcarded locations (since Semeru has wildcard locations, not sure it’s safe to assume these would always be online Maven dependency files)

  • Although not ideal, we would not be opposed to requiring this step include a test build run of all current IBM and Customer Java code to identify what is “current”.

• Provide a way to compare the files included in the deck due to wildcards with the the list of currently required files.

  • Optionally generate @@DELETE cards in the deck to remove old and unnecessary files to cleanup online TPF.

• Provide a way to allow each TPF_ROOT to be searched to identify any files in wildcarded locations that are NOT in the current list of required files

  • Optionally archive them to another location and/or delete them to allow for linux source library cleanup.