SUPERC Option to Ignore Trailer Record

SuperC is a complex program which can process many types of file in many different ways, and one of its key features is that the amount of storage required is very much smaller than the size of the files being processed, so it can scan and compare extremely large files. We have concluded that the scale of restructure which would be necessary to implement the suggested option means that the effort involved and the risks of making such extensive changes appear to far exceed the potential benefit of this option. We would suggest either writing a simple specific comparison tool, perhaps in REXX, to cover this case (if an equal comparison is all that is required) or to implement a simple procedure to make a copy of the relevant file without the trailer record before using SUPERC for the comparison.

If only it were that simple! In general SUPERC would have to back out any processing performed as a result of the previous record, including any side-effects. And it doesn't just handle line comparisons; it also handles word and byte comparisons and searches. At present, record selection processing checks are always done in advance of processing the record, and there are no restrictions on side-effects so we don't even keep track of what we would have to back out.

This could be solved without a read ahead buffer. Instead, buffer the last write. The fact that there is a mis-match has already been established, so instead of writing the mis-match record out and updating the counters immediately, pause the write of the mis-match record to the output file and the update of the counters until the next read is determined to be not EOF. So, effectively you'd be writing one record behind the read. If EOF is detected and the user specified DPTRL then discard the last mis-match. Easy.

This could be solved without a read ahead buffer. Instead, buffer the last write. The fact that there is a mis-match has already been established, so instead of writing the mis-match record out and updating the counters immediately, pause the write of the mis-match record to the output file and the update of the counters until the next read is determined to be not EOF. So, effectively you'd be writing one record behind the read. If EOF is detected and the user specified DPTRL then discard the last mis-match. Easy.

Although this is a very neat idea for improving usability, it unfortunately appears to be extremely difficult to implement. This is because it is not possible to tell a given record is the last record until the point where the next record is read, at which point an end-of-file indication is returned. With the existing program structure this does not happen until after the last record has already been processed. It is theoretically possible to introduce a buffering system to read ahead for this purpose, but this would require major changes to the internals of SUPERC which seem out of proportion to the potential benefit, especially as SUPERC has many different record input routines to handle different file organisations and operating systems.

We will have a more detailed look into this soon in case there is something we have missed in this initial quick assessment. However, at present it seems unlikely that we would be able to implement this request because the amount of work and the risks involved in making such a high-impact change to a mature product appear to be out of proportion to the potential benefit.