TPFDF should provide a more efficient SORT command

TPFUG_Req_Number: DF00021
Contact Originator Id: Waseem Majeed
Abstract: TPFDF should provide a more efficient way to SORT sub-files.
Description: The current SORT command internally uses a DBADD to SORT files. In an effort to make the process more efficient the sub-file is sub-divided into batches of (X) blocks, where X is 10 by default. The use of DBADD makes this similar to a selection and insertion SORT. Essentially that means reading through the file to find the correct place to insert the record, moving the rest of the records to make room for the new record and then adding that record. This causes extensive searching and I/O operations as TPFDF moves from one end of the file to the other and back. This process is very I/O intensive and requires a lot of comparisons and data movements. The technique of using batches does little to improve performance as the same process is repeated when merging the sorted batches. The performance can be improved somewhat by using DETAC mode which does cut down on the I/O but not the path length and data compares and moves.

What is needed is a better SORT algorithm and also one that takes advantage of the TPF HEAP storage.
Critical Success Factors: If an economical means to sort large sub-files is not available, then either the process is very inefficient or in some cases not feasible at all. In the latter case, the result is that user requirements cannot be met and some business function is impacted.
Alternative Solutions: Sorting of data is a well researched and studied subject. Many algorithms for sorting have been devised and documented in the literature. One or more of these could be adapted for TPFDF, taking into account the characteristics of the TPF environment.