Transcription
MIGRATION OF LEGACY WSR-88D ALGORITHMS AND PRODUCTGENERATORS TO THE OPEN SYSTEMZhongqi Jingls, Steve Smith’, Michael Jain3and Allen Zahrai3‘CooperativeInstitute for Mesoscale Meteorological Studies,University of Oklahoma, Norman, Oklahoma*NOAA/NWS/WSR-88DOperational Support Facility3NOAA/ERUNationalSevere Storms Laboratorypreprocessor is developed to convert theCONCURRENT FORTRAN code to a standardFORTRAN code, that can be compiled on a UNIXmachine. This converted version is used onlytemporarily at compile time. A set of supportingfunctions, emulating the current RPG API, isdeveloped to provide data flow and processing controlsupport. All inter-process data exchanges areimplemented through message passing in thesupporting functions.“Wrapped” by the newsupporting functions, the tasks turn into independent,data driven and dynamically relocatable componentsthat can be used in a distributed environment.AbstractA substantial part of the current WeatherSurveillance Radar-1988 Doppler (WSR-88D) RadarProduct Genration (RPG) software consists of theproduct generation and meteorological algorithms,which were implemented through about 50 tasks.These tasks, written in FORTRAN withCONCURRENT extensions and running under theCONCURRENT 0 3 3 2 operating system (OS), can notbe directly compiled and run under a UNIX OS. Intertask communication is implemented through usingshared memory segments, requiring all tasks to berunning on the same computer hardware. Tasks mustfit into a predefined data flow network and arecontrolled by a central control module. This makes thesystem sensitive to task failure and difficult to bedynamically reconfigured (redistributing the tasksamong hosts, adding new tasks and so on). These taskscontain sophisticated algorithms and lengthy code.Rewriting these tasks would require a substantialprogramming and verification effort and causemaintenance problems in the transition period whenboth current and the ported versions had to bemaintained. On the other hand,‘these tasks use asimilar coding structure and application programminginterface (API) and requires minimal system support,which makes direct porting feasible. After some pathfinding work, we decided to take a direct portingapproach to migrate these tasks to the Open SytemRPG (ORPG). The porting approach is unique in thatonly a single version of the source code is maintained.The ported version uses the original RPG code withoutany change for all subroutines except the mainsubroutine, which is replaced by a new version. A1. IntroductionAs part of the NEXRAD product improvementeffort [Saffle and Johnson, 19971, the current WSR88D Radar Product Generator (RPG) software is beingmigrated to the open system computing environment. Asubstantial part of the RPG software consists of theproduct generation and meteorological algorithms,which was implemented through more than 50 tasks.These tasks, written in FORTRAN withCONCURRENT extensions and running under theCONCURRENT OS32 operating system (OS), cannotbe directly compiled and run under a UNIX OS.Inter-task communication is implemented through theuse of shared memory segments, requiring all tasks torun on the same computer hardware. Tasks must fit intoa predefined data flow network and are controlled by acentral control module. This design makes the systemsensitive to task failure and difficult to be dynamicallyreconfigured (redistributing the tasks among multiplehosts, adding new tasks, and so on).The WSR-88D product generation andmeteorological algorithm t s contain sophisticatedalgorithms and lengthy code. Rewriting these taskswould require a substantial programming andverification effort and cause maintenance problems inCorresponding author address: Zhongqi Jing,National Severe St o m Laboratory, 13 13 HalleyCircle, Norman, OK 73069-8480; e-mail jing@nsslsun.nssl.uoknor.edu 318CH36015-97/0000-0318 1.0001997 IEEEAuthorized licensed use limited to: University of Oklahoma Libraries. Downloaded on June 12, 2009 at 16:11 from IEEE Xplore. Restrictions apply.
the transition period when both current and the opensystem RPG versions have to be maintained.one in the main function. With this exception, theremaining W G modules are directly ported without orwith minor modifications. By direct porting, we meanthat the modules are simply copied over and compiledon an ORPG host.On the other hand, these tasks use a similar codingstructure and application programming interface (API)and require minimal system support, which makesdirect porting feasible. After some path-finding work,it was decided to take a direct porting approach tomigrate these tasks to the open system RPG. In thispaper we describe the porting approach and discusssome of the related issues.The RPG FORTRAN code contains non-standardextensions that may not be supported by theFORTRAN compiler on an ORPG machine. AFORTRAN preprocessor can be developed to convertthe existing code to an ANSI FORTRAN versionwhich can be compiled by any FORTRAN compilerthat supports the standard. The conversion is conductedat compile time and the converted version is discardedafter compilation. Only the original RPG source codeis maintained. The preprocessor can be ported todifferent platforms while the RPG code does not needto be modified.In the following we designate the existing RPGand the Open System RPG under development by RPGand ORPG respectively.2. The Porting MethodologyThe porting approach we adopted was designed tomeet the following goals:To guarantee that the ported code can still becompiled by the CONCURRENT OS/32 FORTRANcompiler, any modification to the ported code must betransparent to the CONCURRENT compiler. This isdone by defining special preprocessor directives thatmake the modifications appear as comment lines to theCONCURRENT compiler.The modification to the existing code should beminimized such that the amount of verificationwork for the ported code can be significantlyreduced.The ported software should run in a distributedenvironment and fully support dynamic systemreconfiguration. The ported software must be ableto run together with any new ORPGproduct/algorithm tasks.Only a single version of the RPG source codeneeds to be maintained, which means that theported code can still be compiled and run on theCONCURRENT OS132 operating system.The ported source code can be compiled by anyANSI FORTRAN compiler and run on anyPOSIX compliant OS environment withoutsubstantial modification.The ported software should run efficiently in termsof the use of hardware resource.The RPG uses data buffers, allocated in sharedmemory and managed by a buffer control module, forpassing radar data and products among tasks. TheORPG relies on similar data buffers, implemented bya data management tool called Linear Buffer (LB), toperform this task [Jain, et. al., 19971. Functionally theLB is used in the same way as the RPG data buffers areused, which makes the porting easier. One of thedifferences however is that the LB supports distributedapplications while the RPG data buffer mechanismdoes not. An LB is a self-controlled data store andcommunication object that does not require a controlmodule to support it. This characteristics improves therobustness of the system. Moreover, each LB stores asequence of products for a given product type. Thisprovides the potential of dynamically relocating tasksand buffers without loosing data. For more discussionson the Linear Buffer, refer to [Jain, et. al., 19971 .To meet these requirements, it was decided topreserve the current RPG task partition. That is eachcurrent RPG task will be individually ported andcompiled to generate a corresponding ORPG task. Theinputs and outputs of each ported task will be identicalto those of their RPG version.Shared data, such as adaptation data, scansummary information, and product generation controlinformation, are stored in shared common blocks in theRPG. In the ORPG, those are stored in correspondingLBs and can be accessed by all ORPG tasks. For theported tasks, each of them will have their own commonblocks allocated. A mechanism can be built toautomatically update the local common blocks byThe RPG tasks are controlled by a central controlmodule. In the distributed ORPG, however,product/algorithm tasks are running in a data-drivenenvironment [Jain, et. al., 19971. This will requirechanges in the task control part of the code. We thusmust write a new task control routine to replace the old319Authorized licensed use limited to: University of Oklahoma Libraries. Downloaded on June 12, 2009 at 16:11 from IEEE Xplore. Restrictions apply.
IMPLICIT NONEdata from the LBs. This mechanism can bent to the RPG tasks, requiring no modificationto the ported code. INCLUDE rpg port.inc,**rpg portC adaptation common blocks INCLUDE a309adpt.inc,**A3CD7OCA;SITEADP INCLUDE a309adpt.incr**A3CD70C2;CP15ALGRPG tasks also use shared common blocks forpassing data between tasks. These common blocks areidentified as Inter-Task Common blocks (ITCs). Thiskind of interprocess communication must also beby explicit message passing. Additional LBsed for implementing the ITCs. Messagepassing via these LBs, when activated by productgeneration or data flow events, can be automaticallyprocessed without requiring any change to the RPGother cases, function calls may be inserted intocode to activate ITC reading and writing. INCLUDE A309.INC/G,**A3PMOO INCLUDE itc.inc,**A315TRNDinteger paramvalue from A31519 call; * * specify inputs and outputscall input-data (CENTATTR,VOLUME-DATA)call input-data (TRFRCATR,VOLUME-DATA)call output-data (TRENDATR,VOLUME-DATA)call output-data (HAILATTR,VOLUME-DATA)A library of functions, emulating the current RPGAPI, will be developed to provide data flow andprocessing control support as well as automatic updateof local shared common blocks. By linking with thesefunctions, changes to the ported code can be minimizedand in most cases it can be eliminated. Most of thesupporting functions may be conveniently written in C.However, calling conventions between C andFORTRAN functions are non-standard. To maximizeportability, we must keep the set of necessaryconventions to its minimum.; * * register adaptation blockscall register-adpt (SITEADP,SITEADP-FIRST, BEGIN-VOLUME)call register-adpt (CPlSALG,CP15ALG-FIRST, BEGIN-VOLUME);register ITC inputscall itc-input (A315TRND,A315TRNDPFIRST, A315TRND-LAST (2.1,1TRFRCATR)In addition, CONCURRENT OS32 specificstem functions must be implemented to support thecall task-initialize(VOLUME-BASED)3. The Porting Procedure; * * The main loop, which willnever end10 call wait-for-activation( WAIT-ALL )callPorting an W G task involves the following steps:000;returnGet a copy of all RPG modules required forbuilding the task.Identify all input and output buffer types,adaptation blocks and other shared data blocks, allITCs and the data flow timing of the task.Write a new main function which containsstatements providing the information identified inthe previous step to the supporting modules,followed by a main processing loop.Modify remaining RPG modules if necessary.Compile the code using the preprocessor and thelocal FORTRAN compiler and generate theexecutable by linking the compiled object files andthe supporting libraries.A31519-BUFFER-CONTROL(param)got0 10stopendIn the above code listing we can see that some ofthe CONCURRENT FORTRAN features are used: Theinline comment and the INCLUDE directive. Notethat the RPG include files are used and the RPGfunctions are cnclude files are definethat this task has two inputs, CENTATTR andTRFRCATR, which are expected to be available on avolume basis. Calling subroutine input-data will causethe supporting modules to monitor these two inputAs an example, the following is the new mainfunction of the 'hail" task that implements the hailalgorithm:320Authorized licensed use limited to: University of Oklahoma Libraries. Downloaded on June 12, 2009 at 16:11 from IEEE Xplore. Restrictions apply.
products and activate this task for generating its outputswhen the inputs are ready. This task will generate twoproducts called TRENDATR and HAILA'ITR. Bothare volume based products. Because this task needs twoadaptation data blocks, SITEADP and CPl SALG,subroutine register-adpt is called to tell the supportingmodules that the two adaptation data common blocksneed to be automatically updated at the beginning ofevery volume. The hail algorithm needs data providedin a common block called A315TRND. Since the datain this common block are generated by another task, wemust define this common block as an ITC. By callingitc-input we tell the supporting modules that ITCA315TRND is used as an input and its contents need tobe updated when input TRFRCATR is read in. Theinitialization is then completed by calling"task-initialize", which also specifies that this taskprocesses data by volume.Following the initialization is the main processingloop. In this loop, we first call wait-for-activation topass program control to the supporting modules. Whenthis task needs to be activated for processing (theoutputs are scheduled and the inputs are available),wait-for-activation will return and we then callA3 1519-BUFFER-CONTROLto perform theprocessing tasks and generate the outputs. AfterA3 1519-BUFFER-CONTROL finishes, the controlis passed to the supporting modules again throughcalling wai-for-activation. This loop will last until thetask is terminated.The new main function, called hail.ftn, iscompiled, for example on an HP workstation, bycommandswhere I'rpgcm" is a library containing all RPG sharedfunctions (A3CMnn.FTN), "rpg" is the librarycontaining all RPG supporting modules, "lb" is the LBlibrary and "misc" is a library containing some generalORPG utility routines, which are invoked by "rpg" and"lb".4. The PreprocessorThe RPG FORTRAN code contains non-standardCONCURRENT extensions, which can not beprocessed by an ANSI FORTRAN compiler. Some ofthe extensions are useful for improving the codereadability and maintainability. For example the INCLUDE directive is very important in definingglobal constants and variables. We would like to keeptaking advantage of these features. Moreover, weprefer, if possible, to maintain only a single version ofthe RPG code, which can be compiled on both anORPG machine and the CONCURRENT machine.This will make the verification and future RPG versionupgrades easier. Maintaining the same look and feel ofthe FORTRAN RPG code is also desirable foralgorithm developers and maintainers. To reach thesegoals, a preprocessor has been developed forconverting the CONCURRENT FORTRAN code toANSI FORTRAN code at compile time.The preprocessor reads the CONCURRENTFORTRAN code and generates the ANSI FORTRANcode for the ORPG compiler. The ANSI FORTRANcode is discarded after the code is successfullycompiled. The preprocessor processes the followingnon-standard features:0ftnpp -DHPUX -I/users/jing/rpg/include hail.ftnfort77 -c -K U77 hai1.f0where "ftnpp" is the preprocessor, "fort77" is the HPFORTRAN compiler and "hai1.f" is the ANSIFORTRAN file generated by "ftnpp". For this task,there is no change needed for the remaining 9 portedRPG modules (a31509.ftn, a31519.ftn, etc.). Thesemodules are then compiled in a similar fashion. Andfinally the executable "hail" is created by linking theobject files and the supporting libraries:fort77 -0 hail -K U77 hai1.oa31509.0 a31519.0 a31529.0 a31539.0a31549.0 a31559.0 a31569.0 a31579.0a31599.0 -L/users/jing/lib/hpux-1rpgcm -1rpg -1lb -1misc -1m00 INCLUDE files.CONCURRENT specific directives such as INLINE, INSKIP and TCOM.The debugging print statement led by the "X"character.Non-standard hexadecimal constants of formatY'n' and Zn, where "n" is a hexadecimal constant.In-line comment proceeded by ";".The preprocessor resolves INCLUDE directivesrecursively and eliminates any duplicated inclusion.CONCURRENT specific directives such as INLINE, INSKIP and TCOM are simply ignored. The blockdata files are processed as include files because theblock data usage is not explicitly specified in the code,which can be a source of error. Other undocumentedfeatures that are not accepted by an ORPG compilermust also be processed.32 1Authorized licensed use limited to: University of Oklahoma Libraries. Downloaded on June 12, 2009 at 16:11 from IEEE Xplore. Restrictions apply.
identical MI. This is essentially an implementation ofthe existing RPG API in the OWG system.The preprocessor supports special directives formaking necessary changes in the ported code. Theyallow the modifications in the ported code to betransparent to the CONCURRENT compiler.OS/32 specific systemfunctions are emulated to support porting RPGproductlalgorithm tasks. The RPG API functions andthe OS/32 functions are parts of the supporting library.In certain cases an identifier needs to be replacedby a different one. For example, the RPG code may usean intrinsic function called “INT2” which is notsupported by an ORPG compiler. We may then replaceit by “INT.The preprocessor supports identifierreplacement. Any preprocessor features that involvecode changes, however, are used with caution becausethey can introduce inconsistencies between the portedand the original versions. They are only used when theyare absolutely necessary.The following is a brief description of thefunctions in the supporting library.0Inputloutput data type registrationThe input and output data typregistered in the supporting moduis used in processinontrol and input datasynchronization. The 0uses a data driven modelfor product/algorithm task control. Each task isactivated by a unique input called the driving input.Other inputs are used in synchronization with thedriving input. The driving input is processedsequentially. If a discontinuity in the driving input isdetected, a processing abort may result.The preprocessor also supports directives thatallow compile time code segment selection similar to#indef . #else . #endif in the C preprocessor. Thisincreases the portability of the source code.5. The Supporting LibraryA ported RPG task must be able to run in theORPG environment. Specificallyany interaction withthe RPG monitor and control module and the buffercontrol module must be replaced by an interaction withthe ORPG infrastructure. Note that in the ORPG,which uses an IN”ERNET/WEB model; a task can andmust work without direct interaction with other ORPGtasks. For example, if a task needs adaptation data, itmust explicitly read it from the LB storing these data.For a ported RPG productlalgorithm task, it must readin the necessary adaptation data, the product generationcontrol information and other system configuration andcontrol information. It must also find the availabilityinformation of the products it wants to use as inputs.When the inputs are available, it must then read them infor processing. It must determine what to do whenexceptional situations occur. For example, it must abortthe unfinished processing if the volume scan isrestarted or the task generating its input fails.In ORPG, all RPG products as well asintermediate products are treated as ORPG products.They are identified by their RPG buffer type number.Each ORPG product is stored in an individual LB. Theproduct time is used as the message ID in the LB forconvenient product retrieval. Each ORPG product hasan ORPG product header for additional control anddata information.0Adaptation data and the scan summary tableThe RPG adaptation data is organized into severalcommon blocks, e.g. COLRTBL, RDACNT,CP13ALG and so on. The ORPG uses an LB todistribute the RPG adaptation data to tasks. The RPGscan summary table is implemented via an ITC forsupporting ported RPG tasks. Theadaptation data blocksautomatically update sand the scan summaryr the ported tasks.Fortunately all of these ORPG “burdens”, whichare tradeoffs for distributed processing, geaterflexibility and expand ability, as well as fault tolerance,can be performed automatically and hidden inrting library functions.0Task initializationenvironment. This function also establishes the tasktype. The concepttype is used in processing, if a task is ofcontrol. For eVOLUME-BASED type, it can not start processingdata in the middle of a volume scan after task start upor a processing abort.To ease the porting effort and minimize changes tothe RPG code, most of the functions, which the RPGuse to interact with the RPGmodule and the bu fer controlin the ORPG environment with322Authorized licensed use limited to: University of Oklahoma Libraries. Downloaded on June 12, 2009 at 16:11 from IEEE Xplore. Restrictions apply.
0ITC LB. In cases the scheduled automaticProduct generation controlread-idwrite-out is not sufficient, one can customizeITC readindwriting by inserting the itc-readlitc-writecalls into the RPG code.The ORPG uses a product request LB to storeproduct generation control information. For eachproduct, a message containing a list of requests may bestored in the LB. Product request messages areidentified by the product ID numbers. A request mayspecify an elevation at which the product is requested.It may also specify a user defined window if theproduct is window based. All products' generation canbe controlled through requests and each product iscontrolled individually.0A product/algorithm processing procedure has tobe aborted in several circumstances, which includeelevatiordvolume scan restart, load shed and taskfailure. When the supporting modules detect an abortsituation, get-inbuf will return with STATUS set toTERMINATE. This STATUS return will cause thetask to terminate the current unfinished processingprocedure, free allocated resources, clean up and set upthe next processing resumption time.Subroutine wait-for-activation is designed forprocessing scheduling. wait-for-activation suspendsthe processing and keeps track of the input data flowand the user product requests. If an output is requestedor scheduled to be generated and the required inputdata are ready, wait-for-activation will return and theproductlalgorithm task's main processing routine willthen be activated. The processing routine reads theinput by calling get-inbuf and processes the data untilall output products are generated.00OS32 FORTRAN extension and system functionssupportOS32 FORTRAN extension and OS32 system callfunctions must be emulated. Because the number ofthese functions is quite large and some of them may bedifficult to emulate in a different OS, the approach wetook is to implement those that are found to benecessary. Examples of supported functions include:Buffer control supportRPG tasks rely on buffer control functions to getinputs, generate outputs and allocate scratch workingareas. The RPG buffer control' functions,A3 1212 REL INBUF,A3 121LGET-INBUF,A31215 -GET OUTBUFandA3 1216-REL-OUTBUFare supported in thesupporting library. These functions emulate theiroriginal buffer control functions with enhanced abortprocessing support.0RPG control and monitor function supportbtest, bclr, bset, ilbyte and isbytelokon and lokoffdate, iclock and waitsndmsg and queuedeflst, atl, abl, rtl, rbl and lstfun6. SummaryA direct porting approach to migrate the legacyWSR-88D algorithm and product generators to theOpen Systems W G is described. The approachimplements a "true" porting of the RPG FORTRANcode to the Open Systems environment in the sense thatmost of the code is simply copied over, compiled andready to run. Only a single version of the RPG code,which works on both the current legacy hardware andthe distributed open system environment, needs to bemaintained. Since the basic processing routines are notmodified in the porting, a full scale verification of theported code is not necessary. The portedprodudalgorithm tasks offer full advantage of thedistributed processing and are able to run together withnew ORPG product generators and meteorologicalalgorithms.Inter-Task Common block (ITC) supportThe RPG tasks use common blocks in globalshared memory, called ITCs, to exchange data amongtasks. In ORPG, I T data are exchanged through LBs.When data in an ITC are ready, they are written out asa message to an LB and they are then read by othertasks that need the data.Each ITC is assigned a uniqueITC ID number, which is used for identifying the LBthat stores the message and the particular message inthe LB. Multiple ITCs can be implemented with asingle LB.Functions itc-input and itc-output are used forinforming the supporting modules that certain ITCs areused by the task and they need to be automaticallyupdated or written out at scheduled times. Functionsitc-read and itc-write provide explicit accesses to the323Authorized licensed use limited to: University of Oklahoma Libraries. Downloaded on June 12, 2009 at 16:11 from IEEE Xplore. Restrictions apply.
3.ntsauthors would like to acknowkdge the Officeof Systems Development of the National WeatherService for funding this project.8. Referencespapers incIuded ia this49"' IEEE NAECON,324Authorized licensed use limited to: University of Oklahoma Libraries. Downloaded on June 12, 2009 at 16:11 from IEEE Xplore. Restrictions apply.
MIGRATION OF LEGACY WSR-88D ALGORITHMS AND PRODUCT GENERATORS TO THE OPEN SYSTEM Zhongqi Jingls, Steve Smith', Michael Jain3 and Allen Zahrai3 . developed to provide data flow and processing control support. All inter-process data exchanges are implemented through message passing in the supporting functions. "Wrapped" by the new .
