SSCCIP – a technology of high-performance remote sensing data processing on multicomputer

RUSIN E.V.

Content:

1. Introduction

The involvement of high-performance computers to the solution of time-consuming tasks of remote sensing data processing presently leads to the problem of supercomputer "hostility" to user. Typically, multicomputer works under UNIX-like operating system and acts as a computational server interacting with its clients via secured SSH channel; but widespread SSH access software (like PuTTy or SSH Secure Shell) provide user only with text console service. Thus, using multicomputers turns out to be available for those users only who are experienced in UNIX systems and unavailable to average users of MS Windows family systems which are installed on absolute majority of Russian PCs. And even after this, data processing on multicomputer is carried out in several steps: uploading data to multicomputer with SFTP client (SFTP – Secure FTP – secured protocol for data transfer over SSH), launching processing command on multicomputer with SSH terminal, and downloading results from multicomputer with same SFTP client.

Thus, for mass involvement of high-performance computers to the solution of time-consuming image processing tasks, it is necessary to create not only software implementing wide range of image processing methods and algorithms on multicomputers, but also technologies simplifying complicated for average user process of using remote multiprocessor UNIX computer. In the paper we consider an attempt to solve this problem, the experimental SSCCIP software.

2. Requirements

We formulated the following requirements for SSCCIP software:

The latter requirement is especially important because the ease of extension and customization of such systems for the needs of concrete users are the necessary conditions of their further development.

3. Composition

SSCCIP computing system consists of three main parts:

3.1. Client-Server Interaction

The vast majority of multicomputer servers accept only secured SSH client connections. What provides such a connection in SSCCIP system is:

On the server side, system SSH daemon supporting applied SSH-based protocols SFTP (for files transfer) and rexec (for remote execution of commands/programs on the server).

On the client side, CryptLib library by Peter Gutmann from Auckland University, New Zeeland. On the CryptLib basis, we implemented SFTP and rexec applied protocols.

On the higher level, the client component interacts with the server by the following scheme:

3.2. Client Component

Client component is a Windows application SSCCIP_Client implemented in C++. Its source code is organized as a framework providing the ease of introducing new processing operations: code performing the actions which are common for all processing operations (uploading source data and task files to server, launching and tracking task, downloading results from server), forms executeParallelTask() generic subprogram; to code a concrete processing operation, one need to implement only the actions which are specific for the operation: interactive configuration (source and result files selection, processing algorithm parameterization) and visualization of the results. For this aim framework declares IParallelTask interface with two abstract methods, Configure() and Visualize(), which must be implemented by any concrete processing class. Configure() method must return the content of task file, the list of files to upload to the server before calculations, and the list of files to download from the server after calculations. With Visialize()method, which is called by framework after downloading results from the server, a concrete processing operation performs result visualization. To execute a concrete processing operation, one must parameterize executeParallelTask() generic operation with this concrete operation class.

Also, the framework contains the tools for loading and visualizing images in various graphic formats.

3.3. Server Component

Server component is a UNIX application SSCCIP_Server implemented in C++ with the use of SSCC_PIPL library for high-performance image processing on multicomputer which was created by authors earlier (reference to the library!!!). SSCCIP_Server is MPI-based application, executing on each multiprocessor node by mpirun task launcher. SSCCIP_Server source code is also organized as framework. Particularly, generic subprograms was created for some important types of processing procedures like “pixel-to-pixel” and “neighborhood-to-pixel”, implementing the actions which are common for all the procedures (parsing task file, loading source image with the support of various graphic formats, determining parallelization parameters, parallelizing itself, storing result image in graphic file). To implement a concrete processing operation, one must create C++ algorithm class whose interface is compatible with SSCC_PIPL library.

4. Example of Concrete Technology Implementation

Practical testing of SSCCIP framework was carried out via implementation of technology of circle structures detection in space images.

All remaining operations (uploading source image and task file to multicomputer, enqueuing computational task and tracking its status, parsing task file, loading source image and storing result image, parallelizing calculations, and downloading result to the client computer) was implemented in SSCCIP framework code. As one can see, creating new high-performance technology did not require from programmer to deal with parallelism or network communications.

5. Conclusion

The main result of the work on SSCCIP software development is the creation of the framework for high-performance remote sensing data processing on remote multicomputer. The framework allows solving a various range of remote sensing data processing and analysis problems and a developer of concrete high-performance technology within the framework must implement only details specific for the algorithm to be implemented.