uCosminexus Stream Data Platform - Application Framework Setup and Operation Guide

[Contents][Index][Back][Next]

2.2.2 Components of a stream data processing system

This subsection describes the components of a stream data processing system. It also discusses the mode of connection between adaptors and the SDP server and the number of components that can be placed in the stream data processing system.

Organization of this subsection
(1) Components of a stream data processing system
(2) Connection mode between adaptors and SDP server
(3) Number of components that can be placed in a stream data processing system

(1) Components of a stream data processing system

The following figure shows the components of a stream data processing system.

Figure 2-3 Components of a stream data processing system

[Figure]

The following subsections describe the components shown in the figure:

You use definition files to specify the operation of the adaptors and the SDP server. Files, such as log and trace files, are output by the adaptors and the SDP server. The following figure shows the relationships between the components of a stream data processing system and individual files.

Figure 2-4 Relationships between the components of a stream data processing system and individual files

[Figure]

The following describes the files shown in the figure:

(2) Connection mode between adaptors and SDP server

The adaptors and the SDP server operate either all in the same process or in separate processes. The mode of connection that is used when you operate the adaptors and the SDP server in the same process is called an in-process connection, and the mode of connection that is used when you operate them in separate processes is called an RMI connection.

The table below describes the advantages and disadvantages of these two connection modes.

Table 2-1 Advantages and disadvantages of the adaptor and SDP server connection modes

No. Connection mode Advantage Disadvantage
1 In-process connection Cost of communication between SDP server and adaptors is low and the tuple processing latency can be minimized. Memory usage must be monitored because all processing is performed in a single process.
2 RMI connection The effects of an adaptor failure on the stream data processing engine can be minimized. Also, more memory can be allocated to adaptors. The communication cost is higher than with in-process connection because process-to-process communication must be performed.

You must determine how to configure the SDP server and adaptors, taking into account the advantages and disadvantages of the connection modes.

The system configuration in each connection mode depends on whether standard adaptors or custom adaptors are used. For details about these types of system configurations, see 2.2.3 Configuration when standard adaptors are used and 2.2.4 Configuration when custom adaptors are used.

(3) Number of components that can be placed in a stream data processing system

There are limitations to the number of adaptors that can be started by an SDP server and the number of adaptors that can be managed in an adaptor group. The number of components that can be placed in a stream data processing system depends on whether standard adaptors or custom adaptors are used. The table below shows the number of components that can be placed in a stream data processing system.

Table 2-2 Number of components that can be placed in a stream data processing system

No. Item Maximum value when standard adaptors are used Maximum value when custom adaptors are used
1 Number of SDP servers that can be run in one working directory#1 1 1
2 Number of RMI connection adaptors that can be run by one SDP server -- 32#2
3 Number of in-process-connection adaptors that can be registered in one SDP server -- 16#2
4 Number of adaptor groups that can be executed by one SDP server 1 --
5 Number of adaptors that can be managed in one adaptor group 64#2 --
6 Number of query groups that can be executed by one SDP server 8 8
7 Number of streams that can be registered in one SDP server 1,024#3 in total 1,024#3 in total
8 Number of queries that can be registered in one SDP server

Legend:
--: Not applicable

#1
The working directory is a directory used for running the stream data processing system. For details about the configuration of the working directory, see 2.6.2 Evaluating the configuration of the working directories for executing query groups.

#2
This maximum value is a guideline. You should determine the number of adaptors on the basis of the performance of the server on which Stream Data Platform - AF is to be run and performance in the stream data processing system.

#3
Set the numbers of streams and queries so that their total number combined does not exceed 1,024.