susChkSrv.py \- Provider for SAP HANA srHook method srServiceStateChanged().

.PP

susChkSrv.py can be used to provide a script for the SAP HANA srHook method

srServiceStateChanged().

.PP

The SAP HANA nameserver provides a Python-based API ("HA/DR providers"), which

is called at important points of the host auto-failover and system replication

takeover processes. These so called hooks can be used for arbitrary operations

that need to be executed. The method srServiceStateChanged() is called when

HANA processes are failing, starting or stopping.

.PP

Purpose of susChkSrv.py is to detect failing HANA indexserver processes and

trigger a fast takeover to the secondary site. With regular configuration of an

HANA database, the resource agent (RA) for HANA in a Linux cluster does not

trigger a takeover to the secondary site when:

.br

- A software failure causes one or more HANA processes to be restarted in place

by the HANA daemon (hdbdaemon).

.br

- A hardware error (e.g. SIGBUS from an uncorrectable memory error) causes the

indexserver to restart locally.

.br

See also SAPHanaSR(7) or SAPHanaSR-ScaleOut(7).

.PP

The hook script susChkSrv.py is called on any srServiceStateChanged() event.

The script checks for

'isIndexserver and serviceRestart and serviceWasActiveBefore and hostActive and databaseActive'.

If it finds the correct entries, it executes the predefined action. As soon as

the HANA landscapeHostConfiguration status changes to 1, the Linux cluster will

take action. The action depends on HANA system replication status and the RA´s

configuration parameters PREFER_SITE_TAKEOVER, AUTOMATED_REGISTER and ON_FAIL_ACTION. See manual

page ocf_suse_SAPHana(7) or ocf_suse_SAPHanaController(7).

.PP

Customising of HANA daemon timeout parameters might be needed for adapting the

solution to a given environment. Please refer to SAP HANA documentation.

.PP

This hook script needs to be installed, configured and activated on all HANA

nodes.

.PP

* The "HA/DR providers" API accepts the following parameters for the

ha_dr_provider_suschksrv section in global.ini:

.TP

.TP

Mandatory. Must not be changed.

.TP

Mandatory. Delivered within RPM package. Please change only if requested.

.TP

Mandatory. Order might depend on other hook scripts.

.TP

Action to be processed when a lost indexserver is identified.

.br

- \fBignore\fP: do nothing, just write to tracefiles.

.br

- \fBstop\fP: do 'sapcontrol ... StopSystem'.

If this is combined with SAPHanaController RA parameter 'AUTOMATED_REGISTER=true',

HANA needs to release all OS resources prior to the automated registering. See

also manual page ocf_suse_SAPHanaController(7).

.br

- \fBkill\fP: do 'HDB kill-<\fIsignal\fR>'. The signal can be defined by parameter 'kill_signal'.

This action is primarily meant for scale-up, and scale-out with host auto-failover

(aka BW).

.br

On scale-out without host auto-failover (aka ERP), it depends on runtime effects,

whether HANA will try a restart.

.br

If this is combined with SAPHanaController RA parameter 'AUTOMATED_REGISTER=true', HANA

needs to release all OS resources prior to the automated registering.

.br

- \fBfence\fP: do 'crm node fence <\fIhost\fR>'. This needs a Linux cluster

STONITH method and sudo permission.

For scale-out, SAPHanaSR-alert-fencing needs to be configured additionally, see

manual page SAPHanaSR-alert-fencing(8) for details.

.br

Optional. Default is ignore.

.TP

Signal to be used with 'HDB kill-<\fIsignal\fR>'.

.br

Optional. Default is 9.

.TP

How many seconds to wait for 'sapcontrol ... StopSystem' to return.

Should be greater than value of HANA parameter 'forcedterminationtimeout'.

See also SAPHanaSR_basic_cluster(7).

.br

Optional. Default is 20 seconds.

.TP

.TP

.TP

Optional. Default is info. Will be added automatically if not set.

.PP

* The HANA daemon TODO for the daemon section of daemon.ini:

.PP

.TP

.br

See also SAPHanaSR_basic_cluster(7).

Optional. Timeout in milliseconds. Default is 30000.

.TP

.br

See also SAPHanaSR_basic_cluster(7).

Optional. Timeout in milliseconds. Default is 270000.

.PP

* The HANA daemon TODO for the indexserver.<tenant> section of daemon.ini:

.TP

.TP

TODO Should be 6000.

.br

Optional. Timeout in milliseconds. Default is 2000.

.PP

.B 0

Successful program execution.

.br

.B >0

Usage, syntax or execution errors.

.PP

\fB*\fP Example for minimal entry in SAP HANA scale-up global configuration

/hana/shared/$SID/global/hdb/custom/config/global.ini

.PP

In case of a failing indexserver, the event is logged. No action is performed.

The section ha_dr_provider_suschksrv is needed on all HANA nodes.

The HANA has to be stopped before the file can be changed.

.PP

.RS 2

[ha_dr_provider_suschksrv]

.br

provider = susChkSrv

.br

path = /usr/share/SAPHanaSR-angi/

.br

execution_order = 3

.RE

.PP

\fB*\fP Example for entry in SAP HANA scale-up global configuration

/hana/shared/HA1/global/hdb/custom/config/global.ini

.PP

In case of a failing indexserver, the complete node will get fenced by

calling 'crm node fence ...'.

Unlike fence actions from inside the Linux HA cluster, this hook script fence

action will be issued even when the cluster is in maintenance and stonith is

temporarily disabled. The fence request then is queued until the cluster is set

back into active state. See example below for removing queued fence actions.

.br

The section ha_dr_provider_suschksrv is needed on all HANA nodes.

The HANA has to be stopped before the file can be changed.

Alterntively use SAPHanaSR-manageProvider for applying an HA/DR provider hook

configuration, see manual page SAPHanaSR-manageProvider(8).

.PP

.RS 2

[ha_dr_provider_suschksrv]

.br

provider = susChkSrv

.br

path = /usr/share/SAPHanaSR-angi/

.br

execution_order = 2

.br

action_on_lost = fence

.RE

.PP

\fB*\fP Example for entry in SAP HANA scale-out global configuration

/hana/shared/HA1/global/hdb/custom/config/global.ini

.PP

In case of a failing indexserver, the HANA instance will be stopped with

'sacpcontrol ... StopSystem'. HANA timeouts might be adapted to speed up the

stop.

If this is combined with SAPHanaController parameter 'AUTOMATED_REGISTER=true',

HANA needs to release all OS resources prior to the automated registering.

.br

The hook script should wait maximum 25 seconds on the sapcontrol command to

return.

.br

The section ha_dr_provider_suschksrv is needed on all HANA nodes.

The HANA has to be stopped before the file can be changed.

.br

Note: HANA scale-out is supported only with exactly one master nameserver.

No HANA host auto-failover.

.PP

.RS 2

[ha_dr_provider_suschksrv]

.br

provider = susChkSrv

.br

path = /usr/share/SAPHanaSR-angi/

.br

execution_order = 2

.br

action_on_lost = stop

.br

stop_timeout = 25

.RE

.PP

\fB*\fP Example for entry in SAP HANA daemon configuration

/hana/shared/HA1/global/hdb/custom/config/daemon.ini

.PP

TODO

Example SID is HA1, tenant is HA1.

.br

The sections daemon and indexserver.HA1 are needed on all HANA nodes.

The HANA has to be stopped before the file can be changed.

Please refer to SAP documentation before setting this parameters.

.PP

.RS 2

[daemon]

.br

terminationtimeout = 45000

.br

forcedterminationtimeout = 15000

.PP

[indexserver.HA1]

.br

gracetime = 6000

.RE

.PP

\fB*\fP Example for sudo permissions in /etc/sudoers.d/SAPHanaSR .

.PP

SID is HA1. See also manual page SAPHanaSR-hookHelper(8).

.PP

.RS 2

# SAPHanaSR needs for susChkSrv

.br

ha1adm ALL=(ALL) NOPASSWD: /usr/bin/SAPHanaSR-hookHelper --sid=HA1 --case=fenceMe

.RE

.PP

\fB*\fP Example for looking up the sudo permission for the hook script.

.PP

All related files (/etc/sudoers and /etc/sudoers.d/*) are scanned.

Example SID is HA1.

.PP

.RS 2

# sudo -U ha1adm -l | grep "NOPASSWD.*/usr/bin/SAPHanaSR-hookHelper"

.RE

.PP

\fB*\fP Example for checking the HANA tracefiles for srServiceStateChanged() events.

.PP

Example SID is HA1. To be executed on the respective HANA master nameserver.

.br

If the HANA nameserver process is killed, in some cases hook script actions do

not make it into the nameserver tracefile. In such cases the hook script´s own

tracefile might help, see respective example.

.PP

.RS 2

# su - ha1adm

.br

~> cdtrace

.br

~> grep susChkSrv.*srServiceStateChanged nameserver_*.trc

.br

~> grep -C2 Executed.*StopSystem nameserver_*.trc

.RE

.PP

\fB*\fP Example for checking the HANA tracefiles for when the hook script has been loaded.

.PP

Example SID is HA1. To be executed on both sites' master nameservers.

.PP

.RS 2

# su - ha1adm

.br

~> cdtrace

.br

~> grep HADR.*load.*susChkSrv nameserver_*.trc

.br

~> grep susChkSrv.init nameserver_*.trc

.RE

.PP

\fB*\fP Example for checking the hook script tracefile for actions.

.PP

Example SID is HA1. To be executed on all nodes. All incidents are logged on

the node where it happens.

.PP

.RS 2

# su - ha1adm

.br

~> cdtrace

.br

~> egrep '(LOST:|STOP:|START:|DOWN:|init|load|fail)' nameserver_suschksrv.trc

.RE

.PP

\fB*\fP Example for checking the hook script tracefile for node fence actions.

.PP

Example SID is HA1. To be executed on both sites' master nameservers. See also

manual page SAPHanaSR-hookHelper(8).

.PP

.RS 2

# su - ha1adm

.br

~> cdtrace

.br

~> grep fence.node nameserver_suschksrv.trc

.RE

.PP

\fB*\fP Example for revoking a queued fence request from the Linux cluster.

.PP

This could be done if an HANA indexserver failure has triggerd an node fence

action while the Linux cluster is in maintenance. Before revoking a fence request,

be sure it has been issued by the HA/DR provider hook script. See example above

for checking the hook script tracefile for node fence actions.

Example node is node2. To be executed on that node.

See also manual pages SAPHanaSR-hookHelper(8) and crm_attribute(8).

.br

Note: This removes the node attribute terminate=true from the Linux cluster CIB.

It does not touch any fencing device.

.PP

.RS 2

# grep fenced:.termination.was.requested /var/log/pacemaker/pacemaker.log

.br

# crm_attribute -t status -N 'node2' -D -n terminate

.br

# crm_attribute -t status -N 'node2' -G -n terminate

.RE

.PP

\fB*\fR Example for killing HANA hdbindexserver process.

.PP

This could be done for testing the HA/DR provider hook script integration.

Killing HANA processes is dangerous. This test should not be done

on production systems.

Please refer to SAP HANA documentation. See also manual page killall(1).

.br

Note: Understand the impact before trying.

.PP

1. Check HANA and Linux cluster for clean idle state.

.PP

2. On secondary master name server, kill the hdbindexserver process.

.RS 2

# killall -9 hdbindexserver

.RE

.PP

3. Check the nameserver tracefile for srServiceStateChanged() events.

.PP

4. Check HANA and Linux cluster for clean idle state.

.RE

.PP

.TP

the hook provider, delivered with the RPM

.TP

the external script for node fencing

.TP

the sudo permissions configuration

.TP

the on-disk representation of HANA global system configuration

.TP

the on-disk representation of HANA daemon configuration

.TP

path to HANA tracefiles

.TP

HADR provider hook script tracefile

.PP

1. SAP HANA 2.0 SPS05 or later provides the HA/DR provider hook method

srServiceStateChanged() with needed parameters.

.PP

2. No other HADR provider hook script should be configured for the

srServiceStateChanged() method. Hook scripts for other methods, provided in

SAPHanaSR and SAPHanaSR-ScaleOut, can be used in parallel to susChkSrv.py, if

not documented contradictingly.

.PP

3. The user ${sid}adm needs execution permission as user root for the command

SAPHanaSR-hookHelper.

.PP

4. The hook provider needs to be added to the HANA global configuration, in

memory and on disk (in persistence).

.PP

5. HANA daemon timeout TODO

.PP

6. The hook script runs in the HANA nameserver. It runs on the node where the event

srServiceStateChanged() occurs.

.PP

7. HANA scale-out is supported only with exactly one master nameserver. HANA

host auto-failover is not supported. Thus no standby nodes.

.PP

8. A Linux cluster STONITH method for all nodes is needed, particularly if

susChkSrv.py parameter 'action_on_lost=fence' is set.

.PP

9. If susChkSrv.py parameter 'action_on_lost=stop' is set and the RA SAPHana or

SAPHanaController parameter 'AUTOMATED_REGISTER=true' is set, it depends on HANA

to release all OS resources prior to the registering attempt.

.PP

10. For HANA scale-out, the susChkSrv.py parameter 'action_on_lost=fence' should

be used only, if the SAPHanaSR-alert-fencing is configured.

.PP

11. If the hook provider should be pre-compiled, the particular Python version

that comes with SAP HANA has to be used.

In case of any problem, please use your favourite SAP support process to open

a request for the component BC-OP-LNX-SUSE.

Please report any other feedback and suggestions to [email protected].

.PP

\fBSAPHanaSR\fP(7) , \fBSAPHanaSR-ScaleOut\fP(7) , \fBSAPHanaSR.py\fP(7) ,

\fBocf_suse_SAPHanaTopology\fP(7) , \fBocf_suse_SAPHanaController\fP(7) ,

\fBSAPHanaSR-hookHelper\fP(8) , \fBSAPHanaSR-manageProvider\fP(8) ,

\fBSAPHanaSR-alert-fencing\fP(8) ,

\fBcrm\fP(8) , \fBcrm_attribute\fP(8) ,

\fBpython3\fP(8) , \fBkillall\fP(1) ,

.br

https://help.sap.com/docs/SAP_HANA_PLATFORM?locale=en-US

.br

https://help.sap.com/docs/SAP_HANA_PLATFORM/42668af650f84f9384a3337bcd373692/e2064c4aa47f443ab6a107f9ab7f5edd.html?version=2.0.01

.br

https://help.sap.com/docs/SAP_HANA_PLATFORM/6b94445c94ae495c83a19646e7c3fd56/5df2e766549a405e95de4c5d7f2efc2d.html?locale=en-US

.br

SAP note 2177064

.PP

A.Briel, F.Herschel, L.Pinne.

.PP

(c) 2022-2026 SUSE LLC

.br

susChkSrv.py comes with ABSOLUTELY NO WARRANTY.

.br

For details see the GNU General Public License at

http://www.gnu.org/licenses/gpl.html