Manual Recovery Guide for SAP HANA on Azure VMs from Azure NetApp Files snapshot with AzAcSnap

Published Jun 14 2022 08:00 AM 1,645 Views

Table of Contents

Version

Authors

Overview

Assumptions

Terms and Definitions

System status

Recover the database to its most recent state

Recover the database to the following point in time

Recover the database to a specific data (snapshot) backup

Appendix – SAP HANA Data Volume locations

 

Version

This document is for the SAP HANA on Azure NetApp Files using the Microsoft AzAcSnap version 5.0 or later.

 

Authors

Phil Jensen, Principal Software Engineer at Microsoft.

 

Overview

This document provides guidance on using SAP HANA Studio to recover SAP HANA on Azure NetApp Files. This guide has step-by-step screenshots to follow to understand the three primary methods of recovering SAP HANA using HANA Studio from a snapshot taken using the Microsoft provided AzAcSnap tool.

 

The screenshots in this document are from SAP HANA Studio session accessing SAP HANA 2.0SPS04.

 

Disclaimer: This guide and the associated screenshots are taken from an SAP HANA v2.0 system recovery as set up in the Microsoft test environment for SAP HANA on an Azure Virtual Machine with Azure NetApp Files storage. Anyone following this guide is responsible for ensuring the recovery process works in their own environment as expected.

 

Assumptions

The administrator following this guide has experience with SAP HANA and HANA Studio because not all details are provided as screenshots to follow (e.g. logging in to HANA Studio, etc.).

 

The administrator is familiar with SAP HANA backup processes, including the Backup Catalog and Storage Snapshots.

 

The administrator has the appropriate permissions at a Linux shell to copy files as the <sid>adm user into the SAP HANA Data Area.

 

Terms and Definitions

Terms used in this documentation: 

  • SID: A System Identifier for SAP HANA installation, typically 3 characters long. 
  • ANF: Azure NetApp Files.
  • VM: Virtual Machine.

 

System status

The system layout used for this documentation has a “primary” SID (PR1) and another second tenant (PR2). 

The second tenant (PR2) was created using the SQL commands:

 

  • CREATE DATABASE PR2 SYSTEM USER PASSWORD <SomePassword>

The primary data area is under “/hana/data/PR1/mnt00001”.  Further explanation of the SAP HANA persistent data storage area is in the Appendix.

 

Recover the database to its most recent state

In this case the goal is to restore the complete system (SYSTEMDB, PR1, PR2) from a snapshot to the most recent database state, including any log replay.

 

  1. First step is to stop the database

    GeertVanTeylingen_0-1652793226731.png

     

    GeertVanTeylingen_1-1652793251649.png

     

    You may need to provide the <sid>adm user and password to shut-down the database.
    GeertVanTeylingen_2-1652793272542.png

    When this is finished, the Processes tab should display as follows:
    GeertVanTeylingen_3-1652793291511.png

 

  1. Start the recovery process from the menu.

    GeertVanTeylingen_5-1650967057706.png

    Note, the recovery wizard can take several seconds to launch (see the following status):

    GeertVanTeylingen_6-1650967057707.png

 

  1. Choose the recovery type, in this case “Recover the database to its most recent state”

    GeertVanTeylingen_7-1650967057712.png

 

  1. Choose the location of the backup catalog, which is needed for recovery.

    GeertVanTeylingen_90-1650968009273.png

 

  1. The backup catalog will be fetched to display the appropriate backup to recover from (this can take a minute or two to load

    GeertVanTeylingen_9-1650967057725.png

 

  1. The first time the backup catalog is refreshed, its likely no suitable snapshot will be found to restore from.  This is because the administrator will need to copy/restore the files from the snapshot into the data area.

    GeertVanTeylingen_10-1650967057736.png

 

  1. Restore the snapshot files for recovery using either option (a) Revert Volume from Snapshot or option (b) Copy files.  Option (a) is faster.
    1. Revert Volume from Snapshot – this will roll a Volume back to a prior state and any data or snapshots added to the Volume after the Snapshot used to revert to are lost.  At this time Reverting a Volume can only be done using the Azure Portal.

      ! Make sure SAP HANA is not running before reverting any volumes.

      If dealing with multiple Volumes make a note of the snapshot name you are reverting to as they all must have the same snapshot name.
      GeertVanTeylingen_0-1652791885967.pngGeertVanTeylingen_1-1652791907342.png

       

      After completing the Volume Revert it’s a good idea to remount the volumes:
      # mount -o remount /hana/data/PR1/
    2. Copy files - In this example, the files are copied from the “hidden” snapshot location in the filesystem.
      # su - pr1adm
      > cp -pr /hana/data/PR1/mnt00001/.snapshot/PR1_hourly__2022-03-31T220003-7196432Z/* /hana/data/PR1/mnt00001/.
    3. Single file snapshot restore
      Alternatively Azure NetApp Files single file snapshot restore feature can be used to restore files without the need to copy any of them across the network.

 

  1. When the copy is complete, refresh the view of the backup catalog to ensure the snapshot we are restoring from is listed.

    GeertVanTeylingen_13-1650967057750.png

 

  1. Now select the available SNAPSHOT shown in green to recover from.

    GeertVanTeylingen_3-1652792119286.png

 

  1. Choose the location of the Log Backups.

    GeertVanTeylingen_5-1652793489158.png 

  2. Check any appropriate “Other Settings”, the following screen is the defaults

    GeertVanTeylingen_5-1652792236373.png

 

  1. On the summary page, review any final details and press Finish to restore the system database.

     

    GeertVanTeylingen_6-1652793580895.pngGeertVanTeylingen_7-1652793604753.pngGeertVanTeylingen_8-1652792317554.png

     

  1. While the recovery is in progress, the following screen is displayed:

     

    GeertVanTeylingen_9-1652792351131.png

     

  1. When the recovery has finished a Recovery Execution Summary provides details of the recovery.  The following screen shows a completed recovery of the SYSTEMDB.

    GeertVanTeylingen_10-1652792415237.png

     

    ! Note the message stating “recovering the system database from a storage snapshot invalidates all the tenant databases”.  Tenant databases must now be recovered. 

 

  1. Start the recovery of the Tenant database

     

    GeertVanTeylingen_12-1652792453342.png

 

  1. Choose the Tenant to recover from.

     

    GeertVanTeylingen_13-1652792491311.png

 

  1. Choose to recover the tenant database to its most recent state (same as for the system database).

     

    GeertVanTeylingen_14-1652792596462.png

 

  1. Provide the location of the Backup Catalog (same as for the system database)

    GeertVanTeylingen_23-1650967057805.png

     

  1. Allow the tenant database to be stopped for recovery.

    GeertVanTeylingen_8-1652793655198.png

     


  2. Wait for the Backup Catalog to be refreshed and displayed
  1. When recovering the tenant database there should already be a valid snapshot to recover from (unlike the system database where we needed to restore the snapshot files into the data area and refresh the view).  Select this snapshot and click next.

    GeertVanTeylingen_25-1650967057817.png

 

  1. Specify any locations for log backups to include in the recovery process.

     

    GeertVanTeylingen_16-1652792732583.png

 

  1. Check any appropriate “Other Settings”, the following screen is the defaults

    GeertVanTeylingen_9-1652793719308.png

  2. On the summary page, review any final details and press Finish to restore the tenant database. Select Finish to proceed with the recovery.

     

    GeertVanTeylingen_18-1652792790525.png

      

  3. The recovery process can take a few minutes, depending on database size and log files to process.

    GeertVanTeylingen_29-1650967057840.png

 

  1. When the recovery has finished a Recovery Execution Summary provides details of the recovery. The following screen shows a completed recovery of the TENANT DB.

     

    GeertVanTeylingen_19-1652792887489.png

 

  1. The following screenshot shows the database after recovery with some services running.

    GeertVanTeylingen_31-1650967057852.png


    Note, there is no process for PR2 running, this tenant still needs to be recovered

 

Repeat the steps 14-25 to recover any other tenants.

In our example, after recovering tenant PR2, the process list looks like the following:

GeertVanTeylingen_32-1650967057862.png

 

A process listing can also be retrieved form the command line when logged in as the <sid>adm user.

> /usr/sap/hostctrl/exe/sapcontrol -nr 00 -function GetProcessList
01.04.2022 01:19:55
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2022 04 01 01:06:14, 0:13:41, 713
hdbcompileserver, HDB Compileserver, GREEN, Running, 2022 04 01 01:07:04, 0:12:51, 1653
hdbindexserver, HDB Indexserver-PR1, GREEN, Running, 2022 04 01 01:14:54, 0:05:01, 3929
hdbindexserver, HDB Indexserver-PR2, GREEN, Running, 2022 04 01 01:17:39, 0:02:16, 5302
hdbnameserver, HDB Nameserver, GREEN, Running, 2022 04 01 01:06:14, 0:13:41, 731
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2022 04 01 01:07:04, 0:12:51, 1656
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2022 04 01 01:07:05, 0:12:50, 1702
hdbxsengine, HDB XSEngine-PR1, GREEN, Running, 2022 04 01 01:15:11, 0:04:44, 4266

Recover the database to the following point in time

This process allows recovery of the database to a specific point in time, perhaps just prior to an invalid transaction.

 

  1. First step is to stop the database

    GeertVanTeylingen_0-1652797814385.png

     

    GeertVanTeylingen_1-1652797838354.png

     

    When this is finished, the Processes tab should display as follows:

    GeertVanTeylingen_2-1652797861816.png

     

  2. Start the recovery process from the menu.

    GeertVanTeylingen_3-1652797901086.png

    Note, the recovery wizard can take several seconds to launch (see the following status)

    GeertVanTeylingen_4-1652797930124.png

     

  3. Choose the recovery type, in this case “Recover the database to the following point in time”, in this example the time stamp chosen is 01-April-2022 10:30:00 (in 24 hour GMT+13)

    GeertVanTeylingen_5-1652797970418.png

    ! Note the time used is based on UTC/GMT.

  4. Confirm the recovery to continue, noting the potential for lost data.

    GeertVanTeylingen_6-1652798028771.png

     

  5. Choose the location of the backup catalog, which is needed for recovery.

    GeertVanTeylingen_7-1652798062192.png

  6. The backup catalog will be fetched to display the appropriate backup to recover from (this can take a minute or two to load)

    GeertVanTeylingen_8-1652798087829.png

     

  7. The first time the backup catalog is refreshed, its likely no suitable snapshot will be found to restore from.  This is because the administrator will need to copy/restore the files from the snapshot into the data area.

    GeertVanTeylingen_9-1652798122047.png

     

  8. Restore the snapshot files for recovery using either option (a) Revert Volume from Snapshot or option (b) Copy files.  Option (a) is faster.
    1. Revert Volume from Snapshot – this will roll a Volume back to a prior state and any data or snapshots added to the Volume after the Snapshot used to revert to are lost.
      At this time Reverting a Volume can only be done using the Azure Portal.

      ! Make sure SAP HANA is not running before reverting any volumes.

      If dealing with multiple Volumes make a note of the snapshot name you are reverting to as they all must have the same snapshot name.

      We are reverting from the latest snapshot taken BEFORE our target recovery time.

      GeertVanTeylingen_10-1652798305237.png

       

      GeertVanTeylingen_11-1652798430055.png
    2. Copy files - In this example, the files are copied from the “hidden” snapshot location in the filesystem.
      # su - pr1adm
      > cp -pr /hana/data/PR1/mnt00001/.snapshot/PR1_hourly__2022-03-31T210004-1084693Z/* /hana/data/PR1/mnt00001/.
    3. Single file snapshot restore
      Alternatively Azure NetApp Files single file snapshot restore feature can be used to restore files without the need to copy any of them across the network.

  9. When the copy is complete, refresh the view of the backup catalog to ensure the snapshot we are restoring from is listed.

    GeertVanTeylingen_12-1652798563406.png

  10. Now select the available SNAPSHOT shown in green to recover from.

    GeertVanTeylingen_13-1652798591174.png

     

  11. Choose the location of the Log Backups.

    GeertVanTeylingen_14-1652798670690.png

     

  12. Check any appropriate “Other Settings”, the following screen is the defaults

    GeertVanTeylingen_15-1652798695135.png

     

  13. On the summary page, review any final details and press Finish to restore the system database.

    GeertVanTeylingen_16-1652798722847.png

     

  14. When the recovery has finished a Recovery Execution Summary provides details of the recovery.  The following screen shows a completed recovery of the SYSTEMDB.

    GeertVanTeylingen_17-1652798771057.png

     

    ! Note the message stating “recovering the system database from a storage snapshot invalidates all the tenant databases”.  Tenant databases must now be recovered.

  15. Start the recovery of the Tenant database

    GeertVanTeylingen_18-1652798998726.png

     

  16. Choose the Tenant to recover from.  At the time of writing, only a single tenant database is supported by SAP to recover from.

    GeertVanTeylingen_19-1652799030878.png

     

  17. Choose to recover the tenant database to the following point in time (same as for the system database).

    GeertVanTeylingen_20-1652799064367.png

     

  18. Provide the location of the Backup Catalog (same as for the system database)

    GeertVanTeylingen_21-1652799090284.png

     

  19. Allow the tenant database to be stopped for recovery.

    GeertVanTeylingen_22-1652799115960.png

     

  20. Wait for the Backup Catalog to be refreshed and displayed

  21. When recovering the tenant database there should already be a valid snapshot to recover from (unlike the system database where we needed to restore the snapshot files into the data area and refresh the view).  Select this snapshot and click next.

    GeertVanTeylingen_23-1652799152167.png

     

  22. Specify any locations for log backups to include in the recovery process.

    GeertVanTeylingen_24-1652799180524.png

     

  23. Check any appropriate “Other Settings”, the following screen is the defaults.

    GeertVanTeylingen_25-1652799213690.png

     

  24. On the summary page, review any final details and press Finish to restore the tenant database. Select Finish to proceed with the recovery.

    GeertVanTeylingen_26-1652799248506.png

     

  25. The recovery process can take a few minutes, depending on database size and log files to process.

    GeertVanTeylingen_27-1652799283805.png

     

  26. When the recovery has finished a Recovery Execution Summary provides details of the recovery. The following screen shows a completed recovery of the TENANT DB.

    GeertVanTeylingen_28-1652799312943.png

     

  27. The following screenshot shows the database after recovery with some services running.

    GeertVanTeylingen_29-1652799339786.png

     

    Note, there is no process for PR2 running, this tenant still needs to be recovered


Repeat the steps 14-25 to recover any other tenants.

In our example, after recovering tenant PR2, the process list looks like the following:

GeertVanTeylingen_63-1650967058040.png

 

A process listing can also be retrieved form the command line when logged in as the <sid>adm user.

> /usr/sap/hostctrl/exe/sapcontrol -nr 00 -function GetProcessList
01.04.2022 01:19:55
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2022 04 01 01:06:14, 0:13:41, 713
hdbcompileserver, HDB Compileserver, GREEN, Running, 2022 04 01 01:07:04, 0:12:51, 1653
hdbindexserver, HDB Indexserver-PR1, GREEN, Running, 2022 04 01 01:14:54, 0:05:01, 3929
hdbindexserver, HDB Indexserver-PR2, GREEN, Running, 2022 04 01 01:17:39, 0:02:16, 5302
hdbnameserver, HDB Nameserver, GREEN, Running, 2022 04 01 01:06:14, 0:13:41, 731
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2022 04 01 01:07:04, 0:12:51, 1656
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2022 04 01 01:07:05, 0:12:50, 1702
hdbxsengine, HDB XSEngine-PR1, GREEN, Running, 2022 04 01 01:15:11, 0:04:44, 4266

 

Recover the database to a specific data (snapshot) backup

 

This process recovers the database to a specific snapshot only (i.e. no log replay).

 

  1. First step is to stop the database

    GeertVanTeylingen_0-1652801201618.png

     

    GeertVanTeylingen_1-1652801232602.png

     

    When this is finished, the Processes tab should display as follows:

    GeertVanTeylingen_2-1652801252081.png

     

  2. Start the recovery process from the menu.

    GeertVanTeylingen_3-1652801280353.png

    Note, the recovery wizard can take several seconds to launch (see the following status)

    GeertVanTeylingen_4-1652801306271.png

     

  3. Choose the recovery type, in this case “Recover the database to a specific data backup”.

    GeertVanTeylingen_5-1652801329934.png

     

  4. Confirm the recovery to continue, noting the potential for lost data.

    GeertVanTeylingen_6-1652801349255.png

     

  5. As there will be no log replay, continue to “Recover without the backup catalog”.

    GeertVanTeylingen_7-1652801368926.png

     

  6. Specify the Backup to Recover, Destination Type = Snapshot.

    GeertVanTeylingen_8-1652801403482.png

     

  7. Note this restore method will Initialize Log Area. Check any appropriate “Other Settings”, the following screen is the defaults

    GeertVanTeylingen_9-1652801527683.png

     

  8. Restore the snapshot files for recovery using either option (a) Revert Volume from Snapshot or option (b) Copy files.  Option (a) is faster.
    1. Revert Volume from Snapshot – this will roll a Volume back to a prior state and any data or snapshots added to the Volume after the Snapshot used to revert to are lost. At this time Reverting a Volume can only be done using the Azure Portal.

      ! Make sure SAP HANA is not running before reverting any volumes.

      If dealing with multiple Volumes make a note of the snapshot name you are reverting to as they all must have the same snapshot name.

      GeertVanTeylingen_10-1652801708911.png

       

      GeertVanTeylingen_11-1652802051895.png
    2. Copy files - In this example, the files are copied from the “hidden” snapshot location in the filesystem.
      # su - pr1adm
      > cp -pr /hana/data/PR1/mnt00001/.snapshot/PR1_hourly__2022-03-31T210004-1084693Z/* /hana/data/PR1/mnt00001/.
    3. Single file snapshot restore
      Alternatively Azure NetApp Files single file snapshot restore feature can be used to restore files without the need to copy any of them across the network
  9. On the summary page, review any final details.  Make sure you have copied/restored the snapshot files to the data area, if the copy has completed then press Finish to restore the system database.

    GeertVanTeylingen_12-1652802441840.png

     

  10. When the recovery has finished a Recovery Execution Summary provides details of the recovery.  The following screen shows a completed recovery of the SYSTEMDB.

    GeertVanTeylingen_13-1652802487565.png

     

    ! Note the message stating “recovering the system database from a storage snapshot invalidates all the tenant databases”.  Tenant databases must now be recovered.

  11. Start the recovery of the Tenant database

    GeertVanTeylingen_14-1652802522357.png

     

  12. Choose the Tenant to recover from.  At the time of writing, only a single tenant database is supported by SAP to recover from.

    GeertVanTeylingen_15-1652802549794.png

     

  13. Choose to recover the tenant database to a specific data backup.

    GeertVanTeylingen_16-1652802572609.png

     

  14. As there will be no log replay, continue to “Recover without the backup catalog”.

    GeertVanTeylingen_17-1652802596567.png

     

  15. Specify the Backup to Recover, Destination Type = Snapshot.

    GeertVanTeylingen_18-1652802617096.png

     

  16. Note this restore method will Initialize Log Area. Check any appropriate “Other Settings”, the following screen is the defaults

    GeertVanTeylingen_19-1652802639650.png

     

  17. There is no need to restore the snapshot files to the data area as this was done when recovering the system database.

  18. On the summary page, review any final details and press Finish to restore the system database.

    GeertVanTeylingen_20-1652802670515.png

    GeertVanTeylingen_21-1652802719378.png

     

  19. When the recovery has finished a Recovery Execution Summary provides details of the recovery.  The following screen shows a completed recovery of the tenant database.

    GeertVanTeylingen_23-1652802762646.png

     

  20. The following screenshot shows the database after recovery with some services running.

    GeertVanTeylingen_24-1652802812017.png

    Note, there is no process for PR2 running, this tenant still needs to be recovered

 

Repeat the steps 11-20 to recover any other tenants.

In our example, after recovering tenant PR2, the process list looks like the following:

GeertVanTeylingen_88-1650967058183.png

 

A process listing can also be retrieved form the command line when logged in as the <sid>adm user.

> /usr/sap/hostctrl/exe/sapcontrol -nr 00 -function GetProcessList
01.04.2022 01:19:55
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2022 04 01 01:06:14, 0:13:41, 713
hdbcompileserver, HDB Compileserver, GREEN, Running, 2022 04 01 01:07:04, 0:12:51, 1653
hdbindexserver, HDB Indexserver-PR1, GREEN, Running, 2022 04 01 01:14:54, 0:05:01, 3929
hdbindexserver, HDB Indexserver-PR2, GREEN, Running, 2022 04 01 01:17:39, 0:02:16, 5302
hdbnameserver, HDB Nameserver, GREEN, Running, 2022 04 01 01:06:14, 0:13:41, 731
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2022 04 01 01:07:04, 0:12:51, 1656
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2022 04 01 01:07:05, 0:12:50, 1702
hdbxsengine, HDB XSEngine-PR1, GREEN, Running, 2022 04 01 01:15:11, 0:04:44, 4266 

A process listing can also be retrieved form the command line when logged in as the <sid>adm user.

> /usr/sap/hostctrl/exe/sapcontrol -nr 00 -function GetProcessList
15.09.2019 23:51:43
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2019 09 15 23:08:57, 0:42:46, 28998
hdbcompileserver, HDB Compileserver, GREEN, Running, 2019 09 15 23:09:28, 0:42:15, 29598
hdbindexserver, HDB Indexserver-PR1, GREEN, Running, 2019 09 15 23:21:34, 0:30:09, 31935
hdbindexserver, HDB Indexserver-PR2, GREEN, Running, 2019 09 15 23:37:51, 0:13:52, 36538
hdbnameserver, HDB Nameserver, GREEN, Running, 2019 09 15 23:08:57, 0:42:46, 29017
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2019 09 15 23:09:28, 0:42:15, 29601
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2019 09 15 23:09:29, 0:42:14, 29648
hdbxsengine, HDB XSEngine-PR1, GREEN, Running, 2019 09 15 23:21:53, 0:29:50, 32071

Appendix – SAP HANA Data Volume locations

 

A detailed explanation of persistent data storage can be found in the “SAP HANA Administration Guide for SAP HANA Platform” - “Persistent Data Storage in the SAP HANA Database” section.

 

The following diagram is taken from the “Data and Log Volumes” sub-section. This shows the Directory Hierarchy for Persistent Data Storage (System with Multitenant Database Containers) for SAP HANA.  Note the separation of System DB and Tenant DB files into logically grouped sub-directories.  The volume names of tenant databases have a suffix to represent the database. For example, the indexserver volume for the first tenant database is hdb00002.00002, for the second database hdb00002.000003, and so on.  For example, Tenant DB 1 data storage is grouped into both “hdb00002.00003” and “hdb00003.00003” sub-directories for the indexserver and xsengine respectively. 

 

GeertVanTeylingen_89-1650967058184.png

 

 

1 Comment
Version history
Last update:
‎May 17 2022 09:59 AM
Updated by: