Heterogeneous migration Oracle's past and present life -- Huawei cloud + Ugo + DRS

Posted by cbrooks on Wed, 05 Jan 2022 20:08:20 +0100

1, Database localization background

Oracle is still the greatest database by far. The Oracle versions we have seen or heard of in our generation are 8i and 9i. The release dates are 1998 and 2001. The earliest release date of Oracle can be traced back to the 1960s and 1970s. At that time, oracle was called RSi (Relational Software, Inc). Oracle database has changed from simple function to complex function, From unstable to stable, it has experienced the precipitation of time.

Mr. Gai Guoqiang Gai called "the first year of domestic databases" in 2019. Although domestic databases started late, they have a history of about 40 years. With the guidance, support and support of the state in recent years, Internet enterprises have entered the market with high enthusiasm, the participation of emerging independent database start-ups, the participation of head technology enterprises, and the huge user base of local databases, I believe that the development of domestic databases will catch up one day, from trial, usable, to dare to use, easy to use and easy to use.

With the impact of the "trade war" in 2019, a number of technologies are "stuck", especially in the chip industry, resulting in the situation that domestic enterprises have no core available. As one of the three basic software, the database also has the problem of "neck sticking". Under the guidance of national policies, more and more enterprises enter the Bureau, pursue self-control and get rid of foreign monopoly.

In addition, in many segmentation scenarios, Oracle alone can no longer meet the needs of domestic unique business scenarios. Domestic databases have made efforts in distributed and cloud native. At present, they have made good achievements in finance, Internet, operators and other industries, and even opened markets at home and abroad.

The slogan of going to IOE has been mentioned in recent years. With the wave of domestic databases, the "oracle" in going to IOE has been put on the agenda by many large, medium and small enterprises. A common problem is that the more heavily Oracle is applied, the more users use features, and the greater the migration complexity. For example, with the accumulation of time, the PL/SQL function of Oracle, the PL/SQL code of some customers has reached the level of millions or even tens of millions. The human and material resources spent on the code transformation of heterogeneous migration are unacceptable to many customers. This function alone discourages many users who want to go to "O". The current situation of "O" removal is that eight immortals cross the sea to show their powers. There is no good tool to unify the migration process. Huawei cloud recently launched two products UGO+DRS, which I believe can help users complete and simplify the process of "O".

2, What are UGO, DRS, GaussDB

Database and Application Migration UGO (UGO) is a professional cloud service focusing on heterogeneous database structure migration and application SQL transformation. Through UGO's pre migration evaluation and automatic syntax conversion, users can identify migration risks in advance, improve migration efficiency, and minimize users' database migration costs.

Data Replication Service (DRS) is an easy-to-use, stable and efficient cloud service for online database migration and real-time database synchronization. DRS focuses on cloud database, reduces the complexity of data flow between databases, and effectively helps you reduce the cost of data transmission.

GaussDB(for openGauss) is a self-developed enterprise level distributed relational database built by Huawei. The product has the mixed load capacity of enterprise level complex transactions. At the same time, it supports excellent distributed transactions, cross AZ deployment in the same city, data 0 loss, 1000 + expansion capability, PB level mass storage and other enterprise level database features. It has the key capabilities of high availability, high reliability, high security, elastic scalability, one click deployment, rapid backup and recovery, monitoring and alarm, and can provide enterprises with enterprise level database services with comprehensive functions, stability, reliability, strong scalability and superior performance. At the same time, Huawei open source openGauss stand-alone active and standby community version to encourage more partners and developers to jointly prosper China's database ecology.

With UGO+DRS, Huawei has achieved the highest level of industrial standards set by the Institute in terms of business database migration.

3, Migration test

This test simulates the heterogeneous migration from Oracle database in the production environment to Huawei cloud GaussDB(for openGauss), uses UGO to migrate objects, uses DRS to migrate and synchronize data in real time, and evaluates and verifies the migration process and data.

1. Architecture diagram

The test architecture is shown above. The source end is Oracle 19C installed in Docker, and the target end is the GaussDB(for openGauss) instance purchased on Huawei cloud. Ugos and DRS are purchased through Huawei cloud and operated directly in the web console.

2. Build source side environment

Package image

[root@ecs-ugo ~]# git clone https://github.com/oracle/docker-images.git[root@ecs-ugo ~]# cd docker-images-main/OracleDatabase/SingleInstance/dockerfiles/19.3.0/[root@ecs-ugo 19.3.0]# lscheckDBStatus.sh  Checksum.ee   createDB.sh    db_inst.rsp  installDBBinaries.sh          relinkOracleBinary.sh  runUserScripts.sh  setupLinuxEnv.shcheckSpace.sh     Checksum.se2  dbca.rsp.tmpl  Dockerfile   LINUX.X64_193000_db_home.zip  runOracle.sh           setPassword.sh     startDB.sh[root@ecs-ugo 19.3.0]# pwd/root/docker-images-main/OracleDatabase/SingleInstance/dockerfiles/19.3.0[root@ecs-ugo 19.3.0]# cd ..[root@ecs-ugo dockerfiles]# ./buildContainerImage.sh --help./buildContainerImage.sh: illegal option -- -
Usage: buildContainerImage.sh -v [version] -t [image_name:tag] [-e | -s | -x] [-i] [-o] [container build option]Builds a container image for Oracle Database.
Parameters:   -v: version to build       Choose one of:  18.3.0  18.4.0  19.3.0  21.3.0     -t: image_name:tag for the generated docker image   -e: creates image based on 'Enterprise Edition'   -s: creates image based on 'Standard Edition 2'   -x: creates image based on 'Express Edition'   -i: ignores the MD5 checksums   -o: passes on container build option
* select one edition only: -e, -s, or -x
Copyright (c) 2014,2021 Oracle and/or its affiliates.
[root@ecs-ugo dockerfiles]# ./buildContainerImage.sh -v 19.3.0 -i -e[root@ecs-ugo dockerfiles]# docker image lsREPOSITORY                                       TAG         IMAGE ID       CREATED       SIZEoracle/database                                  21.3.0-ee   0855ee2749de   9 days ago    7.94GBswr.cn-north-4.myhuaweicloud.com/oracle/oracle   21.3.0      0855ee2749de   9 days ago    7.94GBoracle/database                                  19.3.0-ee   cc930c1e5ee2   10 days ago   6.53GBswr.cn-north-4.myhuaweicloud.com/oracle/oracle   19.3.0      cc930c1e5ee2   10 days ago   6.53GBswr.cn-north-4.myhuaweicloud.com/oracle/oracle   19c         cc930c1e5ee2   10 days ago   6.53GBoraclelinux                                      7-slim      078d6e3ae75e   5 weeks ago   132MB

Start Oracle 19c

docker run --name oracle21c -d \
-p 15221:1521  \
-e ORACLE_PDB=oracle19c \
-e ORACLE_PWD=Enmotech \

Test connection

[root@ecs-ugo ~]# docker exec -it oracle19c bash[oracle@666d4b0868ff ~]$ export ORACLE_SID=LEE[oracle@666d4b0868ff ~]$ sqlplus / as sysdba
SQL*Plus: Release - Production on Fri Dec 3 05:35:53 2021Version
Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:Oracle Database 19c Enterprise Edition Release - ProductionVersion
SQL> show pdbs;
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED---------- ------------------------------ ---------- ----------         2 PDB$SEED                       READ ONLY  NO         3 ORACLE19C                      READ WRITE NOSQL> Disconnected from Oracle Database 19c Enterprise Edition Release - ProductionVersion[oracle@666d4b0868ff ~]$ lsnrctl status
LSNRCTL for Linux: Version - Production on 03-DEC-2021 05:36:29
Copyright (c) 1991, 2019, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))STATUS of the LISTENER------------------------Alias                     LISTENERVersion                   TNSLSNR for Linux: Version - ProductionStart Date                03-DEC-2021 03:31:42Uptime                    0 days 2 hr. 4 min. 46 secTrace Level               offSecurity                  ON: Local OS AuthenticationSNMP                      OFFListener Parameter File   /opt/oracle/product/19c/dbhome_1/network/admin/listener.oraListener Log File         /opt/oracle/diag/tnslsnr/666d4b0868ff/listener/alert/log.xmlListening Endpoints Summary...  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=666d4b0868ff)(PORT=5500))(Security=(my_wallet_directory=/opt/oracle/admin/LEE/xdb_wallet))(Presentation=HTTP)(Session=RAW))Services Summary...Service "LEE" has 1 instance(s).  Instance "LEE", status READY, has 1 handler(s) for this service...Service "LEEXDB" has 1 instance(s).  Instance "LEE", status READY, has 1 handler(s) for this service...Service "d16cf25e0b510cc0e053020011ace43e" has 1 instance(s).  Instance "LEE", status READY, has 1 handler(s) for this service...Service "oracle19c" has 1 instance(s).  Instance "LEE", status READY, has 1 handler(s) for this service...The command completed successfully

3. Purchase target end GaussDB for openGauss

Select the cloud database GaussDB on the console

On the instance management page, click purchase database instance

On the create instance page, select the billing mode, fill in and select the instance related information, and then click buy now.


4. Test and log in to GaussDB for openGauss

DAS platform

command line

[root@ecs-ugo Euler2.5_X86_64]# gsql -d enmotech -h -Uroot -p8000 -r -WEnmotech2021@gsql ((GaussDB Kernel V500R001C20 build 327f8401) compiled at 2021-09-15 00:29:02 commit 1094 last mr 6379 )SSL connection (cipher: DHE-RSA-AES128-GCM-SHA256, bits: 128)Type "help" for help.
enmotech=> select version();                                                                                         version                                                                                          ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ PostgreSQL 9.2.4 (GaussDB Kernel V500R001C20 build 327f8401) compiled at 2021-09-15 00:28:41 commit 1094 last mr 6379  on aarch64-unknown-linux-gnu, compiled by g++ (GCC) 7.3.0, 64-bit(1 row)

5. UGO object migration

The whole object migration process of UGO is as follows:

Create evaluation project

  • Support multiple versions of Oracle database (11g, 12c, 18c, 19c)

Fill in database connection information

Pre inspection

  • Verify the necessary conditions for migration

Select evaluation scope

  • Support to customize the evaluation of multiple database objects

Task confirmation, creating

Confirm target library evaluation

  • For the selection of target database types for intelligent recommendation, the perspective of user selection is multi-dimensional. UGO provides compatibility recommendation, which can provide key quantitative dimensions.

Source database analysis

  • The details of the portrait in the original database are recorded

Target database evaluation

  • Analyze the compatibility and performance of different kinds of target databases
  • Workload and man day evaluation, statistics of object information, and solving customer pain points
  • List the existing risk points and give corresponding modification suggestions

Summary report

Create migration project

Conversion plan

  • In the industry, UGO generally imitates O racle syntax at the database syntax layer. In fact, this practice is connected. Huawei's scheme is completely switched to an autonomous and controllable ecology, so the transformation of UGO belongs to "true transformation".
  • Huawei also mentioned in the public live broadcast course that UGO training library has unique advantages. It is the top user of Oracle all over the world. It grinds in the core field of IOE and replaces the core business of Oracle Jushi system, that is, class A business. At present, the conversion rate of ten million row PLSQL in landmark projects has reached 95%.

Comparison and modification suggestions before and after SQL

grammatical transformation

Object correction

Prospect of new functions It is understood that the following functions are planned to be supported in the future:

  • You can see the compatibility of 51 Oracle objects at one time, i.e. GaussDB (for openaauss), rather than just showing which objects are currently used in Oracle. Users who plan to replace Oracle in batches can understand the compatibility problem at one time.
  • For the general situation of integrating Oracle system views into business PL/SQL. UGO compares the compatibility relationship between Oracle and GaussDB (for opencauss) system views in detail, and makes targeted design according to the key demands of users in the process of de-O.
  • The attribute level compatibility analysis and comparison for object compatibility solves a problem that can not be answered in the industry: for example, how to prove that an index conversion is successful? This function of UGO can show that the index has several attributes in Oracle and corresponding attributes in GaussDB(for opengauss), so they are equivalent.

Report display

6. DRS data synchronization

The RDS data synchronization process is as follows:

Create synchronization task

Fill in the connection information and create an SMN

Fill in source library and target library information

Edit synchronization task

advanced setting

Data processing

Pre inspection

  • There are many items for pre inspection, which can avoid stepping on many pits.

Task confirmation

Screenshot of console

Source end

Target end

7. DRS data verification

  • Object level comparison: it supports the comparison of database, index, table, view, stored procedure and function, table sorting rules and other objects.
  • Data level comparison: it supports the comparison of the number of rows and contents of a table or set. (content comparison is not open yet)
  • View the comparison report and export the report.

4, Test summary

Traditional Oracle heterogeneous migration requires a large number of people to participate in research, evaluation, testing and verification. The project implementation cycle is also very long, and the average labor cost is about 500000. Under the current trend of "going to O" and going to the cloud, Huawei cloud's UGO and DRS tools can quickly migrate to heterogeneous databases such as Huawei cloud GaussDB(for openGauss) and RDS. The tools completely cover the whole project cycle of heterogeneous object migration, data synchronization and data verification, For the most complex object migration, you only need to rectify some objects according to the rewriting prompt and evaluation person days, which greatly reduces the cost of "de O" and reduces the threshold for domestic databases to replace traditional databases. It is expected that the tool will support more databases and support cloud migration in the future.

Ink Sky Wheel original link: https://www.modb.pro/db/172953?sjhy

About the author

Li Hongda, the committer of openGauss open source community, is mainly engaged in Oracle, openGauss and cloud services. Now it has Oracle OCP, OceanBase OBCA, openGauss OGCA, PostgreSQL PGCA and Huawei cloud HCIP (Solutions Architect) certification.