Ali Cloud OSS and Tencent Cloud COS Migrate

Posted by DWaller10 on Fri, 10 May 2019 21:58:41 +0200

Use the migration tools provided in Ali and Tencent's help documents to test the migration objects for storing data.

1. Tencent to Ali
Document Links: https://help.aliyun.com/document_detail/56990.html?spm=5176.7851628.6.1079.UqBdXK
The OssImport tool migrates data stored locally and in other clouds to OSS with the following features:

The rich data sources supported are local, Qiniu, Baidu BOS, AWS S3, Azure Blob, Tuoyun, Tencent Cloud COS, Jinshan KS3, HTTP, OSS, etc. and can be expanded as needed.
Support breakpoint continuation;
Support flow control;
Supports the migration of files after a specified time, with a specific prefix;
Support parallel data download and upload;
Supports single-machine mode and distributed mode. Single-machine mode deployment is simple and easy to use, and distributed mode is suitable for large-scale data migration.

  • Running Environment
    Install Java 1.7 and above
  • Deployment Mode
    OssImport has two deployment modes, single-machine mode and distributed mode.For small-scale data migration smaller than 30TB, single-machine mode can be completed.For large-scale data migration, use distributed mode.

The file structure in single-machine mode is as follows:

Distributed:

This example is a test click deployment
Download Toolkit Address: http://gosspublic.alicdn.com/ossimport/standalone/ossimport-2.3.1.zip?spm=5176.doc57053.2.4.iSAMVL&file=ossimport-2.3.1.zip
Unzip when downloaded locally and edit the configuration file\ossimport-2.3.1\conf\local_job.cfg

The following configuration is relevant

#Whether to turn on incremental mode, if set to true, will rescan incremental data every interval incrementalModeInterval (unit seconds) and synchronize incremental data to oss. incrementalModeInterval does not recommend configuring values less than 3600 seconds, which wastes a lot of requests and incurs additional overhead (currently configurable minimum interval is 900 seconds)
isIncremental=false
incrementalModeInterval=86400
####################Synchronize Source Settings####################
#Synchronization source type, currently supported:
#1.local (local file, this option only needs to fill in srcPrefix, not srcAccessKey, srcSecretKey, srcDomain, srcBucket)
#2.oss (migrating from one bucket of OSS to another)
#3.qiniu
#4. Bos (cloud storage in Baidu)
#5.ks3 (Golden Mountain's Cloud Storage)
#6.s3 (Amazon s3)
#7.youpai
#8.http (Synchronize data by providing a list of HTTP links, this option does not require filling in srcAccessKey, srcSecretKey, srcDomain, srcBucket, srcPrefix)
#9.cos (Tencent Cloud)
#10.azure (Microsoft blob)
srcType=cos
srcAccessKey=AKIDuhxxxxxxvUt
srcSecretKey=PekfTxxxxxxxxohR
#Source endpoint, sync local files and sync via http address unnecessary
srcDomain=sh
srcBucket=testsh
#Source prefix, empty by default. If srcType=local, this directory is the local directory. If other types are different, it is the prefix of the Object s in the source Bucket that need to be synchronized. Note that if it is a local directory, the full directory path (split with'/'and end with'/', example: c:/example/)
srcPrefix=/123/

#####################Synchronization Destination Settings######################
destAccessKey=LTxxxxxxxxxxxxxxxxxxxxxxxxxxKQ
destSecretKey=edHc1xxxxxxxxxxxxxxxxxxxxxxxPC
#Purpose endpoint, please fill in according to your actual OSS area, default is the domain name of Hangzhou. If you migrate with Aliyun ecs virtual machine, use internal domain name, no charge and no virtual machine bandwidth restriction (non-virtual machine can not be used); example: http://oss-cn-hangzhou-internal.aliyuncs.com
destDomain=http://oss-cn-shanghai.aliyuncs.com
#Destination bucket, no need to add'/'
destBucket=qianyi-test
#Target prefix, empty by default, placed directly under the bucket (if you want to synchronize data to a directory in oss, end with'/'), note: OSS does not support starting with'/'as a file, so destPrefix should not be configured to start with'/'
destPrefix=111/

Run the program after the configuration is correct:
windows to the tool directory, run import.bat, linux run import.sh

View running status

Migration completed

View results on Ali OSS

Note: There are relevant parameter settings in the program, such as the default is to copy all the data in the source object store every time it is executed, and overwrite the copy, all of which can set parameters in the configuration file.

2. Ali to tencent
This migration tool supports migrating files from services such as AWS S3, Ali Cloud OSS, and Qiniuyun to object storage COS; it also supports file list migration, that is, migrating files from a series of given URL s to object storage COS.
Limitations on use: only applicable to COS V4 version
System environment: Linux or Mac system
Software dependency: Python 2.7.x, pip, gcc and python-dev should also be installed.

Installation and configuration:

[source]
type=oss
accesskeyid=LTxxxxxxxQ
accesskeysecret=edHxxxxxxPC
bucket=qianyi-test1
endpoint=oss-cn-qingdao.aliyuncs.com

[destination]
type=cosv4
region=sh
#Note that this region fills in abbreviations
accesskeyid=AKxxxxxxxxovUt
appid=1253961596
accesskeysecret=PxxxxxxxxxxxxxxxxxxR
bucket=testsh


- 6,Run tests
    cos_migrate_tool -c /root/cos_migrate_tool-master/conf/oss-2-cos.conf

![](https://s1.51cto.com/images/blog/201904/18/18182cdb5a582393ba78161ba3b8b904.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=)

//Uninstall: pip uninstall cos_migrate_tool

Topics: Python pip Linux git