Use the migration tools provided in Ali and Tencent's help documents to test the migration objects for storing data.
1. Tencent to Ali
Document Links: https://help.aliyun.com/document_detail/56990.html?spm=5176.7851628.6.1079.UqBdXK
The OssImport tool migrates data stored locally and in other clouds to OSS with the following features:
The rich data sources supported are local, Qiniu, Baidu BOS, AWS S3, Azure Blob, Tuoyun, Tencent Cloud COS, Jinshan KS3, HTTP, OSS, etc. and can be expanded as needed.
Support breakpoint continuation;
Support flow control;
Supports the migration of files after a specified time, with a specific prefix;
Support parallel data download and upload;
Supports single-machine mode and distributed mode. Single-machine mode deployment is simple and easy to use, and distributed mode is suitable for large-scale data migration.
- Running Environment
Install Java 1.7 and above - Deployment Mode
OssImport has two deployment modes, single-machine mode and distributed mode.For small-scale data migration smaller than 30TB, single-machine mode can be completed.For large-scale data migration, use distributed mode.
The file structure in single-machine mode is as follows:
Distributed:
This example is a test click deployment
Download Toolkit Address: http://gosspublic.alicdn.com/ossimport/standalone/ossimport-2.3.1.zip?spm=5176.doc57053.2.4.iSAMVL&file=ossimport-2.3.1.zip
Unzip when downloaded locally and edit the configuration file\ossimport-2.3.1\conf\local_job.cfg
The following configuration is relevant
#Whether to turn on incremental mode, if set to true, will rescan incremental data every interval incrementalModeInterval (unit seconds) and synchronize incremental data to oss. incrementalModeInterval does not recommend configuring values less than 3600 seconds, which wastes a lot of requests and incurs additional overhead (currently configurable minimum interval is 900 seconds) isIncremental=false incrementalModeInterval=86400 ####################Synchronize Source Settings#################### #Synchronization source type, currently supported: #1.local (local file, this option only needs to fill in srcPrefix, not srcAccessKey, srcSecretKey, srcDomain, srcBucket) #2.oss (migrating from one bucket of OSS to another) #3.qiniu #4. Bos (cloud storage in Baidu) #5.ks3 (Golden Mountain's Cloud Storage) #6.s3 (Amazon s3) #7.youpai #8.http (Synchronize data by providing a list of HTTP links, this option does not require filling in srcAccessKey, srcSecretKey, srcDomain, srcBucket, srcPrefix) #9.cos (Tencent Cloud) #10.azure (Microsoft blob) srcType=cos srcAccessKey=AKIDuhxxxxxxvUt srcSecretKey=PekfTxxxxxxxxohR #Source endpoint, sync local files and sync via http address unnecessary srcDomain=sh srcBucket=testsh #Source prefix, empty by default. If srcType=local, this directory is the local directory. If other types are different, it is the prefix of the Object s in the source Bucket that need to be synchronized. Note that if it is a local directory, the full directory path (split with'/'and end with'/', example: c:/example/) srcPrefix=/123/ #####################Synchronization Destination Settings###################### destAccessKey=LTxxxxxxxxxxxxxxxxxxxxxxxxxxKQ destSecretKey=edHc1xxxxxxxxxxxxxxxxxxxxxxxPC #Purpose endpoint, please fill in according to your actual OSS area, default is the domain name of Hangzhou. If you migrate with Aliyun ecs virtual machine, use internal domain name, no charge and no virtual machine bandwidth restriction (non-virtual machine can not be used); example: http://oss-cn-hangzhou-internal.aliyuncs.com destDomain=http://oss-cn-shanghai.aliyuncs.com #Destination bucket, no need to add'/' destBucket=qianyi-test #Target prefix, empty by default, placed directly under the bucket (if you want to synchronize data to a directory in oss, end with'/'), note: OSS does not support starting with'/'as a file, so destPrefix should not be configured to start with'/' destPrefix=111/
Run the program after the configuration is correct:
windows to the tool directory, run import.bat, linux run import.sh
View running status
Migration completed
View results on Ali OSS
Note: There are relevant parameter settings in the program, such as the default is to copy all the data in the source object store every time it is executed, and overwrite the copy, all of which can set parameters in the configuration file.
2. Ali to tencent
This migration tool supports migrating files from services such as AWS S3, Ali Cloud OSS, and Qiniuyun to object storage COS; it also supports file list migration, that is, migrating files from a series of given URL s to object storage COS.
Limitations on use: only applicable to COS V4 version
System environment: Linux or Mac system
Software dependency: Python 2.7.x, pip, gcc and python-dev should also be installed.
Installation and configuration:
- 1. Install Python 2.7.x
wget https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz
tar -zxvf Python-2.7.9.tgz
cd Python-2.7.9
./configure --prefix=/usr/local/python-2.7.9
make&&make install
ln -s /usr/local/python-2.7.9/bin/python /usr/bin/python2.7.9 - 2. Install pip, gcc and python-dev
yum install python-pip python-devel gcc
If there is a problem with the pip installation, install it as follows:
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
pip-V View pip Version -
3. Download Tools
https://github.com/tencentyun/cos_migrate_tool, ZIP can be downloaded locally and uploaded to the linux server.
Download directly to server: git clone https://github.com/tencentyun/cos_migrate_tool.git
If there is no git, the yum install git installation requires a higher version to be compiled and installed. -
4. Installation Tools
pip install -U cos_migrate_tool
Cos_migrate_tool-h Check for successful installation - 5. Configuration
Edit oss-2-cos.conf (this is cos4 version 4) [common] workspace=/tmp/tmp6
[source]
type=oss
accesskeyid=LTxxxxxxxQ
accesskeysecret=edHxxxxxxPC
bucket=qianyi-test1
endpoint=oss-cn-qingdao.aliyuncs.com
[destination]
type=cosv4
region=sh
#Note that this region fills in abbreviations
accesskeyid=AKxxxxxxxxovUt
appid=1253961596
accesskeysecret=PxxxxxxxxxxxxxxxxxxR
bucket=testsh
- 6,Run tests cos_migrate_tool -c /root/cos_migrate_tool-master/conf/oss-2-cos.conf ![](https://s1.51cto.com/images/blog/201904/18/18182cdb5a582393ba78161ba3b8b904.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=) //Uninstall: pip uninstall cos_migrate_tool