Datax导入(from mysql)使用案例

Datax导入(from mysql)

下载安装

#下载解压datax
wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
tar -zvxf datax.tar.gz

# writer推荐使用最新插件,下载地址:https://github.com/StarRocks/DataX/releases
#下载解压doriswrite
wget 'http://dorisdb-release.oss-cn-zhangjiakou.aliyuncs.com/doriswriter.tar.gz?Expires=1988133021&OSSAccessKeyId=LTAI4GFYjbX9e7QmFnAAvkt8&Signature=9RWd5APw26q%2B7aL8sJpu1h2namU%3D' -O doriswriter.tar.gz
tar -zvxf doriswriter.tar.gz
#将dorisdbwriter放至datax插件目录
mv doriswriter datax/plugin/writer/

DDL

StarRocks建表

CREATE TABLE dataxtest(
 name1

tinyint(4) NULL COMMENT “”,

name2 tinyint(4) NULL COMMENT “”

) ENGINE=OLAP

duplicate KEY( name1 )

COMMENT “OLAP”

DISTRIBUTED BY HASH( name1 ) BUCKETS 3

PROPERTIES (

“replication_num” = “1”,

“in_memory” = “false”,

“storage_format” = “DEFAULT”

);

mysql建表

CREATE TABLE dataxtest(
 name1

tinyint(4) NULL COMMENT “”,

name2 tinyint(4) NULL COMMENT “”

) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci COMMENT=‘订单表’

mysql插入数据

Insert into dataxtest values(1,1),(2,2),(3,3),(4,4)

SR创建用户:

CREATE USER datax@'%' IDENTIFIED BY '123456';
grant all on . to 'datax'@'%';

编辑任务:

vim job/mysql.json

{
    "job": {
        "setting": {
            "speed": {
                 "channel": 1
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0
            }
        },
        "content": [
            {
                "reader": {
                    "name": "mysqlreader",
                    "parameter": {
                        "username": "root",
                        "password": "123456",
                        "column": ["name1", "name2"],
                        "connection": [
                            {
                                "table": [ "dataxtest"],
                                "jdbcUrl": [
                                     "jdbc:mysql://127.0.0.1:3306/canal"
                                ]
                            }
                        ]
                    }
                },
               "writer": {
                    "name": "doriswriter",
                    "parameter": {
                        "username": "datax",
                        "password": "123456",
                        "database": "qcy",
                        "table": "dataxtest",
                        "column": ["name1","name2"],
                        "preSql": [],
                        "postSql": [],
                        "jdbcUrl": "jdbc:mysql://127.0.0.1:8012/",
                        "loadUrl": ["127.0.0.1:8011"],
                        "loadProps": {}
                    }
                }
            }
        ]
    }
}

执行任务

python bin/datax.py --jvm="-Xms6G -Xmx6G" --loglevel=debug job/mysql.json

写入成功,查看数据

3赞

默认传入的数据均会被转为字符串,并以 \t 作为列分隔符, \n 作为行分隔符,组成 csv 文件进行StreamLoad导入操作。 如需更改列分隔符,则正确配置 loadProps 即可:

"loadProps": {
    "column_separator": "\\x01",
    "row_delimiter": "\\x02"
}

如果导入字段包含分割符,会导致导入失败,此时可以定义导入格式为json。
如需更改导入格式为 json ,则正确配置 loadProps 即可:

"loadProps": {
    "format": "json",
    "strip_outer_array": true
}

博主 你好 可以把 doriswriter.tar.gz 给我一份吗 798986060@qq.com

参考这里 https://docs.starrocks.io/zh-cn/main/loading/DataX-starrocks-writer,有对应的starrockswriter包

链接打不开 博主

参考这里 https://docs.starrocks.io/zh-cn/main/loading/DataX-starrocks-writer

链接失效了呀,打不开

更新了,https://docs.starrocks.io/zh-cn/main/loading/DataX-starrocks-writer