stream load通过fe端口导入大概率失败，通过be端口导入成功

wtai · 2024年07月18日 01:47

【详述】stream load导入时，使用fe的端口导出会出现【accept bad request: /bad-request, error: invalid version format】及【accept bad request: /bad-request, error: text is empty (possibly HTTP/0.9)】的400报错，但是通过be端口就没问题
【背景】4个线程并发使用stream load导入。
【业务影响】数据导入失败
【是否存算分离】否
【StarRocks版本】例如：3.3.0
【集群规模】例如：3fe（1 follower+2observer）+3be（fe与be混部）
【机器信息】CPU虚拟核/内存/网卡，例如：32C/64G/万兆
【表模型】例如：主键模型
【导入或者导出方式】stream load
【附件】

wtai · 2024年07月18日 03:38

http的debug日志
11:27:52.542 [pool-1-thread-1] DEBUG org.apache.http.client.protocol.RequestAddCookies - CookieSpec selected: default
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.client.protocol.RequestAuthCache - Auth cache not set in the context
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://172.24.0.170:8030][total available: 2; route allocated: 1 of 2; total allocated: 2 of 20]
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://172.24.0.170:8030][total available: 1; route allocated: 1 of 2; total allocated: 2 of 20]
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.impl.execchain.MainClientExec - Executing request PUT /api/tdtest/xiaohongshu_report_offline_creative_keyword/stream_load HTTP/1.1
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> PUT /api/tdtest/xiaohongshu_report_offline_creative_keyword/stream_load HTTP/1.1
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Authorization: Basic cm9vdDo=
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Expect: 100-continue
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> column_separator: ,
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> skip_header: 1
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> enclose: "
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> max_filter_ratio: 1
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> partial_update: true
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> columns: account_id,account_name,action_button_click,advertiser_id,approve,bidding_strategy,build_type,campaign_id,campaign_name,click,collect,comment,commodity_buy_cnt,creativity_id,creativity_image,creativity_name,external_goods_order15,external_goods_order30,external_goods_order7,external_goods_visit7,external_id,external_leads,external_rgmv15,external_rgmv30,external_rgmv7,fee,follow,goods_order,goods_visit,identity_certi_cnt,impression,initiative_message,invoke_app_engagement_cnt,invoke_app_enter_store_cnt,invoke_app_open_cnt,invoke_app_payment_cnt,jd_active_user_num,keyword,keyword_md5,landing_page_visit,leads,leads_button_impression,marketing_target,message,message_consult,message_user,msg_leads_num,note_id,optimize_target,outside_shop_visit,partner_id,pday,phone_call_cnt,phone_call_succ_cnt,phour,pic_save,placement,platform_version,presale_order_gmv7d,presale_order_num7d,promotion_target,purchase_order_gmv7d,removed,reserve_pv,rgmv,screenshot,search_cmt_after_read,search_cmt_click,search_invoke_button_click_cnt,seller_visit,share,shopping_cart_add,success_goods_order,time,unit_id,unit_name,valid_leads,wechat_copy_cnt,wechat_copy_succ_cnt,word_avg_location,word_click_rank_first,word_click_rank_third,word_click_rate_first,word_click_rate_third,word_impression_rank_first,word_impression_rank_third,word_impression_rate_first,word_impression_rate_third,last_modified=current_timestamp
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 3271301
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: 172.24.0.170:8030
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.13 (Java/11.0.23)
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "PUT /api/tdtest/xiaohongshu_report_offline_creative_keyword/stream_load HTTP/1.1[\r][\n]"
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “Authorization: Basic cm9vdDo=[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “Expect: 100-continue[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “column_separator: ,[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “skip_header: 1[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "enclose: “[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “max_filter_ratio: 1[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “partial_update: true[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “columns: account_id,account_name,action_button_click,advertiser_id,approve,bidding_strategy,build_type,campaign_id,campaign_name,click,collect,comment,commodity_buy_cnt,creativity_id,creativity_image,creativity_name,external_goods_order15,external_goods_order30,external_goods_order7,external_goods_visit7,external_id,external_leads,external_rgmv15,external_rgmv30,external_rgmv7,fee,follow,goods_order,goods_visit,identity_certi_cnt,impression,initiative_message,invoke_app_engagement_cnt,invoke_app_enter_store_cnt,invoke_app_open_cnt,invoke_app_payment_cnt,jd_active_user_num,keyword,keyword_md5,landing_page_visit,leads,leads_button_impression,marketing_target,message,message_consult,message_user,msg_leads_num,note_id,optimize_target,outside_shop_visit,partner_id,pday,phone_call_cnt,phone_call_succ_cnt,phour,pic_save,placement,platform_version,presale_order_gmv7d,presale_order_num7d,promotion_target,purchase_order_gmv7d,removed,reserve_pv,rgmv,screenshot,search_cmt_after_read,search_cmt_click,search_invoke_button_click_cnt,seller_visit,share,shopping_cart_add,success_goods_order,time,unit_id,unit_name,valid_leads,wechat_copy_cnt,wechat_copy_succ_cnt,word_avg_location,word_click_rank_first,word_click_rank_third,word_click_rate_first,word_click_rate_third,word_impression_rank_first,word_impression_rank_third,word_impression_rate_first,word_impression_rate_third,last_modified=current_timestamp[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “Content-Length: 3271301[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “Host: 172.24.0.170:8030[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “Connection: Keep-Alive[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “User-Agent: Apache-HttpClient/4.5.13 (Java/11.0.23)[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “Accept-Encoding: gzip,deflate[\r][\n]”
11:27:52.543 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “[\r][\n]”
11:27:55.550 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << “[read] I/O error: Read timed out”
11:27:55.553 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> “ACCOUNTID,ACCOUNTNAME,ACTIONBUTTONCLICK,ADVERTISERID,APPROVE,BIDDINGSTRATEGY,BUILDTYPE,CAMPAIGNID,CAMPAIGNNAME,CLICK,COLLECT,COMMENT,COMMODITYBUYCNT,CREATIVITYID,CREATIVITYIMAGE,CREATIVITYNAME,EXTERNALGOODSORDER15,EXTERNALGOODSORDER30,EXTERNALGOODSORDER7,EXTERNALGOODSVISIT7,EXTERNALID,EXTERNALLEADS,EXTERNALRGMV15,EXTERNALRGMV30,EXTERNALRGMV7,FEE,FOLLOW,GOODSORDER,GOODSVISIT,IDENTITYCERTICNT,IMPRESSION,INITIATIVEMESSAGE,INVOKEAPPENGAGEMENTCNT,INVOKEAPPENTERSTORECNT,INVOKEAPPOPENCNT,INVOKEAPPPAYMENTCNT,JDACTIVEUSERNUM,KEYWORD,KEYWORDMD5,LANDINGPAGEVISIT,LEADS,LEADSBUTTONIMPRESSION,MARKETINGTARGET,MESSAGE,MESSAGECONSULT,MESSAGEUSER,MSGLEADSNUM,NOTEID,OPTIMIZETARGET,OUTSIDESHOPVISIT,PARTNERID,PDAY,PHONECALLCNT,PHONECALLSUCCCNT,PHOUR,PICSAVE,PLACEMENT,PLATFORMVERSION,PRESALEORDERGMV7D,PRESALEORDERNUM7D,PROMOTIONTARGET,PURCHASEORDERGMV7D,REMOVED,RESERVEPV,RGMV,SCREENSHOT,SEARCHCMTAFTERREAD,SEARCHCMTCLICK,SEARCHINVOKEBUTTONCLICKCNT,SELLERVISIT,SHARE,SHOPPINGCARTADD,SUCCESSGOODSORDER,TIME,UNITID,UNITNAME,VALIDLEADS,WECHATCOPYCNT,WECHATCOPYSUCCCNT,WORDAVGLOCATION,WORDCLICKRANKFIRST,WORDCLICKRANKTHIRD,WORDCLICKRATEFIRST,WORDCLICKRATETHIRD,WORDIMPRESSIONRANKFIRST,WORDIMPRESSIONRANKTHIRD,WORDIMPRESSIONRATEFIRST,WORDIMPRESSIONRATETHIRD[\n]”
11:27:55.553 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "12949,[0xe6][0x8b][0xbe][0xe5][0x85][0x89][0xe5][0xae][0x9d][0xe7][0x9b][0x92][0xef][0xbc][0x88]IKV[0xe7][0x88][0xb1][0xe5][0x92][0x94][0xe5][0xa8][0x81][0xef][0xbc][0x89],0,1,0,OCPX[0xe8][0x87][0xaa][0xe5][0x8a][0xa8][0xe5][0x87][0xba][0xe4][0xbb][0xb7],[0xe6][0xa0][0x87][0xe5][0x87][0x86][0xe6][0x90][0xad][0xe5][0xbb][0xba],20994241,1122[0xe6][0xb0][0xb4][0xe6][0x9d][0xaf][0xe9][0x80][0x9a][0xe7][0x94][0xa8][0xe6][0xb5][0x8b][0xe8][0xaf][0x95],1,0,0,0,83325839,http://ci.xiaohongshu.com/1040g2sg30rqfnrad2k0049vsjs7sem23m4g34i8?imageView2/2/w/1080/format/jpg,[0xe5][0x88][0x9b][0xe6][0x84][0x8f]1122[0xe6][0xb0][0xb4][0xe6][0x9d][0xaf][0xe6][0x90][0x9c][0xe7][0xb4][0xa2][0xe7][0x82][0xb9][0xe5][0x87][0xbb][0xe8][0x87][0xaa][0xe5][0x8a][0xa8][0xe5][0x87][0xba][0xe4][0xbb][0xb7][0xe7][0xb2][0xbe][0xe5][0x87][0x86][0xe6][0xb5][0x8b][0xe8][0xaf][0x95]-4,0,0,0,0,643463,0,0.0,0.0,0.0,1.23,0,0,0,0,2,0,0,0,0,0,0,[0xe5][0x85][0xad][0xe4][0xb8][0xaa][0xe6][0x9c][0x88][0xe5][0xae][0x9d][0xe5][0xae][0x9d][0xe5][0xa5][0xb6][0xe7][0x93][0xb6],f0c434d3daa1ad9362a902361270084f,0,0,0,[0xe4][0xba][0xa7][0xe5][0x93][0x81][0xe7][0xa7][0x8d][0xe8][0x8d][0x89],0,0,0,0,651e1b3d000000001f03c72a,[0xe4][0xba][0x92][0xe5][0x8a][0xa8][0xe9][0x87][0x8f],0,358,20240621,0,0,9,0,[0xe6][0x90][0x9c][0xe7][0xb4][0xa2][0xe6][0x8e][0xa8][0xe5][0xb9][0xbf],V1,0.0,0,[0xe7][0xac][0x94][0xe8][0xae][0xb0],0.0,false,0,0,0,0,0,0,0,0,0,0,2024-06-21 09:00 - 09:59,28402688,[0xe5][0x8d][0x95][0xe5][0x85][0x83]1122[0xe6][0xb0][0xb4][0xe6][0x9d][0xaf][0xe6][0x90][0x9c][0xe7][0xb4][0xa2][0xe7][0x82][0xb9][0xe5][0x87][0xbb][0xe8][0x87][0xaa][0xe5][0x8a][0xa8][0xe5][0x87][0xba][0xe4][0xbb][0xb7][0xe7][0xb2][0xbe][0xe5][0x87][0x86][0xe6][0xb5][0x8b][0xe8][0xaf][0x95],0,0,0,1,[\n]"
…
…
11:27:57.293 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << “HTTP/1.1 400 Bad Request[\r][\n]”
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << “content-type: text/html[\r][\n]”
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << “content-length: 253[\r][\n]”
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << “[\r][\n]”
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << “Bad Request.
invalid version format: - 12:59,28397980,[0xc3][0x85][0xc2][0x8d][0xc2][0x95][0xc3][0x85][0xc2][0x85][0xc2][0x83]-1122-[0xc3][0x88][0xc2][0x98][0xc2][0x91][0xc3][0x88][0xc2][0x8f][0xc2][0x87][0xc3][0x87][0xc2][0x89][0xc2][0x99][0xc3][0x88][0xc2][0x83][0xc2][0xb6]-[0xc3][0x86][0xc2][0x90][0xc2][0x9c][0xc3][0x87][0xc2][0xb4][0xc2][0xa2]-[0xc3][0x84][0xc2][0xba][0xc2][0x92][0xc3][0x85][0xc2][0x8a][0xc2][0xa8]-[0xc3][0x88][0xc2][0x87][0xc2][0xaa][0xc3][0x85][0xc2][0x8a][0xc2][0xa8][0xc3][0x85][0xc2][0x87][0xc2][0xba][0xc3][0x84][0xc2][0xbb][0xc2][0xb7]-[0xc3][0x84][0xc2][0xba][0xc2][0x92][0xc3][0x85][0xc2][0x8a][0xc2][0xa8][0xc3][0x86][0xc2][0x88][0xc2][0x90][0xc3][0x86][0xc2][0x9c][0xc2][0xac][0xc3][0x86][0xc2][0x8e][0xc2][0xa7][0xc3][0x85][0xc2][0x88][0xc2][0xb6]-[0xc3][0x85][0xc2][0x85][0xc2][0xb3][0xc3][0x89][0xc2][0x94][0xc2][0xae][0xc3][0x88][0xc2][0xaf][0xc2][0x8d]-[0xc3][0x87][0xc2][0x89][0xc2][0x99][0xc3][0x88][0xc2][0x83][0xc2][0xb6][0xc3][0x86][0xc2][0x96][0xc2][0xb9][0xc3][0x85][0xc2][0x90][0xc2][0x91],0,0,0,1.67,”
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 400 Bad Request
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << content-type: text/html
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << content-length: 253
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.impl.execchain.MainClientExec - Connection can be kept alive indefinitely
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://172.24.0.170:8030] can be kept alive indefinitely
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0
11:27:57.294 [pool-1-thread-1] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://172.24.0.170:8030][total available: 2; route allocated: 1 of 2; total allocated: 2 of 20]
11:27:57.294 [pool-1-thread-1] INFO com.ipinyou.alphadesk.daq.service.common.persistence.Test - code:[400],Response body:[Bad Request.
invalid version format: - 12:59,28397980,ÅÅ-1122-ÈÈÇÈ¶-ÆÇ´¢-ÄºÅ¨-ÈªÅ¨ÅºÄ»·-ÄºÅ¨ÆÆ¬Æ§Å¶-Å³É®È¯-ÇÈ¶Æ¹Å,0,0,0,1.67,]

Star_Cao · 2024年07月18日 04:52

sr版本、导入的命令发一下

wtai · 2024年07月18日 06:08

3.3.0版本，

使用apache-httpclient
CloseableHttpClient client = HttpClients
.custom()
.setRedirectStrategy(new DefaultRedirectStrategy() {
@Override
protected boolean isRedirectable(String method) {
return true;
}
})
.build();
String loadUrl = String.format(“http://%s:%s/api/%s/%s/_stream_load”,
appConfig.getPersistenceConfig().getStarRocksHost(),
appConfig.getPersistenceConfig().getStarRocksHttpPort(),
appConfig.getPersistenceConfig().getStarRocksDb(),
table);
HttpPut httpPut = new HttpPut(loadUrl);

    httpPut.addHeader(HttpHeaders.AUTHORIZATION, basicAuthHeader(appConfig.getPersistenceConfig().getStarRocksUser(), appConfig.getPersistenceConfig().getStarRocksPassword()));
    httpPut.addHeader(HttpHeaders.EXPECT, "100-continue");
    httpPut.addHeader("label", UUID.randomUUID().toString());
    httpPut.addHeader("column_separator", ",");
    httpPut.addHeader("skip_header", "1");
    httpPut.addHeader("enclose", "\"");
    httpPut.addHeader("max_filter_ratio", "1");
    httpPut.addHeader("partial_update", "true");
    columns = new ArrayList<>(columns);
    columns.add("last_modified=current_timestamp");
    httpPut.addHeader(new BasicHeader("columns", String.join(",", columns)));

    FileEntity fileEntity = new FileEntity(filePath.toFile(), ContentType.create("multipart/form-data", StandardCharsets.UTF_8));
    httpPut.setEntity(fileEntity);

    try (CloseableHttpResponse response = client.execute(httpPut)) {
        if (response.getStatusLine().getStatusCode() == 200 && response.getEntity() != null) {
            Response resp = Json.fromJson(EntityUtils.toString(response.getEntity(), StandardCharsets.UTF_8), Response.class);
            log.info("调用stream load接口响应[{}]", Json.toJsonUnindent(resp));
            if (resp != null && "Success".equalsIgnoreCase(resp.getStatus())) {

// Files.deleteIfExists(filePath);
return;
}
}
throw new RuntimeException(String.format(“调用stream load接口异常,code:[%s],response:[%s]”,
response.getStatusLine().getStatusCode(), EntityUtils.toString(response.getEntity(), StandardCharsets.UTF_8)));
} catch (IOException e) {
log.error(“调用stream load接口异常,文件:[{}]”, filePath, e);
throw new RuntimeException(e);
}

Star_Cao · 2024年07月18日 09:22

curl 一下对应的ip和端口，测试一下是否都是正常的

wtai · 2024年07月18日 09:25

curl没问题，只是java代码调用的时候时而成功，时而失败

Star_Cao · 2024年07月18日 10:30

集群是升级到3.3.0版本的还是新安装部署的。如果是升级上来的，检查一下所有节点的版本。报错的时候有检查集群的所有节点状态吗？

wtai · 2024年07月18日 10:32

新部署的，集群都正常，用standalone试了一下也会有这种问题。
新的发现，每次调用时都新创建httpClient就不会有问题，复用httpClient必现

lvlouisaslia · 2024年09月5日 01:25

@wtai 有最小复现测试代码集吗? 这个应该是httpClient里有redirect时, 复用tcp连接, 有可能数据没有正常清理干净.