3.2.16版本报 no delete vector found tablet

💬 StarRocks 用户问答 功能使用相关
#1

为了更快的定位您的问题，请提供以下信息，谢谢
【详述】问题详细描述
问题出现的现象 主键表的tablet delete_vector(标记删除信息的内部属性)丢失 丢失的分区是4月20的分区

问题出现的时间刚好和升级的时间重合 不能判断是否是升级引起的

从日志分析tablet异常的流程

18日-19日之间进行了一次tablet clone(内部tablet的负载均衡) 15号也有一次tablet clone

两次clone动作中都有一个异常的tablet，clone的流程分析

  1. node01节点需要从node00节点克隆tablet

  2. node00节点基于自身的tablet生成snapshot

  3. node01请求node00节点的snapshot复制到当前节点的复制目录

  4. node01从snapshot读取生成tablet完成clone

异常的日志每次打印的tablet的id都不一样 后续检查的时候发现异常的tablet都是单副本

clone日志 从日志分析应该是单副本的tablet在进行clone的时候执行了查询操作，因为只有单副本导致读取的这个tablet有问题

image
image1914×426 152 KB

show create table 副本数是2 实际副本数是1

CREATE TABLE `ods_api_jky_erp_storage_goodsdocout_v2_info_f` (
  `dt` datetime NOT NULL COMMENT "日分区",
  `__id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT "__id",
  `recId` varchar(1048576) NULL COMMENT "出库单ID",
  `goodsdocNo` varchar(1048576) NULL COMMENT "出库单号",
  `billNo` varchar(1048576) NULL COMMENT "上游单据号(关联单号)",
  `inOutDate` varchar(1048576) NULL COMMENT "出库时间",
  `gmtCreate` varchar(1048576) NULL COMMENT "创建时间",
  `inouttype` varchar(1048576) NULL COMMENT "出入库类型(201-销售出库 202调拨出库 204-其他出库 205采购退货 206生产领料 207组装拆卸出库 208翻新出库 209报废出库 210残次品出库 211倒冲领料 212 包材出库 215维修还厂 299可用库存修正 231成本调整出库)",
  `inouttypeName` varchar(1048576) NULL COMMENT "出入库类型(201-销售出库 202调拨出库 204-其他出库 205采购退货 206生产领料 207组装拆卸出库 208翻新出库 209报废出库 210残次品出库 211倒冲领料 212 包材出库 215维修还厂 299可用库存修正 231成本调整出库)",
  `vendCustomerName` varchar(1048576) NULL COMMENT "往来单位名称",
  `currencyCode` varchar(1048576) NULL COMMENT "币种编号",
  `currencyRate` varchar(1048576) NULL COMMENT "币种汇率",
  `userName` varchar(1048576) NULL COMMENT "业务员名字",
  `warehouseCode` varchar(1048576) NULL COMMENT "仓库编号",
  `warehouseName` varchar(1048576) NULL COMMENT "仓库名字",
  `comment` varchar(1048576) NULL COMMENT "备注",
  `memo` varchar(1048576) NULL COMMENT "收货备注",
  `logisticName` varchar(1048576) NULL COMMENT "物流公司",
  `logisticNo` varchar(1048576) NULL COMMENT "物流单号",
  `companyId` varchar(1048576) NULL COMMENT "公司id",
  `companyName` varchar(1048576) NULL COMMENT "公司名称",
  `logisticType` varchar(1048576) NULL COMMENT "物流类型1-普通快递 2-上门自提 3-门店配送5-无需配送 6-线下配送",
  `logisticCode` varchar(1048576) NULL COMMENT "物流公司编码",
  `inOutReason` varchar(1048576) NULL COMMENT "出库原因",
  `sourceBillNo` varchar(1048576) NULL COMMENT "来源单号",
  `channelId` varchar(1048576) NULL COMMENT "渠道ID",
  `channelCode` varchar(1048576) NULL COMMENT "渠道编号",
  `channelName` varchar(1048576) NULL COMMENT "渠道名称",
  `redStatus` varchar(1048576) NULL COMMENT "红冲状态：1蓝单（未红冲）；2被冲蓝单；9红冲单",
  `field1` varchar(1048576) NULL COMMENT "自定义字段1",
  `field2` varchar(1048576) NULL COMMENT "自定义字段2",
  `field3` varchar(1048576) NULL COMMENT "自定义字段3",
  `field4` varchar(1048576) NULL COMMENT "自定义字段4",
  `field5` varchar(1048576) NULL COMMENT "自定义字段5",
  `financeBillStatus` varchar(1048576) NULL COMMENT "财务状态：1：未成本计算；11部分成本计算；21：已成本计算；25已财务记账；31不成本计算",
  `vendCustomerId` varchar(1048576) NULL COMMENT "往来单位的id",
  `applyDepartId` varchar(1048576) NULL COMMENT "申请人部门id",
  `applyDepartName` varchar(1048576) NULL COMMENT "申请人部门名称",
  `outBillNo` varchar(1048576) NULL COMMENT "外部单号",
  `logisticList` varchar(1048576) NULL COMMENT "物流信息",
  `id` varchar(1048576) NULL COMMENT "-",
  `docNo` varchar(1048576) NULL COMMENT "出入库单号",
  `gmtModified` varchar(1048576) NULL COMMENT "最近修改时间",
  `departCode` varchar(1048576) NULL COMMENT "申请人部门编号",
  `applyCompanyName` varchar(1048576) NULL COMMENT "申请人公司名称",
  `applyCompanyCode` varchar(1048576) NULL COMMENT "申请人公司编号",
  `applyCompanyId` varchar(1048576) NULL COMMENT "申请人公司id",
  `applyUserId` varchar(1048576) NULL COMMENT "申请人id",
  `applyUserName` varchar(1048576) NULL COMMENT "申请人名称",
  `companyCode` varchar(1048576) NULL COMMENT "公司编号",
  `goodsDocDetailList` varchar(1048576) NULL COMMENT "���库单详情",
  `goodsId` varchar(1048576) NULL COMMENT "货品ID",
  `goodsNo` varchar(1048576) NULL COMMENT "货品编号",
  `goodsName` varchar(1048576) NULL COMMENT "货品名称",
  `skuId` varchar(1048576) NULL COMMENT "规格ID",
  `skuName` varchar(1048576) NULL COMMENT "规格名称",
  `skuBarcode` varchar(1048576) NULL COMMENT "条码",
  `unitName` varchar(1048576) NULL COMMENT "单位",
  `cuPrice` varchar(1048576) NULL COMMENT "成本单价",
  `cuValue` varchar(1048576) NULL COMMENT "成本金额",
  `caseNumber` varchar(1048576) NULL COMMENT "箱号",
  `rowRemark` varchar(1048576) NULL COMMENT "明细备注",
  `assistUnit` varchar(1048576) NULL COMMENT "辅助单位显示",
  `estCost` varchar(1048576) NULL COMMENT "本位币含税金额",
  `estTax` varchar(1048576) NULL COMMENT "税额",
  `estPrice` varchar(1048576) NULL COMMENT "含税单价",
  `estPriceNoTax` varchar(1048576) NULL COMMENT "无税单价",
  `estCostNoTax` varchar(1048576) NULL COMMENT "无税金额",
  `taxRate` varchar(1048576) NULL COMMENT "税率",
  `batchNo` varchar(1048576) NULL COMMENT "批次编号",
  `serialNo` varchar(1048576) NULL COMMENT "序列号",
  `productionDate` varchar(1048576) NULL COMMENT "生产日期",
  `expirationDate` varchar(1048576) NULL COMMENT "到期日期",
  `shelfLife` varchar(1048576) NULL COMMENT "质保期",
  `shelfLiftUnit` varchar(1048576) NULL COMMENT "质保期单位",
  `sourceDetailId` varchar(1048576) NULL COMMENT "来源单据明细Id",
  `transNoTaxPrice` varchar(1048576) NULL COMMENT "无税单价（原币种）",
  `transNoTaxAmount` varchar(1048576) NULL COMMENT "无税金额（原币种）",
  `transHasTaxPrice` varchar(1048576) NULL COMMENT "含税单价（原币种）",
  `transHasTaxAmount` varchar(1048576) NULL COMMENT "含税金额（原币种）",
  `transTaxAmount` varchar(1048576) NULL COMMENT "税额（原币种）",
  `detailField1` varchar(1048576) NULL COMMENT "自定义字段1",
  `detailField2` varchar(1048576) NULL COMMENT "自定义字段2",
  `detailField3` varchar(1048576) NULL COMMENT "自定义字段3",
  `detailField4` varchar(1048576) NULL COMMENT "自定义字段4",
  `detailField5` varchar(1048576) NULL COMMENT "自定义字段5",
  `isCertified` varchar(1048576) NULL COMMENT "是否正品 0-否 1-是",
  `registrationNumber` varchar(1048576) NULL COMMENT "批准文号",
  `approvalDate` varchar(1048576) NULL COMMENT "批文有效期 ",
  `productionDepart` varchar(1048576) NULL COMMENT "生产单位 ",
  `batchNumber` varchar(1048576) NULL COMMENT "生产批号",
  `quantity` varchar(1048576) NULL COMMENT "数量",
  `cateId` varchar(1048576) NULL COMMENT "分类id",
  `cateName` varchar(1048576) NULL COMMENT "分类名称",
  `volume` varchar(1048576) NULL COMMENT "体积（m3）",
  `skuWeight` varchar(1048576) NULL COMMENT "重量（kg）",
  `detailField6` varchar(1048576) NULL COMMENT "自定义字段6",
  `detailField7` varchar(1048576) NULL COMMENT "自定义字段7",
  `detailField8` varchar(1048576) NULL COMMENT "自定义字段8",
  `detailField9` varchar(1048576) NULL COMMENT "自定义字段9",
  `detailField10` varchar(1048576) NULL COMMENT "自定义字段10",
  `sendCompanyName` varchar(1048576) NULL COMMENT "发件公司名称",
  `send` varchar(1048576) NULL COMMENT "发件人",
  `sendTel` varchar(1048576) NULL COMMENT "发件人电话",
  `sendPhone` varchar(1048576) NULL COMMENT "发件人手机号",
  `sendEmail` varchar(1048576) NULL COMMENT "发件人邮箱",
  `sendCountryName` varchar(1048576) NULL COMMENT "发件人国家",
  `sendProvinceName` varchar(1048576) NULL COMMENT "发件人省",
  `sendCityName` varchar(1048576) NULL COMMENT "发件人市",
  `sendTownName` varchar(1048576) NULL COMMENT "发件人镇，区",
  `sendStreetName` varchar(1048576) NULL COMMENT "发件人街道",
  `sendAddress` varchar(1048576) NULL COMMENT "发件人详细地址",
  `receiveCompanyName` varchar(1048576) NULL COMMENT "收件公司名称",
  `receive` varchar(1048576) NULL COMMENT "收件人",
  `receiveTel` varchar(1048576) NULL COMMENT "收件人电话",
  `receivePhone` varchar(1048576) NULL COMMENT "收件人手机号",
  `receiveEmail` varchar(1048576) NULL COMMENT "收件人邮箱",
  `receiveCountryName` varchar(1048576) NULL COMMENT "收件人国家",
  `receiveProvinceName` varchar(1048576) NULL COMMENT "收件人省",
  `receiveCityName` varchar(1048576) NULL COMMENT "收件人市",
  `receiveTownName` varchar(1048576) NULL COMMENT "收件人镇，区",
  `receiveStreetName` varchar(1048576) NULL COMMENT "收件人街道",
  `receiveAddress` varchar(1048576) NULL COMMENT "收件人详细地址",
  `field6` varchar(1048576) NULL COMMENT "自定义字段6",
  `field7` varchar(1048576) NULL COMMENT "自定义字段7",
  `field8` varchar(1048576) NULL COMMENT "自定义字段8",
  `field9` varchar(1048576) NULL COMMENT "自定义字段9",
  `field10` varchar(1048576) NULL COMMENT "自定义字段10",
  `origin_data` varchar(1048576) NULL COMMENT "原始数据",
  `yuce_cube_shop_id` varchar(1048576) NULL COMMENT "魔方店铺标识"
) ENGINE=OLAP 
PRIMARY KEY(`dt`, `__id`)
COMMENT "ODS_分页查询出库单(吉客云标准API)"
PARTITION BY date_trunc('day', dt)
DISTRIBUTED BY HASH(`dt`, `__id`)
PROPERTIES (
"replication_num" = "2",
"in_memory" = "false",
"enable_persistent_index" = "true",
"replicated_storage" = "true",
"fast_schema_evolution" = "true",
"compression" = "LZ4"
);

image
image5120×642 226 KB

be日志: 只有node-00节点有warn日志
43058498.be.node-00.INFO (151.6 KB) 43058498.be.node-00.WARN (625 字节) 43058498.be.node-01.INFO (146.9 KB) 43058498.be.node-02.INFO (32.8 KB)

image
image1920×1055 300 KB

image
image772×953 64.4 KB

【背景】做过哪些操作？
从3.2.13升级到3.2.16
【业务影响】
【是否存算分离】
【StarRocks版本】例如：3.2.16
【集群规模】例如：3fe（3follower+0observer）+3be（fe与be混部）

【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息，请补充下您的联系方式，例如：社区群16-可乐鸡或者邮箱，谢谢
【附件】

  • fe.log/beINFO/相应截图
  • 慢查询：
    • Profile信息
    • 并行度：show variables like ‘%parallel_fragment_exec_instance_num%’;
    • pipeline是否开启：show variables like ‘%pipeline%’;
    • be节点cpu和内存使用率截图
  • 查询报错：
  • be crash
    • be.out
  • 外表查询报错
    • be.out和fe.warn.log

京ICP备2022026421号-1