示例:
规则-字段对照表(节选)
规则编号 | 目标字段 | 转换类型 | 脚本片段 | 校验SQL
R01 | phone | 格式标准化 | regexp_replace(phone,'[^0-9]','') | SELECT COUNT(*) FROM ods_user WHERE phone RLIKE '[^0-9]'
R02 | gender | 枚举映射 | CASE WHEN gender IN ('男','M') THEN '1' WHEN gender IN ('女','F') THEN '2' ELSE '9' END | SELECT gender,COUNT(*) FROM ods_user GROUP BY gender
R03 | amount | 单位换算 | amount*100 | SELECT COUNT(*) FROM ods_order WHERE amount IS NULL
R04 | user_id | 去重 | row_number() over(partition by user_id order by update_time desc)=1 | SELECT COUNT(*)-COUNT(DISTINCT user_id) FROM ods_user
清洗脚本草稿(示意)
INSERT OVERWRITE TABLE dwd_user PARTITION(dt='${dt}')
SELECT user_id, regexp_replace(phone,'[^0-9]','') AS phone,
CASE WHEN gender IN ('男','M') THEN '1' WHEN gender IN ('女','F') THEN '2' ELSE '9' END AS gender,
amount*100 AS amount_cent, update_time
FROM (SELECT *, row_number() over(partition by user_id order by update_time desc) rn FROM ods_user WHERE dt='${dt}') t
WHERE rn=1;
回刷后校验(示意)
规则命中行数:R01=3200,R02=850,R03=1200,R04=46
清洗前后空值率:phone 6.2%→0.8%,gender 3.1%→0.4%
去重前后行数:100000→99954,差46行