示例成品 · 平台演示，按左边这组点选真跑出来的
直接按这套走。先跑自动全量，再人工抽 20-50 条，别一上来全看。
## 1. 文件结构
```
你的项目/
├── config.yaml
├── data/
│ └── cases.jsonl
├── results/
│ ├── result.jsonl
│ ├── report.json
│ └── human_eval_template.csv
├── run_eval.py
└── requirements.txt
```
---
## 2. 依赖安装
```bash
pip install openai pyyaml
```
---
## 3. 配置文件 `config.yaml`
```yaml
model:
# 如果自己用 vLLM 部署，base_url 填 vLLM 的 OpenAI 兼容地址
base_url: "http://localhost:8000/v1"
api_key: "EMPTY"
model_name: "Qwen-72B-Chat"
temperature: 0
max_tokens: 512
timeout: 60
data:
test_file: "data/cases.jsonl"
output_dir: "results"
# 0 表示全部跑完；如果只想先跑 50 条，写 50
max_cases: 0
tasks:
classification:
labels: ["退货", "换货", "维修", "投诉", "咨询", "物流催单", "其他"]
required_fields: ["category", "confidence", "reason"]
reply_generation:
required_fields: ["reply", "action", "category"]
min_reply_len: 5
max_reply_len: 200
eval:
human_sample_size: 20
```
如果你用阿里云 DashScope，不用自己部署，改这几行：
```yaml
model:
base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1"
api_key: "sk-你的key"
model_name: "qwen2.5-72b-instruct"
```
---
## 4. 测试集格式 `data/cases.jsonl`
每行一个 JSON。`expected` 是你人工标注的标准答案，给自动打分用。
### 分类 Case
字段说明：
- `task`: `classification`
- `input`: 原始工单文本
- `labels`: 可选，这个 Case 的人工标签
- `expected.category`: 标准分类
- `required_fields`: 这条 Case 要求模型必须输出的字段
```jsonl
{"id":"cls_001","task":"classification","input":"用户：我买的手机屏幕有问题，用了三天就黑屏了，我要退货","labels":["退货"],"expected":{"category":"退货"},"required_fields":["category","confidence","reason"]}
{"id":"cls_002","task":"classification","input":"用户：快递显示签收但我没收到，你们怎么回事","labels":["物流催单"],"expected":{"category":"物流催单"},"required_fields":["category","confidence","reason"]}
{"id":"cls_003","task":"classification","input":"用户：刚收到货，打开发现外壳有划痕，能换吗","labels":["换货"],"expected":{"category":"换货"},"required_fields":["category","confidence","reason"]}
```
### 回复生成 Case
`expected.must_contain`：回复中必须覆盖的业务关键点。
`expected.must_not_contain`：红线词或不能对外承诺的话。
```jsonl
{"id":"gen_001","task":"reply_generation","input":{"category":"退货","user_message":"我买的手机屏幕有问题，用了三天就黑屏了，我要退货"},"expected":{"category":"退货","must_contain":["退货","寄回","检测"],"must_not_contain":["一定可以","保证赔偿"]},"required_fields":["reply","action","category"]}
{"id":"gen_002","task":"reply_generation","input":{"category":"物流催单","user_message":"快递显示签收但我没收到，你们怎么回事"},"expected":{"category":"物流催单","must_contain":["核实","物流","重新派送"],"must_not_contain":["一定今天到"]},"required_fields":["reply","action","category"]}
{"id":"gen_003","task":"reply_generation","input":{"category":"换货","user_message":"刚收到货，打开发现外壳有划痕，能换吗"},"expected":{"category":"换货","must_contain":["换货","凭证","寄回"],"must_not_contain":["赔偿现金"]},"required_fields":["reply","action","category"]}
```
---
## 5. Qwen-72B 部署/接入
不要用 base 版，要用 Chat/Instruct 版，否则指令遵循会很差。
### 自己部署推荐 vLLM
72B BF16 权重大约 145GB，建议 4 张 80G 或更好。显存不够就换 AWQ/GPTQ 量化版。
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m vllm.entrypoints.openai.api_server \
--model /export/models/Qwen-72B-Chat \
--tensor-parallel-size 4 \
--max-model-len 8192 \
--gpu-memory-utilization 0.90 \
--port 8000
```
启动后，`config.yaml` 里的 `base_url` 就是：
```yaml
base_url: "http://localhost:8000/v1"
api_key: "EMPTY"
model_name: "Qwen-72B-Chat"
```
---
## 6. 自动评测脚本 `run_eval.py`
下面脚本直接复制使用。它做三件事：
1. 读取 `cases.jsonl`
2. 逐条调 Qwen-72B
3. 自动打分并生成报告 + 人工抽检 CSV
```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Qwen-72B 售后工单分类/回复生成评测脚本
用法：
python run_eval.py --config config.yaml
python r
点左边「开工 · 直接出成品」，出一份你自己的版本（文字免费）