mirror of
https://gitee.com/houhuan/TrendRadar.git
synced 2025-12-21 08:17:17 +08:00
v3.0.0 AI 智能分析功能
This commit is contained in:
sansan
parent
da81d69309
commit
2afc24e6fb
2
.github/workflows/crawler.yml
vendored
2
.github/workflows/crawler.yml
vendored
@ -21,7 +21,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: "3.9"
|
||||
python-version: "3.10"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
|
||||
144
README-Cherry-Studio.md
Normal file
144
README-Cherry-Studio.md
Normal file
@ -0,0 +1,144 @@
|
||||
# TrendRadar × Cherry Studio 部署指南 🍒
|
||||
|
||||
> **适合人群**：零编程基础的用户
|
||||
> **客户端**：Cherry Studio（免费开源 GUI 客户端）
|
||||
|
||||
---
|
||||
|
||||
## 📥 第一步：下载 Cherry Studio
|
||||
|
||||
### Windows 用户
|
||||
|
||||
访问官网下载：https://cherry-ai.com/
|
||||
或直接下载：[Cherry-Studio-Windows.exe](https://github.com/kangfenmao/cherry-studio/releases/latest)
|
||||
|
||||
### Mac 用户
|
||||
|
||||
访问官网下载：https://cherry-ai.com/
|
||||
或直接下载：[Cherry-Studio-Mac.dmg](https://github.com/kangfenmao/cherry-studio/releases/latest)
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 📦 第二步：获取项目代码
|
||||
|
||||
为什么需要获取项目代码？
|
||||
|
||||
AI 分析功能需要读取项目中的新闻数据才能工作。无论你使用 GitHub Actions 还是 Docker 部署，爬虫生成的新闻数据都保存在项目的 output 目录中。因此，在配置 MCP 服务器之前，需要先获取完整的项目代码（包含数据文件）。
|
||||
|
||||
根据你的技术水平，可以选择以下任一方式获取：：
|
||||
|
||||
### 方法一：Git Clone（推荐给技术用户）
|
||||
|
||||
如果你熟悉 Git，可以使用以下命令克隆项目：
|
||||
|
||||
```bash
|
||||
git clone https://github.com/你的用户名/你的项目名.git
|
||||
cd 你的项目名
|
||||
```
|
||||
|
||||
**优点**：
|
||||
|
||||
- 可以随时拉取一个命令就可以更新最新数据到本地了（`git pull`）
|
||||
|
||||
### 方法二：直接下载 ZIP 压缩包（推荐给初学者）
|
||||
|
||||
|
||||
1. **访问 GitHub 项目页面**
|
||||
|
||||
- 项目链接：`https://github.com/你的用户名/你的项目名`
|
||||
|
||||
2. **下载压缩包**
|
||||
|
||||
- 点击绿色的 "Code" 按钮
|
||||
- 选择 "Download ZIP"
|
||||
- 或直接访问：`https://github.com/你的用户名/你的项目名/archive/refs/heads/master.zip`
|
||||
|
||||
|
||||
**注意事项**：
|
||||
|
||||
- 步骤稍微麻烦，后续更新数据需要重复上面步骤，然后覆盖本地数据(output 目录)
|
||||
|
||||
---
|
||||
|
||||
## 🚀 第三步：一键部署 MCP 服务器
|
||||
|
||||
### Windows 用户
|
||||
|
||||
1. **双击运行**项目文件夹中的 `setup-windows.bat`
|
||||
2. **等待安装完成**
|
||||
3. **记录显示的配置信息**（命令路径和参数）
|
||||
|
||||
### Mac 用户
|
||||
|
||||
1. **打开终端**（在启动台搜索"终端"）
|
||||
2. **拖拽**项目文件夹中的 `setup-mac.sh` 到终端窗口
|
||||
3. **按回车键**
|
||||
4. **记录显示的配置信息**
|
||||
|
||||
---
|
||||
|
||||
## 🔧 第四步：配置 Cherry Studio
|
||||
|
||||
### 1. 打开设置
|
||||
|
||||
启动 Cherry Studio，点击右上角 ⚙️ **设置** 按钮
|
||||
|
||||
### 2. 添加 MCP 服务器
|
||||
|
||||
在设置页面找到：**MCP** → 点击 **添加**
|
||||
|
||||
### 3. 填写配置（重要！）
|
||||
|
||||
根据刚才的安装脚本显示的信息填写
|
||||
|
||||
### 4. 保存并启用
|
||||
|
||||
- 点击 **保存** 按钮
|
||||
- 确保 MCP 服务器列表中的开关是 **开启** 状态 ✅
|
||||
|
||||
---
|
||||
|
||||
## ✅ 第五步：验证是否成功
|
||||
|
||||
### 1. 测试连接
|
||||
|
||||
在 Cherry Studio 的对话框中输入：
|
||||
|
||||
```
|
||||
帮我爬取最新的新闻
|
||||
```
|
||||
|
||||
### 2. 成功标志
|
||||
|
||||
如果配置成功，AI 会：
|
||||
|
||||
- ✅ 调用 TrendRadar 工具
|
||||
- ✅ 返回真实的新闻数据
|
||||
- ✅ 显示平台、标题、排名等信息
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 🎯 进阶配置
|
||||
|
||||
### HTTP 模式（可选）
|
||||
|
||||
如果需要远程访问或多客户端共享，可以使用 HTTP 模式：
|
||||
|
||||
#### Windows
|
||||
|
||||
双击运行 `start-http.bat`
|
||||
|
||||
#### Mac
|
||||
|
||||
```bash
|
||||
./start-http.sh
|
||||
```
|
||||
|
||||
然后在 Cherry Studio 中配置：
|
||||
|
||||
```
|
||||
类型: streamableHttp
|
||||
URL: http://localhost:3333/mcp
|
||||
```
|
||||
442
README-MCP-FAQ.md
Normal file
442
README-MCP-FAQ.md
Normal file
@ -0,0 +1,442 @@
|
||||
# TrendRadar MCP 工具使用问答
|
||||
|
||||
> AI 提问指南 - 如何通过对话使用新闻热点分析工具
|
||||
|
||||
## ⚙️ 默认设置说明（重要！）
|
||||
|
||||
默认采用以下优化策略，主要是为了节约 AI token 消耗：
|
||||
|
||||
| 默认设置 | 说明 | 如何调整 |
|
||||
| -------------- | --------------------------------------- | ------------------------------------- |
|
||||
| **限制条数** | 默认返回 50 条新闻 | 对话中说"返回前 10 条"或"给我 100 条" |
|
||||
| **时间范围** | 默认查询今天的数据 | 说"查询昨天"或"最近一周" |
|
||||
| **URL 链接** | 默认不返回链接（节省约 160 tokens/条） | 说"需要链接"或"包含 URL" |
|
||||
| **关键词列表** | 默认不使用 frequency_words.txt 过滤新闻 | 只有调用"趋势话题"工具时才使用 |
|
||||
|
||||
**⚠️ 重要：** AI 模型的选择直接影响工具调用效果，AI 越智能，调用越准确。当你解除上面的限制，比如从今天的查询，放宽到一周的查询，首先你要在本地有一周的数据，其次，token 消耗量可能会倍增（为什么说可能，比如我查询 分析'苹果'最近一周的热度趋势，如果一周中没多少苹果的新闻，那么 token消耗量可能反而很少）
|
||||
|
||||
|
||||
## 💰 AI 模型
|
||||
|
||||
下面我以 **[硅基流动](https://cloud.siliconflow.cn)** 平台作为例子，里面有很多大模型可选择。在开发和测试本项目的过程中，我使用本平台进行了许多的功能测试和验证。
|
||||
|
||||
### 📊 注册方式对比
|
||||
|
||||
| 注册方式 | 无邀请链接直接注册 | 含有邀请链接注册 |
|
||||
|:-------:|:-------:|:-----------------:|
|
||||
| 注册链接 | [siliconflow.cn](https://cloud.siliconflow.cn) | [邀请链接](https://cloud.siliconflow.cn/i/fqnyVaIU) |
|
||||
| 免费额度 | 0 tokens | **2000万 tokens** (≈14元) |
|
||||
| 额外福利 | ❌ | ✅ 邀请者也获得2000万tokens |
|
||||
|
||||
> 💡 **提示**：上面的赠送额度，应该可以询问 **200次以上**
|
||||
|
||||
|
||||
### 🚀 快速开始
|
||||
|
||||
#### 1️⃣ 注册并获取 API 密钥
|
||||
|
||||
1. 使用上方链接完成注册
|
||||
2. 访问 [API 密钥管理页面](https://cloud.siliconflow.cn/me/account/ak)
|
||||
3. 点击「新建 API 密钥」
|
||||
4. 复制生成的密钥（请妥善保管）
|
||||
|
||||
#### 2️⃣ 在 Cherry Studio 中配置
|
||||
|
||||
1. 打开 **Cherry Studio**
|
||||
2. 进入「模型服务」设置
|
||||
3. 找到「硅基流动」
|
||||
4. 将复制的密钥粘贴到 **[API密钥]** 输入框
|
||||
5. 确保右上角勾选框打开后显示为 **绿色** ✅
|
||||
|
||||
---
|
||||
|
||||
### ✨ 配置完成！
|
||||
|
||||
现在你可以开始使用本项目，享受稳定快速的 AI 服务了！
|
||||
|
||||
在你测试一次询问后，请立刻去 [硅基流动账单](https://cloud.siliconflow.cn/me/bills) 查询这一次的消耗量，心底有个估算。
|
||||
|
||||
|
||||
## 基础查询
|
||||
|
||||
### Q1: 如何查看最新的新闻？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "给我看看最新的新闻"
|
||||
- "查询今天的热点新闻"
|
||||
- "获取知乎和微博的最新 10 条新闻"
|
||||
- "查看最新新闻，需要包含链接"
|
||||
|
||||
**调用的工具：** `get_latest_news`
|
||||
|
||||
**工具返回行为：**
|
||||
|
||||
- MCP 工具会返回所有平台的最新 50 条新闻给 AI
|
||||
- 不包含 URL 链接（节省 token）
|
||||
|
||||
**AI 展示行为（重要）：**
|
||||
|
||||
- ⚠️ **AI 通常会自动总结**，只展示部分新闻（如 TOP 10-20 条）
|
||||
- ✅ 如果你想看全部 50 条，需要明确要求："展示所有新闻"或"完整列出所有 50 条"
|
||||
- 💡 这是 AI 模型的自然行为，不是工具的限制
|
||||
|
||||
**可以调整：**
|
||||
|
||||
- 指定平台：如"只看知乎的"
|
||||
- 调整数量：如"返回前 20 条"
|
||||
- 包含链接：如"需要链接"
|
||||
- **要求完整展示**：如"展示全部，不要总结"
|
||||
|
||||
---
|
||||
|
||||
### Q2: 如何查询特定日期的新闻？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "查询昨天的新闻"
|
||||
- "看看 3 天前知乎的新闻"
|
||||
- "2025-10-10 的新闻有哪些"
|
||||
- "上周一的新闻"
|
||||
- "给我看看最新新闻"（自动查询今天）
|
||||
|
||||
**调用的工具：** `get_news_by_date`
|
||||
|
||||
**支持的日期格式：**
|
||||
|
||||
- 相对日期：今天、昨天、前天、3 天前
|
||||
- 星期：上周一、本周三、last monday
|
||||
- 绝对日期：2025-10-10、10 月 10 日
|
||||
|
||||
**工具返回行为：**
|
||||
|
||||
- 不指定日期时自动查询今天（节省 token）
|
||||
- MCP 工具会返回所有平台的 50 条新闻给 AI
|
||||
- 不包含 URL 链接
|
||||
|
||||
**AI 展示行为（重要）：**
|
||||
|
||||
- ⚠️ **AI 通常会自动总结**，只展示部分新闻（如 TOP 10-20 条）
|
||||
- ✅ 如果你想看全部，需要明确要求："展示所有新闻，不要总结"
|
||||
|
||||
---
|
||||
|
||||
### Q3: 如何查看我关注的话题频率统计？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "我关注的词今天出现了多少次"
|
||||
- "看看我的关注词列表中哪些词最热门"
|
||||
- "统计一下 frequency_words.txt 中的关注词频率"
|
||||
|
||||
**调用的工具：** `get_trending_topics`
|
||||
|
||||
**重要说明：**
|
||||
|
||||
- 本工具**不是**自动提取新闻热点
|
||||
- 而是统计你在 `config/frequency_words.txt` 中设置的**个人关注词**
|
||||
- 这是一个**可自定义**的列表，你可以根据兴趣添加关注词
|
||||
|
||||
---
|
||||
|
||||
## 搜索检索
|
||||
|
||||
### Q4: 如何搜索包含特定关键词的新闻？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "搜索包含'人工智能'的新闻"
|
||||
- "查找关于'特斯拉降价'的报道"
|
||||
- "搜索马斯克相关的新闻，返回前 20 条"
|
||||
- "查找'iPhone 16 发布'这条新闻的链接"
|
||||
|
||||
**调用的工具：** `search_news`
|
||||
|
||||
**工具返回行为：**
|
||||
|
||||
- 使用关键词模式搜索
|
||||
- 搜索今天的数据
|
||||
- MCP 工具会返回最多 50 条结果给 AI
|
||||
- 不包含 URL 链接
|
||||
|
||||
**AI 展示行为（重要）：**
|
||||
|
||||
- ⚠️ **AI 通常会自动总结**，只展示部分搜索结果
|
||||
- ✅ 如果你想看全部，需要明确要求："展示所有搜索结果"
|
||||
|
||||
**可以调整：**
|
||||
|
||||
- 指定时间范围：如"搜索最近一周的"
|
||||
- 指定平台：如"只搜索知乎"
|
||||
- 调整排序：如"按权重排序"
|
||||
- 包含链接：如"需要链接"
|
||||
|
||||
---
|
||||
|
||||
### Q5: 如何查找历史相关新闻？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "查找昨天与'人工智能突破'相关的新闻"
|
||||
- "搜索上周关于'特斯拉'的历史报道"
|
||||
- "找出上个月与'ChatGPT'相关的新闻"
|
||||
- "看看'iPhone 发布会'相关的历史新闻"
|
||||
|
||||
**调用的工具：** `search_related_news_history`
|
||||
|
||||
**工具返回行为：**
|
||||
|
||||
- 搜索昨天的数据
|
||||
- 相似度阈值 0.4
|
||||
- MCP 工具会返回最多 50 条结果给 AI
|
||||
- 不包含 URL 链接
|
||||
|
||||
**AI 展示行为（重要）：**
|
||||
|
||||
- ⚠️ **AI 通常会自动总结**，只展示部分相关新闻
|
||||
- ✅ 如果你想看全部，需要明确要求："展示所有相关新闻"
|
||||
|
||||
---
|
||||
|
||||
## 趋势分析
|
||||
|
||||
### Q6: 如何分析话题的热度趋势？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "分析'人工智能'最近一周的热度趋势"
|
||||
- "看看'特斯拉'话题是昙花一现还是持续热点"
|
||||
- "检测今天有哪些突然爆火的话题"
|
||||
- "预测接下来可能的热点话题"
|
||||
|
||||
**调用的工具：** `analyze_topic_trend`
|
||||
|
||||
**工具返回行为：**
|
||||
|
||||
- 热度趋势模式
|
||||
- 分析最近 7 天数据
|
||||
- 按天粒度统计
|
||||
|
||||
**AI 展示行为：**
|
||||
|
||||
- 通常会展示趋势分析结果和图表
|
||||
- AI 可能会总结关键发现
|
||||
|
||||
---
|
||||
|
||||
## 数据洞察
|
||||
|
||||
### Q7: 如何对比不同平台对话题的关注度？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "对比各个平台对'人工智能'话题的关注度"
|
||||
- "看看哪个平台更新最频繁"
|
||||
- "分析一下哪些关键词经常一起出现"
|
||||
|
||||
**调用的工具：** `analyze_data_insights`
|
||||
|
||||
**三种洞察模式：**
|
||||
|
||||
| 模式 | 功能 | 示例问法 |
|
||||
| -------------- | ---------------- | -------------------------- |
|
||||
| **平台对比** | 对比各平台关注度 | "对比各平台对'AI'的关注度" |
|
||||
| **活跃度统计** | 统计平台发布频率 | "看看哪个平台更新最频繁" |
|
||||
| **关键词共现** | 分析关键词关联 | "哪些关键词经常一起出现" |
|
||||
|
||||
**工具返回行为：**
|
||||
|
||||
- 平台对比模式
|
||||
- 分析今天的数据
|
||||
- 关键词共现最小频次 3 次
|
||||
|
||||
**AI 展示行为：**
|
||||
|
||||
- 通常会展示分析结果和统计数据
|
||||
- AI 可能会总结洞察发现
|
||||
|
||||
---
|
||||
|
||||
## 情感分析
|
||||
|
||||
### Q8: 如何分析新闻的情感倾向？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "分析一下今天新闻的情感倾向"
|
||||
- "看看'特斯拉'相关新闻是正面还是负面的"
|
||||
- "分析各平台对'人工智能'的情感态度"
|
||||
- "看看'比特币'一周内的情感倾向，选择前 20 条最重要的"
|
||||
|
||||
**调用的工具：** `analyze_sentiment`
|
||||
|
||||
**工具返回行为：**
|
||||
|
||||
- 分析今天的数据
|
||||
- MCP 工具会返回最多 50 条新闻给 AI
|
||||
- 按权重排序（优先展示重要新闻）
|
||||
- 不包含 URL 链接
|
||||
|
||||
**AI 展示行为（重要）：**
|
||||
|
||||
- ⚠️ 本工具返回 **AI 提示词**，不是直接的情感分析结果
|
||||
- AI 会根据提示词生成情感分析报告
|
||||
- 通常会展示情感分布、关键发现和代表性新闻
|
||||
|
||||
**可以调整：**
|
||||
|
||||
- 指定话题：如"关于'特斯拉'"
|
||||
- 指定时间：如"最近一周"
|
||||
- 调整数量：如"返回前 20 条"
|
||||
|
||||
---
|
||||
|
||||
### Q9: 如何查找相似的新闻报道？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "找出和'特斯拉降价'相似的新闻"
|
||||
- "查找关于 iPhone 发布的类似报道"
|
||||
- "看看有没有和这条新闻相似的报道"
|
||||
- "找相似新闻，需要链接"
|
||||
|
||||
**调用的工具：** `find_similar_news`
|
||||
|
||||
**工具返回行为：**
|
||||
|
||||
- 相似度阈值 0.6
|
||||
- MCP 工具会返回最多 50 条结果给 AI
|
||||
- 不包含 URL 链接
|
||||
|
||||
**AI 展示行为（重要）：**
|
||||
|
||||
- ⚠️ **AI 通常会自动总结**，只展示部分相似新闻
|
||||
- ✅ 如果你想看全部，需要明确要求："展示所有相似新闻"
|
||||
|
||||
---
|
||||
|
||||
### Q10: 如何生成每日或每周的热点摘要？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "生成今天的新闻摘要报告"
|
||||
- "给我一份本周的热点总结"
|
||||
- "生成过去 7 天的新闻分析报告"
|
||||
|
||||
**调用的工具：** `generate_summary_report`
|
||||
|
||||
**报告类型：**
|
||||
|
||||
- 每日摘要：总结当天的热点新闻
|
||||
- 每周摘要：总结一周的热点趋势
|
||||
|
||||
---
|
||||
|
||||
## 系统管理
|
||||
|
||||
### Q11: 如何查看系统配置？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "查看当前系统配置"
|
||||
- "显示配置文件内容"
|
||||
- "有哪些可用的平台？"
|
||||
- "当前的权重配置是什么？"
|
||||
|
||||
**调用的工具：** `get_current_config`
|
||||
|
||||
**可以查询：**
|
||||
|
||||
- 可用平台列表
|
||||
- 爬虫配置（请求间隔、超时设置）
|
||||
- 权重配置（排名权重、频次权重）
|
||||
- 通知配置（钉钉、微信）
|
||||
|
||||
---
|
||||
|
||||
### Q12: 如何检查系统运行状态？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "检查系统状态"
|
||||
- "系统运行正常吗？"
|
||||
- "最后一次爬取是什么时候？"
|
||||
- "有多少天的历史数据？"
|
||||
|
||||
**调用的工具：** `get_system_status`
|
||||
|
||||
**返回信息：**
|
||||
|
||||
- 系统版本和状态
|
||||
- 最后爬取时间
|
||||
- 历史数据天数
|
||||
- 健康检查结果
|
||||
|
||||
---
|
||||
|
||||
### Q13: 如何手动触发爬取任务？
|
||||
|
||||
**你可以这样问：**
|
||||
|
||||
- "请你爬取当前的今日头条的新闻"（临时查询）
|
||||
- "帮我抓取一下知乎和微博的最新新闻并保存"（持久化）
|
||||
- "触发一次爬取并保存数据"（持久化）
|
||||
- "获取 36 氪 的实时数据但不保存"（临时查询）
|
||||
|
||||
**调用的工具：** `trigger_crawl`
|
||||
|
||||
**两种模式：**
|
||||
|
||||
| 模式 | 用途 | 示例 |
|
||||
| -------------- | -------------------- | -------------------- |
|
||||
| **临时爬取** | 只返回数据不保存 | "爬取今日头条的新闻" |
|
||||
| **持久化爬取** | 保存到 output 文件夹 | "抓取并保存知乎新闻" |
|
||||
|
||||
**工具返回行为：**
|
||||
|
||||
- 临时爬取模式（不保存）
|
||||
- 爬取所有平台
|
||||
- 不包含 URL 链接
|
||||
|
||||
**AI 展示行为（重要）：**
|
||||
|
||||
- ⚠️ **AI 通常会总结爬取结果**，只展示部分新闻
|
||||
- ✅ 如果你想看全部，需要明确要求："展示所有爬取的新闻"
|
||||
|
||||
**可以调整：**
|
||||
|
||||
- 指定平台：如"只爬取知乎"
|
||||
- 保存数据：说"并保存"或"保存到本地"
|
||||
- 包含链接：说"需要链接"
|
||||
|
||||
---
|
||||
|
||||
## 💡 使用技巧
|
||||
|
||||
### 1. 如何让 AI 展示全部数据而不是自动总结？
|
||||
|
||||
**背景**: 有时 AI 会自动总结数据，只展示部分内容，即使工具返回了完整的 50 条数据。
|
||||
|
||||
**如果 AI 仍然总结，你可以**:
|
||||
|
||||
- **方法 1 - 明确要求**: "请展示全部新闻，不要总结"
|
||||
- **方法 2 - 指定数量**: "展示所有 50 条新闻"
|
||||
- **方法 3 - 质疑行为**: "为什么只展示了 15 条？我要看全部"
|
||||
- **方法 4 - 提前说明**: "查询今天的新闻，完整展示所有结果"
|
||||
|
||||
**注意**: AI 仍可能根据上下文调整展示方式。
|
||||
|
||||
|
||||
### 2. 如何组合使用多个工具？
|
||||
|
||||
**示例：深度分析某个话题**
|
||||
|
||||
1. 先搜索："搜索'人工智能'相关新闻"
|
||||
2. 再分析趋势："分析'人工智能'的热度趋势"
|
||||
3. 最后情感分析："分析'人工智能'新闻的情感倾向"
|
||||
|
||||
**示例：追踪某个事件**
|
||||
|
||||
1. 查看最新："查询今天关于'iPhone'的新闻"
|
||||
2. 查找历史："查找上周与'iPhone'相关的历史新闻"
|
||||
3. 找相似报道："找出和'iPhone 发布会'相似的新闻"
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 222 KiB After Width: | Height: | Size: 259 KiB |
@ -6,50 +6,55 @@ WORKDIR /app
|
||||
ARG TARGETARCH
|
||||
ENV SUPERCRONIC_VERSION=v0.2.34
|
||||
|
||||
# supercronic + locale
|
||||
RUN set -ex && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends curl ca-certificates && \
|
||||
apt-get install -y --no-install-recommends curl ca-certificates locales && \
|
||||
sed -i -e 's/# zh_CN.UTF-8 UTF-8/zh_CN.UTF-8 UTF-8/' /etc/locale.gen && \
|
||||
sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
|
||||
locale-gen && \
|
||||
# 根据架构选择并下载 supercronic
|
||||
case ${TARGETARCH} in \
|
||||
amd64) \
|
||||
export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-amd64; \
|
||||
export SUPERCRONIC_SHA1SUM=e8631edc1775000d119b70fd40339a7238eece14; \
|
||||
export SUPERCRONIC=supercronic-linux-amd64; \
|
||||
;; \
|
||||
export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-amd64; \
|
||||
export SUPERCRONIC_SHA1SUM=e8631edc1775000d119b70fd40339a7238eece14; \
|
||||
export SUPERCRONIC=supercronic-linux-amd64; \
|
||||
;; \
|
||||
arm64) \
|
||||
export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-arm64; \
|
||||
export SUPERCRONIC_SHA1SUM=4ab6343b52bf9da592e8b4bb7ae6eb5a8e21b71e; \
|
||||
export SUPERCRONIC=supercronic-linux-arm64; \
|
||||
;; \
|
||||
export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-arm64; \
|
||||
export SUPERCRONIC_SHA1SUM=4ab6343b52bf9da592e8b4bb7ae6eb5a8e21b71e; \
|
||||
export SUPERCRONIC=supercronic-linux-arm64; \
|
||||
;; \
|
||||
arm) \
|
||||
export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-arm; \
|
||||
export SUPERCRONIC_SHA1SUM=4ba4cd0da62082056b6def085fa9377d965fbe01; \
|
||||
export SUPERCRONIC=supercronic-linux-arm; \
|
||||
;; \
|
||||
export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-arm; \
|
||||
export SUPERCRONIC_SHA1SUM=4ba4cd0da62082056b6def085fa9377d965fbe01; \
|
||||
export SUPERCRONIC=supercronic-linux-arm; \
|
||||
;; \
|
||||
386) \
|
||||
export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-386; \
|
||||
export SUPERCRONIC_SHA1SUM=80b4fff03a8d7bf2f24a1771f37640337855e949; \
|
||||
export SUPERCRONIC=supercronic-linux-386; \
|
||||
;; \
|
||||
export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-386; \
|
||||
export SUPERCRONIC_SHA1SUM=80b4fff03a8d7bf2f24a1771f37640337855e949; \
|
||||
export SUPERCRONIC=supercronic-linux-386; \
|
||||
;; \
|
||||
*) \
|
||||
echo "Unsupported architecture: ${TARGETARCH}"; \
|
||||
exit 1; \
|
||||
;; \
|
||||
echo "Unsupported architecture: ${TARGETARCH}"; \
|
||||
exit 1; \
|
||||
;; \
|
||||
esac && \
|
||||
echo "Downloading supercronic for ${TARGETARCH} from ${SUPERCRONIC_URL}" && \
|
||||
# 添加重试机制和超时设置
|
||||
for i in 1 2 3 4 5; do \
|
||||
echo "Download attempt $i/5"; \
|
||||
if curl --fail --silent --show-error --location --retry 3 --retry-delay 2 --connect-timeout 30 --max-time 120 -o "$SUPERCRONIC" "$SUPERCRONIC_URL"; then \
|
||||
echo "Download successful"; \
|
||||
break; \
|
||||
else \
|
||||
echo "Download attempt $i failed, exit code: $?"; \
|
||||
if [ $i -eq 5 ]; then \
|
||||
echo "All download attempts failed"; \
|
||||
exit 1; \
|
||||
fi; \
|
||||
sleep $((i * 2)); \
|
||||
fi; \
|
||||
echo "Download attempt $i/5"; \
|
||||
if curl --fail --silent --show-error --location --retry 3 --retry-delay 2 --connect-timeout 30 --max-time 120 -o "$SUPERCRONIC" "$SUPERCRONIC_URL"; then \
|
||||
echo "Download successful"; \
|
||||
break; \
|
||||
else \
|
||||
echo "Download attempt $i failed, exit code: $?"; \
|
||||
if [ $i -eq 5 ]; then \
|
||||
echo "All download attempts failed"; \
|
||||
exit 1; \
|
||||
fi; \
|
||||
sleep $((i * 2)); \
|
||||
fi; \
|
||||
done && \
|
||||
echo "${SUPERCRONIC_SHA1SUM} ${SUPERCRONIC}" | sha1sum -c - && \
|
||||
chmod +x "$SUPERCRONIC" && \
|
||||
@ -57,6 +62,7 @@ RUN set -ex && \
|
||||
ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic && \
|
||||
# 验证安装
|
||||
supercronic -version && \
|
||||
# 清理（保留 locales，只删除 curl）
|
||||
apt-get remove -y curl && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
@ -77,6 +83,10 @@ RUN sed -i 's/\r$//' /entrypoint.sh.tmp && \
|
||||
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
CONFIG_PATH=/app/config/config.yaml \
|
||||
FREQUENCY_WORDS_PATH=/app/config/frequency_words.txt
|
||||
FREQUENCY_WORDS_PATH=/app/config/frequency_words.txt \
|
||||
LANG=zh_CN.UTF-8 \
|
||||
LANGUAGE=zh_CN:zh:en_US:en \
|
||||
LC_ALL=zh_CN.UTF-8 \
|
||||
PYTHONIOENCODING=utf-8
|
||||
|
||||
ENTRYPOINT ["/entrypoint.sh"]
|
||||
7
mcp_server/__init__.py
Normal file
7
mcp_server/__init__.py
Normal file
@ -0,0 +1,7 @@
|
||||
"""
|
||||
TrendRadar MCP Server
|
||||
|
||||
提供基于MCP协议的新闻聚合数据查询和系统管理接口。
|
||||
"""
|
||||
|
||||
__version__ = "1.0.0"
|
||||
657
mcp_server/server.py
Normal file
657
mcp_server/server.py
Normal file
@ -0,0 +1,657 @@
|
||||
"""
|
||||
TrendRadar MCP Server - FastMCP 2.0 实现
|
||||
|
||||
使用 FastMCP 2.0 提供生产级 MCP 工具服务器。
|
||||
支持 stdio 和 HTTP 两种传输模式。
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import List, Optional, Dict
|
||||
|
||||
from fastmcp import FastMCP
|
||||
|
||||
from .tools.data_query import DataQueryTools
|
||||
from .tools.analytics import AnalyticsTools
|
||||
from .tools.search_tools import SearchTools
|
||||
from .tools.config_mgmt import ConfigManagementTools
|
||||
from .tools.system import SystemManagementTools
|
||||
|
||||
|
||||
# 创建 FastMCP 2.0 应用
|
||||
mcp = FastMCP('trendradar-news')
|
||||
|
||||
# 全局工具实例（在第一次请求时初始化）
|
||||
_tools_instances = {}
|
||||
|
||||
|
||||
def _get_tools(project_root: Optional[str] = None):
|
||||
"""获取或创建工具实例（单例模式）"""
|
||||
if not _tools_instances:
|
||||
_tools_instances['data'] = DataQueryTools(project_root)
|
||||
_tools_instances['analytics'] = AnalyticsTools(project_root)
|
||||
_tools_instances['search'] = SearchTools(project_root)
|
||||
_tools_instances['config'] = ConfigManagementTools(project_root)
|
||||
_tools_instances['system'] = SystemManagementTools(project_root)
|
||||
return _tools_instances
|
||||
|
||||
|
||||
# ==================== 数据查询工具 ====================
|
||||
|
||||
@mcp.tool
|
||||
async def get_latest_news(
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: int = 50,
|
||||
include_url: bool = False
|
||||
) -> str:
|
||||
"""
|
||||
获取最新一批爬取的新闻数据，快速了解当前热点
|
||||
|
||||
Args:
|
||||
platforms: 平台ID列表，如 ['zhihu', 'weibo', 'douyin']
|
||||
- 不指定时：使用 config.yaml 中配置的所有平台
|
||||
- 支持的平台来自 config/config.yaml 的 platforms 配置
|
||||
- 每个平台都有对应的name字段（如"知乎"、"微博"），方便AI识别
|
||||
limit: 返回条数限制，默认50，最大1000
|
||||
注意：实际返回数量可能少于请求值，取决于当前可用的新闻总数
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
JSON格式的新闻列表
|
||||
|
||||
**重要：数据展示建议**
|
||||
本工具会返回完整的新闻列表（通常50条）给你。但请注意：
|
||||
- **工具返回**：完整的50条数据 ✅
|
||||
- **建议展示**：向用户展示全部数据，除非用户明确要求总结
|
||||
- **用户期望**：用户可能需要完整数据，请谨慎总结
|
||||
|
||||
**何时可以总结**：
|
||||
- 用户明确说"给我总结一下"或"挑重点说"
|
||||
- 数据量超过100条时，可先展示部分并询问是否查看全部
|
||||
|
||||
**注意**：如果用户询问"为什么只显示了部分"，说明他们需要完整数据
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['data'].get_latest_news(platforms=platforms, limit=limit, include_url=include_url)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
@mcp.tool
|
||||
async def get_trending_topics(
|
||||
top_n: int = 10,
|
||||
mode: str = 'current'
|
||||
) -> str:
|
||||
"""
|
||||
获取个人关注词的新闻出现频率统计（基于 config/frequency_words.txt）
|
||||
|
||||
注意：本工具不是自动提取新闻热点，而是统计你在 config/frequency_words.txt 中
|
||||
设置的个人关注词在新闻中出现的频率。你可以自定义这个关注词列表。
|
||||
|
||||
Args:
|
||||
top_n: 返回TOP N关注词，默认10
|
||||
mode: 模式选择
|
||||
- daily: 当日累计数据统计
|
||||
- current: 最新一批数据统计（默认）
|
||||
|
||||
Returns:
|
||||
JSON格式的关注词频率统计列表
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['data'].get_trending_topics(top_n=top_n, mode=mode)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
@mcp.tool
|
||||
async def get_news_by_date(
|
||||
date_query: Optional[str] = None,
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: int = 50,
|
||||
include_url: bool = False
|
||||
) -> str:
|
||||
"""
|
||||
获取指定日期的新闻数据，用于历史数据分析和对比
|
||||
|
||||
Args:
|
||||
date_query: 日期查询，可选格式:
|
||||
- 自然语言: "今天", "昨天", "前天", "3天前"
|
||||
- 标准日期: "2024-01-15", "2024/01/15"
|
||||
- 默认值: "今天"（节省token）
|
||||
platforms: 平台ID列表，如 ['zhihu', 'weibo', 'douyin']
|
||||
- 不指定时：使用 config.yaml 中配置的所有平台
|
||||
- 支持的平台来自 config/config.yaml 的 platforms 配置
|
||||
- 每个平台都有对应的name字段（如"知乎"、"微博"），方便AI识别
|
||||
limit: 返回条数限制，默认50，最大1000
|
||||
注意：实际返回数量可能少于请求值，取决于指定日期的新闻总数
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
JSON格式的新闻列表，包含标题、平台、排名等信息
|
||||
|
||||
**重要：数据展示建议**
|
||||
本工具会返回完整的新闻列表（通常50条）给你。但请注意：
|
||||
- **工具返回**：完整的50条数据 ✅
|
||||
- **建议展示**：向用户展示全部数据，除非用户明确要求总结
|
||||
- **用户期望**：用户可能需要完整数据，请谨慎总结
|
||||
|
||||
**何时可以总结**：
|
||||
- 用户明确说"给我总结一下"或"挑重点说"
|
||||
- 数据量超过100条时，可先展示部分并询问是否查看全部
|
||||
|
||||
**注意**：如果用户询问"为什么只显示了部分"，说明他们需要完整数据
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['data'].get_news_by_date(
|
||||
date_query=date_query,
|
||||
platforms=platforms,
|
||||
limit=limit,
|
||||
include_url=include_url
|
||||
)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
|
||||
# ==================== 高级数据分析工具 ====================
|
||||
|
||||
@mcp.tool
|
||||
async def analyze_topic_trend(
|
||||
topic: str,
|
||||
analysis_type: str = "trend",
|
||||
time_range: str = "7d",
|
||||
granularity: str = "day",
|
||||
threshold: float = 3.0,
|
||||
time_window: int = 24,
|
||||
lookback_days: int = 7,
|
||||
lookahead_hours: int = 6,
|
||||
confidence_threshold: float = 0.7
|
||||
) -> str:
|
||||
"""
|
||||
统一话题趋势分析工具 - 整合多种趋势分析模式
|
||||
|
||||
Args:
|
||||
topic: 话题关键词（必需）
|
||||
analysis_type: 分析类型，可选值：
|
||||
- "trend": 热度趋势分析（追踪话题的热度变化）
|
||||
- "lifecycle": 生命周期分析（从出现到消失的完整周期）
|
||||
- "viral": 异常热度检测（识别突然爆火的话题）
|
||||
- "predict": 话题预测（预测未来可能的热点）
|
||||
time_range: 时间范围（trend模式），默认"7d"（7d/24h/1w/1m/2m）
|
||||
granularity: 时间粒度（trend模式），默认"day"（仅支持 day，因为底层数据按天聚合）
|
||||
threshold: 热度突增倍数阈值（viral模式），默认3.0
|
||||
time_window: 检测时间窗口小时数（viral模式），默认24
|
||||
lookback_days: 回溯天数（lifecycle模式），默认7
|
||||
lookahead_hours: 预测未来小时数（predict模式），默认6
|
||||
confidence_threshold: 置信度阈值（predict模式），默认0.7
|
||||
|
||||
Returns:
|
||||
JSON格式的趋势分析结果
|
||||
|
||||
Examples:
|
||||
- analyze_topic_trend(topic="人工智能", analysis_type="trend", time_range="7d")
|
||||
- analyze_topic_trend(topic="特斯拉", analysis_type="lifecycle", lookback_days=7)
|
||||
- analyze_topic_trend(topic="比特币", analysis_type="viral", threshold=3.0)
|
||||
- analyze_topic_trend(topic="ChatGPT", analysis_type="predict", lookahead_hours=6)
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['analytics'].analyze_topic_trend_unified(
|
||||
topic=topic,
|
||||
analysis_type=analysis_type,
|
||||
time_range=time_range,
|
||||
granularity=granularity,
|
||||
threshold=threshold,
|
||||
time_window=time_window,
|
||||
lookback_days=lookback_days,
|
||||
lookahead_hours=lookahead_hours,
|
||||
confidence_threshold=confidence_threshold
|
||||
)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
@mcp.tool
|
||||
async def analyze_data_insights(
|
||||
insight_type: str = "platform_compare",
|
||||
topic: Optional[str] = None,
|
||||
date_range: Optional[Dict[str, str]] = None,
|
||||
min_frequency: int = 3,
|
||||
top_n: int = 20
|
||||
) -> str:
|
||||
"""
|
||||
统一数据洞察分析工具 - 整合多种数据分析模式
|
||||
|
||||
Args:
|
||||
insight_type: 洞察类型，可选值：
|
||||
- "platform_compare": 平台对比分析（对比不同平台对话题的关注度）
|
||||
- "platform_activity": 平台活跃度统计（统计各平台发布频率和活跃时间）
|
||||
- "keyword_cooccur": 关键词共现分析（分析关键词同时出现的模式）
|
||||
topic: 话题关键词（可选，platform_compare模式适用）
|
||||
date_range: 日期范围，格式: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}
|
||||
min_frequency: 最小共现频次（keyword_cooccur模式），默认3
|
||||
top_n: 返回TOP N结果（keyword_cooccur模式），默认20
|
||||
|
||||
Returns:
|
||||
JSON格式的数据洞察分析结果
|
||||
|
||||
Examples:
|
||||
- analyze_data_insights(insight_type="platform_compare", topic="人工智能")
|
||||
- analyze_data_insights(insight_type="platform_activity", date_range={...})
|
||||
- analyze_data_insights(insight_type="keyword_cooccur", min_frequency=5, top_n=15)
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['analytics'].analyze_data_insights_unified(
|
||||
insight_type=insight_type,
|
||||
topic=topic,
|
||||
date_range=date_range,
|
||||
min_frequency=min_frequency,
|
||||
top_n=top_n
|
||||
)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
@mcp.tool
|
||||
async def analyze_sentiment(
|
||||
topic: Optional[str] = None,
|
||||
platforms: Optional[List[str]] = None,
|
||||
date_range: Optional[Dict[str, str]] = None,
|
||||
limit: int = 50,
|
||||
sort_by_weight: bool = True,
|
||||
include_url: bool = False
|
||||
) -> str:
|
||||
"""
|
||||
分析新闻的情感倾向和热度趋势
|
||||
|
||||
Args:
|
||||
keywords: 关键词列表，如 ["AI", "人工智能"]
|
||||
date_range: 日期范围（天数），如 7 表示最近7天，默认3天
|
||||
platforms: 平台ID列表，如 ['zhihu', 'weibo', 'douyin']
|
||||
- 不指定时：使用 config.yaml 中配置的所有平台
|
||||
- 支持的平台来自 config/config.yaml 的 platforms 配置
|
||||
- 每个平台都有对应的name字段（如"知乎"、"微博"），方便AI识别
|
||||
limit: 返回新闻数量，默认50，最大100
|
||||
注意：本工具会对新闻标题进行去重（同一标题在不同平台只保留一次），
|
||||
因此实际返回数量可能少于请求的 limit 值
|
||||
sort_by_weight: 是否按热度权重排序，默认True
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
JSON格式的分析结果，包含情感分布、热度趋势和相关新闻
|
||||
|
||||
**重要：数据展示策略**
|
||||
- 本工具返回完整的分析结果和新闻列表
|
||||
- **默认展示方式**：展示完整的分析结果（包括所有新闻）
|
||||
- 仅在用户明确要求"总结"或"挑重点"时才进行筛选
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['analytics'].analyze_sentiment(
|
||||
topic=topic,
|
||||
platforms=platforms,
|
||||
date_range=date_range,
|
||||
limit=limit,
|
||||
sort_by_weight=sort_by_weight,
|
||||
include_url=include_url
|
||||
)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
@mcp.tool
|
||||
async def find_similar_news(
|
||||
reference_title: str,
|
||||
threshold: float = 0.6,
|
||||
limit: int = 50,
|
||||
include_url: bool = False
|
||||
) -> str:
|
||||
"""
|
||||
查找与指定新闻标题相似的其他新闻
|
||||
|
||||
Args:
|
||||
title: 新闻标题（完整或部分）
|
||||
threshold: 相似度阈值，0-1之间，默认0.6
|
||||
注意：阈值越高匹配越严格，返回结果越少
|
||||
limit: 返回条数限制，默认50，最大100
|
||||
注意：实际返回数量取决于相似度匹配结果，可能少于请求值
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
JSON格式的相似新闻列表，包含相似度分数
|
||||
|
||||
**重要：数据展示策略**
|
||||
- 本工具返回完整的相似新闻列表
|
||||
- **默认展示方式**：展示全部返回的新闻（包括相似度分数）
|
||||
- 仅在用户明确要求"总结"或"挑重点"时才进行筛选
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['analytics'].find_similar_news(
|
||||
reference_title=reference_title,
|
||||
threshold=threshold,
|
||||
limit=limit,
|
||||
include_url=include_url
|
||||
)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
@mcp.tool
|
||||
async def generate_summary_report(
|
||||
report_type: str = "daily",
|
||||
date_range: Optional[Dict[str, str]] = None
|
||||
) -> str:
|
||||
"""
|
||||
每日/每周摘要生成器 - 自动生成热点摘要报告
|
||||
|
||||
Args:
|
||||
report_type: 报告类型（daily/weekly）
|
||||
date_range: 自定义日期范围（可选）
|
||||
|
||||
Returns:
|
||||
JSON格式的摘要报告，包含Markdown格式内容
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['analytics'].generate_summary_report(
|
||||
report_type=report_type,
|
||||
date_range=date_range
|
||||
)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
# ==================== 智能检索工具 ====================
|
||||
|
||||
@mcp.tool
|
||||
async def search_news(
|
||||
query: str,
|
||||
search_mode: str = "keyword",
|
||||
date_range: Optional[Dict[str, str]] = None,
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: int = 50,
|
||||
sort_by: str = "relevance",
|
||||
threshold: float = 0.6,
|
||||
include_url: bool = False
|
||||
) -> str:
|
||||
"""
|
||||
统一搜索接口，支持多种搜索模式
|
||||
|
||||
Args:
|
||||
query: 搜索关键词或内容片段
|
||||
search_mode: 搜索模式，可选值：
|
||||
- "keyword": 精确关键词匹配（默认，适合搜索特定话题）
|
||||
- "fuzzy": 模糊内容匹配（适合搜索内容片段，会过滤相似度低于阈值的结果）
|
||||
- "entity": 实体名称搜索（适合搜索人物/地点/机构）
|
||||
threshold: 相似度阈值（仅fuzzy模式有效），0-1之间，默认0.6
|
||||
注意：阈值越高匹配越严格，返回结果越少
|
||||
platforms: 平台ID列表，如 ['zhihu', 'weibo', 'douyin']
|
||||
- 不指定时：使用 config.yaml 中配置的所有平台
|
||||
- 支持的平台来自 config/config.yaml 的 platforms 配置
|
||||
- 每个平台都有对应的name字段（如"知乎"、"微博"），方便AI识别
|
||||
lookback_days: 回溯天数，默认7天，最大30天
|
||||
limit: 返回条数限制，默认50，最大1000
|
||||
注意：实际返回数量取决于搜索匹配结果（特别是 fuzzy 模式下会过滤低相似度结果）
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
JSON格式的搜索结果，包含标题、平台、排名等信息
|
||||
|
||||
**重要：数据展示策略**
|
||||
- 本工具返回完整的搜索结果列表
|
||||
- **默认展示方式**：展示全部返回的新闻，无需总结或筛选
|
||||
- 仅在用户明确要求"总结"或"挑重点"时才进行筛选
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['search'].search_news_unified(
|
||||
query=query,
|
||||
search_mode=search_mode,
|
||||
date_range=date_range,
|
||||
platforms=platforms,
|
||||
limit=limit,
|
||||
sort_by=sort_by,
|
||||
threshold=threshold,
|
||||
include_url=include_url
|
||||
)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
@mcp.tool
|
||||
async def search_related_news_history(
|
||||
reference_text: str,
|
||||
time_range: str = "yesterday",
|
||||
threshold: float = 0.4,
|
||||
limit: int = 50,
|
||||
include_url: bool = False
|
||||
) -> str:
|
||||
"""
|
||||
基于种子新闻，在历史数据中搜索相关新闻
|
||||
|
||||
Args:
|
||||
seed_news_title: 种子新闻标题（完整或部分）
|
||||
lookback_days: 向前查找的天数范围，默认7天，最大30天
|
||||
threshold: 相关性阈值，0-1之间，默认0.4
|
||||
注意：综合相似度计算（70%关键词重合 + 30%文本相似度）
|
||||
阈值越高匹配越严格，返回结果越少
|
||||
platforms: 平台ID列表。不指定则搜索所有平台
|
||||
limit: 返回条数限制，默认50，最大100
|
||||
注意：实际返回数量取决于相关性匹配结果，可能少于请求值
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
JSON格式的相关新闻列表，包含相关性分数和时间分布
|
||||
|
||||
**重要：数据展示策略**
|
||||
- 本工具返回完整的相关新闻列表
|
||||
- **默认展示方式**：展示全部返回的新闻（包括相关性分数）
|
||||
- 仅在用户明确要求"总结"或"挑重点"时才进行筛选
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['search'].search_related_news_history(
|
||||
reference_text=reference_text,
|
||||
time_range=time_range,
|
||||
threshold=threshold,
|
||||
limit=limit,
|
||||
include_url=include_url
|
||||
)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
# ==================== 配置与系统管理工具 ====================
|
||||
|
||||
@mcp.tool
|
||||
async def get_current_config(
|
||||
section: str = "all"
|
||||
) -> str:
|
||||
"""
|
||||
获取当前系统配置
|
||||
|
||||
Args:
|
||||
section: 配置节，可选值：
|
||||
- "all": 所有配置（默认）
|
||||
- "crawler": 爬虫配置
|
||||
- "push": 推送配置
|
||||
- "keywords": 关键词配置
|
||||
- "weights": 权重配置
|
||||
|
||||
Returns:
|
||||
JSON格式的配置信息
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['config'].get_current_config(section=section)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
@mcp.tool
|
||||
async def get_system_status() -> str:
|
||||
"""
|
||||
获取系统运行状态和健康检查信息
|
||||
|
||||
返回系统版本、数据统计、缓存状态等信息
|
||||
|
||||
Returns:
|
||||
JSON格式的系统状态信息
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['system'].get_system_status()
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
@mcp.tool
|
||||
async def trigger_crawl(
|
||||
platforms: Optional[List[str]] = None,
|
||||
save_to_local: bool = False,
|
||||
include_url: bool = False
|
||||
) -> str:
|
||||
"""
|
||||
手动触发一次爬取任务（可选持久化）
|
||||
|
||||
Args:
|
||||
platforms: 指定平台ID列表，如 ['zhihu', 'weibo', 'douyin']
|
||||
- 不指定时：使用 config.yaml 中配置的所有平台
|
||||
- 支持的平台来自 config/config.yaml 的 platforms 配置
|
||||
- 每个平台都有对应的name字段（如"知乎"、"微博"），方便AI识别
|
||||
- 注意：失败的平台会在返回结果的 failed_platforms 字段中列出
|
||||
save_to_local: 是否保存到本地 output 目录，默认 False
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
JSON格式的任务状态信息，包含：
|
||||
- platforms: 成功爬取的平台列表
|
||||
- failed_platforms: 失败的平台列表（如有）
|
||||
- total_news: 爬取的新闻总数
|
||||
- data: 新闻数据
|
||||
|
||||
Examples:
|
||||
- 临时爬取: trigger_crawl(platforms=['zhihu'])
|
||||
- 爬取并保存: trigger_crawl(platforms=['weibo'], save_to_local=True)
|
||||
- 使用默认平台: trigger_crawl() # 爬取config.yaml中配置的所有平台
|
||||
"""
|
||||
tools = _get_tools()
|
||||
result = tools['system'].trigger_crawl(platforms=platforms, save_to_local=save_to_local, include_url=include_url)
|
||||
return json.dumps(result, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
# ==================== 启动入口 ====================
|
||||
|
||||
def run_server(
|
||||
project_root: Optional[str] = None,
|
||||
transport: str = 'stdio',
|
||||
host: str = '0.0.0.0',
|
||||
port: int = 3333
|
||||
):
|
||||
"""
|
||||
启动 MCP 服务器
|
||||
|
||||
Args:
|
||||
project_root: 项目根目录路径
|
||||
transport: 传输模式，'stdio' 或 'http'
|
||||
host: HTTP模式的监听地址，默认 0.0.0.0
|
||||
port: HTTP模式的监听端口，默认 3333
|
||||
"""
|
||||
# 初始化工具实例
|
||||
_get_tools(project_root)
|
||||
|
||||
# 打印启动信息
|
||||
print()
|
||||
print("=" * 60)
|
||||
print(" TrendRadar MCP Server - FastMCP 2.0")
|
||||
print("=" * 60)
|
||||
print(f" 传输模式: {transport.upper()}")
|
||||
|
||||
if transport == 'stdio':
|
||||
print(" 协议: MCP over stdio (标准输入输出)")
|
||||
print(" 说明: 通过标准输入输出与 MCP 客户端通信")
|
||||
elif transport == 'http':
|
||||
print(f" 监听地址: http://{host}:{port}")
|
||||
print(f" HTTP端点: http://{host}:{port}/mcp")
|
||||
print(" 协议: MCP over HTTP (生产环境)")
|
||||
|
||||
if project_root:
|
||||
print(f" 项目目录: {project_root}")
|
||||
else:
|
||||
print(" 项目目录: 当前目录")
|
||||
|
||||
print()
|
||||
print(" 已注册的工具:")
|
||||
print(" === 基础数据查询（P0核心）===")
|
||||
print(" 1. get_latest_news - 获取最新新闻")
|
||||
print(" 2. get_news_by_date - 按日期查询新闻（支持自然语言）")
|
||||
print(" 3. get_trending_topics - 获取趋势话题")
|
||||
print()
|
||||
print(" === 智能检索工具 ===")
|
||||
print(" 4. search_news - 统一新闻搜索（关键词/模糊/实体）")
|
||||
print(" 5. search_related_news_history - 历史相关新闻检索")
|
||||
print()
|
||||
print(" === 高级数据分析 ===")
|
||||
print(" 6. analyze_topic_trend - 统一话题趋势分析（热度/生命周期/爆火/预测）")
|
||||
print(" 7. analyze_data_insights - 统一数据洞察分析（平台对比/活跃度/关键词共现）")
|
||||
print(" 8. analyze_sentiment - 情感倾向分析")
|
||||
print(" 9. find_similar_news - 相似新闻查找")
|
||||
print(" 10. generate_summary_report - 每日/每周摘要生成")
|
||||
print()
|
||||
print(" === 配置与系统管理 ===")
|
||||
print(" 11. get_current_config - 获取当前系统配置")
|
||||
print(" 12. get_system_status - 获取系统运行状态")
|
||||
print(" 13. trigger_crawl - 手动触发爬取任务")
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
# 根据传输模式运行服务器
|
||||
if transport == 'stdio':
|
||||
mcp.run(transport='stdio')
|
||||
elif transport == 'http':
|
||||
# HTTP 模式（生产推荐）
|
||||
mcp.run(
|
||||
transport='http',
|
||||
host=host,
|
||||
port=port,
|
||||
path='/mcp' # HTTP 端点路径
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"不支持的传输模式: {transport}")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
import sys
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description='TrendRadar MCP Server - 新闻热点聚合 MCP 工具服务器',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
使用示例:
|
||||
# STDIO 模式（用于 Cherry Studio）
|
||||
uv run python mcp_server/server.py
|
||||
|
||||
# HTTP 模式（适合远程访问）
|
||||
uv run python mcp_server/server.py --transport http --port 3333
|
||||
|
||||
Cherry Studio 配置示例:
|
||||
设置 > MCP Servers > 添加服务器
|
||||
- 名称: TrendRadar
|
||||
- 类型: STDIO
|
||||
- 命令: [UV的完整路径]
|
||||
- 参数: --directory [项目路径] run python mcp_server/server.py
|
||||
|
||||
详细配置教程请查看: README-Cherry-Studio.md
|
||||
"""
|
||||
)
|
||||
parser.add_argument(
|
||||
'--transport',
|
||||
choices=['stdio', 'http'],
|
||||
default='stdio',
|
||||
help='传输模式：stdio (默认) 或 http (生产环境)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--host',
|
||||
default='0.0.0.0',
|
||||
help='HTTP模式的监听地址，默认 0.0.0.0'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--port',
|
||||
type=int,
|
||||
default=3333,
|
||||
help='HTTP模式的监听端口，默认 3333'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--project-root',
|
||||
help='项目根目录路径'
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
run_server(
|
||||
project_root=args.project_root,
|
||||
transport=args.transport,
|
||||
host=args.host,
|
||||
port=args.port
|
||||
)
|
||||
5
mcp_server/services/__init__.py
Normal file
5
mcp_server/services/__init__.py
Normal file
@ -0,0 +1,5 @@
|
||||
"""
|
||||
服务层模块
|
||||
|
||||
提供数据访问、缓存、解析等核心服务。
|
||||
"""
|
||||
136
mcp_server/services/cache_service.py
Normal file
136
mcp_server/services/cache_service.py
Normal file
@ -0,0 +1,136 @@
|
||||
"""
|
||||
缓存服务
|
||||
|
||||
实现TTL缓存机制，提升数据访问性能。
|
||||
"""
|
||||
|
||||
import time
|
||||
from typing import Any, Optional
|
||||
from threading import Lock
|
||||
|
||||
|
||||
class CacheService:
|
||||
"""缓存服务类"""
|
||||
|
||||
def __init__(self):
|
||||
"""初始化缓存服务"""
|
||||
self._cache = {}
|
||||
self._timestamps = {}
|
||||
self._lock = Lock()
|
||||
|
||||
def get(self, key: str, ttl: int = 900) -> Optional[Any]:
|
||||
"""
|
||||
获取缓存数据
|
||||
|
||||
Args:
|
||||
key: 缓存键
|
||||
ttl: 存活时间（秒），默认15分钟
|
||||
|
||||
Returns:
|
||||
缓存的值，如果不存在或已过期则返回None
|
||||
"""
|
||||
with self._lock:
|
||||
if key in self._cache:
|
||||
# 检查是否过期
|
||||
if time.time() - self._timestamps[key] < ttl:
|
||||
return self._cache[key]
|
||||
else:
|
||||
# 已过期，删除缓存
|
||||
del self._cache[key]
|
||||
del self._timestamps[key]
|
||||
return None
|
||||
|
||||
def set(self, key: str, value: Any) -> None:
|
||||
"""
|
||||
设置缓存数据
|
||||
|
||||
Args:
|
||||
key: 缓存键
|
||||
value: 缓存值
|
||||
"""
|
||||
with self._lock:
|
||||
self._cache[key] = value
|
||||
self._timestamps[key] = time.time()
|
||||
|
||||
def delete(self, key: str) -> bool:
|
||||
"""
|
||||
删除缓存
|
||||
|
||||
Args:
|
||||
key: 缓存键
|
||||
|
||||
Returns:
|
||||
是否成功删除
|
||||
"""
|
||||
with self._lock:
|
||||
if key in self._cache:
|
||||
del self._cache[key]
|
||||
del self._timestamps[key]
|
||||
return True
|
||||
return False
|
||||
|
||||
def clear(self) -> None:
|
||||
"""清空所有缓存"""
|
||||
with self._lock:
|
||||
self._cache.clear()
|
||||
self._timestamps.clear()
|
||||
|
||||
def cleanup_expired(self, ttl: int = 900) -> int:
|
||||
"""
|
||||
清理过期缓存
|
||||
|
||||
Args:
|
||||
ttl: 存活时间（秒）
|
||||
|
||||
Returns:
|
||||
清理的条目数量
|
||||
"""
|
||||
with self._lock:
|
||||
current_time = time.time()
|
||||
expired_keys = [
|
||||
key for key, timestamp in self._timestamps.items()
|
||||
if current_time - timestamp >= ttl
|
||||
]
|
||||
|
||||
for key in expired_keys:
|
||||
del self._cache[key]
|
||||
del self._timestamps[key]
|
||||
|
||||
return len(expired_keys)
|
||||
|
||||
def get_stats(self) -> dict:
|
||||
"""
|
||||
获取缓存统计信息
|
||||
|
||||
Returns:
|
||||
统计信息字典
|
||||
"""
|
||||
with self._lock:
|
||||
return {
|
||||
"total_entries": len(self._cache),
|
||||
"oldest_entry_age": (
|
||||
time.time() - min(self._timestamps.values())
|
||||
if self._timestamps else 0
|
||||
),
|
||||
"newest_entry_age": (
|
||||
time.time() - max(self._timestamps.values())
|
||||
if self._timestamps else 0
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
# 全局缓存实例
|
||||
_global_cache = None
|
||||
|
||||
|
||||
def get_cache() -> CacheService:
|
||||
"""
|
||||
获取全局缓存实例
|
||||
|
||||
Returns:
|
||||
全局缓存服务实例
|
||||
"""
|
||||
global _global_cache
|
||||
if _global_cache is None:
|
||||
_global_cache = CacheService()
|
||||
return _global_cache
|
||||
564
mcp_server/services/data_service.py
Normal file
564
mcp_server/services/data_service.py
Normal file
@ -0,0 +1,564 @@
|
||||
"""
|
||||
数据访问服务
|
||||
|
||||
提供统一的数据查询接口,封装数据访问逻辑。
|
||||
"""
|
||||
|
||||
import re
|
||||
from collections import Counter
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
from .cache_service import get_cache
|
||||
from .parser_service import ParserService
|
||||
from ..utils.errors import DataNotFoundError
|
||||
|
||||
|
||||
class DataService:
|
||||
"""数据访问服务类"""
|
||||
|
||||
def __init__(self, project_root: str = None):
|
||||
"""
|
||||
初始化数据服务
|
||||
|
||||
Args:
|
||||
project_root: 项目根目录
|
||||
"""
|
||||
self.parser = ParserService(project_root)
|
||||
self.cache = get_cache()
|
||||
|
||||
def get_latest_news(
|
||||
self,
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: int = 50,
|
||||
include_url: bool = False
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
获取最新一批爬取的新闻数据
|
||||
|
||||
Args:
|
||||
platforms: 平台ID列表,None表示所有平台
|
||||
limit: 返回条数限制
|
||||
include_url: 是否包含URL链接,默认False(节省token)
|
||||
|
||||
Returns:
|
||||
新闻列表
|
||||
|
||||
Raises:
|
||||
DataNotFoundError: 数据不存在
|
||||
"""
|
||||
# 尝试从缓存获取
|
||||
cache_key = f"latest_news:{','.join(platforms or [])}:{limit}:{include_url}"
|
||||
cached = self.cache.get(cache_key, ttl=900) # 15分钟缓存
|
||||
if cached:
|
||||
return cached
|
||||
|
||||
# 读取今天的数据
|
||||
all_titles, id_to_name, timestamps = self.parser.read_all_titles_for_date(
|
||||
date=None,
|
||||
platform_ids=platforms
|
||||
)
|
||||
|
||||
# 获取最新的文件时间
|
||||
if timestamps:
|
||||
latest_timestamp = max(timestamps.values())
|
||||
fetch_time = datetime.fromtimestamp(latest_timestamp)
|
||||
else:
|
||||
fetch_time = datetime.now()
|
||||
|
||||
# 转换为新闻列表
|
||||
news_list = []
|
||||
for platform_id, titles in all_titles.items():
|
||||
platform_name = id_to_name.get(platform_id, platform_id)
|
||||
|
||||
for title, info in titles.items():
|
||||
# 取第一个排名
|
||||
rank = info["ranks"][0] if info["ranks"] else 0
|
||||
|
||||
news_item = {
|
||||
"title": title,
|
||||
"platform": platform_id,
|
||||
"platform_name": platform_name,
|
||||
"rank": rank,
|
||||
"timestamp": fetch_time.strftime("%Y-%m-%d %H:%M:%S")
|
||||
}
|
||||
|
||||
# 条件性添加 URL 字段
|
||||
if include_url:
|
||||
news_item["url"] = info.get("url", "")
|
||||
news_item["mobileUrl"] = info.get("mobileUrl", "")
|
||||
|
||||
news_list.append(news_item)
|
||||
|
||||
# 按排名排序
|
||||
news_list.sort(key=lambda x: x["rank"])
|
||||
|
||||
# 限制返回数量
|
||||
result = news_list[:limit]
|
||||
|
||||
# 缓存结果
|
||||
self.cache.set(cache_key, result)
|
||||
|
||||
return result
|
||||
|
||||
def get_news_by_date(
|
||||
self,
|
||||
target_date: datetime,
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: int = 50,
|
||||
include_url: bool = False
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
按指定日期获取新闻
|
||||
|
||||
Args:
|
||||
target_date: 目标日期
|
||||
platforms: 平台ID列表,None表示所有平台
|
||||
limit: 返回条数限制
|
||||
include_url: 是否包含URL链接,默认False(节省token)
|
||||
|
||||
Returns:
|
||||
新闻列表
|
||||
|
||||
Raises:
|
||||
DataNotFoundError: 数据不存在
|
||||
|
||||
Examples:
|
||||
>>> service = DataService()
|
||||
>>> news = service.get_news_by_date(
|
||||
... target_date=datetime(2025, 10, 10),
|
||||
... platforms=['zhihu'],
|
||||
... limit=20
|
||||
... )
|
||||
"""
|
||||
# 尝试从缓存获取
|
||||
date_str = target_date.strftime("%Y-%m-%d")
|
||||
cache_key = f"news_by_date:{date_str}:{','.join(platforms or [])}:{limit}:{include_url}"
|
||||
cached = self.cache.get(cache_key, ttl=1800) # 30分钟缓存
|
||||
if cached:
|
||||
return cached
|
||||
|
||||
# 读取指定日期的数据
|
||||
all_titles, id_to_name, timestamps = self.parser.read_all_titles_for_date(
|
||||
date=target_date,
|
||||
platform_ids=platforms
|
||||
)
|
||||
|
||||
# 转换为新闻列表
|
||||
news_list = []
|
||||
for platform_id, titles in all_titles.items():
|
||||
platform_name = id_to_name.get(platform_id, platform_id)
|
||||
|
||||
for title, info in titles.items():
|
||||
# 计算平均排名
|
||||
avg_rank = sum(info["ranks"]) / len(info["ranks"]) if info["ranks"] else 0
|
||||
|
||||
news_item = {
|
||||
"title": title,
|
||||
"platform": platform_id,
|
||||
"platform_name": platform_name,
|
||||
"rank": info["ranks"][0] if info["ranks"] else 0,
|
||||
"avg_rank": round(avg_rank, 2),
|
||||
"count": len(info["ranks"]),
|
||||
"date": date_str
|
||||
}
|
||||
|
||||
# 条件性添加 URL 字段
|
||||
if include_url:
|
||||
news_item["url"] = info.get("url", "")
|
||||
news_item["mobileUrl"] = info.get("mobileUrl", "")
|
||||
|
||||
news_list.append(news_item)
|
||||
|
||||
# 按排名排序
|
||||
news_list.sort(key=lambda x: x["rank"])
|
||||
|
||||
# 限制返回数量
|
||||
result = news_list[:limit]
|
||||
|
||||
# 缓存结果(历史数据缓存更久)
|
||||
self.cache.set(cache_key, result)
|
||||
|
||||
return result
|
||||
|
||||
def search_news_by_keyword(
|
||||
self,
|
||||
keyword: str,
|
||||
date_range: Optional[Tuple[datetime, datetime]] = None,
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: Optional[int] = None
|
||||
) -> Dict:
|
||||
"""
|
||||
按关键词搜索新闻
|
||||
|
||||
Args:
|
||||
keyword: 搜索关键词
|
||||
date_range: 日期范围 (start_date, end_date)
|
||||
platforms: 平台过滤列表
|
||||
limit: 返回条数限制(可选)
|
||||
|
||||
Returns:
|
||||
搜索结果字典
|
||||
|
||||
Raises:
|
||||
DataNotFoundError: 数据不存在
|
||||
"""
|
||||
# 确定搜索日期范围
|
||||
if date_range:
|
||||
start_date, end_date = date_range
|
||||
else:
|
||||
# 默认搜索今天
|
||||
start_date = end_date = datetime.now()
|
||||
|
||||
# 收集所有匹配的新闻
|
||||
results = []
|
||||
platform_distribution = Counter()
|
||||
|
||||
# 遍历日期范围
|
||||
current_date = start_date
|
||||
while current_date <= end_date:
|
||||
try:
|
||||
all_titles, id_to_name, _ = self.parser.read_all_titles_for_date(
|
||||
date=current_date,
|
||||
platform_ids=platforms
|
||||
)
|
||||
|
||||
# 搜索包含关键词的标题
|
||||
for platform_id, titles in all_titles.items():
|
||||
platform_name = id_to_name.get(platform_id, platform_id)
|
||||
|
||||
for title, info in titles.items():
|
||||
if keyword.lower() in title.lower():
|
||||
# 计算平均排名
|
||||
avg_rank = sum(info["ranks"]) / len(info["ranks"]) if info["ranks"] else 0
|
||||
|
||||
results.append({
|
||||
"title": title,
|
||||
"platform": platform_id,
|
||||
"platform_name": platform_name,
|
||||
"ranks": info["ranks"],
|
||||
"count": len(info["ranks"]),
|
||||
"avg_rank": round(avg_rank, 2),
|
||||
"url": info.get("url", ""),
|
||||
"mobileUrl": info.get("mobileUrl", ""),
|
||||
"date": current_date.strftime("%Y-%m-%d")
|
||||
})
|
||||
|
||||
platform_distribution[platform_id] += 1
|
||||
|
||||
except DataNotFoundError:
|
||||
# 该日期没有数据,继续下一天
|
||||
pass
|
||||
|
||||
# 下一天
|
||||
current_date += timedelta(days=1)
|
||||
|
||||
if not results:
|
||||
raise DataNotFoundError(
|
||||
f"未找到包含关键词 '{keyword}' 的新闻",
|
||||
suggestion="请尝试其他关键词或扩大日期范围"
|
||||
)
|
||||
|
||||
# 计算统计信息
|
||||
total_ranks = []
|
||||
for item in results:
|
||||
total_ranks.extend(item["ranks"])
|
||||
|
||||
avg_rank = sum(total_ranks) / len(total_ranks) if total_ranks else 0
|
||||
|
||||
# 限制返回数量(如果指定)
|
||||
total_found = len(results)
|
||||
if limit is not None and limit > 0:
|
||||
results = results[:limit]
|
||||
|
||||
return {
|
||||
"results": results,
|
||||
"total": len(results),
|
||||
"total_found": total_found,
|
||||
"statistics": {
|
||||
"platform_distribution": dict(platform_distribution),
|
||||
"avg_rank": round(avg_rank, 2),
|
||||
"keyword": keyword
|
||||
}
|
||||
}
|
||||
|
||||
def get_trending_topics(
|
||||
self,
|
||||
top_n: int = 10,
|
||||
mode: str = "current"
|
||||
) -> Dict:
|
||||
"""
|
||||
获取个人关注词的新闻出现频率统计
|
||||
|
||||
注意:本工具基于 config/frequency_words.txt 中的个人关注词列表进行统计,
|
||||
而不是自动从新闻中提取热点话题。用户可以自定义这个关注词列表。
|
||||
|
||||
Args:
|
||||
top_n: 返回TOP N关注词
|
||||
mode: 模式 - daily(当日累计), current(最新一批)
|
||||
|
||||
Returns:
|
||||
关注词频率统计字典
|
||||
|
||||
Raises:
|
||||
DataNotFoundError: 数据不存在
|
||||
"""
|
||||
# 尝试从缓存获取
|
||||
cache_key = f"trending_topics:{top_n}:{mode}"
|
||||
cached = self.cache.get(cache_key, ttl=1800) # 30分钟缓存
|
||||
if cached:
|
||||
return cached
|
||||
|
||||
# 读取今天的数据
|
||||
all_titles, id_to_name, timestamps = self.parser.read_all_titles_for_date()
|
||||
|
||||
if not all_titles:
|
||||
raise DataNotFoundError(
|
||||
"未找到今天的新闻数据",
|
||||
suggestion="请确保爬虫已经运行并生成了数据"
|
||||
)
|
||||
|
||||
# 加载关键词配置
|
||||
word_groups = self.parser.parse_frequency_words()
|
||||
|
||||
# 根据mode选择要处理的标题数据
|
||||
titles_to_process = {}
|
||||
|
||||
if mode == "daily":
|
||||
# daily模式:处理当天所有累计数据
|
||||
titles_to_process = all_titles
|
||||
|
||||
elif mode == "current":
|
||||
# current模式:只处理最新一批数据(最新时间戳的文件)
|
||||
if timestamps:
|
||||
# 找出最新的时间戳
|
||||
latest_timestamp = max(timestamps.values())
|
||||
|
||||
# 重新读取,只获取最新时间的数据
|
||||
# 这里我们通过timestamps字典反查找最新文件对应的平台
|
||||
latest_titles, _, _ = self.parser.read_all_titles_for_date()
|
||||
|
||||
# 由于read_all_titles_for_date返回所有文件的合并数据,
|
||||
# 我们需要通过timestamps来过滤出最新批次
|
||||
# 简化实现:使用当前所有数据作为最新批次
|
||||
# (更精确的实现需要解析服务支持按时间过滤)
|
||||
titles_to_process = latest_titles
|
||||
else:
|
||||
titles_to_process = all_titles
|
||||
|
||||
else:
|
||||
raise ValueError(
|
||||
f"不支持的模式: {mode}。支持的模式: daily, current"
|
||||
)
|
||||
|
||||
# 统计词频
|
||||
word_frequency = Counter()
|
||||
keyword_to_news = {}
|
||||
|
||||
# 遍历要处理的标题
|
||||
for platform_id, titles in titles_to_process.items():
|
||||
for title in titles.keys():
|
||||
# 对每个关键词组进行匹配
|
||||
for group in word_groups:
|
||||
all_words = group.get("required", []) + group.get("normal", [])
|
||||
|
||||
for word in all_words:
|
||||
if word and word in title:
|
||||
word_frequency[word] += 1
|
||||
|
||||
if word not in keyword_to_news:
|
||||
keyword_to_news[word] = []
|
||||
keyword_to_news[word].append(title)
|
||||
|
||||
# 获取TOP N关键词
|
||||
top_keywords = word_frequency.most_common(top_n)
|
||||
|
||||
# 构建话题列表
|
||||
topics = []
|
||||
for keyword, frequency in top_keywords:
|
||||
matched_news = keyword_to_news.get(keyword, [])
|
||||
|
||||
topics.append({
|
||||
"keyword": keyword,
|
||||
"frequency": frequency,
|
||||
"matched_news": len(set(matched_news)), # 去重后的新闻数量
|
||||
"trend": "stable", # TODO: 需要历史数据来计算趋势
|
||||
"weight_score": 0.0 # TODO: 需要实现权重计算
|
||||
})
|
||||
|
||||
# 构建结果
|
||||
result = {
|
||||
"topics": topics,
|
||||
"generated_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"mode": mode,
|
||||
"total_keywords": len(word_frequency),
|
||||
"description": self._get_mode_description(mode)
|
||||
}
|
||||
|
||||
# 缓存结果
|
||||
self.cache.set(cache_key, result)
|
||||
|
||||
return result
|
||||
|
||||
def _get_mode_description(self, mode: str) -> str:
|
||||
"""获取模式描述"""
|
||||
descriptions = {
|
||||
"daily": "当日累计统计",
|
||||
"current": "最新一批统计"
|
||||
}
|
||||
return descriptions.get(mode, "未知模式")
|
||||
|
||||
def get_current_config(self, section: str = "all") -> Dict:
|
||||
"""
|
||||
获取当前系统配置
|
||||
|
||||
Args:
|
||||
section: 配置节 - all/crawler/push/keywords/weights
|
||||
|
||||
Returns:
|
||||
配置字典
|
||||
|
||||
Raises:
|
||||
FileParseError: 配置文件解析错误
|
||||
"""
|
||||
# 尝试从缓存获取
|
||||
cache_key = f"config:{section}"
|
||||
cached = self.cache.get(cache_key, ttl=3600) # 1小时缓存
|
||||
if cached:
|
||||
return cached
|
||||
|
||||
# 解析配置文件
|
||||
config_data = self.parser.parse_yaml_config()
|
||||
word_groups = self.parser.parse_frequency_words()
|
||||
|
||||
# 根据section返回对应配置
|
||||
if section == "all" or section == "crawler":
|
||||
crawler_config = {
|
||||
"enable_crawler": config_data.get("crawler", {}).get("enable_crawler", True),
|
||||
"use_proxy": config_data.get("crawler", {}).get("use_proxy", False),
|
||||
"request_interval": config_data.get("crawler", {}).get("request_interval", 1),
|
||||
"retry_times": 3,
|
||||
"platforms": [p["id"] for p in config_data.get("platforms", [])]
|
||||
}
|
||||
|
||||
if section == "all" or section == "push":
|
||||
push_config = {
|
||||
"enable_notification": config_data.get("notification", {}).get("enable_notification", True),
|
||||
"enabled_channels": [],
|
||||
"message_batch_size": config_data.get("notification", {}).get("message_batch_size", 20),
|
||||
"push_window": config_data.get("notification", {}).get("push_window", {})
|
||||
}
|
||||
|
||||
# 检测已配置的通知渠道
|
||||
webhooks = config_data.get("notification", {}).get("webhooks", {})
|
||||
if webhooks.get("feishu_url"):
|
||||
push_config["enabled_channels"].append("feishu")
|
||||
if webhooks.get("dingtalk_url"):
|
||||
push_config["enabled_channels"].append("dingtalk")
|
||||
if webhooks.get("wework_url"):
|
||||
push_config["enabled_channels"].append("wework")
|
||||
|
||||
if section == "all" or section == "keywords":
|
||||
keywords_config = {
|
||||
"word_groups": word_groups,
|
||||
"total_groups": len(word_groups)
|
||||
}
|
||||
|
||||
if section == "all" or section == "weights":
|
||||
weights_config = {
|
||||
"rank_weight": config_data.get("weight", {}).get("rank_weight", 0.6),
|
||||
"frequency_weight": config_data.get("weight", {}).get("frequency_weight", 0.3),
|
||||
"hotness_weight": config_data.get("weight", {}).get("hotness_weight", 0.1)
|
||||
}
|
||||
|
||||
# 组装结果
|
||||
if section == "all":
|
||||
result = {
|
||||
"crawler": crawler_config,
|
||||
"push": push_config,
|
||||
"keywords": keywords_config,
|
||||
"weights": weights_config
|
||||
}
|
||||
elif section == "crawler":
|
||||
result = crawler_config
|
||||
elif section == "push":
|
||||
result = push_config
|
||||
elif section == "keywords":
|
||||
result = keywords_config
|
||||
elif section == "weights":
|
||||
result = weights_config
|
||||
else:
|
||||
result = {}
|
||||
|
||||
# 缓存结果
|
||||
self.cache.set(cache_key, result)
|
||||
|
||||
return result
|
||||
|
||||
def get_system_status(self) -> Dict:
|
||||
"""
|
||||
获取系统运行状态
|
||||
|
||||
Returns:
|
||||
系统状态字典
|
||||
"""
|
||||
# 获取数据统计
|
||||
output_dir = self.parser.project_root / "output"
|
||||
|
||||
total_storage = 0
|
||||
oldest_record = None
|
||||
latest_record = None
|
||||
total_news = 0
|
||||
|
||||
if output_dir.exists():
|
||||
# 遍历日期文件夹
|
||||
for date_folder in output_dir.iterdir():
|
||||
if date_folder.is_dir():
|
||||
# 解析日期
|
||||
try:
|
||||
date_str = date_folder.name
|
||||
# 格式: YYYY年MM月DD日
|
||||
date_match = re.match(r'(\d{4})年(\d{2})月(\d{2})日', date_str)
|
||||
if date_match:
|
||||
folder_date = datetime(
|
||||
int(date_match.group(1)),
|
||||
int(date_match.group(2)),
|
||||
int(date_match.group(3))
|
||||
)
|
||||
|
||||
if oldest_record is None or folder_date < oldest_record:
|
||||
oldest_record = folder_date
|
||||
if latest_record is None or folder_date > latest_record:
|
||||
latest_record = folder_date
|
||||
|
||||
except:
|
||||
pass
|
||||
|
||||
# 计算存储大小
|
||||
for item in date_folder.rglob("*"):
|
||||
if item.is_file():
|
||||
total_storage += item.stat().st_size
|
||||
|
||||
# 读取版本信息
|
||||
version_file = self.parser.project_root / "version"
|
||||
version = "unknown"
|
||||
if version_file.exists():
|
||||
try:
|
||||
with open(version_file, "r") as f:
|
||||
version = f.read().strip()
|
||||
except:
|
||||
pass
|
||||
|
||||
return {
|
||||
"system": {
|
||||
"version": version,
|
||||
"project_root": str(self.parser.project_root)
|
||||
},
|
||||
"data": {
|
||||
"total_storage": f"{total_storage / 1024 / 1024:.2f} MB",
|
||||
"oldest_record": oldest_record.strftime("%Y-%m-%d") if oldest_record else None,
|
||||
"latest_record": latest_record.strftime("%Y-%m-%d") if latest_record else None,
|
||||
},
|
||||
"cache": self.cache.get_stats(),
|
||||
"health": "healthy"
|
||||
}
|
||||
355
mcp_server/services/parser_service.py
Normal file
355
mcp_server/services/parser_service.py
Normal file
@ -0,0 +1,355 @@
|
||||
"""
|
||||
文件解析服务
|
||||
|
||||
提供txt格式新闻数据和YAML配置文件的解析功能。
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Tuple, Optional
|
||||
from datetime import datetime
|
||||
|
||||
import yaml
|
||||
|
||||
from ..utils.errors import FileParseError, DataNotFoundError
|
||||
from .cache_service import get_cache
|
||||
|
||||
|
||||
class ParserService:
|
||||
"""文件解析服务类"""
|
||||
|
||||
def __init__(self, project_root: str = None):
|
||||
"""
|
||||
初始化解析服务
|
||||
|
||||
Args:
|
||||
project_root: 项目根目录，默认为当前目录的父目录
|
||||
"""
|
||||
if project_root is None:
|
||||
# 获取当前文件所在目录的父目录的父目录
|
||||
current_file = Path(__file__)
|
||||
self.project_root = current_file.parent.parent.parent
|
||||
else:
|
||||
self.project_root = Path(project_root)
|
||||
|
||||
# 初始化缓存服务
|
||||
self.cache = get_cache()
|
||||
|
||||
@staticmethod
|
||||
def clean_title(title: str) -> str:
|
||||
"""
|
||||
清理标题文本
|
||||
|
||||
Args:
|
||||
title: 原始标题
|
||||
|
||||
Returns:
|
||||
清理后的标题
|
||||
"""
|
||||
# 移除多余空白
|
||||
title = re.sub(r'\s+', ' ', title)
|
||||
# 移除特殊字符
|
||||
title = title.strip()
|
||||
return title
|
||||
|
||||
def parse_txt_file(self, file_path: Path) -> Tuple[Dict, Dict]:
|
||||
"""
|
||||
解析单个txt文件的标题数据
|
||||
|
||||
Args:
|
||||
file_path: txt文件路径
|
||||
|
||||
Returns:
|
||||
(titles_by_id, id_to_name) 元组
|
||||
- titles_by_id: {platform_id: {title: {ranks, url, mobileUrl}}}
|
||||
- id_to_name: {platform_id: platform_name}
|
||||
|
||||
Raises:
|
||||
FileParseError: 文件解析错误
|
||||
"""
|
||||
if not file_path.exists():
|
||||
raise FileParseError(str(file_path), "文件不存在")
|
||||
|
||||
titles_by_id = {}
|
||||
id_to_name = {}
|
||||
|
||||
try:
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
sections = content.split("\n\n")
|
||||
|
||||
for section in sections:
|
||||
if not section.strip() or "==== 以下ID请求失败 ====" in section:
|
||||
continue
|
||||
|
||||
lines = section.strip().split("\n")
|
||||
if len(lines) < 2:
|
||||
continue
|
||||
|
||||
# 解析header: id | name 或 id
|
||||
header_line = lines[0].strip()
|
||||
if " | " in header_line:
|
||||
parts = header_line.split(" | ", 1)
|
||||
source_id = parts[0].strip()
|
||||
name = parts[1].strip()
|
||||
id_to_name[source_id] = name
|
||||
else:
|
||||
source_id = header_line
|
||||
id_to_name[source_id] = source_id
|
||||
|
||||
titles_by_id[source_id] = {}
|
||||
|
||||
# 解析标题行
|
||||
for line in lines[1:]:
|
||||
if line.strip():
|
||||
try:
|
||||
title_part = line.strip()
|
||||
rank = None
|
||||
|
||||
# 提取排名
|
||||
if ". " in title_part and title_part.split(". ")[0].isdigit():
|
||||
rank_str, title_part = title_part.split(". ", 1)
|
||||
rank = int(rank_str)
|
||||
|
||||
# 提取 MOBILE URL
|
||||
mobile_url = ""
|
||||
if " [MOBILE:" in title_part:
|
||||
title_part, mobile_part = title_part.rsplit(" [MOBILE:", 1)
|
||||
if mobile_part.endswith("]"):
|
||||
mobile_url = mobile_part[:-1]
|
||||
|
||||
# 提取 URL
|
||||
url = ""
|
||||
if " [URL:" in title_part:
|
||||
title_part, url_part = title_part.rsplit(" [URL:", 1)
|
||||
if url_part.endswith("]"):
|
||||
url = url_part[:-1]
|
||||
|
||||
title = self.clean_title(title_part.strip())
|
||||
ranks = [rank] if rank is not None else [1]
|
||||
|
||||
titles_by_id[source_id][title] = {
|
||||
"ranks": ranks,
|
||||
"url": url,
|
||||
"mobileUrl": mobile_url,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
# 忽略单行解析错误
|
||||
continue
|
||||
|
||||
except Exception as e:
|
||||
raise FileParseError(str(file_path), str(e))
|
||||
|
||||
return titles_by_id, id_to_name
|
||||
|
||||
def get_date_folder_name(self, date: datetime = None) -> str:
|
||||
"""
|
||||
获取日期文件夹名称
|
||||
|
||||
Args:
|
||||
date: 日期对象，默认为今天
|
||||
|
||||
Returns:
|
||||
文件夹名称，格式: YYYY年MM月DD日
|
||||
"""
|
||||
if date is None:
|
||||
date = datetime.now()
|
||||
return date.strftime("%Y年%m月%d日")
|
||||
|
||||
def read_all_titles_for_date(
|
||||
self,
|
||||
date: datetime = None,
|
||||
platform_ids: Optional[List[str]] = None
|
||||
) -> Tuple[Dict, Dict, Dict]:
|
||||
"""
|
||||
读取指定日期的所有标题文件（带缓存）
|
||||
|
||||
Args:
|
||||
date: 日期对象，默认为今天
|
||||
platform_ids: 平台ID列表，None表示所有平台
|
||||
|
||||
Returns:
|
||||
(all_titles, id_to_name, all_timestamps) 元组
|
||||
- all_titles: {platform_id: {title: {ranks, url, mobileUrl, ...}}}
|
||||
- id_to_name: {platform_id: platform_name}
|
||||
- all_timestamps: {filename: timestamp}
|
||||
|
||||
Raises:
|
||||
DataNotFoundError: 数据不存在
|
||||
"""
|
||||
# 生成缓存键
|
||||
date_str = self.get_date_folder_name(date)
|
||||
platform_key = ','.join(sorted(platform_ids)) if platform_ids else 'all'
|
||||
cache_key = f"read_all_titles:{date_str}:{platform_key}"
|
||||
|
||||
# 尝试从缓存获取
|
||||
# 对于历史数据（非今天），使用更长的缓存时间（1小时）
|
||||
# 对于今天的数据，使用较短的缓存时间（15分钟），因为可能有新数据
|
||||
is_today = (date is None) or (date.date() == datetime.now().date())
|
||||
ttl = 900 if is_today else 3600 # 15分钟 vs 1小时
|
||||
|
||||
cached = self.cache.get(cache_key, ttl=ttl)
|
||||
if cached:
|
||||
return cached
|
||||
|
||||
# 缓存未命中，读取文件
|
||||
date_folder = self.get_date_folder_name(date)
|
||||
txt_dir = self.project_root / "output" / date_folder / "txt"
|
||||
|
||||
if not txt_dir.exists():
|
||||
raise DataNotFoundError(
|
||||
f"未找到 {date_folder} 的数据目录",
|
||||
suggestion="请先运行爬虫或检查日期是否正确"
|
||||
)
|
||||
|
||||
all_titles = {}
|
||||
id_to_name = {}
|
||||
all_timestamps = {}
|
||||
|
||||
# 读取所有txt文件
|
||||
txt_files = sorted(txt_dir.glob("*.txt"))
|
||||
|
||||
if not txt_files:
|
||||
raise DataNotFoundError(
|
||||
f"{date_folder} 没有数据文件",
|
||||
suggestion="请等待爬虫任务完成"
|
||||
)
|
||||
|
||||
for txt_file in txt_files:
|
||||
try:
|
||||
titles_by_id, file_id_to_name = self.parse_txt_file(txt_file)
|
||||
|
||||
# 更新id_to_name
|
||||
id_to_name.update(file_id_to_name)
|
||||
|
||||
# 合并标题数据
|
||||
for platform_id, titles in titles_by_id.items():
|
||||
# 如果指定了平台过滤
|
||||
if platform_ids and platform_id not in platform_ids:
|
||||
continue
|
||||
|
||||
if platform_id not in all_titles:
|
||||
all_titles[platform_id] = {}
|
||||
|
||||
for title, info in titles.items():
|
||||
if title in all_titles[platform_id]:
|
||||
# 合并排名
|
||||
all_titles[platform_id][title]["ranks"].extend(info["ranks"])
|
||||
else:
|
||||
all_titles[platform_id][title] = info.copy()
|
||||
|
||||
# 记录文件时间戳
|
||||
all_timestamps[txt_file.name] = txt_file.stat().st_mtime
|
||||
|
||||
except Exception as e:
|
||||
# 忽略单个文件的解析错误，继续处理其他文件
|
||||
print(f"Warning: 解析文件 {txt_file} 失败: {e}")
|
||||
continue
|
||||
|
||||
if not all_titles:
|
||||
raise DataNotFoundError(
|
||||
f"{date_folder} 没有有效的数据",
|
||||
suggestion="请检查数据文件格式或重新运行爬虫"
|
||||
)
|
||||
|
||||
# 缓存结果
|
||||
result = (all_titles, id_to_name, all_timestamps)
|
||||
self.cache.set(cache_key, result)
|
||||
|
||||
return result
|
||||
|
||||
def parse_yaml_config(self, config_path: str = None) -> dict:
|
||||
"""
|
||||
解析YAML配置文件
|
||||
|
||||
Args:
|
||||
config_path: 配置文件路径，默认为 config/config.yaml
|
||||
|
||||
Returns:
|
||||
配置字典
|
||||
|
||||
Raises:
|
||||
FileParseError: 配置文件解析错误
|
||||
"""
|
||||
if config_path is None:
|
||||
config_path = self.project_root / "config" / "config.yaml"
|
||||
else:
|
||||
config_path = Path(config_path)
|
||||
|
||||
if not config_path.exists():
|
||||
raise FileParseError(str(config_path), "配置文件不存在")
|
||||
|
||||
try:
|
||||
with open(config_path, "r", encoding="utf-8") as f:
|
||||
config_data = yaml.safe_load(f)
|
||||
return config_data
|
||||
except Exception as e:
|
||||
raise FileParseError(str(config_path), str(e))
|
||||
|
||||
def parse_frequency_words(self, words_file: str = None) -> List[Dict]:
|
||||
"""
|
||||
解析关键词配置文件
|
||||
|
||||
Args:
|
||||
words_file: 关键词文件路径，默认为 config/frequency_words.txt
|
||||
|
||||
Returns:
|
||||
词组列表
|
||||
|
||||
Raises:
|
||||
FileParseError: 文件解析错误
|
||||
"""
|
||||
if words_file is None:
|
||||
words_file = self.project_root / "config" / "frequency_words.txt"
|
||||
else:
|
||||
words_file = Path(words_file)
|
||||
|
||||
if not words_file.exists():
|
||||
return []
|
||||
|
||||
word_groups = []
|
||||
|
||||
try:
|
||||
with open(words_file, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
|
||||
# 使用 | 分隔符
|
||||
parts = [p.strip() for p in line.split("|")]
|
||||
if not parts:
|
||||
continue
|
||||
|
||||
group = {
|
||||
"required": [],
|
||||
"normal": [],
|
||||
"filter_words": []
|
||||
}
|
||||
|
||||
for part in parts:
|
||||
if not part:
|
||||
continue
|
||||
|
||||
words = [w.strip() for w in part.split(",")]
|
||||
for word in words:
|
||||
if not word:
|
||||
continue
|
||||
if word.endswith("+"):
|
||||
# 必须词
|
||||
group["required"].append(word[:-1])
|
||||
elif word.endswith("!"):
|
||||
# 过滤词
|
||||
group["filter_words"].append(word[:-1])
|
||||
else:
|
||||
# 普通词
|
||||
group["normal"].append(word)
|
||||
|
||||
if group["required"] or group["normal"]:
|
||||
word_groups.append(group)
|
||||
|
||||
except Exception as e:
|
||||
raise FileParseError(str(words_file), str(e))
|
||||
|
||||
return word_groups
|
||||
5
mcp_server/tools/__init__.py
Normal file
5
mcp_server/tools/__init__.py
Normal file
@ -0,0 +1,5 @@
|
||||
"""
|
||||
MCP 工具模块
|
||||
|
||||
包含所有MCP工具的实现。
|
||||
"""
|
||||
1989
mcp_server/tools/analytics.py
Normal file
1989
mcp_server/tools/analytics.py
Normal file
File diff suppressed because it is too large
Load Diff
66
mcp_server/tools/config_mgmt.py
Normal file
66
mcp_server/tools/config_mgmt.py
Normal file
@ -0,0 +1,66 @@
|
||||
"""
|
||||
配置管理工具
|
||||
|
||||
实现配置查询和管理功能。
|
||||
"""
|
||||
|
||||
from typing import Dict, Optional
|
||||
|
||||
from ..services.data_service import DataService
|
||||
from ..utils.validators import validate_config_section
|
||||
from ..utils.errors import MCPError
|
||||
|
||||
|
||||
class ConfigManagementTools:
|
||||
"""配置管理工具类"""
|
||||
|
||||
def __init__(self, project_root: str = None):
|
||||
"""
|
||||
初始化配置管理工具
|
||||
|
||||
Args:
|
||||
project_root: 项目根目录
|
||||
"""
|
||||
self.data_service = DataService(project_root)
|
||||
|
||||
def get_current_config(self, section: Optional[str] = None) -> Dict:
|
||||
"""
|
||||
获取当前系统配置
|
||||
|
||||
Args:
|
||||
section: 配置节 - all/crawler/push/keywords/weights，默认all
|
||||
|
||||
Returns:
|
||||
配置字典
|
||||
|
||||
Example:
|
||||
>>> tools = ConfigManagementTools()
|
||||
>>> result = tools.get_current_config(section="crawler")
|
||||
>>> print(result['crawler']['platforms'])
|
||||
"""
|
||||
try:
|
||||
# 参数验证
|
||||
section = validate_config_section(section)
|
||||
|
||||
# 获取配置
|
||||
config = self.data_service.get_current_config(section=section)
|
||||
|
||||
return {
|
||||
"config": config,
|
||||
"section": section,
|
||||
"success": True
|
||||
}
|
||||
|
||||
except MCPError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": e.to_dict()
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": {
|
||||
"code": "INTERNAL_ERROR",
|
||||
"message": str(e)
|
||||
}
|
||||
}
|
||||
284
mcp_server/tools/data_query.py
Normal file
284
mcp_server/tools/data_query.py
Normal file
@ -0,0 +1,284 @@
|
||||
"""
|
||||
数据查询工具
|
||||
|
||||
实现P0核心的数据查询工具。
|
||||
"""
|
||||
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from ..services.data_service import DataService
|
||||
from ..utils.validators import (
|
||||
validate_platforms,
|
||||
validate_limit,
|
||||
validate_keyword,
|
||||
validate_date_range,
|
||||
validate_top_n,
|
||||
validate_mode,
|
||||
validate_date_query
|
||||
)
|
||||
from ..utils.errors import MCPError
|
||||
|
||||
|
||||
class DataQueryTools:
|
||||
"""数据查询工具类"""
|
||||
|
||||
def __init__(self, project_root: str = None):
|
||||
"""
|
||||
初始化数据查询工具
|
||||
|
||||
Args:
|
||||
project_root: 项目根目录
|
||||
"""
|
||||
self.data_service = DataService(project_root)
|
||||
|
||||
def get_latest_news(
|
||||
self,
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: Optional[int] = None,
|
||||
include_url: bool = False
|
||||
) -> Dict:
|
||||
"""
|
||||
获取最新一批爬取的新闻数据
|
||||
|
||||
Args:
|
||||
platforms: 平台ID列表，如 ['zhihu', 'weibo']
|
||||
limit: 返回条数限制，默认20
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
新闻列表字典
|
||||
|
||||
Example:
|
||||
>>> tools = DataQueryTools()
|
||||
>>> result = tools.get_latest_news(platforms=['zhihu'], limit=10)
|
||||
>>> print(result['total'])
|
||||
10
|
||||
"""
|
||||
try:
|
||||
# 参数验证
|
||||
platforms = validate_platforms(platforms)
|
||||
limit = validate_limit(limit, default=50)
|
||||
|
||||
# 获取数据
|
||||
news_list = self.data_service.get_latest_news(
|
||||
platforms=platforms,
|
||||
limit=limit,
|
||||
include_url=include_url
|
||||
)
|
||||
|
||||
return {
|
||||
"news": news_list,
|
||||
"total": len(news_list),
|
||||
"platforms": platforms,
|
||||
"success": True
|
||||
}
|
||||
|
||||
except MCPError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": e.to_dict()
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": {
|
||||
"code": "INTERNAL_ERROR",
|
||||
"message": str(e)
|
||||
}
|
||||
}
|
||||
|
||||
def search_news_by_keyword(
|
||||
self,
|
||||
keyword: str,
|
||||
date_range: Optional[Dict] = None,
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: Optional[int] = None
|
||||
) -> Dict:
|
||||
"""
|
||||
按关键词搜索历史新闻
|
||||
|
||||
Args:
|
||||
keyword: 搜索关键词（必需）
|
||||
date_range: 日期范围，格式: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}
|
||||
platforms: 平台过滤列表
|
||||
limit: 返回条数限制（可选，默认返回所有）
|
||||
|
||||
Returns:
|
||||
搜索结果字典
|
||||
|
||||
Example:
|
||||
>>> tools = DataQueryTools()
|
||||
>>> result = tools.search_news_by_keyword(
|
||||
... keyword="人工智能",
|
||||
... date_range={"start": "2025-10-01", "end": "2025-10-11"},
|
||||
... limit=50
|
||||
... )
|
||||
>>> print(result['total'])
|
||||
"""
|
||||
try:
|
||||
# 参数验证
|
||||
keyword = validate_keyword(keyword)
|
||||
date_range_tuple = validate_date_range(date_range)
|
||||
platforms = validate_platforms(platforms)
|
||||
|
||||
if limit is not None:
|
||||
limit = validate_limit(limit, default=100)
|
||||
|
||||
# 搜索数据
|
||||
search_result = self.data_service.search_news_by_keyword(
|
||||
keyword=keyword,
|
||||
date_range=date_range_tuple,
|
||||
platforms=platforms,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
return {
|
||||
**search_result,
|
||||
"success": True
|
||||
}
|
||||
|
||||
except MCPError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": e.to_dict()
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": {
|
||||
"code": "INTERNAL_ERROR",
|
||||
"message": str(e)
|
||||
}
|
||||
}
|
||||
|
||||
def get_trending_topics(
|
||||
self,
|
||||
top_n: Optional[int] = None,
|
||||
mode: Optional[str] = None
|
||||
) -> Dict:
|
||||
"""
|
||||
获取个人关注词的新闻出现频率统计
|
||||
|
||||
注意：本工具基于 config/frequency_words.txt 中的个人关注词列表进行统计，
|
||||
而不是自动从新闻中提取热点话题。这是一个个人可定制的关注词列表，
|
||||
用户可以根据自己的兴趣添加或删除关注词。
|
||||
|
||||
Args:
|
||||
top_n: 返回TOP N关注词，默认10
|
||||
mode: 模式 - daily(当日累计), current(最新一批), incremental(增量)
|
||||
|
||||
Returns:
|
||||
关注词频率统计字典，包含每个关注词在新闻中出现的次数
|
||||
|
||||
Example:
|
||||
>>> tools = DataQueryTools()
|
||||
>>> result = tools.get_trending_topics(top_n=5, mode="current")
|
||||
>>> print(len(result['topics']))
|
||||
5
|
||||
>>> # 返回的是你在 frequency_words.txt 中设置的关注词的频率统计
|
||||
"""
|
||||
try:
|
||||
# 参数验证
|
||||
top_n = validate_top_n(top_n, default=10)
|
||||
valid_modes = ["daily", "current", "incremental"]
|
||||
mode = validate_mode(mode, valid_modes, default="current")
|
||||
|
||||
# 获取趋势话题
|
||||
trending_result = self.data_service.get_trending_topics(
|
||||
top_n=top_n,
|
||||
mode=mode
|
||||
)
|
||||
|
||||
return {
|
||||
**trending_result,
|
||||
"success": True
|
||||
}
|
||||
|
||||
except MCPError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": e.to_dict()
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": {
|
||||
"code": "INTERNAL_ERROR",
|
||||
"message": str(e)
|
||||
}
|
||||
}
|
||||
|
||||
def get_news_by_date(
|
||||
self,
|
||||
date_query: Optional[str] = None,
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: Optional[int] = None,
|
||||
include_url: bool = False
|
||||
) -> Dict:
|
||||
"""
|
||||
按日期查询新闻，支持自然语言日期
|
||||
|
||||
Args:
|
||||
date_query: 日期查询字符串（可选，默认"今天"），支持：
|
||||
- 相对日期：今天、昨天、前天、3天前、yesterday、3 days ago
|
||||
- 星期：上周一、本周三、last monday、this friday
|
||||
- 绝对日期：2025-10-10、10月10日、2025年10月10日
|
||||
platforms: 平台ID列表，如 ['zhihu', 'weibo']
|
||||
limit: 返回条数限制，默认50
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
新闻列表字典
|
||||
|
||||
Example:
|
||||
>>> tools = DataQueryTools()
|
||||
>>> # 不指定日期，默认查询今天
|
||||
>>> result = tools.get_news_by_date(platforms=['zhihu'], limit=20)
|
||||
>>> # 指定日期
|
||||
>>> result = tools.get_news_by_date(
|
||||
... date_query="昨天",
|
||||
... platforms=['zhihu'],
|
||||
... limit=20
|
||||
... )
|
||||
>>> print(result['total'])
|
||||
20
|
||||
"""
|
||||
try:
|
||||
# 参数验证 - 默认今天
|
||||
if date_query is None:
|
||||
date_query = "今天"
|
||||
target_date = validate_date_query(date_query)
|
||||
platforms = validate_platforms(platforms)
|
||||
limit = validate_limit(limit, default=50)
|
||||
|
||||
# 获取数据
|
||||
news_list = self.data_service.get_news_by_date(
|
||||
target_date=target_date,
|
||||
platforms=platforms,
|
||||
limit=limit,
|
||||
include_url=include_url
|
||||
)
|
||||
|
||||
return {
|
||||
"news": news_list,
|
||||
"total": len(news_list),
|
||||
"date": target_date.strftime("%Y-%m-%d"),
|
||||
"date_query": date_query,
|
||||
"platforms": platforms,
|
||||
"success": True
|
||||
}
|
||||
|
||||
except MCPError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": e.to_dict()
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": {
|
||||
"code": "INTERNAL_ERROR",
|
||||
"message": str(e)
|
||||
}
|
||||
}
|
||||
|
||||
664
mcp_server/tools/search_tools.py
Normal file
664
mcp_server/tools/search_tools.py
Normal file
@ -0,0 +1,664 @@
|
||||
"""
|
||||
智能新闻检索工具
|
||||
|
||||
提供模糊搜索、链接查询、历史相关新闻检索等高级搜索功能。
|
||||
"""
|
||||
|
||||
import re
|
||||
from collections import Counter
|
||||
from datetime import datetime, timedelta
|
||||
from difflib import SequenceMatcher
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
|
||||
from ..services.data_service import DataService
|
||||
from ..utils.validators import validate_keyword, validate_limit
|
||||
from ..utils.errors import MCPError, InvalidParameterError, DataNotFoundError
|
||||
|
||||
|
||||
class SearchTools:
|
||||
"""智能新闻检索工具类"""
|
||||
|
||||
def __init__(self, project_root: str = None):
|
||||
"""
|
||||
初始化智能检索工具
|
||||
|
||||
Args:
|
||||
project_root: 项目根目录
|
||||
"""
|
||||
self.data_service = DataService(project_root)
|
||||
# 中文停用词列表
|
||||
self.stopwords = {
|
||||
'的', '了', '在', '是', '我', '有', '和', '就', '不', '人', '都', '一',
|
||||
'一个', '上', '也', '很', '到', '说', '要', '去', '你', '会', '着', '没有',
|
||||
'看', '好', '自己', '这', '那', '来', '被', '与', '为', '对', '将', '从',
|
||||
'以', '及', '等', '但', '或', '而', '于', '中', '由', '可', '可以', '已',
|
||||
'已经', '还', '更', '最', '再', '因为', '所以', '如果', '虽然', '然而'
|
||||
}
|
||||
|
||||
def search_news_unified(
|
||||
self,
|
||||
query: str,
|
||||
search_mode: str = "keyword",
|
||||
date_range: Optional[Dict[str, str]] = None,
|
||||
platforms: Optional[List[str]] = None,
|
||||
limit: int = 50,
|
||||
sort_by: str = "relevance",
|
||||
threshold: float = 0.6,
|
||||
include_url: bool = False
|
||||
) -> Dict:
|
||||
"""
|
||||
统一新闻搜索工具 - 整合多种搜索模式
|
||||
|
||||
Args:
|
||||
query: 查询内容（必需）- 关键词、内容片段或实体名称
|
||||
search_mode: 搜索模式，可选值：
|
||||
- "keyword": 精确关键词匹配（默认）
|
||||
- "fuzzy": 模糊内容匹配（使用相似度算法）
|
||||
- "entity": 实体名称搜索（自动按权重排序）
|
||||
date_range: 日期范围，格式: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}
|
||||
不指定则默认查询今天
|
||||
platforms: 平台过滤列表，如 ['zhihu', 'weibo']
|
||||
limit: 返回条数限制，默认50
|
||||
sort_by: 排序方式，可选值：
|
||||
- "relevance": 按相关度排序（默认）
|
||||
- "weight": 按新闻权重排序
|
||||
- "date": 按日期排序
|
||||
threshold: 相似度阈值（仅fuzzy模式有效），0-1之间，默认0.6
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
搜索结果字典，包含匹配的新闻列表
|
||||
|
||||
Examples:
|
||||
- search_news_unified(query="人工智能", search_mode="keyword")
|
||||
- search_news_unified(query="特斯拉降价", search_mode="fuzzy", threshold=0.4)
|
||||
- search_news_unified(query="马斯克", search_mode="entity", limit=20)
|
||||
- search_news_unified(query="iPhone 16发布", search_mode="keyword")
|
||||
"""
|
||||
try:
|
||||
# 参数验证
|
||||
query = validate_keyword(query)
|
||||
|
||||
if search_mode not in ["keyword", "fuzzy", "entity"]:
|
||||
raise InvalidParameterError(
|
||||
f"无效的搜索模式: {search_mode}",
|
||||
suggestion="支持的模式: keyword, fuzzy, entity"
|
||||
)
|
||||
|
||||
if sort_by not in ["relevance", "weight", "date"]:
|
||||
raise InvalidParameterError(
|
||||
f"无效的排序方式: {sort_by}",
|
||||
suggestion="支持的排序: relevance, weight, date"
|
||||
)
|
||||
|
||||
limit = validate_limit(limit, default=50)
|
||||
threshold = max(0.0, min(1.0, threshold))
|
||||
|
||||
# 处理日期范围
|
||||
if date_range:
|
||||
from ..utils.validators import validate_date_range
|
||||
date_range_tuple = validate_date_range(date_range)
|
||||
start_date, end_date = date_range_tuple
|
||||
else:
|
||||
# 默认今天
|
||||
start_date = end_date = datetime.now()
|
||||
|
||||
# 收集所有匹配的新闻
|
||||
all_matches = []
|
||||
current_date = start_date
|
||||
|
||||
while current_date <= end_date:
|
||||
try:
|
||||
all_titles, id_to_name, timestamps = self.data_service.parser.read_all_titles_for_date(
|
||||
date=current_date,
|
||||
platform_ids=platforms
|
||||
)
|
||||
|
||||
# 根据搜索模式执行不同的搜索逻辑
|
||||
if search_mode == "keyword":
|
||||
matches = self._search_by_keyword_mode(
|
||||
query, all_titles, id_to_name, current_date, include_url
|
||||
)
|
||||
elif search_mode == "fuzzy":
|
||||
matches = self._search_by_fuzzy_mode(
|
||||
query, all_titles, id_to_name, current_date, threshold, include_url
|
||||
)
|
||||
else: # entity
|
||||
matches = self._search_by_entity_mode(
|
||||
query, all_titles, id_to_name, current_date, include_url
|
||||
)
|
||||
|
||||
all_matches.extend(matches)
|
||||
|
||||
except DataNotFoundError:
|
||||
# 该日期没有数据，继续下一天
|
||||
pass
|
||||
|
||||
current_date += timedelta(days=1)
|
||||
|
||||
if not all_matches:
|
||||
time_desc = "今天" if start_date == end_date else f"{start_date.strftime('%Y-%m-%d')} 至 {end_date.strftime('%Y-%m-%d')}"
|
||||
return {
|
||||
"success": True,
|
||||
"results": [],
|
||||
"total": 0,
|
||||
"query": query,
|
||||
"search_mode": search_mode,
|
||||
"time_range": time_desc,
|
||||
"message": f"未找到匹配的新闻（{time_desc}）"
|
||||
}
|
||||
|
||||
# 统一排序逻辑
|
||||
if sort_by == "relevance":
|
||||
all_matches.sort(key=lambda x: x.get("similarity_score", 1.0), reverse=True)
|
||||
elif sort_by == "weight":
|
||||
from .analytics import calculate_news_weight
|
||||
all_matches.sort(key=lambda x: calculate_news_weight(x), reverse=True)
|
||||
elif sort_by == "date":
|
||||
all_matches.sort(key=lambda x: x.get("date", ""), reverse=True)
|
||||
|
||||
# 限制返回数量
|
||||
results = all_matches[:limit]
|
||||
|
||||
# 构建时间范围描述
|
||||
if start_date == end_date:
|
||||
time_range_desc = start_date.strftime("%Y-%m-%d")
|
||||
else:
|
||||
time_range_desc = f"{start_date.strftime('%Y-%m-%d')} 至 {end_date.strftime('%Y-%m-%d')}"
|
||||
|
||||
result = {
|
||||
"success": True,
|
||||
"summary": {
|
||||
"total_found": len(all_matches),
|
||||
"returned_count": len(results),
|
||||
"requested_limit": limit,
|
||||
"search_mode": search_mode,
|
||||
"query": query,
|
||||
"platforms": platforms or "所有平台",
|
||||
"time_range": time_range_desc,
|
||||
"sort_by": sort_by
|
||||
},
|
||||
"results": results
|
||||
}
|
||||
|
||||
if search_mode == "fuzzy":
|
||||
result["summary"]["threshold"] = threshold
|
||||
if len(all_matches) < limit:
|
||||
result["note"] = f"模糊搜索模式下，相似度阈值 {threshold} 仅匹配到 {len(all_matches)} 条结果"
|
||||
|
||||
return result
|
||||
|
||||
except MCPError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": e.to_dict()
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": {
|
||||
"code": "INTERNAL_ERROR",
|
||||
"message": str(e)
|
||||
}
|
||||
}
|
||||
|
||||
def _search_by_keyword_mode(
|
||||
self,
|
||||
query: str,
|
||||
all_titles: Dict,
|
||||
id_to_name: Dict,
|
||||
current_date: datetime,
|
||||
include_url: bool
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
关键词搜索模式（精确匹配）
|
||||
|
||||
Args:
|
||||
query: 搜索关键词
|
||||
all_titles: 所有标题字典
|
||||
id_to_name: 平台ID到名称映射
|
||||
current_date: 当前日期
|
||||
|
||||
Returns:
|
||||
匹配的新闻列表
|
||||
"""
|
||||
matches = []
|
||||
query_lower = query.lower()
|
||||
|
||||
for platform_id, titles in all_titles.items():
|
||||
platform_name = id_to_name.get(platform_id, platform_id)
|
||||
|
||||
for title, info in titles.items():
|
||||
# 精确包含判断
|
||||
if query_lower in title.lower():
|
||||
news_item = {
|
||||
"title": title,
|
||||
"platform": platform_id,
|
||||
"platform_name": platform_name,
|
||||
"date": current_date.strftime("%Y-%m-%d"),
|
||||
"similarity_score": 1.0, # 精确匹配，相似度为1
|
||||
"ranks": info.get("ranks", []),
|
||||
"count": len(info.get("ranks", [])),
|
||||
"rank": info["ranks"][0] if info["ranks"] else 999
|
||||
}
|
||||
|
||||
# 条件性添加 URL 字段
|
||||
if include_url:
|
||||
news_item["url"] = info.get("url", "")
|
||||
news_item["mobileUrl"] = info.get("mobileUrl", "")
|
||||
|
||||
matches.append(news_item)
|
||||
|
||||
return matches
|
||||
|
||||
def _search_by_fuzzy_mode(
|
||||
self,
|
||||
query: str,
|
||||
all_titles: Dict,
|
||||
id_to_name: Dict,
|
||||
current_date: datetime,
|
||||
threshold: float,
|
||||
include_url: bool
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
模糊搜索模式（使用相似度算法）
|
||||
|
||||
Args:
|
||||
query: 搜索内容
|
||||
all_titles: 所有标题字典
|
||||
id_to_name: 平台ID到名称映射
|
||||
current_date: 当前日期
|
||||
threshold: 相似度阈值
|
||||
|
||||
Returns:
|
||||
匹配的新闻列表
|
||||
"""
|
||||
matches = []
|
||||
|
||||
for platform_id, titles in all_titles.items():
|
||||
platform_name = id_to_name.get(platform_id, platform_id)
|
||||
|
||||
for title, info in titles.items():
|
||||
# 模糊匹配
|
||||
is_match, similarity = self._fuzzy_match(query, title, threshold)
|
||||
|
||||
if is_match:
|
||||
news_item = {
|
||||
"title": title,
|
||||
"platform": platform_id,
|
||||
"platform_name": platform_name,
|
||||
"date": current_date.strftime("%Y-%m-%d"),
|
||||
"similarity_score": round(similarity, 4),
|
||||
"ranks": info.get("ranks", []),
|
||||
"count": len(info.get("ranks", [])),
|
||||
"rank": info["ranks"][0] if info["ranks"] else 999
|
||||
}
|
||||
|
||||
# 条件性添加 URL 字段
|
||||
if include_url:
|
||||
news_item["url"] = info.get("url", "")
|
||||
news_item["mobileUrl"] = info.get("mobileUrl", "")
|
||||
|
||||
matches.append(news_item)
|
||||
|
||||
return matches
|
||||
|
||||
def _search_by_entity_mode(
|
||||
self,
|
||||
query: str,
|
||||
all_titles: Dict,
|
||||
id_to_name: Dict,
|
||||
current_date: datetime,
|
||||
include_url: bool
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
实体搜索模式（自动按权重排序）
|
||||
|
||||
Args:
|
||||
query: 实体名称
|
||||
all_titles: 所有标题字典
|
||||
id_to_name: 平台ID到名称映射
|
||||
current_date: 当前日期
|
||||
|
||||
Returns:
|
||||
匹配的新闻列表
|
||||
"""
|
||||
matches = []
|
||||
|
||||
for platform_id, titles in all_titles.items():
|
||||
platform_name = id_to_name.get(platform_id, platform_id)
|
||||
|
||||
for title, info in titles.items():
|
||||
# 实体搜索：精确包含实体名称
|
||||
if query in title:
|
||||
news_item = {
|
||||
"title": title,
|
||||
"platform": platform_id,
|
||||
"platform_name": platform_name,
|
||||
"date": current_date.strftime("%Y-%m-%d"),
|
||||
"similarity_score": 1.0,
|
||||
"ranks": info.get("ranks", []),
|
||||
"count": len(info.get("ranks", [])),
|
||||
"rank": info["ranks"][0] if info["ranks"] else 999
|
||||
}
|
||||
|
||||
# 条件性添加 URL 字段
|
||||
if include_url:
|
||||
news_item["url"] = info.get("url", "")
|
||||
news_item["mobileUrl"] = info.get("mobileUrl", "")
|
||||
|
||||
matches.append(news_item)
|
||||
|
||||
return matches
|
||||
|
||||
def _calculate_similarity(self, text1: str, text2: str) -> float:
|
||||
"""
|
||||
计算两个文本的相似度
|
||||
|
||||
Args:
|
||||
text1: 文本1
|
||||
text2: 文本2
|
||||
|
||||
Returns:
|
||||
相似度分数 (0-1之间)
|
||||
"""
|
||||
# 使用 difflib.SequenceMatcher 计算序列相似度
|
||||
return SequenceMatcher(None, text1.lower(), text2.lower()).ratio()
|
||||
|
||||
def _fuzzy_match(self, query: str, text: str, threshold: float = 0.3) -> Tuple[bool, float]:
|
||||
"""
|
||||
模糊匹配函数
|
||||
|
||||
Args:
|
||||
query: 查询文本
|
||||
text: 待匹配文本
|
||||
threshold: 匹配阈值
|
||||
|
||||
Returns:
|
||||
(是否匹配, 相似度分数)
|
||||
"""
|
||||
# 直接包含判断
|
||||
if query.lower() in text.lower():
|
||||
return True, 1.0
|
||||
|
||||
# 计算整体相似度
|
||||
similarity = self._calculate_similarity(query, text)
|
||||
if similarity >= threshold:
|
||||
return True, similarity
|
||||
|
||||
# 分词后的部分匹配
|
||||
query_words = set(self._extract_keywords(query))
|
||||
text_words = set(self._extract_keywords(text))
|
||||
|
||||
if not query_words or not text_words:
|
||||
return False, 0.0
|
||||
|
||||
# 计算关键词重合度
|
||||
common_words = query_words & text_words
|
||||
keyword_overlap = len(common_words) / len(query_words)
|
||||
|
||||
if keyword_overlap >= 0.5: # 50%的关键词重合
|
||||
return True, keyword_overlap
|
||||
|
||||
return False, similarity
|
||||
|
||||
def _extract_keywords(self, text: str, min_length: int = 2) -> List[str]:
|
||||
"""
|
||||
从文本中提取关键词
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
min_length: 最小词长
|
||||
|
||||
Returns:
|
||||
关键词列表
|
||||
"""
|
||||
# 移除URL和特殊字符
|
||||
text = re.sub(r'http[s]?://\S+', '', text)
|
||||
text = re.sub(r'\[.*?\]', '', text) # 移除方括号内容
|
||||
|
||||
# 使用正则表达式分词（中文和英文）
|
||||
words = re.findall(r'[\w]+', text)
|
||||
|
||||
# 过滤停用词和短词
|
||||
keywords = [
|
||||
word for word in words
|
||||
if word and len(word) >= min_length and word not in self.stopwords
|
||||
]
|
||||
|
||||
return keywords
|
||||
|
||||
def _calculate_keyword_overlap(self, keywords1: List[str], keywords2: List[str]) -> float:
|
||||
"""
|
||||
计算两个关键词列表的重合度
|
||||
|
||||
Args:
|
||||
keywords1: 关键词列表1
|
||||
keywords2: 关键词列表2
|
||||
|
||||
Returns:
|
||||
重合度分数 (0-1之间)
|
||||
"""
|
||||
if not keywords1 or not keywords2:
|
||||
return 0.0
|
||||
|
||||
set1 = set(keywords1)
|
||||
set2 = set(keywords2)
|
||||
|
||||
# Jaccard 相似度
|
||||
intersection = len(set1 & set2)
|
||||
union = len(set1 | set2)
|
||||
|
||||
if union == 0:
|
||||
return 0.0
|
||||
|
||||
return intersection / union
|
||||
|
||||
def search_related_news_history(
|
||||
self,
|
||||
reference_text: str,
|
||||
time_range: str = "yesterday",
|
||||
start_date: Optional[datetime] = None,
|
||||
end_date: Optional[datetime] = None,
|
||||
threshold: float = 0.4,
|
||||
limit: int = 50,
|
||||
include_url: bool = False
|
||||
) -> Dict:
|
||||
"""
|
||||
在历史数据中搜索与给定新闻相关的新闻
|
||||
|
||||
Args:
|
||||
reference_text: 参考新闻标题或内容
|
||||
time_range: 时间范围预设值，可选：
|
||||
- "yesterday": 昨天
|
||||
- "last_week": 上周 (7天)
|
||||
- "last_month": 上个月 (30天)
|
||||
- "custom": 自定义日期范围（需要提供 start_date 和 end_date）
|
||||
start_date: 自定义开始日期（仅当 time_range="custom" 时有效）
|
||||
end_date: 自定义结束日期（仅当 time_range="custom" 时有效）
|
||||
threshold: 相似度阈值 (0-1之间)，默认0.4
|
||||
limit: 返回条数限制，默认50
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
搜索结果字典，包含相关新闻列表
|
||||
|
||||
Example:
|
||||
>>> tools = SearchTools()
|
||||
>>> result = tools.search_related_news_history(
|
||||
... reference_text="人工智能技术突破",
|
||||
... time_range="last_week",
|
||||
... threshold=0.4,
|
||||
... limit=50
|
||||
... )
|
||||
>>> for news in result['results']:
|
||||
... print(f"{news['date']}: {news['title']} (相似度: {news['similarity_score']})")
|
||||
"""
|
||||
try:
|
||||
# 参数验证
|
||||
reference_text = validate_keyword(reference_text)
|
||||
threshold = max(0.0, min(1.0, threshold))
|
||||
limit = validate_limit(limit, default=50)
|
||||
|
||||
# 确定查询日期范围
|
||||
today = datetime.now()
|
||||
|
||||
if time_range == "yesterday":
|
||||
search_start = today - timedelta(days=1)
|
||||
search_end = today - timedelta(days=1)
|
||||
elif time_range == "last_week":
|
||||
search_start = today - timedelta(days=7)
|
||||
search_end = today - timedelta(days=1)
|
||||
elif time_range == "last_month":
|
||||
search_start = today - timedelta(days=30)
|
||||
search_end = today - timedelta(days=1)
|
||||
elif time_range == "custom":
|
||||
if not start_date or not end_date:
|
||||
raise InvalidParameterError(
|
||||
"自定义时间范围需要提供 start_date 和 end_date",
|
||||
suggestion="请提供 start_date 和 end_date 参数"
|
||||
)
|
||||
search_start = start_date
|
||||
search_end = end_date
|
||||
else:
|
||||
raise InvalidParameterError(
|
||||
f"不支持的时间范围: {time_range}",
|
||||
suggestion="请使用 'yesterday', 'last_week', 'last_month' 或 'custom'"
|
||||
)
|
||||
|
||||
# 提取参考文本的关键词
|
||||
reference_keywords = self._extract_keywords(reference_text)
|
||||
|
||||
if not reference_keywords:
|
||||
raise InvalidParameterError(
|
||||
"无法从参考文本中提取关键词",
|
||||
suggestion="请提供更详细的文本内容"
|
||||
)
|
||||
|
||||
# 收集所有相关新闻
|
||||
all_related_news = []
|
||||
current_date = search_start
|
||||
|
||||
while current_date <= search_end:
|
||||
try:
|
||||
# 读取该日期的数据
|
||||
all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date(current_date)
|
||||
|
||||
# 搜索相关新闻
|
||||
for platform_id, titles in all_titles.items():
|
||||
platform_name = id_to_name.get(platform_id, platform_id)
|
||||
|
||||
for title, info in titles.items():
|
||||
# 计算标题相似度
|
||||
title_similarity = self._calculate_similarity(reference_text, title)
|
||||
|
||||
# 提取标题关键词
|
||||
title_keywords = self._extract_keywords(title)
|
||||
|
||||
# 计算关键词重合度
|
||||
keyword_overlap = self._calculate_keyword_overlap(
|
||||
reference_keywords,
|
||||
title_keywords
|
||||
)
|
||||
|
||||
# 综合相似度 (70% 关键词重合 + 30% 文本相似度)
|
||||
combined_score = keyword_overlap * 0.7 + title_similarity * 0.3
|
||||
|
||||
if combined_score >= threshold:
|
||||
news_item = {
|
||||
"title": title,
|
||||
"platform": platform_id,
|
||||
"platform_name": platform_name,
|
||||
"date": current_date.strftime("%Y-%m-%d"),
|
||||
"similarity_score": round(combined_score, 4),
|
||||
"keyword_overlap": round(keyword_overlap, 4),
|
||||
"text_similarity": round(title_similarity, 4),
|
||||
"common_keywords": list(set(reference_keywords) & set(title_keywords)),
|
||||
"rank": info["ranks"][0] if info["ranks"] else 0
|
||||
}
|
||||
|
||||
# 条件性添加 URL 字段
|
||||
if include_url:
|
||||
news_item["url"] = info.get("url", "")
|
||||
news_item["mobileUrl"] = info.get("mobileUrl", "")
|
||||
|
||||
all_related_news.append(news_item)
|
||||
|
||||
except DataNotFoundError:
|
||||
# 该日期没有数据，继续下一天
|
||||
pass
|
||||
except Exception as e:
|
||||
# 记录错误但继续处理其他日期
|
||||
print(f"Warning: 处理日期 {current_date.strftime('%Y-%m-%d')} 时出错: {e}")
|
||||
|
||||
# 移动到下一天
|
||||
current_date += timedelta(days=1)
|
||||
|
||||
if not all_related_news:
|
||||
return {
|
||||
"success": True,
|
||||
"results": [],
|
||||
"total": 0,
|
||||
"query": reference_text,
|
||||
"time_range": time_range,
|
||||
"date_range": {
|
||||
"start": search_start.strftime("%Y-%m-%d"),
|
||||
"end": search_end.strftime("%Y-%m-%d")
|
||||
},
|
||||
"message": "未找到相关新闻"
|
||||
}
|
||||
|
||||
# 按相似度排序
|
||||
all_related_news.sort(key=lambda x: x["similarity_score"], reverse=True)
|
||||
|
||||
# 限制返回数量
|
||||
results = all_related_news[:limit]
|
||||
|
||||
# 统计信息
|
||||
platform_distribution = Counter([news["platform"] for news in all_related_news])
|
||||
date_distribution = Counter([news["date"] for news in all_related_news])
|
||||
|
||||
result = {
|
||||
"success": True,
|
||||
"summary": {
|
||||
"total_found": len(all_related_news),
|
||||
"returned_count": len(results),
|
||||
"requested_limit": limit,
|
||||
"threshold": threshold,
|
||||
"reference_text": reference_text,
|
||||
"reference_keywords": reference_keywords,
|
||||
"time_range": time_range,
|
||||
"date_range": {
|
||||
"start": search_start.strftime("%Y-%m-%d"),
|
||||
"end": search_end.strftime("%Y-%m-%d")
|
||||
}
|
||||
},
|
||||
"results": results,
|
||||
"statistics": {
|
||||
"platform_distribution": dict(platform_distribution),
|
||||
"date_distribution": dict(date_distribution),
|
||||
"avg_similarity": round(
|
||||
sum([news["similarity_score"] for news in all_related_news]) / len(all_related_news),
|
||||
4
|
||||
) if all_related_news else 0.0
|
||||
}
|
||||
}
|
||||
|
||||
if len(all_related_news) < limit:
|
||||
result["note"] = f"相关性阈值 {threshold} 下仅找到 {len(all_related_news)} 条相关新闻"
|
||||
|
||||
return result
|
||||
|
||||
except MCPError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": e.to_dict()
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": {
|
||||
"code": "INTERNAL_ERROR",
|
||||
"message": str(e)
|
||||
}
|
||||
}
|
||||
465
mcp_server/tools/system.py
Normal file
465
mcp_server/tools/system.py
Normal file
@ -0,0 +1,465 @@
|
||||
"""
|
||||
系统管理工具
|
||||
|
||||
实现系统状态查询和爬虫触发功能。
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from ..services.data_service import DataService
|
||||
from ..utils.validators import validate_platforms
|
||||
from ..utils.errors import MCPError, CrawlTaskError
|
||||
|
||||
|
||||
class SystemManagementTools:
|
||||
"""系统管理工具类"""
|
||||
|
||||
def __init__(self, project_root: str = None):
|
||||
"""
|
||||
初始化系统管理工具
|
||||
|
||||
Args:
|
||||
project_root: 项目根目录
|
||||
"""
|
||||
self.data_service = DataService(project_root)
|
||||
if project_root:
|
||||
self.project_root = Path(project_root)
|
||||
else:
|
||||
# 获取项目根目录
|
||||
current_file = Path(__file__)
|
||||
self.project_root = current_file.parent.parent.parent
|
||||
|
||||
def get_system_status(self) -> Dict:
|
||||
"""
|
||||
获取系统运行状态和健康检查信息
|
||||
|
||||
Returns:
|
||||
系统状态字典
|
||||
|
||||
Example:
|
||||
>>> tools = SystemManagementTools()
|
||||
>>> result = tools.get_system_status()
|
||||
>>> print(result['system']['version'])
|
||||
"""
|
||||
try:
|
||||
# 获取系统状态
|
||||
status = self.data_service.get_system_status()
|
||||
|
||||
return {
|
||||
**status,
|
||||
"success": True
|
||||
}
|
||||
|
||||
except MCPError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": e.to_dict()
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": {
|
||||
"code": "INTERNAL_ERROR",
|
||||
"message": str(e)
|
||||
}
|
||||
}
|
||||
|
||||
def trigger_crawl(self, platforms: Optional[List[str]] = None, save_to_local: bool = False, include_url: bool = False) -> Dict:
|
||||
"""
|
||||
手动触发一次临时爬取任务（可选持久化）
|
||||
|
||||
Args:
|
||||
platforms: 指定平台列表，为空则爬取所有平台
|
||||
save_to_local: 是否保存到本地 output 目录，默认 False
|
||||
include_url: 是否包含URL链接，默认False（节省token）
|
||||
|
||||
Returns:
|
||||
爬取结果字典，包含新闻数据和保存路径（如果保存）
|
||||
|
||||
Example:
|
||||
>>> tools = SystemManagementTools()
|
||||
>>> # 临时爬取，不保存
|
||||
>>> result = tools.trigger_crawl(platforms=['zhihu', 'weibo'])
|
||||
>>> print(result['data'])
|
||||
>>> # 爬取并保存到本地
|
||||
>>> result = tools.trigger_crawl(platforms=['zhihu'], save_to_local=True)
|
||||
>>> print(result['saved_files'])
|
||||
"""
|
||||
try:
|
||||
import json
|
||||
import time
|
||||
import random
|
||||
import requests
|
||||
from datetime import datetime
|
||||
import pytz
|
||||
import yaml
|
||||
|
||||
# 参数验证
|
||||
platforms = validate_platforms(platforms)
|
||||
|
||||
# 加载配置文件
|
||||
config_path = self.project_root / "config" / "config.yaml"
|
||||
if not config_path.exists():
|
||||
raise CrawlTaskError(
|
||||
"配置文件不存在",
|
||||
suggestion=f"请确保配置文件存在: {config_path}"
|
||||
)
|
||||
|
||||
# 读取配置
|
||||
with open(config_path, "r", encoding="utf-8") as f:
|
||||
config_data = yaml.safe_load(f)
|
||||
|
||||
# 获取平台配置
|
||||
all_platforms = config_data.get("platforms", [])
|
||||
if not all_platforms:
|
||||
raise CrawlTaskError(
|
||||
"配置文件中没有平台配置",
|
||||
suggestion="请检查 config/config.yaml 中的 platforms 配置"
|
||||
)
|
||||
|
||||
# 过滤平台
|
||||
if platforms:
|
||||
target_platforms = [p for p in all_platforms if p["id"] in platforms]
|
||||
if not target_platforms:
|
||||
raise CrawlTaskError(
|
||||
f"指定的平台不存在: {platforms}",
|
||||
suggestion=f"可用平台: {[p['id'] for p in all_platforms]}"
|
||||
)
|
||||
else:
|
||||
target_platforms = all_platforms
|
||||
|
||||
# 获取请求间隔
|
||||
request_interval = config_data.get("crawler", {}).get("request_interval", 100)
|
||||
|
||||
# 构建平台ID列表
|
||||
ids = []
|
||||
for platform in target_platforms:
|
||||
if "name" in platform:
|
||||
ids.append((platform["id"], platform["name"]))
|
||||
else:
|
||||
ids.append(platform["id"])
|
||||
|
||||
print(f"开始临时爬取，平台: {[p.get('name', p['id']) for p in target_platforms]}")
|
||||
|
||||
# 爬取数据
|
||||
results = {}
|
||||
id_to_name = {}
|
||||
failed_ids = []
|
||||
|
||||
for i, id_info in enumerate(ids):
|
||||
if isinstance(id_info, tuple):
|
||||
id_value, name = id_info
|
||||
else:
|
||||
id_value = id_info
|
||||
name = id_value
|
||||
|
||||
id_to_name[id_value] = name
|
||||
|
||||
# 构建请求URL
|
||||
url = f"https://newsnow.busiyi.world/api/s?id={id_value}&latest"
|
||||
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
|
||||
"Accept": "application/json, text/plain, */*",
|
||||
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
||||
"Connection": "keep-alive",
|
||||
"Cache-Control": "no-cache",
|
||||
}
|
||||
|
||||
# 重试机制
|
||||
max_retries = 2
|
||||
retries = 0
|
||||
success = False
|
||||
|
||||
while retries <= max_retries and not success:
|
||||
try:
|
||||
response = requests.get(url, headers=headers, timeout=10)
|
||||
response.raise_for_status()
|
||||
|
||||
data_text = response.text
|
||||
data_json = json.loads(data_text)
|
||||
|
||||
status = data_json.get("status", "未知")
|
||||
if status not in ["success", "cache"]:
|
||||
raise ValueError(f"响应状态异常: {status}")
|
||||
|
||||
status_info = "最新数据" if status == "success" else "缓存数据"
|
||||
print(f"获取 {id_value} 成功（{status_info}）")
|
||||
|
||||
# 解析数据
|
||||
results[id_value] = {}
|
||||
for index, item in enumerate(data_json.get("items", []), 1):
|
||||
title = item["title"]
|
||||
url_link = item.get("url", "")
|
||||
mobile_url = item.get("mobileUrl", "")
|
||||
|
||||
if title in results[id_value]:
|
||||
results[id_value][title]["ranks"].append(index)
|
||||
else:
|
||||
results[id_value][title] = {
|
||||
"ranks": [index],
|
||||
"url": url_link,
|
||||
"mobileUrl": mobile_url,
|
||||
}
|
||||
|
||||
success = True
|
||||
|
||||
except Exception as e:
|
||||
retries += 1
|
||||
if retries <= max_retries:
|
||||
wait_time = random.uniform(3, 5)
|
||||
print(f"请求 {id_value} 失败: {e}. {wait_time:.2f}秒后重试...")
|
||||
time.sleep(wait_time)
|
||||
else:
|
||||
print(f"请求 {id_value} 失败: {e}")
|
||||
failed_ids.append(id_value)
|
||||
|
||||
# 请求间隔
|
||||
if i < len(ids) - 1:
|
||||
actual_interval = request_interval + random.randint(-10, 20)
|
||||
actual_interval = max(50, actual_interval)
|
||||
time.sleep(actual_interval / 1000)
|
||||
|
||||
# 格式化返回数据
|
||||
news_data = []
|
||||
for platform_id, titles_data in results.items():
|
||||
platform_name = id_to_name.get(platform_id, platform_id)
|
||||
for title, info in titles_data.items():
|
||||
news_item = {
|
||||
"platform_id": platform_id,
|
||||
"platform_name": platform_name,
|
||||
"title": title,
|
||||
"ranks": info["ranks"]
|
||||
}
|
||||
|
||||
# 条件性添加 URL 字段
|
||||
if include_url:
|
||||
news_item["url"] = info.get("url", "")
|
||||
news_item["mobile_url"] = info.get("mobileUrl", "")
|
||||
|
||||
news_data.append(news_item)
|
||||
|
||||
# 获取北京时间
|
||||
beijing_tz = pytz.timezone("Asia/Shanghai")
|
||||
now = datetime.now(beijing_tz)
|
||||
|
||||
# 构建返回结果
|
||||
result = {
|
||||
"success": True,
|
||||
"task_id": f"crawl_{int(time.time())}",
|
||||
"status": "completed",
|
||||
"crawl_time": now.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"platforms": list(results.keys()),
|
||||
"total_news": len(news_data),
|
||||
"failed_platforms": failed_ids,
|
||||
"data": news_data,
|
||||
"saved_to_local": save_to_local
|
||||
}
|
||||
|
||||
# 如果需要持久化，调用保存逻辑
|
||||
if save_to_local:
|
||||
try:
|
||||
import re
|
||||
|
||||
# 辅助函数：清理标题
|
||||
def clean_title(title: str) -> str:
|
||||
"""清理标题中的特殊字符"""
|
||||
if not isinstance(title, str):
|
||||
title = str(title)
|
||||
cleaned_title = title.replace("\n", " ").replace("\r", " ")
|
||||
cleaned_title = re.sub(r"\s+", " ", cleaned_title)
|
||||
cleaned_title = cleaned_title.strip()
|
||||
return cleaned_title
|
||||
|
||||
# 辅助函数：创建目录
|
||||
def ensure_directory_exists(directory: str):
|
||||
"""确保目录存在"""
|
||||
Path(directory).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 格式化日期和时间
|
||||
date_folder = now.strftime("%Y年%m月%d日")
|
||||
time_filename = now.strftime("%H时%M分")
|
||||
|
||||
# 创建 txt 文件路径
|
||||
txt_dir = self.project_root / "output" / date_folder / "txt"
|
||||
ensure_directory_exists(str(txt_dir))
|
||||
txt_file_path = txt_dir / f"{time_filename}.txt"
|
||||
|
||||
# 创建 html 文件路径
|
||||
html_dir = self.project_root / "output" / date_folder / "html"
|
||||
ensure_directory_exists(str(html_dir))
|
||||
html_file_path = html_dir / f"{time_filename}.html"
|
||||
|
||||
# 保存 txt 文件（按照 main.py 的格式）
|
||||
with open(txt_file_path, "w", encoding="utf-8") as f:
|
||||
for id_value, title_data in results.items():
|
||||
# id | name 或 id
|
||||
name = id_to_name.get(id_value)
|
||||
if name and name != id_value:
|
||||
f.write(f"{id_value} | {name}\n")
|
||||
else:
|
||||
f.write(f"{id_value}\n")
|
||||
|
||||
# 按排名排序标题
|
||||
sorted_titles = []
|
||||
for title, info in title_data.items():
|
||||
cleaned = clean_title(title)
|
||||
if isinstance(info, dict):
|
||||
ranks = info.get("ranks", [])
|
||||
url = info.get("url", "")
|
||||
mobile_url = info.get("mobileUrl", "")
|
||||
else:
|
||||
ranks = info if isinstance(info, list) else []
|
||||
url = ""
|
||||
mobile_url = ""
|
||||
|
||||
rank = ranks[0] if ranks else 1
|
||||
sorted_titles.append((rank, cleaned, url, mobile_url))
|
||||
|
||||
sorted_titles.sort(key=lambda x: x[0])
|
||||
|
||||
for rank, cleaned, url, mobile_url in sorted_titles:
|
||||
line = f"{rank}. {cleaned}"
|
||||
if url:
|
||||
line += f" [URL:{url}]"
|
||||
if mobile_url:
|
||||
line += f" [MOBILE:{mobile_url}]"
|
||||
f.write(line + "\n")
|
||||
|
||||
f.write("\n")
|
||||
|
||||
if failed_ids:
|
||||
f.write("==== 以下ID请求失败 ====\n")
|
||||
for id_value in failed_ids:
|
||||
f.write(f"{id_value}\n")
|
||||
|
||||
# 保存 html 文件（简化版）
|
||||
html_content = self._generate_simple_html(results, id_to_name, failed_ids, now)
|
||||
with open(html_file_path, "w", encoding="utf-8") as f:
|
||||
f.write(html_content)
|
||||
|
||||
print(f"数据已保存到:")
|
||||
print(f" TXT: {txt_file_path}")
|
||||
print(f" HTML: {html_file_path}")
|
||||
|
||||
result["saved_files"] = {
|
||||
"txt": str(txt_file_path),
|
||||
"html": str(html_file_path)
|
||||
}
|
||||
result["note"] = "数据已持久化到 output 文件夹"
|
||||
|
||||
except Exception as e:
|
||||
print(f"保存文件失败: {e}")
|
||||
result["save_error"] = str(e)
|
||||
result["note"] = "爬取成功但保存失败，数据仅在内存中"
|
||||
else:
|
||||
result["note"] = "临时爬取结果，未持久化到output文件夹"
|
||||
|
||||
return result
|
||||
|
||||
except MCPError as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": e.to_dict()
|
||||
}
|
||||
except Exception as e:
|
||||
import traceback
|
||||
return {
|
||||
"success": False,
|
||||
"error": {
|
||||
"code": "INTERNAL_ERROR",
|
||||
"message": str(e),
|
||||
"traceback": traceback.format_exc()
|
||||
}
|
||||
}
|
||||
|
||||
def _generate_simple_html(self, results: Dict, id_to_name: Dict, failed_ids: List, now) -> str:
|
||||
"""生成简化的 HTML 报告"""
|
||||
html = """<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>MCP 爬取结果</title>
|
||||
<style>
|
||||
body { font-family: Arial, sans-serif; margin: 20px; background: #f5f5f5; }
|
||||
.container { max-width: 900px; margin: 0 auto; background: white; padding: 20px; border-radius: 8px; }
|
||||
h1 { color: #333; border-bottom: 2px solid #4CAF50; padding-bottom: 10px; }
|
||||
.platform { margin-bottom: 30px; }
|
||||
.platform-name { background: #4CAF50; color: white; padding: 10px; border-radius: 5px; margin-bottom: 10px; }
|
||||
.news-item { padding: 8px; border-bottom: 1px solid #eee; }
|
||||
.rank { color: #666; font-weight: bold; margin-right: 10px; }
|
||||
.title { color: #333; }
|
||||
.link { color: #1976D2; text-decoration: none; margin-left: 10px; font-size: 0.9em; }
|
||||
.link:hover { text-decoration: underline; }
|
||||
.failed { background: #ffebee; padding: 10px; border-radius: 5px; margin-top: 20px; }
|
||||
.failed h3 { color: #c62828; margin-top: 0; }
|
||||
.timestamp { color: #666; font-size: 0.9em; text-align: right; margin-top: 20px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<h1>MCP 爬取结果</h1>
|
||||
"""
|
||||
|
||||
# 添加时间戳
|
||||
html += f' <p class="timestamp">爬取时间: {now.strftime("%Y-%m-%d %H:%M:%S")}</p>\n\n'
|
||||
|
||||
# 遍历每个平台
|
||||
for platform_id, titles_data in results.items():
|
||||
platform_name = id_to_name.get(platform_id, platform_id)
|
||||
html += f' <div class="platform">\n'
|
||||
html += f' <div class="platform-name">{platform_name}</div>\n'
|
||||
|
||||
# 排序标题
|
||||
sorted_items = []
|
||||
for title, info in titles_data.items():
|
||||
ranks = info.get("ranks", [])
|
||||
url = info.get("url", "")
|
||||
mobile_url = info.get("mobileUrl", "")
|
||||
rank = ranks[0] if ranks else 999
|
||||
sorted_items.append((rank, title, url, mobile_url))
|
||||
|
||||
sorted_items.sort(key=lambda x: x[0])
|
||||
|
||||
# 显示新闻
|
||||
for rank, title, url, mobile_url in sorted_items:
|
||||
html += f' <div class="news-item">\n'
|
||||
html += f' <span class="rank">{rank}.</span>\n'
|
||||
html += f' <span class="title">{self._html_escape(title)}</span>\n'
|
||||
if url:
|
||||
html += f' <a class="link" href="{self._html_escape(url)}" target="_blank">链接</a>\n'
|
||||
if mobile_url and mobile_url != url:
|
||||
html += f' <a class="link" href="{self._html_escape(mobile_url)}" target="_blank">移动版</a>\n'
|
||||
html += ' </div>\n'
|
||||
|
||||
html += ' </div>\n\n'
|
||||
|
||||
# 失败的平台
|
||||
if failed_ids:
|
||||
html += ' <div class="failed">\n'
|
||||
html += ' <h3>请求失败的平台</h3>\n'
|
||||
html += ' <ul>\n'
|
||||
for platform_id in failed_ids:
|
||||
html += f' <li>{self._html_escape(platform_id)}</li>\n'
|
||||
html += ' </ul>\n'
|
||||
html += ' </div>\n'
|
||||
|
||||
html += """ </div>
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
return html
|
||||
|
||||
def _html_escape(self, text: str) -> str:
|
||||
"""HTML 转义"""
|
||||
if not isinstance(text, str):
|
||||
text = str(text)
|
||||
return (
|
||||
text.replace("&", "&")
|
||||
.replace("<", "<")
|
||||
.replace(">", ">")
|
||||
.replace('"', """)
|
||||
.replace("'", "'")
|
||||
)
|
||||
5
mcp_server/utils/__init__.py
Normal file
5
mcp_server/utils/__init__.py
Normal file
@ -0,0 +1,5 @@
|
||||
"""
|
||||
工具类模块
|
||||
|
||||
提供参数验证、错误处理等辅助功能。
|
||||
"""
|
||||
278
mcp_server/utils/date_parser.py
Normal file
278
mcp_server/utils/date_parser.py
Normal file
@ -0,0 +1,278 @@
|
||||
"""
|
||||
日期解析工具
|
||||
|
||||
支持多种自然语言日期格式解析，包括相对日期和绝对日期。
|
||||
"""
|
||||
|
||||
import re
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from .errors import InvalidParameterError
|
||||
|
||||
|
||||
class DateParser:
|
||||
"""日期解析器类"""
|
||||
|
||||
# 中文日期映射
|
||||
CN_DATE_MAPPING = {
|
||||
"今天": 0,
|
||||
"昨天": 1,
|
||||
"前天": 2,
|
||||
"大前天": 3,
|
||||
}
|
||||
|
||||
# 英文日期映射
|
||||
EN_DATE_MAPPING = {
|
||||
"today": 0,
|
||||
"yesterday": 1,
|
||||
}
|
||||
|
||||
# 星期映射
|
||||
WEEKDAY_CN = {
|
||||
"一": 0, "二": 1, "三": 2, "四": 3,
|
||||
"五": 4, "六": 5, "日": 6, "天": 6
|
||||
}
|
||||
|
||||
WEEKDAY_EN = {
|
||||
"monday": 0, "tuesday": 1, "wednesday": 2, "thursday": 3,
|
||||
"friday": 4, "saturday": 5, "sunday": 6
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def parse_date_query(date_query: str) -> datetime:
|
||||
"""
|
||||
解析日期查询字符串
|
||||
|
||||
支持的格式：
|
||||
- 相对日期（中文）：今天、昨天、前天、大前天、N天前
|
||||
- 相对日期（英文）：today、yesterday、N days ago
|
||||
- 星期（中文）：上周一、上周二、本周三
|
||||
- 星期（英文）：last monday、this friday
|
||||
- 绝对日期：2025-10-10、10月10日、2025年10月10日
|
||||
|
||||
Args:
|
||||
date_query: 日期查询字符串
|
||||
|
||||
Returns:
|
||||
datetime对象
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 日期格式无法识别
|
||||
|
||||
Examples:
|
||||
>>> DateParser.parse_date_query("今天")
|
||||
datetime(2025, 10, 11)
|
||||
>>> DateParser.parse_date_query("昨天")
|
||||
datetime(2025, 10, 10)
|
||||
>>> DateParser.parse_date_query("3天前")
|
||||
datetime(2025, 10, 8)
|
||||
>>> DateParser.parse_date_query("2025-10-10")
|
||||
datetime(2025, 10, 10)
|
||||
"""
|
||||
if not date_query or not isinstance(date_query, str):
|
||||
raise InvalidParameterError(
|
||||
"日期查询字符串不能为空",
|
||||
suggestion="请提供有效的日期查询，如：今天、昨天、2025-10-10"
|
||||
)
|
||||
|
||||
date_query = date_query.strip().lower()
|
||||
|
||||
# 1. 尝试解析中文常用相对日期
|
||||
if date_query in DateParser.CN_DATE_MAPPING:
|
||||
days_ago = DateParser.CN_DATE_MAPPING[date_query]
|
||||
return datetime.now() - timedelta(days=days_ago)
|
||||
|
||||
# 2. 尝试解析英文常用相对日期
|
||||
if date_query in DateParser.EN_DATE_MAPPING:
|
||||
days_ago = DateParser.EN_DATE_MAPPING[date_query]
|
||||
return datetime.now() - timedelta(days=days_ago)
|
||||
|
||||
# 3. 尝试解析 "N天前" 或 "N days ago"
|
||||
cn_days_ago_match = re.match(r'(\d+)\s*天前', date_query)
|
||||
if cn_days_ago_match:
|
||||
days = int(cn_days_ago_match.group(1))
|
||||
if days > 365:
|
||||
raise InvalidParameterError(
|
||||
f"天数过大: {days}天",
|
||||
suggestion="请使用小于365天的相对日期或使用绝对日期"
|
||||
)
|
||||
return datetime.now() - timedelta(days=days)
|
||||
|
||||
en_days_ago_match = re.match(r'(\d+)\s*days?\s+ago', date_query)
|
||||
if en_days_ago_match:
|
||||
days = int(en_days_ago_match.group(1))
|
||||
if days > 365:
|
||||
raise InvalidParameterError(
|
||||
f"天数过大: {days}天",
|
||||
suggestion="请使用小于365天的相对日期或使用绝对日期"
|
||||
)
|
||||
return datetime.now() - timedelta(days=days)
|
||||
|
||||
# 4. 尝试解析星期（中文）：上周一、本周三
|
||||
cn_weekday_match = re.match(r'(上|本)周([一二三四五六日天])', date_query)
|
||||
if cn_weekday_match:
|
||||
week_type = cn_weekday_match.group(1) # 上 或 本
|
||||
weekday_str = cn_weekday_match.group(2)
|
||||
target_weekday = DateParser.WEEKDAY_CN[weekday_str]
|
||||
return DateParser._get_date_by_weekday(target_weekday, week_type == "上")
|
||||
|
||||
# 5. 尝试解析星期（英文）：last monday、this friday
|
||||
en_weekday_match = re.match(r'(last|this)\s+(monday|tuesday|wednesday|thursday|friday|saturday|sunday)', date_query)
|
||||
if en_weekday_match:
|
||||
week_type = en_weekday_match.group(1) # last 或 this
|
||||
weekday_str = en_weekday_match.group(2)
|
||||
target_weekday = DateParser.WEEKDAY_EN[weekday_str]
|
||||
return DateParser._get_date_by_weekday(target_weekday, week_type == "last")
|
||||
|
||||
# 6. 尝试解析绝对日期：YYYY-MM-DD
|
||||
iso_date_match = re.match(r'(\d{4})-(\d{1,2})-(\d{1,2})', date_query)
|
||||
if iso_date_match:
|
||||
year = int(iso_date_match.group(1))
|
||||
month = int(iso_date_match.group(2))
|
||||
day = int(iso_date_match.group(3))
|
||||
try:
|
||||
return datetime(year, month, day)
|
||||
except ValueError as e:
|
||||
raise InvalidParameterError(
|
||||
f"无效的日期: {date_query}",
|
||||
suggestion=f"日期值错误: {str(e)}"
|
||||
)
|
||||
|
||||
# 7. 尝试解析中文日期：MM月DD日 或 YYYY年MM月DD日
|
||||
cn_date_match = re.match(r'(?:(\d{4})年)?(\d{1,2})月(\d{1,2})日', date_query)
|
||||
if cn_date_match:
|
||||
year_str = cn_date_match.group(1)
|
||||
month = int(cn_date_match.group(2))
|
||||
day = int(cn_date_match.group(3))
|
||||
|
||||
# 如果没有年份，使用当前年份
|
||||
if year_str:
|
||||
year = int(year_str)
|
||||
else:
|
||||
year = datetime.now().year
|
||||
# 如果月份大于当前月份，说明是去年
|
||||
current_month = datetime.now().month
|
||||
if month > current_month:
|
||||
year -= 1
|
||||
|
||||
try:
|
||||
return datetime(year, month, day)
|
||||
except ValueError as e:
|
||||
raise InvalidParameterError(
|
||||
f"无效的日期: {date_query}",
|
||||
suggestion=f"日期值错误: {str(e)}"
|
||||
)
|
||||
|
||||
# 8. 尝试解析斜杠格式：YYYY/MM/DD 或 MM/DD
|
||||
slash_date_match = re.match(r'(?:(\d{4})/)?(\d{1,2})/(\d{1,2})', date_query)
|
||||
if slash_date_match:
|
||||
year_str = slash_date_match.group(1)
|
||||
month = int(slash_date_match.group(2))
|
||||
day = int(slash_date_match.group(3))
|
||||
|
||||
if year_str:
|
||||
year = int(year_str)
|
||||
else:
|
||||
year = datetime.now().year
|
||||
current_month = datetime.now().month
|
||||
if month > current_month:
|
||||
year -= 1
|
||||
|
||||
try:
|
||||
return datetime(year, month, day)
|
||||
except ValueError as e:
|
||||
raise InvalidParameterError(
|
||||
f"无效的日期: {date_query}",
|
||||
suggestion=f"日期值错误: {str(e)}"
|
||||
)
|
||||
|
||||
# 如果所有格式都不匹配
|
||||
raise InvalidParameterError(
|
||||
f"无法识别的日期格式: {date_query}",
|
||||
suggestion=(
|
||||
"支持的格式:\n"
|
||||
"- 相对日期: 今天、昨天、前天、3天前、today、yesterday、3 days ago\n"
|
||||
"- 星期: 上周一、本周三、last monday、this friday\n"
|
||||
"- 绝对日期: 2025-10-10、10月10日、2025年10月10日"
|
||||
)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _get_date_by_weekday(target_weekday: int, is_last_week: bool) -> datetime:
|
||||
"""
|
||||
根据星期几获取日期
|
||||
|
||||
Args:
|
||||
target_weekday: 目标星期 (0=周一, 6=周日)
|
||||
is_last_week: 是否是上周
|
||||
|
||||
Returns:
|
||||
datetime对象
|
||||
"""
|
||||
today = datetime.now()
|
||||
current_weekday = today.weekday()
|
||||
|
||||
# 计算天数差
|
||||
if is_last_week:
|
||||
# 上周的某一天
|
||||
days_diff = current_weekday - target_weekday + 7
|
||||
else:
|
||||
# 本周的某一天
|
||||
days_diff = current_weekday - target_weekday
|
||||
if days_diff < 0:
|
||||
days_diff += 7
|
||||
|
||||
return today - timedelta(days=days_diff)
|
||||
|
||||
@staticmethod
|
||||
def format_date_folder(date: datetime) -> str:
|
||||
"""
|
||||
将日期格式化为文件夹名称
|
||||
|
||||
Args:
|
||||
date: datetime对象
|
||||
|
||||
Returns:
|
||||
文件夹名称，格式: YYYY年MM月DD日
|
||||
|
||||
Examples:
|
||||
>>> DateParser.format_date_folder(datetime(2025, 10, 11))
|
||||
'2025年10月11日'
|
||||
"""
|
||||
return date.strftime("%Y年%m月%d日")
|
||||
|
||||
@staticmethod
|
||||
def validate_date_not_future(date: datetime) -> None:
|
||||
"""
|
||||
验证日期不在未来
|
||||
|
||||
Args:
|
||||
date: 待验证的日期
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 日期在未来
|
||||
"""
|
||||
if date.date() > datetime.now().date():
|
||||
raise InvalidParameterError(
|
||||
f"不能查询未来的日期: {date.strftime('%Y-%m-%d')}",
|
||||
suggestion="请使用今天或过去的日期"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def validate_date_not_too_old(date: datetime, max_days: int = 365) -> None:
|
||||
"""
|
||||
验证日期不太久远
|
||||
|
||||
Args:
|
||||
date: 待验证的日期
|
||||
max_days: 最大天数
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 日期太久远
|
||||
"""
|
||||
days_ago = (datetime.now().date() - date.date()).days
|
||||
if days_ago > max_days:
|
||||
raise InvalidParameterError(
|
||||
f"日期太久远: {date.strftime('%Y-%m-%d')} ({days_ago}天前)",
|
||||
suggestion=f"请查询{max_days}天内的数据"
|
||||
)
|
||||
93
mcp_server/utils/errors.py
Normal file
93
mcp_server/utils/errors.py
Normal file
@ -0,0 +1,93 @@
|
||||
"""
|
||||
自定义错误类
|
||||
|
||||
定义MCP Server使用的所有自定义异常类型。
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
|
||||
class MCPError(Exception):
|
||||
"""MCP工具错误基类"""
|
||||
|
||||
def __init__(self, message: str, code: str = "MCP_ERROR", suggestion: Optional[str] = None):
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.suggestion = suggestion
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
"""转换为字典格式"""
|
||||
error_dict = {
|
||||
"code": self.code,
|
||||
"message": self.message
|
||||
}
|
||||
if self.suggestion:
|
||||
error_dict["suggestion"] = self.suggestion
|
||||
return error_dict
|
||||
|
||||
|
||||
class DataNotFoundError(MCPError):
|
||||
"""数据不存在错误"""
|
||||
|
||||
def __init__(self, message: str, suggestion: Optional[str] = None):
|
||||
super().__init__(
|
||||
message=message,
|
||||
code="DATA_NOT_FOUND",
|
||||
suggestion=suggestion or "请检查日期范围或等待爬取任务完成"
|
||||
)
|
||||
|
||||
|
||||
class InvalidParameterError(MCPError):
|
||||
"""参数无效错误"""
|
||||
|
||||
def __init__(self, message: str, suggestion: Optional[str] = None):
|
||||
super().__init__(
|
||||
message=message,
|
||||
code="INVALID_PARAMETER",
|
||||
suggestion=suggestion or "请检查参数格式是否正确"
|
||||
)
|
||||
|
||||
|
||||
class ConfigurationError(MCPError):
|
||||
"""配置错误"""
|
||||
|
||||
def __init__(self, message: str, suggestion: Optional[str] = None):
|
||||
super().__init__(
|
||||
message=message,
|
||||
code="CONFIGURATION_ERROR",
|
||||
suggestion=suggestion or "请检查配置文件是否正确"
|
||||
)
|
||||
|
||||
|
||||
class PlatformNotSupportedError(MCPError):
|
||||
"""平台不支持错误"""
|
||||
|
||||
def __init__(self, platform: str):
|
||||
super().__init__(
|
||||
message=f"平台 '{platform}' 不受支持",
|
||||
code="PLATFORM_NOT_SUPPORTED",
|
||||
suggestion="支持的平台: zhihu, weibo, douyin, bilibili, baidu, toutiao, qq, 36kr, sspai, hellogithub, thepaper"
|
||||
)
|
||||
|
||||
|
||||
class CrawlTaskError(MCPError):
|
||||
"""爬取任务错误"""
|
||||
|
||||
def __init__(self, message: str, suggestion: Optional[str] = None):
|
||||
super().__init__(
|
||||
message=message,
|
||||
code="CRAWL_TASK_ERROR",
|
||||
suggestion=suggestion or "请稍后重试或查看日志"
|
||||
)
|
||||
|
||||
|
||||
class FileParseError(MCPError):
|
||||
"""文件解析错误"""
|
||||
|
||||
def __init__(self, file_path: str, reason: str):
|
||||
super().__init__(
|
||||
message=f"解析文件 {file_path} 失败: {reason}",
|
||||
code="FILE_PARSE_ERROR",
|
||||
suggestion="请检查文件格式是否正确"
|
||||
)
|
||||
324
mcp_server/utils/validators.py
Normal file
324
mcp_server/utils/validators.py
Normal file
@ -0,0 +1,324 @@
|
||||
"""
|
||||
参数验证工具
|
||||
|
||||
提供统一的参数验证功能。
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
import os
|
||||
import yaml
|
||||
|
||||
from .errors import InvalidParameterError
|
||||
from .date_parser import DateParser
|
||||
|
||||
|
||||
def get_supported_platforms() -> List[str]:
|
||||
"""
|
||||
从 config.yaml 动态获取支持的平台列表
|
||||
|
||||
Returns:
|
||||
平台ID列表
|
||||
|
||||
Note:
|
||||
- 读取失败时返回空列表，允许所有平台通过（降级策略）
|
||||
- 平台列表来自 config/config.yaml 中的 platforms 配置
|
||||
"""
|
||||
try:
|
||||
# 获取 config.yaml 路径（相对于当前文件）
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
config_path = os.path.join(current_dir, "..", "..", "config", "config.yaml")
|
||||
config_path = os.path.normpath(config_path)
|
||||
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
config = yaml.safe_load(f)
|
||||
platforms = config.get('platforms', [])
|
||||
return [p['id'] for p in platforms if 'id' in p]
|
||||
except Exception as e:
|
||||
# 降级方案：返回空列表，允许所有平台
|
||||
print(f"警告：无法加载平台配置 ({config_path}): {e}")
|
||||
return []
|
||||
|
||||
|
||||
def validate_platforms(platforms: Optional[List[str]]) -> List[str]:
|
||||
"""
|
||||
验证平台列表
|
||||
|
||||
Args:
|
||||
platforms: 平台ID列表，None表示使用 config.yaml 中配置的所有平台
|
||||
|
||||
Returns:
|
||||
验证后的平台列表
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 平台不支持
|
||||
|
||||
Note:
|
||||
- platforms=None 时，返回 config.yaml 中配置的平台列表
|
||||
- 会验证平台ID是否在 config.yaml 的 platforms 配置中
|
||||
- 配置加载失败时，允许所有平台通过（降级策略）
|
||||
"""
|
||||
supported_platforms = get_supported_platforms()
|
||||
|
||||
if platforms is None:
|
||||
# 返回配置文件中的平台列表（用户的默认配置）
|
||||
return supported_platforms if supported_platforms else []
|
||||
|
||||
if not isinstance(platforms, list):
|
||||
raise InvalidParameterError("platforms 参数必须是列表类型")
|
||||
|
||||
if not platforms:
|
||||
# 空列表时，返回配置文件中的平台列表
|
||||
return supported_platforms if supported_platforms else []
|
||||
|
||||
# 如果配置加载失败（supported_platforms为空），允许所有平台通过
|
||||
if not supported_platforms:
|
||||
print("警告：平台配置未加载，跳过平台验证")
|
||||
return platforms
|
||||
|
||||
# 验证每个平台是否在配置中
|
||||
invalid_platforms = [p for p in platforms if p not in supported_platforms]
|
||||
if invalid_platforms:
|
||||
raise InvalidParameterError(
|
||||
f"不支持的平台: {', '.join(invalid_platforms)}",
|
||||
suggestion=f"支持的平台（来自config.yaml）: {', '.join(supported_platforms)}"
|
||||
)
|
||||
|
||||
return platforms
|
||||
|
||||
|
||||
def validate_limit(limit: Optional[int], default: int = 20, max_limit: int = 1000) -> int:
|
||||
"""
|
||||
验证数量限制参数
|
||||
|
||||
Args:
|
||||
limit: 限制数量
|
||||
default: 默认值
|
||||
max_limit: 最大限制
|
||||
|
||||
Returns:
|
||||
验证后的限制值
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 参数无效
|
||||
"""
|
||||
if limit is None:
|
||||
return default
|
||||
|
||||
if not isinstance(limit, int):
|
||||
raise InvalidParameterError("limit 参数必须是整数类型")
|
||||
|
||||
if limit <= 0:
|
||||
raise InvalidParameterError("limit 必须大于0")
|
||||
|
||||
if limit > max_limit:
|
||||
raise InvalidParameterError(
|
||||
f"limit 不能超过 {max_limit}",
|
||||
suggestion=f"请使用分页或降低limit值"
|
||||
)
|
||||
|
||||
return limit
|
||||
|
||||
|
||||
def validate_date(date_str: str) -> datetime:
|
||||
"""
|
||||
验证日期格式
|
||||
|
||||
Args:
|
||||
date_str: 日期字符串 (YYYY-MM-DD)
|
||||
|
||||
Returns:
|
||||
datetime对象
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 日期格式错误
|
||||
"""
|
||||
try:
|
||||
return datetime.strptime(date_str, "%Y-%m-%d")
|
||||
except ValueError:
|
||||
raise InvalidParameterError(
|
||||
f"日期格式错误: {date_str}",
|
||||
suggestion="请使用 YYYY-MM-DD 格式，例如: 2025-10-11"
|
||||
)
|
||||
|
||||
|
||||
def validate_date_range(date_range: Optional[dict]) -> Optional[tuple]:
|
||||
"""
|
||||
验证日期范围
|
||||
|
||||
Args:
|
||||
date_range: 日期范围字典 {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}
|
||||
|
||||
Returns:
|
||||
(start_date, end_date) 元组，或 None
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 日期范围无效
|
||||
"""
|
||||
if date_range is None:
|
||||
return None
|
||||
|
||||
if not isinstance(date_range, dict):
|
||||
raise InvalidParameterError("date_range 必须是字典类型")
|
||||
|
||||
start_str = date_range.get("start")
|
||||
end_str = date_range.get("end")
|
||||
|
||||
if not start_str or not end_str:
|
||||
raise InvalidParameterError(
|
||||
"date_range 必须包含 start 和 end 字段",
|
||||
suggestion='例如: {"start": "2025-10-01", "end": "2025-10-11"}'
|
||||
)
|
||||
|
||||
start_date = validate_date(start_str)
|
||||
end_date = validate_date(end_str)
|
||||
|
||||
if start_date > end_date:
|
||||
raise InvalidParameterError(
|
||||
"开始日期不能晚于结束日期",
|
||||
suggestion=f"start: {start_str}, end: {end_str}"
|
||||
)
|
||||
|
||||
return (start_date, end_date)
|
||||
|
||||
|
||||
def validate_keyword(keyword: str) -> str:
|
||||
"""
|
||||
验证关键词
|
||||
|
||||
Args:
|
||||
keyword: 搜索关键词
|
||||
|
||||
Returns:
|
||||
处理后的关键词
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 关键词无效
|
||||
"""
|
||||
if not keyword:
|
||||
raise InvalidParameterError("keyword 不能为空")
|
||||
|
||||
if not isinstance(keyword, str):
|
||||
raise InvalidParameterError("keyword 必须是字符串类型")
|
||||
|
||||
keyword = keyword.strip()
|
||||
|
||||
if not keyword:
|
||||
raise InvalidParameterError("keyword 不能为空白字符")
|
||||
|
||||
if len(keyword) > 100:
|
||||
raise InvalidParameterError(
|
||||
"keyword 长度不能超过100个字符",
|
||||
suggestion="请使用更简洁的关键词"
|
||||
)
|
||||
|
||||
return keyword
|
||||
|
||||
|
||||
def validate_top_n(top_n: Optional[int], default: int = 10) -> int:
|
||||
"""
|
||||
验证TOP N参数
|
||||
|
||||
Args:
|
||||
top_n: TOP N数量
|
||||
default: 默认值
|
||||
|
||||
Returns:
|
||||
验证后的值
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 参数无效
|
||||
"""
|
||||
return validate_limit(top_n, default=default, max_limit=100)
|
||||
|
||||
|
||||
def validate_mode(mode: Optional[str], valid_modes: List[str], default: str) -> str:
|
||||
"""
|
||||
验证模式参数
|
||||
|
||||
Args:
|
||||
mode: 模式字符串
|
||||
valid_modes: 有效模式列表
|
||||
default: 默认模式
|
||||
|
||||
Returns:
|
||||
验证后的模式
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 模式无效
|
||||
"""
|
||||
if mode is None:
|
||||
return default
|
||||
|
||||
if not isinstance(mode, str):
|
||||
raise InvalidParameterError("mode 必须是字符串类型")
|
||||
|
||||
if mode not in valid_modes:
|
||||
raise InvalidParameterError(
|
||||
f"无效的模式: {mode}",
|
||||
suggestion=f"支持的模式: {', '.join(valid_modes)}"
|
||||
)
|
||||
|
||||
return mode
|
||||
|
||||
|
||||
def validate_config_section(section: Optional[str]) -> str:
|
||||
"""
|
||||
验证配置节参数
|
||||
|
||||
Args:
|
||||
section: 配置节名称
|
||||
|
||||
Returns:
|
||||
验证后的配置节
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 配置节无效
|
||||
"""
|
||||
valid_sections = ["all", "crawler", "push", "keywords", "weights"]
|
||||
return validate_mode(section, valid_sections, "all")
|
||||
|
||||
|
||||
def validate_date_query(
|
||||
date_query: str,
|
||||
allow_future: bool = False,
|
||||
max_days_ago: int = 365
|
||||
) -> datetime:
|
||||
"""
|
||||
验证并解析日期查询字符串
|
||||
|
||||
Args:
|
||||
date_query: 日期查询字符串
|
||||
allow_future: 是否允许未来日期
|
||||
max_days_ago: 允许查询的最大天数
|
||||
|
||||
Returns:
|
||||
解析后的datetime对象
|
||||
|
||||
Raises:
|
||||
InvalidParameterError: 日期查询无效
|
||||
|
||||
Examples:
|
||||
>>> validate_date_query("昨天")
|
||||
datetime(2025, 10, 10)
|
||||
>>> validate_date_query("2025-10-10")
|
||||
datetime(2025, 10, 10)
|
||||
"""
|
||||
if not date_query:
|
||||
raise InvalidParameterError(
|
||||
"日期查询字符串不能为空",
|
||||
suggestion="请提供日期查询，如：今天、昨天、2025-10-10"
|
||||
)
|
||||
|
||||
# 使用DateParser解析日期
|
||||
parsed_date = DateParser.parse_date_query(date_query)
|
||||
|
||||
# 验证日期不在未来
|
||||
if not allow_future:
|
||||
DateParser.validate_date_not_future(parsed_date)
|
||||
|
||||
# 验证日期不太久远
|
||||
DateParser.validate_date_not_too_old(parsed_date, max_days=max_days_ago)
|
||||
|
||||
return parsed_date
|
||||
|
||||
25
pyproject.toml
Normal file
25
pyproject.toml
Normal file
@ -0,0 +1,25 @@
|
||||
[project]
|
||||
name = "trendradar-mcp"
|
||||
version = "1.0.0"
|
||||
description = "TrendRadar MCP Server - 新闻热点聚合工具"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"requests>=2.32.5",
|
||||
"pytz>=2025.2",
|
||||
"PyYAML>=6.0.3",
|
||||
"fastmcp>=2.12.0",
|
||||
"websockets>=13.0,<14.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
trendradar = "mcp_server.server:run_server"
|
||||
|
||||
[dependency-groups]
|
||||
dev = []
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["mcp_server"]
|
||||
111
readme.md
111
readme.md
@ -1,5 +1,3 @@
|
||||
> 预告：下一次更新是 v3.0.0 版本，主要是 **AI** 有关的各种分析新闻资讯的功能
|
||||
|
||||
<div align="center" id="trendradar">
|
||||
|
||||
<a href="https://github.com/sansan0/TrendRadar" title="TrendRadar">
|
||||
@ -13,7 +11,7 @@
|
||||
[](https://github.com/sansan0/TrendRadar/stargazers)
|
||||
[](https://github.com/sansan0/TrendRadar/network/members)
|
||||
[](LICENSE)
|
||||
[](https://github.com/sansan0/TrendRadar)
|
||||
[](https://github.com/sansan0/TrendRadar)
|
||||
|
||||
[](https://work.weixin.qq.com/)
|
||||
[](https://telegram.org/)
|
||||
@ -22,16 +20,16 @@
|
||||
[](#)
|
||||
[](https://github.com/binwiederhier/ntfy)
|
||||
|
||||
|
||||
[](https://github.com/sansan0/TrendRadar)
|
||||
[](https://sansan0.github.io/TrendRadar)
|
||||
[](https://hub.docker.com/)
|
||||
[](https://modelcontextprotocol.io/)
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
> 本项目以轻量，易部署为目标
|
||||
>
|
||||
> 开源路上，感谢有你~😉
|
||||
> 本项目以轻量，易部署为目标。开源路上，感谢有你~😉
|
||||
|
||||
- 感谢**耐心反馈 bug** 的贡献者,你们的每一条反馈让项目更加完善;
|
||||
- 感谢**为项目点 star** 的观众们,你们的每一个 star 都是对开源精神最好的支持;
|
||||
@ -39,7 +37,7 @@
|
||||
- 感谢**给予资金支持** 的朋友们,你们的慷慨已化身为键盘旁的零食饮料,陪伴着项目的每一次迭代。
|
||||
|
||||
<details>
|
||||
<summary>👉 点击查看<strong>致谢名单</strong> (当前 <strong>🔥32🔥</strong> 位)</summary>
|
||||
<summary>👉 点击查看<strong>致谢名单</strong> (当前 <strong>🔥39🔥</strong> 位)</summary>
|
||||
|
||||
### 数据支持
|
||||
|
||||
@ -59,6 +57,13 @@
|
||||
|
||||
| 点赞人 | 金额 | 日期 | 备注 |
|
||||
| :-------------------------: | :----: | :----: | :-----------------------: |
|
||||
| P*n | 1 | 2025.10.20 | |
|
||||
| *杰 | 1 | 2025.10.19 | |
|
||||
| *徐 | 1 | 2025.10.18 | |
|
||||
| *志 | 1 | 2025.10.17 | |
|
||||
| *😀 | 10 | 2025.10.16 | 点赞 |
|
||||
| **杰 | 10 | 2025.10.16 | |
|
||||
| *啸 | 10 | 2025.10.16 | |
|
||||
| *纪 | 5 | 2025.10.14 | TrendRadar |
|
||||
| J*d | 1 | 2025.10.14 | 谢谢你的工具，很好玩... |
|
||||
| *H | 1 | 2025.10.14 | |
|
||||
@ -336,7 +341,7 @@ OPPO
|
||||
- **持续性分析**：区分一次性热点话题和持续发酵的深度新闻
|
||||
- **跨平台对比**：同一新闻在不同平台的排名表现，看出媒体关注度差异
|
||||
|
||||
**实际效果**：不再错过重要新闻的完整发展过程，从话题萌芽到高峰热议，全程掌握
|
||||
> 不再错过重要新闻的完整发展过程，从话题萌芽到高峰热议，全程掌握
|
||||
|
||||
<details>
|
||||
<summary><strong>👉 推送格式说明</strong></summary>
|
||||
@ -403,9 +408,7 @@ OPPO
|
||||
- **关注持续出现的话题**（占30%）：反复出现的新闻更重要
|
||||
- **考虑排名质量**（占10%）：不仅多次出现，还经常排在前列
|
||||
|
||||
**实际效果**：把分散在各个平台的热搜合并起来，按照你关心的热度重新排序
|
||||
|
||||
> 这三个比例可以选择适合自己的场景进行调整
|
||||
> 把分散在各个平台的热搜合并起来，按照你关心的热度重新排序，这三个比例可以选择适合自己的场景进行调整
|
||||
|
||||
<details>
|
||||
<summary><strong>👉 热点权重调整</strong></summary>
|
||||
@ -451,6 +454,21 @@ weight:
|
||||
- **Docker部署**：支持多架构容器化运行
|
||||
- **数据持久化**：HTML/TXT多格式历史记录保存
|
||||
|
||||
|
||||
### **AI 智能分析（v3.0.0 新增）**
|
||||
|
||||
基于 MCP (Model Context Protocol) 协议的 AI 对话分析系统，让你用自然语言深度挖掘新闻数据
|
||||
|
||||
- **对话式查询**：用自然语言提问，如"查询昨天知乎的热点"、"分析比特币最近的热度趋势"
|
||||
- **13 种分析工具**：涵盖基础查询、智能检索、趋势分析、数据洞察、情感分析等
|
||||
- **多客户端支持**：Cherry Studio（GUI 配置）、Claude Desktop、Cursor、Cline 等
|
||||
- **深度分析能力**：
|
||||
- 话题趋势追踪（热度变化、生命周期、爆火检测、趋势预测）
|
||||
- 跨平台数据对比（活跃度统计、关键词共现）
|
||||
- 智能摘要生成、相似新闻查找、历史关联检索
|
||||
|
||||
> 告别手动翻阅数据文件，AI 助手帮你秒懂新闻背后的故事
|
||||
|
||||
### **零技术门槛部署**
|
||||
|
||||
GitHub 一键 Fork 即可使用，无需编程基础。
|
||||
@ -482,6 +500,35 @@ GitHub 一键 Fork 即可使用，无需编程基础。
|
||||
- **小版本更新**：从 v2.x 升级到 v2.y, 用本项目的 `main.py` 代码替换你 fork 仓库中的对应文件
|
||||
- **大版本升级**：从 v1.x 升级到 v2.y, 建议删除现有 fork 后重新 fork，这样更省力且避免配置冲突
|
||||
|
||||
### 2025/10/20 - v3.0.0
|
||||
|
||||
**重大更新 - AI 分析功能上线** 🤖
|
||||
|
||||
- **核心功能**：
|
||||
- 新增基于 MCP (Model Context Protocol) 的 AI 分析服务器
|
||||
- 支持13种智能分析工具：基础查询、智能检索、高级分析、系统管理
|
||||
- 自然语言交互：通过对话方式查询和分析新闻数据
|
||||
- 多客户端支持：Claude Desktop、Cherry Studio、Cursor、Cline 等
|
||||
|
||||
- **快速部署方案**：
|
||||
- Cherry Studio 一键部署（推荐新手）：GUI 配置，5分钟完成
|
||||
- Claude Desktop 配置（技术用户）：JSON 配置文件
|
||||
- 完整的部署文档和使用指南
|
||||
|
||||
- **分析能力**：
|
||||
- 话题趋势分析（热度追踪、生命周期、爆火检测、趋势预测）
|
||||
- 数据洞察（平台对比、活跃度统计、关键词共现）
|
||||
- 情感分析、相似新闻查找、智能摘要生成
|
||||
- 历史相关新闻检索、多模式搜索
|
||||
|
||||
- **更新提示**：
|
||||
- 这是独立的 AI 分析功能，不影响现有的推送功能
|
||||
- 可选择性使用，无需升级现有部署
|
||||
|
||||
|
||||
<details>
|
||||
<summary><strong>👉 历史更新</strong></summary>
|
||||
|
||||
### 2025/10/15 - v2.4.4
|
||||
|
||||
- **更新内容**：
|
||||
@ -492,10 +539,6 @@ GitHub 一键 Fork 即可使用，无需编程基础。
|
||||
- 建议【小版本升级】
|
||||
|
||||
|
||||
<details>
|
||||
<summary><strong>👉 历史更新</strong></summary>
|
||||
|
||||
|
||||
### 2025/10/10 - v2.4.3
|
||||
|
||||
> 感谢 [nidaye996](https://github.com/sansan0/TrendRadar/issues/98) 发现的体验问题
|
||||
@ -1236,12 +1279,48 @@ docker exec -it trend-radar ls -la /app/config/
|
||||
</details>
|
||||
|
||||
|
||||
## 🤖 AI 智能分析部署
|
||||
|
||||
TrendRadar v3.0.0 新增了基于 **MCP (Model Context Protocol)** 的 AI 分析功能，让你可以通过自然语言与新闻数据对话，进行深度分析。
|
||||
|
||||
### 快速部署
|
||||
|
||||
Cherry Studio 提供 GUI 配置界面，可快速部署。
|
||||
|
||||
**详细教程**：[README-Cherry-Studio.md](README-Cherry-Studio.md)
|
||||
|
||||
|
||||
### 与 AI 对话的姿势
|
||||
|
||||
**基础查询**：
|
||||
```
|
||||
"给我看看最新的新闻"
|
||||
"查询昨天知乎的热点"
|
||||
"我关注的词今天出现了多少次"
|
||||
```
|
||||
|
||||
**趋势分析**：
|
||||
```
|
||||
"分析'比特币'最近一周的热度趋势"
|
||||
"看看'iPhone'话题是昙花一现还是持续热点"
|
||||
```
|
||||
|
||||
**详细教程**：[README-MCP-FAQ.md](README-MCP-FAQ.md)
|
||||
|
||||
>如果还有配置部署方面的问题，后续我会根据反馈出个**图文教程**，届时会更新到我的公众号上
|
||||
|
||||
## ☕问题答疑与1元点赞
|
||||
|
||||
> 心意到就行，收到的**点赞**用于提高开发者开源的积极性。**点赞**已收录于**致谢名单**
|
||||
|
||||
- **GitHub Issues**：适合针对性强的解答。提问时请提供完整信息（截图、错误日志、系统环境等）。
|
||||
- **公众号交流**：适合快速咨询和使用疑问。可以在文章下留言或私信交流。
|
||||
- **公众号交流**：适合快速咨询。尽量别私信，建议优先在相关文章下的公共留言区交流，有个收敛度
|
||||
|
||||
> 对于极少数私信中表达的那种'摸索了好久都部署不了'的焦急心态，我能理解，但我认为这不足以成为我必须立刻抽空帮你解决的理由
|
||||
>
|
||||
> 请舒缓情绪后再表达，别让我觉得'我分享了项目，你部署不了反而首先怪我'，对于这种非商业化的个人开源小作品，多一点担待~好不？😉
|
||||
>
|
||||
> 项目的 issues 和留言我还算卖力解决，除了有自己写的项目哪怕再烂也得含泪😢维护的念头，主要还是因为有上面感谢的那群人的支持😘
|
||||
|
||||
|公众号关注 |微信点赞 | 支付宝点赞 |
|
||||
|:---:|:---:|:---:|
|
||||
|
||||
@ -1,3 +1,5 @@
|
||||
requests==2.32.4
|
||||
pytz==2025.2
|
||||
PyYAML==6.0.2
|
||||
requests>=2.32.5
|
||||
pytz>=2025.2
|
||||
PyYAML>=6.0.3
|
||||
fastmcp>=2.12.0
|
||||
websockets>=13.0,<14.0
|
||||
118
setup-mac.sh
Normal file
118
setup-mac.sh
Normal file
@ -0,0 +1,118 @@
|
||||
#!/bin/bash
|
||||
|
||||
# 颜色定义
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
BOLD='\033[1m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo -e "${BOLD}╔════════════════════════════════════════╗${NC}"
|
||||
echo -e "${BOLD}║ TrendRadar MCP 一键部署 (Mac) ║${NC}"
|
||||
echo -e "${BOLD}╚════════════════════════════════════════╝${NC}"
|
||||
echo ""
|
||||
|
||||
# 获取项目根目录
|
||||
PROJECT_ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
|
||||
echo -e "📍 项目目录: ${BLUE}${PROJECT_ROOT}${NC}"
|
||||
echo ""
|
||||
|
||||
# 检查 UV 是否已安装
|
||||
if ! command -v uv &> /dev/null; then
|
||||
echo -e "${YELLOW}[1/3] 🔧 UV 未安装，正在自动安装...${NC}"
|
||||
echo "提示: UV 是一个快速的 Python 包管理器，只需安装一次"
|
||||
echo ""
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
|
||||
echo ""
|
||||
echo "正在刷新 PATH 环境变量..."
|
||||
echo ""
|
||||
|
||||
# 添加 UV 到 PATH
|
||||
export PATH="$HOME/.cargo/bin:$PATH"
|
||||
|
||||
# 验证 UV 是否真正可用
|
||||
if ! command -v uv &> /dev/null; then
|
||||
echo -e "${RED}❌ [错误] UV 安装失败${NC}"
|
||||
echo ""
|
||||
echo "可能的原因："
|
||||
echo " 1. 网络连接问题，无法下载安装脚本"
|
||||
echo " 2. 安装路径权限不足"
|
||||
echo " 3. 安装脚本执行异常"
|
||||
echo ""
|
||||
echo "解决方案："
|
||||
echo " 1. 检查网络连接是否正常"
|
||||
echo " 2. 手动安装: https://docs.astral.sh/uv/getting-started/installation/"
|
||||
echo " 3. 或运行: curl -LsSf https://astral.sh/uv/install.sh | sh"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}✅ [成功] UV 已安装${NC}"
|
||||
echo -e "${YELLOW}⚠️ 请重新运行此脚本以继续${NC}"
|
||||
exit 0
|
||||
else
|
||||
echo -e "${GREEN}[1/3] ✅ UV 已安装${NC}"
|
||||
uv --version
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "[2/3] 📦 安装项目依赖..."
|
||||
echo "提示: 这可能需要 1-2 分钟，请耐心等待"
|
||||
echo ""
|
||||
|
||||
# 创建虚拟环境并安装依赖
|
||||
uv sync
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
echo ""
|
||||
echo -e "${RED}❌ [错误] 依赖安装失败${NC}"
|
||||
echo "请检查网络连接后重试"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}[3/3] ✅ 检查配置文件...${NC}"
|
||||
echo ""
|
||||
|
||||
# 检查配置文件
|
||||
if [ ! -f "config/config.yaml" ]; then
|
||||
echo -e "${YELLOW}⚠️ [警告] 未找到配置文件: config/config.yaml${NC}"
|
||||
echo "请确保配置文件存在"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# 添加执行权限
|
||||
chmod +x start-http.sh 2>/dev/null || true
|
||||
|
||||
# 获取 UV 路径
|
||||
UV_PATH=$(which uv)
|
||||
|
||||
echo ""
|
||||
echo -e "${BOLD}╔════════════════════════════════════════╗${NC}"
|
||||
echo -e "${BOLD}║ 部署完成！ ║${NC}"
|
||||
echo -e "${BOLD}╚════════════════════════════════════════╝${NC}"
|
||||
echo ""
|
||||
echo "📋 下一步操作:"
|
||||
echo ""
|
||||
echo " 1️⃣ 打开 Cherry Studio"
|
||||
echo " 2️⃣ 进入 设置 > MCP Servers > 添加服务器"
|
||||
echo " 3️⃣ 填入以下配置:"
|
||||
echo ""
|
||||
echo " 名称: TrendRadar"
|
||||
echo " 描述: 新闻热点聚合工具"
|
||||
echo " 类型: STDIO"
|
||||
echo -e " 命令: ${BLUE}${UV_PATH}${NC}"
|
||||
echo " 参数（每个占一行）:"
|
||||
echo -e " ${BLUE}--directory${NC}"
|
||||
echo -e " ${BLUE}${PROJECT_ROOT}${NC}"
|
||||
echo -e " ${BLUE}run${NC}"
|
||||
echo -e " ${BLUE}python${NC}"
|
||||
echo -e " ${BLUE}-m${NC}"
|
||||
echo -e " ${BLUE}mcp_server.server${NC}"
|
||||
echo ""
|
||||
echo " 4️⃣ 保存并启用 MCP 开关"
|
||||
echo ""
|
||||
echo "📖 详细教程请查看: README-Cherry-Studio.md，本窗口别关，待会儿用于填入参数"
|
||||
echo ""
|
||||
114
setup-windows.bat
Normal file
114
setup-windows.bat
Normal file
@ -0,0 +1,114 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
setlocal enabledelayedexpansion
|
||||
|
||||
echo ╔════════════════════════════════════════╗
|
||||
echo ║ TrendRadar MCP 一键部署 (Windows) ║
|
||||
echo ╚════════════════════════════════════════╝
|
||||
echo.
|
||||
|
||||
REM 获取当前目录作为项目根目录
|
||||
set "PROJECT_ROOT=%CD%"
|
||||
|
||||
echo 📍 项目目录: %PROJECT_ROOT%
|
||||
echo.
|
||||
|
||||
REM 检查 UV 是否已安装
|
||||
where uv >nul 2>&1
|
||||
if %errorlevel% neq 0 (
|
||||
echo [1/3] 🔧 UV 未安装，正在自动安装...
|
||||
echo 提示: UV 是一个快速的 Python 包管理器，只需安装一次
|
||||
echo.
|
||||
powershell -Command "irm https://astral.sh/uv/install.ps1 | iex"
|
||||
|
||||
echo.
|
||||
echo 🔄 刷新环境变量并检测 UV 安装状态...
|
||||
echo.
|
||||
|
||||
REM 刷新 PATH 环境变量
|
||||
for /f "tokens=2*" %%a in ('reg query "HKCU\Environment" /v PATH 2^>nul') do set "USER_PATH=%%b"
|
||||
for /f "tokens=2*" %%a in ('reg query "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment" /v PATH 2^>nul') do set "SYSTEM_PATH=%%b"
|
||||
set "PATH=%USER_PATH%;%SYSTEM_PATH%"
|
||||
|
||||
REM 再次检查 UV 是否可用
|
||||
where uv >nul 2>&1
|
||||
if %errorlevel% neq 0 (
|
||||
echo ❌ [错误] UV 安装失败 - 无法找到 UV 命令
|
||||
echo 可能的原因:
|
||||
echo - 网络连接问题，安装脚本未成功下载
|
||||
echo - 安装路径未正确添加到 PATH
|
||||
echo.
|
||||
echo 解决方案:
|
||||
echo 1. 请关闭此窗口，重新打开命令提示符后再次运行本脚本
|
||||
echo 2. 或手动安装: https://docs.astral.sh/uv/getting-started/installation/
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo ✅ [成功] UV 已安装
|
||||
echo ⚠️ 请关闭此窗口，重新运行本脚本以继续安装依赖
|
||||
pause
|
||||
exit /b 0
|
||||
) else (
|
||||
echo [1/3] ✅ UV 已安装
|
||||
uv --version
|
||||
)
|
||||
|
||||
echo.
|
||||
echo [2/3] 📦 安装项目依赖...
|
||||
echo 提示: 这可能需要 1-2 分钟，请耐心等待
|
||||
echo.
|
||||
|
||||
REM 创建虚拟环境并安装依赖
|
||||
uv sync
|
||||
|
||||
if %errorlevel% neq 0 (
|
||||
echo.
|
||||
echo ❌ [错误] 依赖安装失败
|
||||
echo 请检查网络连接后重试
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo.
|
||||
echo [3/3] ✅ 检查配置文件...
|
||||
echo.
|
||||
|
||||
REM 检查配置文件
|
||||
if not exist "config\config.yaml" (
|
||||
echo ⚠️ [警告] 未找到配置文件: config\config.yaml
|
||||
echo 请确保配置文件存在
|
||||
echo.
|
||||
)
|
||||
|
||||
REM 获取 UV 的完整路径
|
||||
for /f "tokens=*" %%i in ('where uv') do set "UV_PATH=%%i"
|
||||
|
||||
echo.
|
||||
echo ╔════════════════════════════════════════╗
|
||||
echo ║ 部署完成！ ║
|
||||
echo ╚════════════════════════════════════════╝
|
||||
echo.
|
||||
echo 📋 下一步操作:
|
||||
echo.
|
||||
echo 1️⃣ 打开 Cherry Studio
|
||||
echo 2️⃣ 进入 设置 ^> MCP Servers ^> 添加服务器
|
||||
echo 3️⃣ 填入以下配置:
|
||||
echo.
|
||||
echo 名称: TrendRadar
|
||||
echo 描述: 新闻热点聚合工具
|
||||
echo 类型: STDIO
|
||||
echo 命令: %UV_PATH%
|
||||
echo 参数（每个占一行）:
|
||||
echo --directory
|
||||
echo %PROJECT_ROOT%
|
||||
echo run
|
||||
echo python
|
||||
echo -m
|
||||
echo mcp_server.server
|
||||
echo.
|
||||
echo 4️⃣ 保存并启用 MCP 开关
|
||||
echo.
|
||||
echo 📖 详细教程请查看: README-Cherry-Studio.md，本窗口别关，待会儿用于填入参数
|
||||
echo.
|
||||
pause
|
||||
25
start-http.bat
Normal file
25
start-http.bat
Normal file
@ -0,0 +1,25 @@
|
||||
@echo off
|
||||
chcp 65001 >nul
|
||||
|
||||
echo ╔════════════════════════════════════════╗
|
||||
echo ║ TrendRadar MCP Server (HTTP 模式) ║
|
||||
echo ╚════════════════════════════════════════╝
|
||||
echo.
|
||||
|
||||
REM 检查虚拟环境
|
||||
if not exist ".venv\Scripts\python.exe" (
|
||||
echo ❌ [错误] 虚拟环境未找到
|
||||
echo 请先运行 setup-windows.bat 进行部署
|
||||
echo.
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo [模式] HTTP (适合远程访问)
|
||||
echo [地址] http://localhost:3333/mcp
|
||||
echo [提示] 按 Ctrl+C 停止服务
|
||||
echo.
|
||||
|
||||
uv run python mcp_server/server.py --transport http --host 0.0.0.0 --port 3333
|
||||
|
||||
pause
|
||||
21
start-http.sh
Normal file
21
start-http.sh
Normal file
@ -0,0 +1,21 @@
|
||||
#!/bin/bash
|
||||
|
||||
echo "╔════════════════════════════════════════╗"
|
||||
echo "║ TrendRadar MCP Server (HTTP 模式) ║"
|
||||
echo "╚════════════════════════════════════════╝"
|
||||
echo ""
|
||||
|
||||
# 检查虚拟环境
|
||||
if [ ! -d ".venv" ]; then
|
||||
echo "❌ [错误] 虚拟环境未找到"
|
||||
echo "请先运行 ./setup-mac.sh 进行部署"
|
||||
echo ""
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[模式] HTTP (适合远程访问)"
|
||||
echo "[地址] http://localhost:3333/mcp"
|
||||
echo "[提示] 按 Ctrl+C 停止服务"
|
||||
echo ""
|
||||
|
||||
uv run python mcp_server/server.py --transport http --host 0.0.0.0 --port 3333
|
||||
Loading…
Reference in New Issue
Block a user