init: 初始化 AssetX 项目仓库
包含 webapp（Next.js 用户端）、webapp-back（Go 后端）、 antdesign（管理后台）、landingpage（营销落地页）、 数据库 SQL 和配置文件。
This commit is contained in:
157
webapp-back/holders/README-CN.md
Normal file
157
webapp-back/holders/README-CN.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# Holder Scanner 使用说明
|
||||
|
||||
## 问题诊断
|
||||
|
||||
你遇到的问题是：**Scanner（扫描器）没有运行**，所以数据库中没有数据。
|
||||
|
||||
当前状态：
|
||||
- ✅ API Server 正常运行
|
||||
- ❌ Scanner 未启动（需要启动才能从区块链获取数据）
|
||||
- ❌ 部署区块号可能不正确（需要更新）
|
||||
|
||||
## 解决步骤
|
||||
|
||||
### 第一步：获取正确的部署区块号
|
||||
|
||||
你数据库中的合约地址是：
|
||||
- YT-A: `0x7f9eEA491eE53045594ee4669327f0355aCd0e58`
|
||||
- YT-B: `0x20B94C5E5b7361552E0548161a58696aA6FeDBd4`
|
||||
- YT-C: `0x0EF308D70cf35460E26a3Eb42F3442Ff28cbE07C`
|
||||
|
||||
**重要**：你需要找到这些合约的**实际部署区块号**。
|
||||
|
||||
#### 方法 1：通过 Arbiscan 查询
|
||||
|
||||
1. 访问 [Arbiscan Sepolia](https://sepolia.arbiscan.io/)
|
||||
2. 搜索合约地址（例如：`0x7f9eEA491eE53045594ee4669327f0355aCd0e58`）
|
||||
3. 查看 "Contract Creation" 或 "Contract Creator" 信息
|
||||
4. 找到 "Block" 号码
|
||||
|
||||
#### 方法 2：使用 Web3 工具查询
|
||||
|
||||
```bash
|
||||
# 使用 cast (foundry) 工具
|
||||
cast block-number --rpc-url https://sepolia-rollup.arbitrum.io/rpc
|
||||
|
||||
# 或者使用 etherscan API
|
||||
curl "https://api-sepolia.arbiscan.io/api?module=contract&action=getcontractcreation&contractaddresses=0x7f9eEA491eE53045594ee4669327f0355aCd0e58"
|
||||
```
|
||||
|
||||
#### 方法 3：临时方案（不推荐）
|
||||
|
||||
如果暂时无法获取准确的部署区块号，可以使用一个较新的区块号（例如最新区块 - 10000），但这会导致：
|
||||
- ⚠️ 可能遗漏早期的持有者数据
|
||||
- ⚠️ 首次扫描时间更短，但数据不完整
|
||||
|
||||
### 第二步：更新 `.env` 配置
|
||||
|
||||
编辑 `.env` 文件，更新部署区块号：
|
||||
|
||||
```bash
|
||||
# Deployment Block Numbers
|
||||
YT_VAULTS_DEPLOY_BLOCK=123456789 # 替换为 YT-A/B/C 的实际部署区块
|
||||
YTLP_DEPLOY_BLOCK=123456789 # 替换为 ytLP 的实际部署区块
|
||||
LENDING_DEPLOY_BLOCK=123456789 # 替换为 Lending 的实际部署区块
|
||||
```
|
||||
|
||||
### 第三步：启动 Scanner
|
||||
|
||||
#### 方法 1：使用快速启动脚本（推荐）
|
||||
|
||||
```bash
|
||||
# Linux/Mac
|
||||
./run-scanner.sh
|
||||
|
||||
# Windows (Git Bash)
|
||||
bash run-scanner.sh
|
||||
```
|
||||
|
||||
#### 方法 2：手动编译并运行
|
||||
|
||||
```bash
|
||||
# 编译
|
||||
go build -o bin/holder-scanner cmd/scanner/main.go
|
||||
|
||||
# 运行
|
||||
./bin/holder-scanner
|
||||
```
|
||||
|
||||
#### 方法 3：使用 start-holders.sh（交互式）
|
||||
|
||||
```bash
|
||||
./start-holders.sh
|
||||
# 选择: 2. Scanner only
|
||||
```
|
||||
|
||||
### 第四步：验证数据
|
||||
|
||||
启动 Scanner 后，你应该看到类似的日志：
|
||||
|
||||
```
|
||||
=== Holder Scanner Service ===
|
||||
✓ Configuration loaded
|
||||
✓ Database tables checked
|
||||
🔗 Chain ID: 421614
|
||||
📚 从数据库加载合约地址配置...
|
||||
✅ [Scanner] 加载了 3 个 YT 资产
|
||||
✓ YT-A: 0x7f9eEA491eE53045594ee4669327f0355aCd0e58
|
||||
✓ YT-B: 0x20B94C5E5b7361552E0548161a58696aA6FeDBd4
|
||||
✓ YT-C: 0x0EF308D70cf35460E26a3Eb42F3442Ff28cbE07C
|
||||
...
|
||||
📊 开始首次扫描...
|
||||
1. Scanning YT Vaults...
|
||||
正在查询 YT-A (0x7f9eEA491eE53045594ee4669327f0355aCd0e58)...
|
||||
发现 X 个新地址
|
||||
余额>0: X 个持有者 ✅
|
||||
```
|
||||
|
||||
等待首次扫描完成后，再次访问 API：
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/api/holders/YT-A
|
||||
```
|
||||
|
||||
应该能看到持有者数据。
|
||||
|
||||
## 常见问题
|
||||
|
||||
### Q1: Scanner 报错 "failed to query logs"
|
||||
**原因**：部署区块号设置太早，或 RPC 节点限制
|
||||
|
||||
**解决**：
|
||||
1. 检查部署区块号是否正确
|
||||
2. 尝试使用更小的 `BatchSize`（在代码中默认 9999）
|
||||
3. 更换 RPC 节点
|
||||
|
||||
### Q2: 扫描速度很慢
|
||||
**原因**：区块范围太大
|
||||
|
||||
**解决**：
|
||||
1. 确保使用准确的部署区块号（不要从太早的区块开始）
|
||||
2. 检查网络连接
|
||||
3. 考虑使用付费的 RPC 节点（更高的速率限制）
|
||||
|
||||
### Q3: 余额都是 0
|
||||
**原因**：合约地址不正确，或合约不是 ERC20
|
||||
|
||||
**解决**：
|
||||
1. 验证合约地址是否正确
|
||||
2. 在区块浏览器上检查合约是否是 ERC20 代币
|
||||
3. 确认合约在对应的链上（Arbitrum Sepolia）
|
||||
|
||||
## 配置参数说明
|
||||
|
||||
| 环境变量 | 说明 | 默认值 |
|
||||
|---------|------|--------|
|
||||
| `CHAIN_ID` | 链 ID（421614=Arbitrum Sepolia, 97=BSC Testnet） | 421614 |
|
||||
| `YT_VAULTS_DEPLOY_BLOCK` | YT 代币合约部署区块号 | 227339300 |
|
||||
| `YTLP_DEPLOY_BLOCK` | ytLP 合约部署区块号 | 227230270 |
|
||||
| `LENDING_DEPLOY_BLOCK` | Lending 合约部署区块号 | 227746053 |
|
||||
|
||||
## 技术支持
|
||||
|
||||
如果遇到问题，请提供：
|
||||
1. Scanner 完整日志
|
||||
2. 合约地址
|
||||
3. 部署区块号
|
||||
4. 错误信息
|
||||
454
webapp-back/holders/README.md
Normal file
454
webapp-back/holders/README.md
Normal file
@@ -0,0 +1,454 @@
|
||||
# Holders API - 持有者统计服务
|
||||
|
||||
## 📋 概述
|
||||
|
||||
这个模块提供了区块链代币持有者的统计和查询功能，包括：
|
||||
- **YT 代币持有者** (YT-A, YT-B, YT-C)
|
||||
- **ytLP 代币持有者**
|
||||
- **Lending 提供者**
|
||||
|
||||
## 🏗️ 架构
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────┐ ┌──────────────┐
|
||||
│ Frontend │─────▶│ API Server │─────▶│ Database │
|
||||
│ (Ant Design) │ │ (Gin + Gorm) │ │ (MySQL/PG) │
|
||||
└─────────────────┘ └──────────────────┘ └──────────────┘
|
||||
│ ▲
|
||||
│ │
|
||||
▼ │
|
||||
┌──────────────────┐ │
|
||||
│ Scanner Service │────────────┘
|
||||
│ (Blockchain) │
|
||||
└──────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Arbitrum RPC │
|
||||
│ (Sepolia) │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
## 📁 文件结构
|
||||
|
||||
```
|
||||
holders/
|
||||
├── routers.go # API 路由和处理函数
|
||||
├── scanner.go # 区块链扫描器核心逻辑
|
||||
└── README.md # 本文档
|
||||
|
||||
cmd/scanner/
|
||||
└── main.go # 扫描器服务启动入口
|
||||
|
||||
models/
|
||||
└── holder.go # 数据库模型定义
|
||||
```
|
||||
|
||||
## 🔧 安装依赖
|
||||
|
||||
### 1. 安装 Go 依赖
|
||||
|
||||
```bash
|
||||
cd /home/coder/myprojects/assetx/golang-gin-realworld
|
||||
|
||||
# 添加 ethereum 依赖
|
||||
go get github.com/ethereum/go-ethereum
|
||||
go get github.com/ethereum/go-ethereum/ethclient
|
||||
go get github.com/ethereum/go-ethereum/accounts/abi
|
||||
go get github.com/ethereum/go-ethereum/common
|
||||
go get github.com/ethereum/go-ethereum/core/types
|
||||
go get github.com/ethereum/go-ethereum/crypto
|
||||
|
||||
# 更新依赖
|
||||
go mod tidy
|
||||
```
|
||||
|
||||
### 2. 数据库迁移
|
||||
|
||||
数据库表会在启动时自动创建（通过 GORM AutoMigrate）：
|
||||
|
||||
```sql
|
||||
CREATE TABLE holder_snapshots (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
holder_address VARCHAR(42) NOT NULL,
|
||||
token_type VARCHAR(50) NOT NULL,
|
||||
token_address VARCHAR(42) NOT NULL,
|
||||
balance VARCHAR(78) NOT NULL,
|
||||
chain_id INTEGER NOT NULL,
|
||||
first_seen BIGINT NOT NULL,
|
||||
last_updated BIGINT NOT NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
|
||||
INDEX idx_holder_token (holder_address, token_type),
|
||||
INDEX idx_token_time (token_type, last_updated)
|
||||
);
|
||||
```
|
||||
|
||||
## 🚀 启动服务
|
||||
|
||||
### 方法一：分别启动（推荐开发环境）
|
||||
|
||||
**1. 启动 API 服务器**
|
||||
```bash
|
||||
cd /home/coder/myprojects/assetx/golang-gin-realworld
|
||||
|
||||
# 设置环境变量
|
||||
export DB_TYPE=mysql
|
||||
export DB_HOST=localhost
|
||||
export DB_PORT=3306
|
||||
export DB_USER=root
|
||||
export DB_PASSWORD=your_password
|
||||
export DB_NAME=assetx
|
||||
export GIN_MODE=release
|
||||
export PORT=8080
|
||||
|
||||
# 启动 API 服务
|
||||
go run main.go
|
||||
```
|
||||
|
||||
**2. 启动扫描器服务**
|
||||
```bash
|
||||
cd /home/coder/myprojects/assetx/golang-gin-realworld
|
||||
|
||||
# 设置 RPC URL（可选，有默认值）
|
||||
export RPC_URL=https://api.zan.top/node/v1/arb/sepolia/baf84c429d284bb5b676cb8c9ca21c07
|
||||
|
||||
# 启动扫描器
|
||||
go run cmd/scanner/main.go
|
||||
```
|
||||
|
||||
### 方法二：使用 PM2（推荐生产环境）
|
||||
|
||||
**1. 创建 PM2 配置文件**
|
||||
```json
|
||||
// ecosystem.holders.config.js
|
||||
module.exports = {
|
||||
apps: [
|
||||
{
|
||||
name: 'assetx-api',
|
||||
script: './golang-gin-realworld-example-app',
|
||||
env: {
|
||||
DB_TYPE: 'mysql',
|
||||
DB_HOST: 'localhost',
|
||||
DB_PORT: '3306',
|
||||
DB_USER: 'root',
|
||||
DB_PASSWORD: 'your_password',
|
||||
DB_NAME: 'assetx',
|
||||
GIN_MODE: 'release',
|
||||
PORT: '8080'
|
||||
}
|
||||
},
|
||||
{
|
||||
name: 'holder-scanner',
|
||||
script: 'go',
|
||||
args: 'run cmd/scanner/main.go',
|
||||
env: {
|
||||
DB_TYPE: 'mysql',
|
||||
DB_HOST: 'localhost',
|
||||
DB_PORT: '3306',
|
||||
DB_USER: 'root',
|
||||
DB_PASSWORD: 'your_password',
|
||||
DB_NAME: 'assetx',
|
||||
RPC_URL: 'https://api.zan.top/node/v1/arb/sepolia/baf84c429d284bb5b676cb8c9ca21c07'
|
||||
}
|
||||
}
|
||||
]
|
||||
};
|
||||
```
|
||||
|
||||
**2. 启动服务**
|
||||
```bash
|
||||
# 先编译 API 服务器
|
||||
go build -o golang-gin-realworld-example-app
|
||||
|
||||
# 使用 PM2 启动
|
||||
pm2 start ecosystem.holders.config.js
|
||||
pm2 logs
|
||||
pm2 status
|
||||
```
|
||||
|
||||
## 📡 API 接口
|
||||
|
||||
### 1. GET /api/holders/stats
|
||||
|
||||
获取所有代币类型的统计信息。
|
||||
|
||||
**请求**
|
||||
```bash
|
||||
curl http://localhost:8080/api/holders/stats
|
||||
```
|
||||
|
||||
**响应**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": [
|
||||
{
|
||||
"token_type": "YT-A",
|
||||
"holder_count": 156,
|
||||
"total_balance": "1250000000000000000000"
|
||||
},
|
||||
{
|
||||
"token_type": "YT-B",
|
||||
"holder_count": 89,
|
||||
"total_balance": "780000000000000000000"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. GET /api/holders/:tokenType
|
||||
|
||||
获取特定代币的持有者列表。
|
||||
|
||||
**参数**
|
||||
- `tokenType`: `YT-A` | `YT-B` | `YT-C` | `ytLP` | `Lending`
|
||||
|
||||
**请求**
|
||||
```bash
|
||||
curl http://localhost:8080/api/holders/YT-A
|
||||
```
|
||||
|
||||
**响应**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": [
|
||||
{
|
||||
"id": 1,
|
||||
"holder_address": "0x1234567890abcdef1234567890abcdef12345678",
|
||||
"token_type": "YT-A",
|
||||
"token_address": "0x97204190B35D9895a7a47aa7BaC61ac08De3cF05",
|
||||
"balance": "100000000000000000000",
|
||||
"chain_id": 421614,
|
||||
"first_seen": 1707800000,
|
||||
"last_updated": 1707900000
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 3. POST /api/holders/update
|
||||
|
||||
手动触发数据更新（需要管理员权限）。
|
||||
|
||||
**请求**
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/api/holders/update \
|
||||
-H "Authorization: Bearer YOUR_ADMIN_TOKEN"
|
||||
```
|
||||
|
||||
**响应**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Update triggered successfully",
|
||||
"timestamp": 1707900000
|
||||
}
|
||||
```
|
||||
|
||||
## ⚙️ 配置说明
|
||||
|
||||
### 扫描器配置
|
||||
|
||||
在 `cmd/scanner/main.go` 中可以修改以下配置：
|
||||
|
||||
```go
|
||||
scannerConfig := holders.Config{
|
||||
RPCURL: "your_rpc_url", // RPC 节点地址
|
||||
PollInterval: 10 * time.Second, // 轮询间隔（10秒）
|
||||
BatchSize: 9999, // 每批次查询的区块数
|
||||
|
||||
// 代币合约地址
|
||||
YTVaults: []holders.VaultConfig{
|
||||
{Name: "YT-A", Address: "0x..."},
|
||||
{Name: "YT-B", Address: "0x..."},
|
||||
{Name: "YT-C", Address: "0x..."},
|
||||
},
|
||||
YTLPAddress: "0x...",
|
||||
LendingAddress: "0x...",
|
||||
|
||||
// 部署区块号（从这里开始扫描）
|
||||
DeploymentBlocks: holders.DeploymentBlocks{
|
||||
YTVaults: 227339300,
|
||||
YTLP: 227230270,
|
||||
Lending: 227746053,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### 环境变量
|
||||
|
||||
| 变量名 | 说明 | 默认值 |
|
||||
|--------|------|--------|
|
||||
| `RPC_URL` | Arbitrum RPC 节点地址 | zan.top URL |
|
||||
| `DB_TYPE` | 数据库类型 | `sqlite` |
|
||||
| `DB_HOST` | 数据库主机 | `localhost` |
|
||||
| `DB_PORT` | 数据库端口 | `3306` |
|
||||
| `DB_USER` | 数据库用户名 | `root` |
|
||||
| `DB_PASSWORD` | 数据库密码 | - |
|
||||
| `DB_NAME` | 数据库名称 | `assetx` |
|
||||
|
||||
## 🔍 工作原理
|
||||
|
||||
### 扫描流程
|
||||
|
||||
1. **初始扫描**
|
||||
- 从合约部署区块开始
|
||||
- 扫描所有历史 Transfer/Supply 事件
|
||||
- 记录所有曾经持有代币的地址
|
||||
- 查询当前余额并保存到数据库
|
||||
|
||||
2. **增量扫描**
|
||||
- 每 10 秒检查一次新区块
|
||||
- 只扫描自上次扫描后的新区块
|
||||
- 更新新地址和余额变化
|
||||
- 持续追踪地址首次出现时间
|
||||
|
||||
3. **数据存储**
|
||||
- 使用 `holder_snapshots` 表存储快照
|
||||
- 余额以 wei 格式字符串存储（避免精度丢失）
|
||||
- 记录首次持有时间和最后更新时间
|
||||
- 支持唯一约束（address + token_type）
|
||||
|
||||
### 性能优化
|
||||
|
||||
- ✅ 批量查询区块事件（每次最多 9999 个区块）
|
||||
- ✅ 增量扫描（只查询新区块）
|
||||
- ✅ 地址缓存（减少重复查询）
|
||||
- ✅ 并发控制（防止重复扫描）
|
||||
- ✅ 速率限制（避免 RPC 限流）
|
||||
|
||||
## 📊 监控和日志
|
||||
|
||||
### 查看扫描器日志
|
||||
|
||||
```bash
|
||||
# 实时查看日志
|
||||
pm2 logs holder-scanner
|
||||
|
||||
# 或者直接运行时查看
|
||||
go run cmd/scanner/main.go
|
||||
```
|
||||
|
||||
### 日志示例
|
||||
|
||||
```
|
||||
=== Holder Scanner Service ===
|
||||
✓ Configuration loaded
|
||||
✓ Database tables checked
|
||||
🚀 Starting blockchain scanner...
|
||||
=== Holder Scanner Started ===
|
||||
RPC: https://api.zan.top/node/v1/arb/sepolia/...
|
||||
Poll Interval: 10s
|
||||
📊 Starting initial scan...
|
||||
Current block: 227950000
|
||||
|
||||
1. Scanning YT Vaults...
|
||||
Scanning YT-A (0x9720...)...
|
||||
Querying blocks 227339300 -> 227950000 (total: 610700 blocks)
|
||||
Querying blocks 227339300 - 227349299...
|
||||
✓ Got 45 events
|
||||
...
|
||||
Found 23 new addresses, total tracking: 156
|
||||
YT-A: 156 holders saved
|
||||
|
||||
2. Scanning ytLP...
|
||||
ytLP: 67 holders saved
|
||||
|
||||
3. Scanning Lending...
|
||||
Lending: 43 holders saved
|
||||
|
||||
📌 Last scanned block: 227950000
|
||||
✓ Initial scan completed in 2m34s
|
||||
|
||||
⏰ Starting polling every 10s...
|
||||
⏰ [15:30:45] No new blocks (current: 227950000)
|
||||
⏰ [15:30:55] Found new blocks
|
||||
🔄 Incremental scan: blocks 227950001 -> 227950015
|
||||
✓ Incremental scan completed in 1.2s
|
||||
```
|
||||
|
||||
## 🐛 故障排查
|
||||
|
||||
### 常见问题
|
||||
|
||||
**1. RPC 连接失败**
|
||||
```
|
||||
Error: failed to connect to Ethereum client
|
||||
```
|
||||
**解决**: 检查 `RPC_URL` 是否正确，网络是否通畅
|
||||
|
||||
**2. 数据库连接失败**
|
||||
```
|
||||
Error: failed to connect to database
|
||||
```
|
||||
**解决**: 检查数据库配置和权限
|
||||
|
||||
**3. 查询超时**
|
||||
```
|
||||
Error: context deadline exceeded
|
||||
```
|
||||
**解决**: 减小 `BatchSize` 参数（如改为 5000）
|
||||
|
||||
**4. 内存占用过高**
|
||||
```
|
||||
OOM or high memory usage
|
||||
```
|
||||
**解决**: 增加批次间隔时间或减小批次大小
|
||||
|
||||
### 数据验证
|
||||
|
||||
**检查数据库中的数据**
|
||||
```sql
|
||||
-- 查看所有代币类型的持有者数量
|
||||
SELECT token_type, COUNT(*) as count
|
||||
FROM holder_snapshots
|
||||
GROUP BY token_type;
|
||||
|
||||
-- 查看 YT-A 的前 10 名持有者
|
||||
SELECT holder_address, balance
|
||||
FROM holder_snapshots
|
||||
WHERE token_type = 'YT-A'
|
||||
ORDER BY CAST(balance AS DECIMAL) DESC
|
||||
LIMIT 10;
|
||||
|
||||
-- 查看最近更新的记录
|
||||
SELECT * FROM holder_snapshots
|
||||
ORDER BY last_updated DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
## 🔐 安全注意事项
|
||||
|
||||
1. **RPC 密钥保护**
|
||||
- 不要在代码中硬编码 RPC URL
|
||||
- 使用环境变量或配置文件
|
||||
- 限制 RPC API Key 的权限
|
||||
|
||||
2. **数据库安全**
|
||||
- 使用强密码
|
||||
- 限制数据库访问权限
|
||||
- 定期备份数据
|
||||
|
||||
3. **API 权限**
|
||||
- `/update` 接口需要管理员权限
|
||||
- 建议添加请求频率限制
|
||||
- 记录操作日志
|
||||
|
||||
## 📝 TODO
|
||||
|
||||
- [ ] 添加 WebSocket 支持（实时推送）
|
||||
- [ ] 实现手动触发扫描的实际逻辑
|
||||
- [ ] 添加 Prometheus 监控指标
|
||||
- [ ] 支持多链配置
|
||||
- [ ] 添加数据导出功能
|
||||
|
||||
## 📞 联系方式
|
||||
|
||||
如有问题请联系开发团队或查看项目文档。
|
||||
|
||||
---
|
||||
|
||||
**最后更新**: 2024-02-13
|
||||
125
webapp-back/holders/db_config.go
Normal file
125
webapp-back/holders/db_config.go
Normal file
@@ -0,0 +1,125 @@
|
||||
package holders
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"github.com/gothinkster/golang-gin-realworld-example-app/common"
|
||||
)
|
||||
|
||||
// Asset represents a product/asset in the database
|
||||
type Asset struct {
|
||||
ID int64 `gorm:"column:id"`
|
||||
AssetCode string `gorm:"column:asset_code"`
|
||||
Name string `gorm:"column:name"`
|
||||
TokenRole string `gorm:"column:token_role"`
|
||||
ChainID int `gorm:"column:chain_id"`
|
||||
ContractAddress string `gorm:"column:contract_address"`
|
||||
DeployBlock *uint64 `gorm:"column:deploy_block"`
|
||||
}
|
||||
|
||||
|
||||
const zeroAddress = "0x0000000000000000000000000000000000000000"
|
||||
|
||||
// LoadConfigFromDB loads contract addresses and deploy blocks from database based on chain ID
|
||||
func LoadConfigFromDB(chainID int64) (Config, error) {
|
||||
db := common.GetDB()
|
||||
|
||||
log.Printf("📚 [Scanner] 从数据库加载配置 - Chain ID: %d", chainID)
|
||||
|
||||
switch chainID {
|
||||
case 97, 421614:
|
||||
// supported
|
||||
default:
|
||||
return Config{}, fmt.Errorf("unsupported chain ID: %d", chainID)
|
||||
}
|
||||
|
||||
// Load YT assets by token_role filtered by chain_id
|
||||
var assets []Asset
|
||||
if err := db.Table("assets").
|
||||
Where("token_role = ? AND chain_id = ? AND is_active = ?", "yt_token", chainID, true).
|
||||
Find(&assets).Error; err != nil {
|
||||
return Config{}, fmt.Errorf("failed to load assets: %w", err)
|
||||
}
|
||||
|
||||
ytVaults := make([]VaultConfig, 0, len(assets))
|
||||
for _, asset := range assets {
|
||||
if asset.ContractAddress == "" || asset.ContractAddress == zeroAddress {
|
||||
log.Printf("⚠️ [Scanner] 跳过 %s (地址未配置)", asset.AssetCode)
|
||||
continue
|
||||
}
|
||||
if asset.DeployBlock == nil || *asset.DeployBlock == 0 {
|
||||
log.Printf("⚠️ [Scanner] 跳过 %s (deploy_block 未配置)", asset.AssetCode)
|
||||
continue
|
||||
}
|
||||
|
||||
ytVaults = append(ytVaults, VaultConfig{
|
||||
Name: asset.AssetCode,
|
||||
Address: asset.ContractAddress,
|
||||
DeployBlock: *asset.DeployBlock,
|
||||
})
|
||||
log.Printf(" ✓ %s: %s (部署区块: %d)", asset.AssetCode, asset.ContractAddress, *asset.DeployBlock)
|
||||
}
|
||||
log.Printf("✅ [Scanner] 加载了 %d 个 YT Vault", len(ytVaults))
|
||||
|
||||
// Load YTLPToken address from system_contracts
|
||||
var ytLPContract struct {
|
||||
Address string `gorm:"column:address"`
|
||||
DeployBlock *uint64 `gorm:"column:deploy_block"`
|
||||
}
|
||||
ytLPAddress := ""
|
||||
var ytLPDeployBlock uint64
|
||||
err := db.Table("system_contracts").
|
||||
Where("name = ? AND chain_id = ? AND is_active = ?", "YTLPToken", chainID, 1).
|
||||
Select("address, deploy_block").
|
||||
First(&ytLPContract).Error
|
||||
if err != nil {
|
||||
log.Printf("⚠️ [Scanner] 未找到 YTLPToken 配置")
|
||||
} else if ytLPContract.Address == "" || ytLPContract.Address == zeroAddress {
|
||||
log.Printf("⚠️ [Scanner] 跳过 ytLP (地址未配置)")
|
||||
} else if ytLPContract.DeployBlock == nil || *ytLPContract.DeployBlock == 0 {
|
||||
log.Printf("⚠️ [Scanner] 跳过 ytLP (deploy_block 未配置)")
|
||||
} else {
|
||||
ytLPAddress = ytLPContract.Address
|
||||
ytLPDeployBlock = *ytLPContract.DeployBlock
|
||||
log.Printf("✅ [Scanner] ytLP: %s (部署区块: %d)", ytLPAddress, ytLPDeployBlock)
|
||||
}
|
||||
|
||||
rpcURL := getRPCURLForChain(chainID)
|
||||
|
||||
config := Config{
|
||||
ChainID: int(chainID),
|
||||
RPCURL: rpcURL,
|
||||
YTVaults: ytVaults,
|
||||
YTLPAddress: ytLPAddress,
|
||||
DeploymentBlocks: DeploymentBlocks{
|
||||
YTLP: ytLPDeployBlock,
|
||||
},
|
||||
PollInterval: 30 * time.Second,
|
||||
BatchSize: 9999,
|
||||
}
|
||||
|
||||
log.Printf("📊 [Scanner] 配置加载完成: YT Vaults=%d, ytLP=%s, RPC=%s",
|
||||
len(config.YTVaults), config.YTLPAddress, config.RPCURL)
|
||||
|
||||
return config, nil
|
||||
}
|
||||
|
||||
// getRPCURLForChain returns the RPC URL for the given chain ID
|
||||
func getRPCURLForChain(chainID int64) string {
|
||||
switch chainID {
|
||||
case 421614:
|
||||
return "https://api.zan.top/node/v1/arb/sepolia/baf84c429d284bb5b676cb8c9ca21c07"
|
||||
case 97:
|
||||
return "https://api.zan.top/node/v1/bsc/testnet/baf84c429d284bb5b676cb8c9ca21c07"
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// TableName sets the table name for GORM
|
||||
func (Asset) TableName() string {
|
||||
return "assets"
|
||||
}
|
||||
|
||||
132
webapp-back/holders/routers.go
Normal file
132
webapp-back/holders/routers.go
Normal file
@@ -0,0 +1,132 @@
|
||||
package holders
|
||||
|
||||
import (
|
||||
"log"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/gothinkster/golang-gin-realworld-example-app/common"
|
||||
"github.com/gothinkster/golang-gin-realworld-example-app/models"
|
||||
)
|
||||
|
||||
// RegisterRoutes registers holders API routes
|
||||
func RegisterRoutes(router *gin.RouterGroup) {
|
||||
router.GET("/stats", GetStats)
|
||||
router.GET("/:tokenType", GetHoldersByType)
|
||||
router.POST("/update", UpdateHolders)
|
||||
}
|
||||
|
||||
// GetStats returns aggregated statistics for all token types
|
||||
func GetStats(c *gin.Context) {
|
||||
db := common.GetDB()
|
||||
|
||||
log.Printf("📊 [API] 收到获取统计数据请求 - IP: %s", c.ClientIP())
|
||||
|
||||
var stats []models.HolderStats
|
||||
err := db.Raw(`
|
||||
SELECT
|
||||
token_type,
|
||||
COUNT(DISTINCT holder_address) as holder_count
|
||||
FROM holder_snapshots
|
||||
WHERE balance != '0' AND balance != '' AND LENGTH(balance) > 0
|
||||
GROUP BY token_type
|
||||
`).Scan(&stats).Error
|
||||
|
||||
if err != nil {
|
||||
log.Printf("❌ [API] 查询统计数据失败: %v", err)
|
||||
c.JSON(http.StatusInternalServerError, gin.H{
|
||||
"success": false,
|
||||
"error": "Failed to fetch stats",
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
log.Printf("✅ [API] 成功返回统计数据 - %d 个token类型", len(stats))
|
||||
for _, stat := range stats {
|
||||
log.Printf(" - %s: %d 个持有者", stat.TokenType, stat.HolderCount)
|
||||
}
|
||||
|
||||
c.JSON(http.StatusOK, gin.H{
|
||||
"success": true,
|
||||
"data": stats,
|
||||
})
|
||||
}
|
||||
|
||||
// GetHoldersByType returns holders for a specific token type
|
||||
func GetHoldersByType(c *gin.Context) {
|
||||
tokenType := c.Param("tokenType")
|
||||
db := common.GetDB()
|
||||
|
||||
// 记录请求日志
|
||||
log.Printf("🔍 [API] 收到获取持有者请求 - Token类型: %s, IP: %s", tokenType, c.ClientIP())
|
||||
|
||||
var holders []models.HolderSnapshot
|
||||
err := db.Where("token_type = ? AND balance != ? AND balance != '' AND LENGTH(balance) > 0", tokenType, "0").
|
||||
Order("LENGTH(balance) DESC, balance DESC").
|
||||
Find(&holders).Error
|
||||
|
||||
if err != nil {
|
||||
log.Printf("❌ [API] 查询持有者失败 - Token类型: %s, 错误: %v", tokenType, err)
|
||||
c.JSON(http.StatusInternalServerError, gin.H{
|
||||
"success": false,
|
||||
"error": "Failed to fetch holders",
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
log.Printf("✅ [API] 成功返回 %d 个持有者 - Token类型: %s", len(holders), tokenType)
|
||||
|
||||
c.JSON(http.StatusOK, gin.H{
|
||||
"success": true,
|
||||
"data": holders,
|
||||
})
|
||||
}
|
||||
|
||||
// Global scanner instance
|
||||
var globalScanner *Scanner
|
||||
|
||||
// StartAllScanners 为所有支持的链启动 holder 扫描器（后台常驻）
|
||||
func StartAllScanners() {
|
||||
supportedChains := []int64{97, 421614}
|
||||
for _, chainID := range supportedChains {
|
||||
config, err := LoadConfigFromDB(chainID)
|
||||
if err != nil {
|
||||
log.Printf("[HolderScanner] Chain %d 配置加载失败: %v", chainID, err)
|
||||
continue
|
||||
}
|
||||
if len(config.YTVaults) == 0 && config.YTLPAddress == "" {
|
||||
log.Printf("[HolderScanner] Chain %d 无活跃资产，跳过", chainID)
|
||||
continue
|
||||
}
|
||||
go func(cfg Config) {
|
||||
log.Printf("[HolderScanner] Chain %d 启动，YTVaults=%d ytLP=%s",
|
||||
cfg.ChainID, len(cfg.YTVaults), cfg.YTLPAddress)
|
||||
if err := Start(cfg); err != nil {
|
||||
log.Printf("[HolderScanner] Chain %d 异常退出: %v", cfg.ChainID, err)
|
||||
}
|
||||
}(config)
|
||||
}
|
||||
}
|
||||
|
||||
// UpdateHolders triggers blockchain data update (admin only)
|
||||
func UpdateHolders(c *gin.Context) {
|
||||
if globalScanner == nil {
|
||||
c.JSON(http.StatusOK, gin.H{
|
||||
"success": true,
|
||||
"message": "Scanner is running in background. It will process new blocks on next tick.",
|
||||
"timestamp": time.Now().Unix(),
|
||||
})
|
||||
return
|
||||
}
|
||||
go func() {
|
||||
if err := globalScanner.incrementalScan(c.Request.Context()); err != nil {
|
||||
log.Printf("[HolderScanner] Manual update error: %v", err)
|
||||
}
|
||||
}()
|
||||
c.JSON(http.StatusOK, gin.H{
|
||||
"success": true,
|
||||
"message": "Manual scan triggered.",
|
||||
"timestamp": time.Now().Unix(),
|
||||
})
|
||||
}
|
||||
616
webapp-back/holders/scanner.go
Normal file
616
webapp-back/holders/scanner.go
Normal file
@@ -0,0 +1,616 @@
|
||||
package holders
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"math/big"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/ethereum/go-ethereum"
|
||||
"github.com/ethereum/go-ethereum/accounts/abi"
|
||||
"github.com/ethereum/go-ethereum/common"
|
||||
"github.com/ethereum/go-ethereum/core/types"
|
||||
"github.com/ethereum/go-ethereum/crypto"
|
||||
"github.com/ethereum/go-ethereum/ethclient"
|
||||
"github.com/gothinkster/golang-gin-realworld-example-app/models"
|
||||
db "github.com/gothinkster/golang-gin-realworld-example-app/common"
|
||||
)
|
||||
|
||||
// Config holds scanner configuration
|
||||
type Config struct {
|
||||
ChainID int
|
||||
RPCURL string
|
||||
YTVaults []VaultConfig
|
||||
YTLPAddress string
|
||||
DeploymentBlocks DeploymentBlocks
|
||||
PollInterval time.Duration
|
||||
BatchSize int64
|
||||
}
|
||||
|
||||
type VaultConfig struct {
|
||||
Name string
|
||||
Address string
|
||||
DeployBlock uint64
|
||||
}
|
||||
|
||||
type DeploymentBlocks struct {
|
||||
YTLP uint64
|
||||
}
|
||||
|
||||
// Scanner handles blockchain event scanning
|
||||
type Scanner struct {
|
||||
config Config
|
||||
client *ethclient.Client
|
||||
lastScannedBlock uint64
|
||||
isScanning bool
|
||||
mu sync.Mutex
|
||||
|
||||
// Address tracking
|
||||
ytAddresses map[string]map[string]int64 // vault -> address -> firstSeen
|
||||
lpAddresses map[string]int64 // address -> firstSeen
|
||||
}
|
||||
|
||||
// Event topics
|
||||
var (
|
||||
transferEventTopic = crypto.Keccak256Hash([]byte("Transfer(address,address,uint256)"))
|
||||
)
|
||||
|
||||
// NewScanner creates a new blockchain scanner
|
||||
func NewScanner(config Config) (*Scanner, error) {
|
||||
client, err := ethclient.Dial(config.RPCURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to connect to Ethereum client: %w", err)
|
||||
}
|
||||
|
||||
return &Scanner{
|
||||
config: config,
|
||||
client: client,
|
||||
ytAddresses: make(map[string]map[string]int64),
|
||||
lpAddresses: make(map[string]int64),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// loadStateFromDB loads the last scanned block from database
|
||||
func (s *Scanner) loadStateFromDB() uint64 {
|
||||
database := db.GetDB()
|
||||
var state models.ScannerState
|
||||
result := database.Where("chain_id = ? AND scanner_type = ?", s.config.ChainID, "holder").First(&state)
|
||||
if result.Error != nil {
|
||||
return 0
|
||||
}
|
||||
return state.LastScannedBlock
|
||||
}
|
||||
|
||||
// saveStateToDB persists the last scanned block to database
|
||||
func (s *Scanner) saveStateToDB(block uint64) {
|
||||
database := db.GetDB()
|
||||
state := models.ScannerState{
|
||||
ScannerType: "holder",
|
||||
ChainID: s.config.ChainID,
|
||||
LastScannedBlock: block,
|
||||
}
|
||||
database.Where("chain_id = ? AND scanner_type = ?", s.config.ChainID, "holder").Assign(state).FirstOrCreate(&state)
|
||||
database.Model(&state).Update("last_scanned_block", block)
|
||||
}
|
||||
|
||||
// loadAddressesFromDB restores in-memory address maps from holder_snapshots
|
||||
func (s *Scanner) loadAddressesFromDB() {
|
||||
database := db.GetDB()
|
||||
var snapshots []models.HolderSnapshot
|
||||
database.Where("chain_id = ?", s.config.ChainID).Find(&snapshots)
|
||||
|
||||
for _, snap := range snapshots {
|
||||
switch snap.TokenType {
|
||||
case "ytLP":
|
||||
s.lpAddresses[snap.HolderAddress] = snap.FirstSeen
|
||||
default:
|
||||
for _, vault := range s.config.YTVaults {
|
||||
if vault.Name == snap.TokenType {
|
||||
if s.ytAddresses[vault.Address] == nil {
|
||||
s.ytAddresses[vault.Address] = make(map[string]int64)
|
||||
}
|
||||
s.ytAddresses[vault.Address][snap.HolderAddress] = snap.FirstSeen
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
log.Printf("▶️ 从数据库加载了 %d 个历史地址", len(snapshots))
|
||||
}
|
||||
|
||||
// Start begins the scanning process
|
||||
func Start(config Config) error {
|
||||
scanner, err := NewScanner(config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Println("=== Holder Scanner Started ===")
|
||||
log.Printf("RPC: %s\n", config.RPCURL)
|
||||
log.Printf("Poll Interval: %v\n", config.PollInterval)
|
||||
|
||||
// Check if we can resume from a previous scan
|
||||
lastBlock := scanner.loadStateFromDB()
|
||||
if lastBlock > 0 {
|
||||
log.Printf("▶️ 发现上次扫描记录，从区块 %d 续扫...", lastBlock)
|
||||
scanner.loadAddressesFromDB()
|
||||
scanner.lastScannedBlock = lastBlock
|
||||
} else {
|
||||
// Fresh start: full initial scan from deploy blocks
|
||||
log.Println("📊 首次扫描，从部署区块开始...")
|
||||
startTime := time.Now()
|
||||
if err := scanner.scanAll(context.Background(), true); err != nil {
|
||||
return fmt.Errorf("initial scan failed: %w", err)
|
||||
}
|
||||
log.Printf("✓ 初始扫描完成，耗时 %v\n", time.Since(startTime))
|
||||
}
|
||||
|
||||
// Start polling
|
||||
ticker := time.NewTicker(config.PollInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
log.Printf("⏰ 开始轮询，每 %v 扫一次新区块...\n", config.PollInterval)
|
||||
|
||||
for range ticker.C {
|
||||
if err := scanner.incrementalScan(context.Background()); err != nil {
|
||||
log.Printf("✗ Incremental scan error: %v\n", err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// minUint64 returns the smaller of two uint64 values.
|
||||
func minUint64(a, b uint64) uint64 {
|
||||
if a < b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// scanBatch scans ALL contract types for a single block range [fromBlock, toBlock].
|
||||
// The range must be ≤ BatchSize. After this call, every contract has been scanned to toBlock.
|
||||
func (s *Scanner) scanBatch(ctx context.Context, fromBlock, toBlock uint64) {
|
||||
log.Printf(" [Batch] Scanning blocks %d → %d\n", fromBlock, toBlock)
|
||||
s.scanYTVaultTransfers(ctx, fromBlock, toBlock)
|
||||
s.scanYTLPTransfers(ctx, fromBlock, toBlock)
|
||||
s.scanSwapEvents(ctx, fromBlock, toBlock)
|
||||
|
||||
// Checkpoint: all contracts have been scanned up to toBlock
|
||||
s.mu.Lock()
|
||||
s.lastScannedBlock = toBlock
|
||||
s.mu.Unlock()
|
||||
s.saveStateToDB(toBlock)
|
||||
}
|
||||
|
||||
// scanAll performs a full scan from deployment blocks, saving a checkpoint after every batch.
|
||||
// All contract types are scanned together per batch, so the checkpoint is always safe.
|
||||
func (s *Scanner) scanAll(ctx context.Context, isInitial bool) error {
|
||||
s.mu.Lock()
|
||||
if s.isScanning {
|
||||
s.mu.Unlock()
|
||||
return fmt.Errorf("scan already in progress")
|
||||
}
|
||||
s.isScanning = true
|
||||
s.mu.Unlock()
|
||||
|
||||
defer func() {
|
||||
s.mu.Lock()
|
||||
s.isScanning = false
|
||||
s.mu.Unlock()
|
||||
}()
|
||||
|
||||
latestBlock, err := s.client.BlockNumber(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
log.Printf("Current block: %d\n", latestBlock)
|
||||
|
||||
// Determine the earliest block we need to scan across all contracts
|
||||
startBlock := s.lastScannedBlock + 1
|
||||
if isInitial {
|
||||
startBlock = latestBlock // will be lowered below
|
||||
for _, v := range s.config.YTVaults {
|
||||
startBlock = minUint64(startBlock, v.DeployBlock)
|
||||
}
|
||||
if s.config.YTLPAddress != "" && s.config.DeploymentBlocks.YTLP > 0 {
|
||||
startBlock = minUint64(startBlock, s.config.DeploymentBlocks.YTLP)
|
||||
}
|
||||
}
|
||||
if startBlock > latestBlock {
|
||||
log.Printf("📌 No new blocks to scan (startBlock %d > latestBlock %d)\n", startBlock, latestBlock)
|
||||
return nil
|
||||
}
|
||||
|
||||
log.Printf("📊 Scanning blocks %d → %d in batches of %d\n", startBlock, latestBlock, s.config.BatchSize)
|
||||
|
||||
// Outer loop: one checkpoint per batch, all contracts scanned together
|
||||
for batchFrom := startBlock; batchFrom <= latestBlock; {
|
||||
batchTo := minUint64(batchFrom+uint64(s.config.BatchSize)-1, latestBlock)
|
||||
s.scanBatch(ctx, batchFrom, batchTo)
|
||||
batchFrom = batchTo + 1
|
||||
|
||||
// Rate limiting between batches
|
||||
if batchFrom <= latestBlock {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// Balance snapshots: run once after all event scanning is done
|
||||
log.Printf("📊 Updating balance snapshots...\n")
|
||||
for _, vault := range s.config.YTVaults {
|
||||
if err := s.saveYTHolders(ctx, vault); err != nil {
|
||||
log.Printf(" [Snapshot] %s error: %v", vault.Name, err)
|
||||
}
|
||||
}
|
||||
if err := s.saveYTLPHolders(ctx); err != nil {
|
||||
log.Printf(" [Snapshot] ytLP error: %v", err)
|
||||
}
|
||||
|
||||
log.Printf("📌 Scan complete. Last scanned block: %d\n", latestBlock)
|
||||
return nil
|
||||
}
|
||||
|
||||
// incrementalScan scans new blocks since last scan.
|
||||
// Incremental ranges are small (seconds of blocks), so a single batch suffices.
|
||||
func (s *Scanner) incrementalScan(ctx context.Context) error {
|
||||
s.mu.Lock()
|
||||
if s.isScanning {
|
||||
s.mu.Unlock()
|
||||
log.Println("⏰ Skipping scan (previous scan still running)")
|
||||
return nil
|
||||
}
|
||||
s.isScanning = true
|
||||
lastBlock := s.lastScannedBlock
|
||||
s.mu.Unlock()
|
||||
|
||||
defer func() {
|
||||
s.mu.Lock()
|
||||
s.isScanning = false
|
||||
s.mu.Unlock()
|
||||
}()
|
||||
|
||||
latestBlock, err := s.client.BlockNumber(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if latestBlock <= lastBlock {
|
||||
log.Printf("⏰ [%s] No new blocks (current: %d)\n", time.Now().Format("15:04:05"), latestBlock)
|
||||
return nil
|
||||
}
|
||||
|
||||
log.Printf("\n%s\n", strings.Repeat("=", 60))
|
||||
log.Printf("⏰ [%s] Found new blocks %d → %d\n", time.Now().Format("15:04:05"), lastBlock+1, latestBlock)
|
||||
log.Printf("%s\n", strings.Repeat("=", 60))
|
||||
|
||||
startTime := time.Now()
|
||||
fromBlock := lastBlock + 1
|
||||
|
||||
// Incremental range may exceed BatchSize if server was down for a while; use batches
|
||||
for batchFrom := fromBlock; batchFrom <= latestBlock; {
|
||||
batchTo := minUint64(batchFrom+uint64(s.config.BatchSize)-1, latestBlock)
|
||||
s.scanBatch(ctx, batchFrom, batchTo)
|
||||
batchFrom = batchTo + 1
|
||||
if batchFrom <= latestBlock {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// Update balance snapshots
|
||||
for _, vault := range s.config.YTVaults {
|
||||
if err := s.saveYTHolders(ctx, vault); err != nil {
|
||||
log.Printf(" [Snapshot] %s error: %v", vault.Name, err)
|
||||
}
|
||||
}
|
||||
if err := s.saveYTLPHolders(ctx); err != nil {
|
||||
log.Printf(" [Snapshot] ytLP error: %v", err)
|
||||
}
|
||||
|
||||
log.Printf("✓ Incremental scan completed in %v\n", time.Since(startTime))
|
||||
return nil
|
||||
}
|
||||
|
||||
// scanYTVaultTransfers scans Transfer events for all YT vaults in the given block range.
|
||||
// fromBlock/toBlock must already be a single batch (≤ BatchSize blocks).
|
||||
// Each vault skips blocks before its own deployBlock.
|
||||
// Does NOT save balance snapshots (call saveYTHolders separately after all batches complete).
|
||||
func (s *Scanner) scanYTVaultTransfers(ctx context.Context, fromBlock, toBlock uint64) error {
|
||||
for _, vault := range s.config.YTVaults {
|
||||
// Skip if vault not yet deployed in this range
|
||||
if vault.DeployBlock > toBlock {
|
||||
continue
|
||||
}
|
||||
effectiveFrom := fromBlock
|
||||
if vault.DeployBlock > effectiveFrom {
|
||||
effectiveFrom = vault.DeployBlock
|
||||
}
|
||||
|
||||
logs, err := s.queryBlockRange(ctx, common.HexToAddress(vault.Address), transferEventTopic, effectiveFrom, toBlock)
|
||||
if err != nil {
|
||||
log.Printf(" [Transfer] %s query error: %v", vault.Name, err)
|
||||
continue
|
||||
}
|
||||
|
||||
if s.ytAddresses[vault.Address] == nil {
|
||||
s.ytAddresses[vault.Address] = make(map[string]int64)
|
||||
}
|
||||
|
||||
for _, l := range logs {
|
||||
toAddress := common.BytesToAddress(l.Topics[2].Bytes()).Hex()
|
||||
if toAddress == "0x0000000000000000000000000000000000000000" {
|
||||
continue
|
||||
}
|
||||
if _, exists := s.ytAddresses[vault.Address][toAddress]; !exists {
|
||||
block, err := s.client.BlockByNumber(ctx, big.NewInt(int64(l.BlockNumber)))
|
||||
if err == nil {
|
||||
s.ytAddresses[vault.Address][toAddress] = int64(block.Time())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// scanYTLPTransfers scans Transfer events for the YT LP token in the given block range.
|
||||
// Does NOT save balance snapshots (call saveYTLPHolders separately after all batches complete).
|
||||
func (s *Scanner) scanYTLPTransfers(ctx context.Context, fromBlock, toBlock uint64) error {
|
||||
if s.config.YTLPAddress == "" {
|
||||
return nil
|
||||
}
|
||||
lpDeployBlock := s.config.DeploymentBlocks.YTLP
|
||||
if lpDeployBlock > toBlock {
|
||||
return nil
|
||||
}
|
||||
effectiveFrom := fromBlock
|
||||
if lpDeployBlock > effectiveFrom {
|
||||
effectiveFrom = lpDeployBlock
|
||||
}
|
||||
|
||||
logs, err := s.queryBlockRange(ctx, common.HexToAddress(s.config.YTLPAddress), transferEventTopic, effectiveFrom, toBlock)
|
||||
if err != nil {
|
||||
log.Printf(" [Transfer] ytLP query error: %v", err)
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, l := range logs {
|
||||
toAddress := common.BytesToAddress(l.Topics[2].Bytes()).Hex()
|
||||
if toAddress == "0x0000000000000000000000000000000000000000" {
|
||||
continue
|
||||
}
|
||||
if _, exists := s.lpAddresses[toAddress]; !exists {
|
||||
block, err := s.client.BlockByNumber(ctx, big.NewInt(int64(l.BlockNumber)))
|
||||
if err == nil {
|
||||
s.lpAddresses[toAddress] = int64(block.Time())
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
// queryBlockRange queries logs for a single block range (no internal batching).
|
||||
// The caller is responsible for keeping the range within RPC limits (BatchSize).
|
||||
func (s *Scanner) queryBlockRange(ctx context.Context, contractAddr common.Address, topic common.Hash, fromBlock, toBlock uint64) ([]types.Log, error) {
|
||||
query := ethereum.FilterQuery{
|
||||
FromBlock: big.NewInt(int64(fromBlock)),
|
||||
ToBlock: big.NewInt(int64(toBlock)),
|
||||
Addresses: []common.Address{contractAddr},
|
||||
Topics: [][]common.Hash{{topic}},
|
||||
}
|
||||
logs, err := s.client.FilterLogs(ctx, query)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to query logs [%d-%d]: %w", fromBlock, toBlock, err)
|
||||
}
|
||||
return logs, nil
|
||||
}
|
||||
|
||||
// queryLogsInBatches queries logs in batches to avoid RPC limits.
|
||||
// Used only by incremental scans where range size is unpredictable.
|
||||
func (s *Scanner) queryLogsInBatches(ctx context.Context, contractAddr common.Address, topic common.Hash, fromBlock, toBlock uint64) ([]types.Log, error) {
|
||||
var allLogs []types.Log
|
||||
currentBlock := fromBlock
|
||||
|
||||
log.Printf(" Querying blocks %d -> %d (total: %d blocks)\n", fromBlock, toBlock, toBlock-fromBlock+1)
|
||||
|
||||
for currentBlock <= toBlock {
|
||||
endBlock := currentBlock + uint64(s.config.BatchSize)
|
||||
if endBlock > toBlock {
|
||||
endBlock = toBlock
|
||||
}
|
||||
|
||||
log.Printf(" Querying blocks %d - %d...\n", currentBlock, endBlock)
|
||||
|
||||
logs, err := s.queryBlockRange(ctx, contractAddr, topic, currentBlock, endBlock)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
allLogs = append(allLogs, logs...)
|
||||
log.Printf(" ✓ Got %d events\n", len(logs))
|
||||
|
||||
currentBlock = endBlock + 1
|
||||
|
||||
// Rate limiting
|
||||
if currentBlock <= toBlock {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf(" Total: %d events\n\n", len(allLogs))
|
||||
return allLogs, nil
|
||||
}
|
||||
|
||||
// saveYTHolders queries balances and saves to database
|
||||
func (s *Scanner) saveYTHolders(ctx context.Context, vault VaultConfig) error {
|
||||
// ERC20 balanceOf ABI
|
||||
balanceOfABI, _ := abi.JSON(strings.NewReader(`[{"constant":true,"inputs":[{"name":"account","type":"address"}],"name":"balanceOf","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"}]`))
|
||||
|
||||
database := db.GetDB()
|
||||
now := time.Now().Unix()
|
||||
holders := 0
|
||||
totalAddresses := len(s.ytAddresses[vault.Address])
|
||||
|
||||
contractAddr := common.HexToAddress(vault.Address)
|
||||
log.Printf("📞 [%s] 开始查询 %d 个地址的余额", vault.Name, totalAddresses)
|
||||
log.Printf("📍 [%s] 合约地址: %s", vault.Name, vault.Address)
|
||||
|
||||
processedCount := 0
|
||||
errorCount := 0
|
||||
zeroBalanceCount := 0
|
||||
|
||||
for address, firstSeen := range s.ytAddresses[vault.Address] {
|
||||
processedCount++
|
||||
|
||||
// Call balanceOf
|
||||
data, err := balanceOfABI.Pack("balanceOf", common.HexToAddress(address))
|
||||
if err != nil {
|
||||
log.Printf("❌ [%s] Pack balanceOf 失败: %s - %v", vault.Name, address, err)
|
||||
errorCount++
|
||||
continue
|
||||
}
|
||||
|
||||
result, err := s.client.CallContract(ctx, ethereum.CallMsg{
|
||||
To: &contractAddr,
|
||||
Data: data,
|
||||
}, nil)
|
||||
if err != nil {
|
||||
log.Printf("❌ [%s] CallContract 失败: %s - %v", vault.Name, address, err)
|
||||
errorCount++
|
||||
continue
|
||||
}
|
||||
|
||||
balance := new(big.Int).SetBytes(result)
|
||||
|
||||
// Log balance query result
|
||||
if processedCount <= 5 || balance.Cmp(big.NewInt(0)) > 0 {
|
||||
log.Printf(" [%s] 地址: %s, 余额: %s", vault.Name, address[:10]+"...", balance.String())
|
||||
}
|
||||
|
||||
if balance.Cmp(big.NewInt(0)) == 0 {
|
||||
zeroBalanceCount++
|
||||
continue
|
||||
}
|
||||
|
||||
holders++
|
||||
|
||||
// Upsert to database: create if not exists, only update balance/last_updated if exists
|
||||
holder := models.HolderSnapshot{
|
||||
HolderAddress: address,
|
||||
TokenType: vault.Name,
|
||||
TokenAddress: vault.Address,
|
||||
Balance: balance.String(),
|
||||
ChainID: s.config.ChainID,
|
||||
FirstSeen: firstSeen,
|
||||
LastUpdated: now,
|
||||
}
|
||||
var existing models.HolderSnapshot
|
||||
res := database.Where("holder_address = ? AND token_type = ?", address, vault.Name).First(&existing)
|
||||
if res.Error != nil {
|
||||
database.Create(&holder)
|
||||
} else {
|
||||
database.Model(&existing).Updates(map[string]interface{}{
|
||||
"balance": balance.String(),
|
||||
"last_updated": now,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
|
||||
log.Printf("📊 [%s] 查询完成统计:", vault.Name)
|
||||
log.Printf(" • 总地址数: %d", totalAddresses)
|
||||
log.Printf(" • 已处理: %d", processedCount)
|
||||
log.Printf(" • 余额>0: %d ✅", holders)
|
||||
log.Printf(" • 余额=0: %d", zeroBalanceCount)
|
||||
log.Printf(" • 错误数: %d", errorCount)
|
||||
log.Printf(" • 已保存到数据库: %d", holders)
|
||||
log.Printf("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
// saveYTLPHolders saves ytLP holders to database
|
||||
func (s *Scanner) saveYTLPHolders(ctx context.Context) error {
|
||||
balanceOfABI, _ := abi.JSON(strings.NewReader(`[{"constant":true,"inputs":[{"name":"account","type":"address"}],"name":"balanceOf","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"}]`))
|
||||
|
||||
database := db.GetDB()
|
||||
now := time.Now().Unix()
|
||||
holders := 0
|
||||
totalAddresses := len(s.lpAddresses)
|
||||
|
||||
contractAddr := common.HexToAddress(s.config.YTLPAddress)
|
||||
log.Printf("📞 [ytLP] 开始查询 %d 个地址的余额", totalAddresses)
|
||||
log.Printf("📍 [ytLP] 合约地址: %s", s.config.YTLPAddress)
|
||||
|
||||
processedCount := 0
|
||||
errorCount := 0
|
||||
zeroBalanceCount := 0
|
||||
|
||||
for address, firstSeen := range s.lpAddresses {
|
||||
processedCount++
|
||||
|
||||
data, err := balanceOfABI.Pack("balanceOf", common.HexToAddress(address))
|
||||
if err != nil {
|
||||
log.Printf("❌ [ytLP] Pack balanceOf 失败: %s - %v", address, err)
|
||||
errorCount++
|
||||
continue
|
||||
}
|
||||
|
||||
result, err := s.client.CallContract(ctx, ethereum.CallMsg{
|
||||
To: &contractAddr,
|
||||
Data: data,
|
||||
}, nil)
|
||||
if err != nil {
|
||||
log.Printf("❌ [ytLP] CallContract 失败: %s - %v", address, err)
|
||||
errorCount++
|
||||
continue
|
||||
}
|
||||
|
||||
balance := new(big.Int).SetBytes(result)
|
||||
|
||||
// Log balance query result
|
||||
if processedCount <= 5 || balance.Cmp(big.NewInt(0)) > 0 {
|
||||
log.Printf(" [ytLP] 地址: %s, 余额: %s", address[:10]+"...", balance.String())
|
||||
}
|
||||
|
||||
if balance.Cmp(big.NewInt(0)) == 0 {
|
||||
zeroBalanceCount++
|
||||
continue
|
||||
}
|
||||
|
||||
holders++
|
||||
|
||||
holder := models.HolderSnapshot{
|
||||
HolderAddress: address,
|
||||
TokenType: "ytLP",
|
||||
TokenAddress: s.config.YTLPAddress,
|
||||
Balance: balance.String(),
|
||||
ChainID: s.config.ChainID,
|
||||
FirstSeen: firstSeen,
|
||||
LastUpdated: now,
|
||||
}
|
||||
var existing models.HolderSnapshot
|
||||
res := database.Where("holder_address = ? AND token_type = ?", address, "ytLP").First(&existing)
|
||||
if res.Error != nil {
|
||||
database.Create(&holder)
|
||||
} else {
|
||||
database.Model(&existing).Updates(map[string]interface{}{
|
||||
"balance": balance.String(),
|
||||
"last_updated": now,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
|
||||
log.Printf("📊 [ytLP] 查询完成统计:")
|
||||
log.Printf(" • 总地址数: %d", totalAddresses)
|
||||
log.Printf(" • 已处理: %d", processedCount)
|
||||
log.Printf(" • 余额>0: %d ✅", holders)
|
||||
log.Printf(" • 余额=0: %d", zeroBalanceCount)
|
||||
log.Printf(" • 错误数: %d", errorCount)
|
||||
log.Printf(" • 已保存到数据库: %d", holders)
|
||||
log.Printf("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
99
webapp-back/holders/swap_scanner.go
Normal file
99
webapp-back/holders/swap_scanner.go
Normal file
@@ -0,0 +1,99 @@
|
||||
package holders
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"math/big"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/ethereum/go-ethereum/common"
|
||||
"github.com/ethereum/go-ethereum/crypto"
|
||||
db "github.com/gothinkster/golang-gin-realworld-example-app/common"
|
||||
"github.com/gothinkster/golang-gin-realworld-example-app/models"
|
||||
)
|
||||
|
||||
// Swap(address account, address tokenIn, address tokenOut, uint256 amountIn, uint256 amountOut, uint256 feeBasisPoints)
|
||||
// account/tokenIn/tokenOut are indexed → topics[1..3]
|
||||
// amountIn/amountOut/feeBasisPoints are non-indexed → data[0..95]
|
||||
var swapEventTopic = crypto.Keccak256Hash([]byte("Swap(address,address,address,uint256,uint256,uint256)"))
|
||||
|
||||
// scanSwapEvents scans Swap events from all YT vaults and upserts into yt_swap_records.
|
||||
func (s *Scanner) scanSwapEvents(ctx context.Context, fromBlock, toBlock uint64) error {
|
||||
if toBlock < fromBlock {
|
||||
return nil
|
||||
}
|
||||
|
||||
database := db.GetDB()
|
||||
totalNew := 0
|
||||
|
||||
// Cache block timestamps to avoid repeated RPC calls for the same block
|
||||
blockTimeCache := make(map[uint64]time.Time)
|
||||
|
||||
for _, vault := range s.config.YTVaults {
|
||||
log.Printf(" [SwapScanner] %s blocks %d→%d", vault.Name, fromBlock, toBlock)
|
||||
|
||||
logs, err := s.queryBlockRange(ctx, common.HexToAddress(vault.Address), swapEventTopic, fromBlock, toBlock)
|
||||
if err != nil {
|
||||
log.Printf(" [SwapScanner] Error scanning %s: %v", vault.Name, err)
|
||||
continue
|
||||
}
|
||||
|
||||
for _, l := range logs {
|
||||
// Validate topics and data length
|
||||
if len(l.Topics) < 4 || len(l.Data) < 64 {
|
||||
continue
|
||||
}
|
||||
|
||||
account := common.BytesToAddress(l.Topics[1].Bytes()).Hex()
|
||||
tokenIn := common.BytesToAddress(l.Topics[2].Bytes()).Hex()
|
||||
tokenOut := common.BytesToAddress(l.Topics[3].Bytes()).Hex()
|
||||
amountIn := new(big.Int).SetBytes(l.Data[0:32])
|
||||
amountOut := new(big.Int).SetBytes(l.Data[32:64])
|
||||
|
||||
// Get block time (cached per block number)
|
||||
blockTime, ok := blockTimeCache[l.BlockNumber]
|
||||
if !ok {
|
||||
blk, err := s.client.BlockByNumber(ctx, big.NewInt(int64(l.BlockNumber)))
|
||||
if err == nil {
|
||||
blockTime = time.Unix(int64(blk.Time()), 0)
|
||||
blockTimeCache[l.BlockNumber] = blockTime
|
||||
}
|
||||
}
|
||||
|
||||
record := models.YTSwapRecord{
|
||||
TxHash: l.TxHash.Hex(),
|
||||
LogIndex: uint(l.Index),
|
||||
ChainID: s.config.ChainID,
|
||||
BlockNumber: l.BlockNumber,
|
||||
BlockTime: blockTime,
|
||||
VaultAddr: vault.Address,
|
||||
Account: account,
|
||||
TokenIn: strings.ToLower(tokenIn),
|
||||
TokenOut: strings.ToLower(tokenOut),
|
||||
AmountIn: amountIn.String(),
|
||||
AmountOut: amountOut.String(),
|
||||
}
|
||||
|
||||
// Skip if already exists (idempotent)
|
||||
var existing models.YTSwapRecord
|
||||
res := database.Where("tx_hash = ? AND log_index = ?", record.TxHash, record.LogIndex).First(&existing)
|
||||
if res.Error == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if err := database.Create(&record).Error; err != nil {
|
||||
if !strings.Contains(err.Error(), "Duplicate") {
|
||||
log.Printf(" [SwapScanner] Save failed %s:%d: %v", record.TxHash, record.LogIndex, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
totalNew++
|
||||
}
|
||||
}
|
||||
|
||||
if totalNew > 0 {
|
||||
log.Printf(" [SwapScanner] Saved %d new swap records", totalNew)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
Reference in New Issue
Block a user