From 35b84787adcea98941ace4fc236a588cd6b2c25c Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sat, 15 Mar 2025 13:42:19 +0000 Subject: [PATCH 01/24] doc: GitBook - No subject --- doc/en/SUMMARY.md | 3 +-- doc/en/about/scope-of-inclusion.md | 18 +++++++++++++++--- doc/en/architecure/artificial-intelligence.md | 4 ++++ .../architecure/database-structure/README.md | 2 ++ doc/en/architecure/message-queue.md | 7 +++++++ doc/en/architecure/message-queue/README.md | 2 -- .../message-queue/videotagsqueue.md | 11 ----------- doc/en/architecure/overview.md | 11 ++++++++++- 8 files changed, 39 insertions(+), 19 deletions(-) create mode 100644 doc/en/architecure/message-queue.md delete mode 100644 doc/en/architecure/message-queue/README.md delete mode 100644 doc/en/architecure/message-queue/videotagsqueue.md diff --git a/doc/en/SUMMARY.md b/doc/en/SUMMARY.md index 5137229..536f1f7 100644 --- a/doc/en/SUMMARY.md +++ b/doc/en/SUMMARY.md @@ -12,8 +12,7 @@ * [Overview](architecure/overview.md) * [Database Structure](architecure/database-structure/README.md) * [Type of Song](architecure/database-structure/type-of-song.md) -* [Message Queue](architecure/message-queue/README.md) - * [VideoTagsQueue](architecure/message-queue/videotagsqueue.md) +* [Message Queue](architecure/message-queue.md) * [Artificial Intelligence](architecure/artificial-intelligence.md) ## API Doc diff --git a/doc/en/about/scope-of-inclusion.md b/doc/en/about/scope-of-inclusion.md index d893e33..214b141 100644 --- a/doc/en/about/scope-of-inclusion.md +++ b/doc/en/about/scope-of-inclusion.md @@ -6,11 +6,23 @@ For a **song**, it must meet the following conditions to be included in CVSA: ### Category 30 -In principle, the songs featured in CVSA must be included in a video categorized under VOCALOID·UTAU (ID 30) that is posted on Bilibili. In some special cases, this rule may not be enforced. +In principle, the songs must be featured in a video that is categorized under the VOCALOID·UTAU (ID 30) category in [Bilibili](https://en.wikipedia.org/wiki/Bilibili) in order to be observed by our [automation program](../architecure/overview.md#crawler). We welcome editors to manually add songs that have not been uploaded to bilibili / categorized under this category. -### At Leats One Line of Chinese +#### NEWS -The lyrics of the song must contain at least one line in Chinese. This means that even if a voicebank that only supports Chinese is used, if the lyrics of the song do not contain Chinese, it will not be included in the CVSA. +Recently, Bilibili seems to be offlining the sub-category. This means the VOCALOID·UTAU category can no longer be entered from the frontend, and producers can no longer upload videos to this category (instead, they can only choose the parent category "Music"). + +According to our experiments, Bilibili still retains the code logic of sub-categories in the backend, and newly published songs may still be in the VOCALOID·UTAU sub-category, and the related APIs can still work normally. However, there are [reports](https://www.bilibili.com/opus/1041223385394184199) that some of the new songs have been placed under the "Music General" sub-category.\ +We are still waiting for Bilibili's follow-up actions, and in the future, we may adjust the scope of our automated program's crawling. + +### At Leats One Line of Chinese / Chinese Virtual Singer + +The lyrics of the song must contain at least one line in Chinese. Otherwise, if the lyrics of the song do not contain Chinese, it will only be included in the CVSA only if a Chinese virtual singer has been used. + +We define a **Chinese virtual singer** as follows: + +1. The singer primarily uses Chinese voicebank (i.e. the most widely used voickbank for the singer is Chinese) +2. The singer is operated by a company, organization, individual or group located in Mainland China, Hong Kong, Macau or Taiwan. ### Using Vocal Synthesizer diff --git a/doc/en/architecure/artificial-intelligence.md b/doc/en/architecure/artificial-intelligence.md index 849cb27..1d560d9 100644 --- a/doc/en/architecure/artificial-intelligence.md +++ b/doc/en/architecure/artificial-intelligence.md @@ -11,3 +11,7 @@ Located at `/filter/` under project root dir, it classifies a video in the [cate * 0: Not related to Chinese vocal synthesis * 1: A original song with Chinese vocal synthesis * 2: A cover/remix song with Chinese vocal synthesis + +### The Predictor + +Located at `/pred/`under the project root dir, it predicts the future views of a video. This is a regression model that takes historical view trends of a video, other contextual information (such as the current time), and future time points to be predicted as feature inputs, and outputs the increment in the video's view count from "now" to the specified future time point. diff --git a/doc/en/architecure/database-structure/README.md b/doc/en/architecure/database-structure/README.md index 96704b7..f9f738e 100644 --- a/doc/en/architecure/database-structure/README.md +++ b/doc/en/architecure/database-structure/README.md @@ -8,4 +8,6 @@ All public data of CVSA (excluding users' personal data) is stored in a database * bili\_user: stores snapshots of Bilibili user information * all\_data: metadata of all videos in [category 30](../../about/scope-of-inclusion.md#category-30). * labelling\_result: Contains label of videos in `all_data`tagged by our [AI system](../artificial-intelligence.md#the-filter). +* video\_snapshot: Statistical data of videos that are fetched regularly (e.g., number of views, etc.), we call this fetch process as "snapshot". +* snapshot\_schedule: The scheduling information for video snapshots. diff --git a/doc/en/architecure/message-queue.md b/doc/en/architecure/message-queue.md new file mode 100644 index 0000000..4fa4877 --- /dev/null +++ b/doc/en/architecure/message-queue.md @@ -0,0 +1,7 @@ +# Message Queue + +We rely on message queues to manage the various tasks that [the cralwer ](overview.md#crawler)needs to perform. + +### Code Path + +Currently, the code related to message queues are located at `lib/mq` and `src`. diff --git a/doc/en/architecure/message-queue/README.md b/doc/en/architecure/message-queue/README.md deleted file mode 100644 index d0a8349..0000000 --- a/doc/en/architecure/message-queue/README.md +++ /dev/null @@ -1,2 +0,0 @@ -# Message Queue - diff --git a/doc/en/architecure/message-queue/videotagsqueue.md b/doc/en/architecure/message-queue/videotagsqueue.md deleted file mode 100644 index bdddddb..0000000 --- a/doc/en/architecure/message-queue/videotagsqueue.md +++ /dev/null @@ -1,11 +0,0 @@ -# VideoTagsQueue - -### Jobs - -The VideoTagsQueue contains two jobs: `getVideoTags`and `getVideosTags`. The former is used to fetch the tags of a video, and the latter is responsible for scheduling the former. - -### Return value - -The return values across two jobs follows the following table: - -
Return ValueDescription
0In getVideoTags: the tags was successfully fetched
In getVideosTags: all null-tags videos have a corresponding job successfully queued.
1Used in getVideoTags: occured fetcherror during the job
2Used in getVideoTags: we've reached the rate limit set in NetScheduler
3Used in getVideoTags: did't provide aid in the job data
4Used ingetVideosTags: There's no video with NULL as `tags`
1xxUsed ingetVideosTags: the number of tasks in the queue has exceeded the limit, thus getVideosTags stops adding tasks. xx is the number of jobs added to the queue during execution.
diff --git a/doc/en/architecure/overview.md b/doc/en/architecure/overview.md index d80036e..e46c887 100644 --- a/doc/en/architecure/overview.md +++ b/doc/en/architecure/overview.md @@ -1,5 +1,4 @@ --- -icon: globe-pointer layout: title: visible: true @@ -15,4 +14,14 @@ layout: # Overview +The whole CVSA system can be sperate into three different parts: + +* Frontend +* API +* Crawler + +The frontend is driven by [Astro](https://astro.build/) and is used to display the final CVSA page. The API is driven by [Hono](https://hono.dev) and is used to query the database and provide REST/GraphQL APIs that can be called by out website, applications, or third parties. The crawler is our automatic data collector, used to automatically collect new songs from bilibili, track their statistics, etc. + +### Crawler + Automation is the biggest highlight of CVSA's technical design. To achieve this, we use a message queue powered by [BullMQ](https://bullmq.io/) to concurrently process various tasks in the data collection life cycle. From f401417ce2f75757a632420ce6390a4e5b69ba54 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sat, 29 Mar 2025 19:02:48 +0800 Subject: [PATCH 02/24] update: remove refactor guide --- README-refactor.md | 65 ---------------------------------------------- 1 file changed, 65 deletions(-) delete mode 100644 README-refactor.md diff --git a/README-refactor.md b/README-refactor.md deleted file mode 100644 index 75ffdb9..0000000 --- a/README-refactor.md +++ /dev/null @@ -1,65 +0,0 @@ -# 项目重构方案 - -## 目标架构 -采用monorepo结构管理三个独立部分： -1. `packages/crawler` - 现有爬虫功能 -2. `packages/frontend` - 基于Astro的前端 -3. `packages/backend` - 基于Hono的API后端 - -## 目录结构调整方案 - -### 新结构 -``` -. -├── packages/ -│ ├── crawler/ # 爬虫组件 -│ ├── frontend/ # Astro前端 -│ ├── backend/ # Hono后端API -│ └── core/ # 共享代码(未来提取) -├── docs/ # 文档 -├── scripts/ # 项目脚本 -└── package.json # 根项目配置 -``` - -### 具体迁移方案 - -#### 1. 爬虫部分(crawler) -保留以下目录/文件： -- `lib/` (除前端相关) -- `src/db/raw/` -- `src/filterWorker.ts` -- `src/worker.ts` -- `test/` -- `deno.json` -- `.gitignore` - -需要移除： -- Fresh框架相关文件 -- 前端组件(`components/`) -- 静态资源(`static/`) - -#### 2. 前端部分(frontend) -全新创建Astro项目，不保留任何现有前端代码 - -#### 3. 后端部分(backend) -全新创建Hono项目 - -#### 4. 共享代码(core) -未来可从爬虫中提取以下内容到core package： -- 数据库相关：`lib/db/` -- 消息队列：`lib/mq/` -- 网络请求：`lib/net/` -- 工具函数：`lib/utils/` - -## 重构步骤建议 - -1. 初始化monorepo结构 -2. 迁移爬虫代码到`packages/crawler` -3. 创建新的Astro项目在`packages/frontend` -4. 创建新的Hono项目在`packages/backend` -5. 逐步提取共享代码到`packages/core` - -## 注意事项 -- 机器学习相关代码(`pred/`, `filter/`, `lab/`)保持现状 -- 文档(`doc/`)可以迁移到`docs/`目录 -- 需要更新CI/CD流程支持monorepo \ No newline at end of file From 291a21d82a59468feb01d0b09eb55a4d9c991ec5 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sat, 29 Mar 2025 19:10:52 +0800 Subject: [PATCH 03/24] init: astro frontend --- packages/frontend/.gitignore | 24 ++ packages/frontend/README.md | 48 ++++ packages/frontend/astro.config.mjs | 5 + packages/frontend/deno.json | 0 packages/frontend/package.json | 14 ++ packages/frontend/public/favicon.svg | 9 + packages/frontend/src/assets/astro.svg | 1 + packages/frontend/src/assets/background.svg | 1 + .../frontend/src/components/Welcome.astro | 210 ++++++++++++++++++ packages/frontend/src/layouts/Layout.astro | 22 ++ packages/frontend/src/pages/index.astro | 11 + packages/frontend/tsconfig.json | 5 + 12 files changed, 350 insertions(+) create mode 100644 packages/frontend/.gitignore create mode 100644 packages/frontend/README.md create mode 100644 packages/frontend/astro.config.mjs delete mode 100644 packages/frontend/deno.json create mode 100644 packages/frontend/package.json create mode 100644 packages/frontend/public/favicon.svg create mode 100644 packages/frontend/src/assets/astro.svg create mode 100644 packages/frontend/src/assets/background.svg create mode 100644 packages/frontend/src/components/Welcome.astro create mode 100644 packages/frontend/src/layouts/Layout.astro create mode 100644 packages/frontend/src/pages/index.astro create mode 100644 packages/frontend/tsconfig.json diff --git a/packages/frontend/.gitignore b/packages/frontend/.gitignore new file mode 100644 index 0000000..016b59e --- /dev/null +++ b/packages/frontend/.gitignore @@ -0,0 +1,24 @@ +# build output +dist/ + +# generated types +.astro/ + +# dependencies +node_modules/ + +# logs +npm-debug.log* +yarn-debug.log* +yarn-error.log* +pnpm-debug.log* + +# environment variables +.env +.env.production + +# macOS-specific files +.DS_Store + +# jetbrains setting folder +.idea/ diff --git a/packages/frontend/README.md b/packages/frontend/README.md new file mode 100644 index 0000000..588f744 --- /dev/null +++ b/packages/frontend/README.md @@ -0,0 +1,48 @@ +# Astro Starter Kit: Basics + +```sh +deno create astro@latest -- --template basics +``` + +[![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/github/withastro/astro/tree/latest/examples/basics) +[![Open with CodeSandbox](https://assets.codesandbox.io/github/button-edit-lime.svg)](https://codesandbox.io/p/sandbox/github/withastro/astro/tree/latest/examples/basics) +[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/withastro/astro?devcontainer_path=.devcontainer/basics/devcontainer.json) + +> 🧑‍🚀 **Seasoned astronaut?** Delete this file. Have fun! + +![just-the-basics](https://github.com/withastro/astro/assets/2244813/a0a5533c-a856-4198-8470-2d67b1d7c554) + +## 🚀 Project Structure + +Inside of your Astro project, you'll see the following folders and files: + +```text +/ +├── public/ +│ └── favicon.svg +├── src/ +│ ├── layouts/ +│ │ └── Layout.astro +│ └── pages/ +│ └── index.astro +└── package.json +``` + +To learn more about the folder structure of an Astro project, refer to [our guide on project structure](https://docs.astro.build/en/basics/project-structure/). + +## 🧞 Commands + +All commands are run from the root of the project, from a terminal: + +| Command | Action | +| :------------------------ | :----------------------------------------------- | +| `deno install` | Installs dependencies | +| `deno dev` | Starts local dev server at `localhost:4321` | +| `deno build` | Build your production site to `./dist/` | +| `deno preview` | Preview your build locally, before deploying | +| `deno astro ...` | Run CLI commands like `astro add`, `astro check` | +| `deno astro -- --help` | Get help using the Astro CLI | + +## 👀 Want to learn more? + +Feel free to check [our documentation](https://docs.astro.build) or jump into our [Discord server](https://astro.build/chat). diff --git a/packages/frontend/astro.config.mjs b/packages/frontend/astro.config.mjs new file mode 100644 index 0000000..e762ba5 --- /dev/null +++ b/packages/frontend/astro.config.mjs @@ -0,0 +1,5 @@ +// @ts-check +import { defineConfig } from 'astro/config'; + +// https://astro.build/config +export default defineConfig({}); diff --git a/packages/frontend/deno.json b/packages/frontend/deno.json deleted file mode 100644 index e69de29..0000000 diff --git a/packages/frontend/package.json b/packages/frontend/package.json new file mode 100644 index 0000000..00ecd25 --- /dev/null +++ b/packages/frontend/package.json @@ -0,0 +1,14 @@ +{ + "name": "frontend", + "type": "module", + "version": "0.0.1", + "scripts": { + "dev": "astro dev", + "build": "astro build", + "preview": "astro preview", + "astro": "astro" + }, + "dependencies": { + "astro": "^5.5.5" + } +} \ No newline at end of file diff --git a/packages/frontend/public/favicon.svg b/packages/frontend/public/favicon.svg new file mode 100644 index 0000000..f157bd1 --- /dev/null +++ b/packages/frontend/public/favicon.svg @@ -0,0 +1,9 @@ + + + + diff --git a/packages/frontend/src/assets/astro.svg b/packages/frontend/src/assets/astro.svg new file mode 100644 index 0000000..8cf8fb0 --- /dev/null +++ b/packages/frontend/src/assets/astro.svg @@ -0,0 +1 @@ + diff --git a/packages/frontend/src/assets/background.svg b/packages/frontend/src/assets/background.svg new file mode 100644 index 0000000..4b2be0a --- /dev/null +++ b/packages/frontend/src/assets/background.svg @@ -0,0 +1 @@ + diff --git a/packages/frontend/src/components/Welcome.astro b/packages/frontend/src/components/Welcome.astro new file mode 100644 index 0000000..52e0333 --- /dev/null +++ b/packages/frontend/src/components/Welcome.astro @@ -0,0 +1,210 @@ +--- +import astroLogo from '../assets/astro.svg'; +import background from '../assets/background.svg'; +--- + + + + diff --git a/packages/frontend/src/layouts/Layout.astro b/packages/frontend/src/layouts/Layout.astro new file mode 100644 index 0000000..e455c61 --- /dev/null +++ b/packages/frontend/src/layouts/Layout.astro @@ -0,0 +1,22 @@ + + + + + + + + Astro Basics + + + + + + + diff --git a/packages/frontend/src/pages/index.astro b/packages/frontend/src/pages/index.astro new file mode 100644 index 0000000..c04f360 --- /dev/null +++ b/packages/frontend/src/pages/index.astro @@ -0,0 +1,11 @@ +--- +import Welcome from '../components/Welcome.astro'; +import Layout from '../layouts/Layout.astro'; + +// Welcome to Astro! Wondering what to do next? Check out the Astro documentation at https://docs.astro.build +// Don't want to use any of this? Delete everything in this file, the `assets`, `components`, and `layouts` directories, and start fresh. +--- + + + + diff --git a/packages/frontend/tsconfig.json b/packages/frontend/tsconfig.json new file mode 100644 index 0000000..8bf91d3 --- /dev/null +++ b/packages/frontend/tsconfig.json @@ -0,0 +1,5 @@ +{ + "extends": "astro/tsconfigs/strict", + "include": [".astro/types.d.ts", "**/*"], + "exclude": ["dist"] +} From e0776a452e79f2a17506180583cb75f110774931 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sat, 29 Mar 2025 22:23:31 +0800 Subject: [PATCH 04/24] ref: format --- deno.json | 18 +++++-- packages/crawler/db/snapshotSchedule.ts | 27 +++++++--- packages/crawler/deno.json | 23 +++------ packages/crawler/main.ts | 2 +- packages/crawler/mq/exec/snapshotTick.ts | 5 +- packages/crawler/net/bilibili.d.ts | 5 +- packages/crawler/net/bulkGetVideoStats.ts | 2 +- packages/crawler/src/worker.ts | 6 +-- packages/frontend/README.md | 12 +++-- packages/frontend/astro.config.mjs | 12 ++++- packages/frontend/package.json | 26 +++++----- packages/frontend/tsconfig.json | 6 +-- src/db/raw/fetchAids.ts | 21 ++++++-- src/db/raw/insertAidsToDB.ts | 63 +++++++++++++++++++---- src/db/raw/videoInfo.ts | 14 +++-- 15 files changed, 164 insertions(+), 78 deletions(-) diff --git a/deno.json b/deno.json index 2a573a5..43d84de 100644 --- a/deno.json +++ b/deno.json @@ -1,8 +1,20 @@ { - "lock": false, - "workspace": ["./packages/crawler", "./packages/frontend", "./packages/backend", "./packages/core"], - "nodeModulesDir": "auto", + "lock": false, + "workspace": [ + "./packages/crawler", + "./packages/frontend", + "./packages/backend", + "./packages/core" + ], + "nodeModulesDir": "auto", "tasks": { "crawler": "deno task --filter 'crawler' all" + }, + "fmt": { + "useTabs": true, + "lineWidth": 120, + "indentWidth": 4, + "semiColons": true, + "proseWrap": "always" } } diff --git a/packages/crawler/db/snapshotSchedule.ts b/packages/crawler/db/snapshotSchedule.ts index b98f900..b8aec48 100644 --- a/packages/crawler/db/snapshotSchedule.ts +++ b/packages/crawler/db/snapshotSchedule.ts @@ -28,7 +28,7 @@ export async function refreshSnapshotWindowCounts(client: Client, redisClient: R WHERE started_at >= NOW() AND status = 'pending' AND started_at <= NOW() + INTERVAL '10 days' GROUP BY 1 ORDER BY window_start - ` + `; await redisClient.del(REDIS_KEY); @@ -36,7 +36,7 @@ export async function refreshSnapshotWindowCounts(client: Client, redisClient: R for (const row of result.rows) { const targetOffset = Math.floor((row.window_start.getTime() - startTime) / (5 * MINUTE)); - const offset = (currentWindow + targetOffset); + const offset = currentWindow + targetOffset; if (offset >= 0) { await redisClient.hset(REDIS_KEY, offset.toString(), Number(row.count)); } @@ -186,7 +186,13 @@ export async function getSnapshotScheduleCountWithinRange(client: Client, start: * @param aid The aid of the video. * @param targetTime Scheduled time for snapshot. (Timestamp in milliseconds) */ -export async function scheduleSnapshot(client: Client, aid: number, type: string, targetTime: number, force: boolean = false) { +export async function scheduleSnapshot( + client: Client, + aid: number, + type: string, + targetTime: number, + force: boolean = false, +) { if (await videoHasActiveSchedule(client, aid) && !force) return; let adjustedTime = new Date(targetTime); if (type !== "milestone" && type !== "new") { @@ -199,7 +205,13 @@ export async function scheduleSnapshot(client: Client, aid: number, type: string ); } -export async function bulkScheduleSnapshot(client: Client, aids: number[], type: string, targetTime: number, force: boolean = false) { +export async function bulkScheduleSnapshot( + client: Client, + aids: number[], + type: string, + targetTime: number, + force: boolean = false, +) { for (const aid of aids) { await scheduleSnapshot(client, aid, type, targetTime, force); } @@ -237,12 +249,12 @@ export async function adjustSnapshotTime( if (delayedDate.getTime() < now.getTime()) { const elapsed = performance.now() - t; - timePerIteration = elapsed / (i+1); + timePerIteration = elapsed / (i + 1); logger.log(`${timePerIteration.toFixed(3)}ms * ${iters} iterations`, "perf", "fn:adjustSnapshotTime"); return now; } const elapsed = performance.now() - t; - timePerIteration = elapsed / (i+1); + timePerIteration = elapsed / (i + 1); logger.log(`${timePerIteration.toFixed(3)}ms * ${iters} iterations`, "perf", "fn:adjustSnapshotTime"); return delayedDate; } @@ -253,7 +265,6 @@ export async function adjustSnapshotTime( return expectedStartTime; } - export async function getSnapshotsInNextSecond(client: Client) { const query = ` SELECT * @@ -272,7 +283,7 @@ export async function getSnapshotsInNextSecond(client: Client) { } export async function getBulkSnapshotsInNextSecond(client: Client) { - const query = ` + const query = ` SELECT * FROM snapshot_schedule WHERE started_at <= NOW() + INTERVAL '15 seconds' AND status = 'pending' AND type = 'normal' diff --git a/packages/crawler/deno.json b/packages/crawler/deno.json index 4f95bb9..7c10eae 100644 --- a/packages/crawler/deno.json +++ b/packages/crawler/deno.json @@ -1,5 +1,5 @@ { - "name": "@cvsa/crawler", + "name": "@cvsa/crawler", "tasks": { "crawl-raw-bili": "deno --allow-env --allow-ffi --allow-read --allow-net --allow-write --allow-run src/db/raw/insertAidsToDB.ts", "crawl-bili-aids": "deno --allow-env --allow-ffi --allow-read --allow-net --allow-write --allow-run src/db/raw/fetchAids.ts", @@ -26,11 +26,11 @@ "@huggingface/transformers": "npm:@huggingface/transformers@3.0.0", "bullmq": "npm:bullmq", "mq/": "./mq/", - "db/": "./db/", - "log/": "./log/", - "net/": "./net/", - "ml/": "./ml/", - "utils/": "./utils/", + "db/": "./db/", + "log/": "./log/", + "net/": "./net/", + "ml/": "./ml/", + "utils/": "./utils/", "ioredis": "npm:ioredis", "@bull-board/api": "npm:@bull-board/api", "@bull-board/express": "npm:@bull-board/express", @@ -39,12 +39,5 @@ "onnxruntime": "npm:onnxruntime-node@1.19.2", "chalk": "npm:chalk" }, - "fmt": { - "useTabs": true, - "lineWidth": 120, - "indentWidth": 4, - "semiColons": true, - "proseWrap": "always" - }, - "exports": "./main.ts" -} \ No newline at end of file + "exports": "./main.ts" +} diff --git a/packages/crawler/main.ts b/packages/crawler/main.ts index 3cdc0f4..f31ba12 100644 --- a/packages/crawler/main.ts +++ b/packages/crawler/main.ts @@ -4,4 +4,4 @@ // SO HERE'S A PLACHOLDER EXPORT FOR DENO: export const DENO = "FUCK YOU DENO"; // Oh, maybe export the version is a good idea -export const VERSION = "1.0.13"; \ No newline at end of file +export const VERSION = "1.0.13"; diff --git a/packages/crawler/mq/exec/snapshotTick.ts b/packages/crawler/mq/exec/snapshotTick.ts index 876e05a..defe21a 100644 --- a/packages/crawler/mq/exec/snapshotTick.ts +++ b/packages/crawler/mq/exec/snapshotTick.ts @@ -7,6 +7,7 @@ import { bulkSetSnapshotStatus, findClosestSnapshot, findSnapshotBefore, + getBulkSnapshotsInNextSecond, getLatestSnapshot, getSnapshotsInNextSecond, getVideosWithoutActiveSnapshotSchedule, @@ -15,7 +16,6 @@ import { setSnapshotStatus, snapshotScheduleExists, videoHasProcessingSchedule, - getBulkSnapshotsInNextSecond } from "db/snapshotSchedule.ts"; import { Client } from "https://deno.land/x/postgres@v0.19.3/mod.ts"; import { HOUR, MINUTE, SECOND, WEEK } from "$std/datetime/constants.ts"; @@ -282,8 +282,7 @@ export const takeBulkSnapshotForVideosWorker = async (job: Job) => { } logger.error(e as Error, "mq", "fn:takeBulkSnapshotForVideosWorker"); await bulkSetSnapshotStatus(client, ids, "failed"); - } - finally { + } finally { client.release(); } }; diff --git a/packages/crawler/net/bilibili.d.ts b/packages/crawler/net/bilibili.d.ts index 19e1ba2..4327f51 100644 --- a/packages/crawler/net/bilibili.d.ts +++ b/packages/crawler/net/bilibili.d.ts @@ -13,10 +13,9 @@ export type MediaListInfoResponse = BaseResponse; export type MediaListInfoData = MediaListInfoItem[]; - export interface MediaListInfoItem { attr: number; - bvid: string; + bvid: string; id: number; cnt_info: { coin: number; @@ -26,7 +25,7 @@ export interface MediaListInfoItem { reply: number; share: number; thumb_up: number; - } + }; } interface VideoInfoData { diff --git a/packages/crawler/net/bulkGetVideoStats.ts b/packages/crawler/net/bulkGetVideoStats.ts index 2b0c7f2..3ed518c 100644 --- a/packages/crawler/net/bulkGetVideoStats.ts +++ b/packages/crawler/net/bulkGetVideoStats.ts @@ -13,7 +13,7 @@ import logger from "log/logger.ts"; */ export async function bulkGetVideoStats(aids: number[]): Promise { const baseURL = `https://api.bilibili.com/medialist/gateway/base/resource/infos?resources=`; - let url = baseURL; + let url = baseURL; for (const aid of aids) { url += `${aid}:2,`; } diff --git a/packages/crawler/src/worker.ts b/packages/crawler/src/worker.ts index e240a0b..6507d63 100644 --- a/packages/crawler/src/worker.ts +++ b/packages/crawler/src/worker.ts @@ -6,13 +6,13 @@ import { lockManager } from "mq/lockManager.ts"; import { WorkerError } from "mq/schema.ts"; import { getVideoInfoWorker } from "mq/exec/getLatestVideos.ts"; import { + bulkSnapshotTickWorker, collectMilestoneSnapshotsWorker, regularSnapshotsWorker, - snapshotTickWorker, - takeSnapshotForVideoWorker, scheduleCleanupWorker, + snapshotTickWorker, takeBulkSnapshotForVideosWorker, - bulkSnapshotTickWorker + takeSnapshotForVideoWorker, } from "mq/exec/snapshotTick.ts"; Deno.addSignalListener("SIGINT", async () => { diff --git a/packages/frontend/README.md b/packages/frontend/README.md index 588f744..0eb9af9 100644 --- a/packages/frontend/README.md +++ b/packages/frontend/README.md @@ -28,15 +28,16 @@ Inside of your Astro project, you'll see the following folders and files: └── package.json ``` -To learn more about the folder structure of an Astro project, refer to [our guide on project structure](https://docs.astro.build/en/basics/project-structure/). +To learn more about the folder structure of an Astro project, refer to +[our guide on project structure](https://docs.astro.build/en/basics/project-structure/). ## 🧞 Commands All commands are run from the root of the project, from a terminal: -| Command | Action | -| :------------------------ | :----------------------------------------------- | -| `deno install` | Installs dependencies | +| Command | Action | +| :--------------------- | :----------------------------------------------- | +| `deno install` | Installs dependencies | | `deno dev` | Starts local dev server at `localhost:4321` | | `deno build` | Build your production site to `./dist/` | | `deno preview` | Preview your build locally, before deploying | @@ -45,4 +46,5 @@ All commands are run from the root of the project, from a terminal: ## 👀 Want to learn more? -Feel free to check [our documentation](https://docs.astro.build) or jump into our [Discord server](https://astro.build/chat). +Feel free to check [our documentation](https://docs.astro.build) or jump into our +[Discord server](https://astro.build/chat). diff --git a/packages/frontend/astro.config.mjs b/packages/frontend/astro.config.mjs index e762ba5..e6619b2 100644 --- a/packages/frontend/astro.config.mjs +++ b/packages/frontend/astro.config.mjs @@ -1,5 +1,13 @@ // @ts-check -import { defineConfig } from 'astro/config'; +import { defineConfig } from "astro/config"; // https://astro.build/config -export default defineConfig({}); +export default defineConfig({ + vite: { + server: { + fs: { + allow: [".", "../../"], + }, + }, + }, +}); diff --git a/packages/frontend/package.json b/packages/frontend/package.json index 00ecd25..8761ba2 100644 --- a/packages/frontend/package.json +++ b/packages/frontend/package.json @@ -1,14 +1,14 @@ { - "name": "frontend", - "type": "module", - "version": "0.0.1", - "scripts": { - "dev": "astro dev", - "build": "astro build", - "preview": "astro preview", - "astro": "astro" - }, - "dependencies": { - "astro": "^5.5.5" - } -} \ No newline at end of file + "name": "frontend", + "type": "module", + "version": "0.0.1", + "scripts": { + "dev": "astro dev", + "build": "astro build", + "preview": "astro preview", + "astro": "astro" + }, + "dependencies": { + "astro": "^5.5.5" + } +} diff --git a/packages/frontend/tsconfig.json b/packages/frontend/tsconfig.json index 8bf91d3..a9210e6 100644 --- a/packages/frontend/tsconfig.json +++ b/packages/frontend/tsconfig.json @@ -1,5 +1,5 @@ { - "extends": "astro/tsconfigs/strict", - "include": [".astro/types.d.ts", "**/*"], - "exclude": ["dist"] + "extends": "astro/tsconfigs/strict", + "include": [".astro/types.d.ts", "**/*"], + "exclude": ["dist"] } diff --git a/src/db/raw/fetchAids.ts b/src/db/raw/fetchAids.ts index 10770d5..01b27d1 100644 --- a/src/db/raw/fetchAids.ts +++ b/src/db/raw/fetchAids.ts @@ -14,14 +14,20 @@ const db = new Database(DATABASE_PATH, { int64: true }); // 设置日志 async function setupLogging() { await ensureDir(LOG_DIR); - const logStream = await Deno.open(LOG_FILE, { write: true, create: true, append: true }); + const logStream = await Deno.open(LOG_FILE, { + write: true, + create: true, + append: true, + }); const redirectConsole = // deno-lint-ignore no-explicit-any (originalConsole: (...args: any[]) => void) => // deno-lint-ignore no-explicit-any (...args: any[]) => { - const message = args.map((arg) => (typeof arg === "object" ? JSON.stringify(arg) : arg)).join(" "); + const message = args.map(( + arg, + ) => (typeof arg === "object" ? JSON.stringify(arg) : arg)).join(" "); originalConsole(message); logStream.write(new TextEncoder().encode(message + "\n")); }; @@ -38,14 +44,17 @@ interface Metadata { // 获取最后一次更新的时间 function getLastUpdate(): Date { - const result = db.prepare("SELECT value FROM metadata WHERE key = 'fetchAid-lastUpdate'").get() as Metadata; + const result = db.prepare( + "SELECT value FROM metadata WHERE key = 'fetchAid-lastUpdate'", + ).get() as Metadata; return result ? new Date(result.value as string) : new Date(0); } // 更新最后更新时间 function updateLastUpdate() { const now = new Date().toISOString(); - db.prepare("UPDATE metadata SET value = ? WHERE key = 'fetchAid-lastUpdate'").run(now); + db.prepare("UPDATE metadata SET value = ? WHERE key = 'fetchAid-lastUpdate'") + .run(now); } // 辅助函数：获取数据 @@ -66,7 +75,9 @@ async function fetchData(pn: number, retries = MAX_RETRIES): Promise { // 插入 aid 到数据库 function insertAid(aid: number) { - db.prepare("INSERT OR IGNORE INTO bili_info_crawl (aid, status) VALUES (?, 'pending')").run(aid); + db.prepare( + "INSERT OR IGNORE INTO bili_info_crawl (aid, status) VALUES (?, 'pending')", + ).run(aid); } // 主函数 diff --git a/src/db/raw/insertAidsToDB.ts b/src/db/raw/insertAidsToDB.ts index b067d69..b5855c7 100644 --- a/src/db/raw/insertAidsToDB.ts +++ b/src/db/raw/insertAidsToDB.ts @@ -5,7 +5,16 @@ import { ensureDir } from "https://deno.land/std@0.113.0/fs/mod.ts"; const aidPath = "./data/2025010104_c30_aids.txt"; const db = new Database("./data/main.db", { int64: true }); -const regions = ["shanghai", "hangzhou", "qingdao", "beijing", "zhangjiakou", "chengdu", "shenzhen", "hohhot"]; +const regions = [ + "shanghai", + "hangzhou", + "qingdao", + "beijing", + "zhangjiakou", + "chengdu", + "shenzhen", + "hohhot", +]; const logDir = "./logs/bili-info-crawl"; const logFile = path.join(logDir, `run-${Date.now() / 1000}.log`); const shouldReadTextFile = false; @@ -26,14 +35,20 @@ const requestQueue: number[] = []; async function setupLogging() { await ensureDir(logDir); - const logStream = await Deno.open(logFile, { write: true, create: true, append: true }); + const logStream = await Deno.open(logFile, { + write: true, + create: true, + append: true, + }); const redirectConsole = // deno-lint-ignore no-explicit-any (originalConsole: (...args: any[]) => void) => // deno-lint-ignore no-explicit-any (...args: any[]) => { - const message = args.map((arg) => (typeof arg === "object" ? JSON.stringify(arg) : arg)).join(" "); + const message = args.map(( + arg, + ) => (typeof arg === "object" ? JSON.stringify(arg) : arg)).join(" "); originalConsole(message); logStream.write(new TextEncoder().encode(message + "\n")); }; @@ -78,7 +93,9 @@ async function readFromText() { const newAids = aids.filter((aid) => !existingAidsSet.has(aid)); // 插入这些新条目 - const insertStmt = db.prepare("INSERT OR IGNORE INTO bili_info_crawl (aid, status) VALUES (?, 'pending')"); + const insertStmt = db.prepare( + "INSERT OR IGNORE INTO bili_info_crawl (aid, status) VALUES (?, 'pending')", + ); newAids.forEach((aid) => insertStmt.run(aid)); } @@ -88,7 +105,9 @@ async function insertAidsToDB() { } const aidsInDB = db - .prepare("SELECT aid FROM bili_info_crawl WHERE status = 'pending' OR status = 'failed'") + .prepare( + "SELECT aid FROM bili_info_crawl WHERE status = 'pending' OR status = 'failed'", + ) .all() .map((row) => row.aid) as number[]; @@ -98,13 +117,21 @@ async function insertAidsToDB() { const processAid = async (aid: number) => { try { - const res = await getBiliBiliVideoInfo(aid, regions[processedAids % regions.length]); + const res = await getBiliBiliVideoInfo( + aid, + regions[processedAids % regions.length], + ); if (res === null) { updateAidStatus(aid, "failed"); } else { const rawData = JSON.parse(res); if (rawData.code === 0) { - updateAidStatus(aid, "success", rawData.data.View.bvid, JSON.stringify(rawData.data)); + updateAidStatus( + aid, + "success", + rawData.data.View.bvid, + JSON.stringify(rawData.data), + ); } else { updateAidStatus(aid, "error", undefined, res); } @@ -136,7 +163,12 @@ async function insertAidsToDB() { console.log("Starting to process aids..."); } -function updateAidStatus(aid: number, status: string, bvid?: string, data?: string) { +function updateAidStatus( + aid: number, + status: string, + bvid?: string, + data?: string, +) { const stmt = db.prepare(` UPDATE bili_info_crawl SET status = ?, @@ -145,11 +177,22 @@ function updateAidStatus(aid: number, status: string, bvid?: string, data?: stri timestamp = ? WHERE aid = ? `); - const params = [status, ...(bvid ? [bvid] : []), ...(data ? [data] : []), Date.now() / 1000, aid]; + const params = [ + status, + ...(bvid ? [bvid] : []), + ...(data ? [data] : []), + Date.now() / 1000, + aid, + ]; stmt.run(...params); } -function logProgress(aid: number, processedAids: number, totalAids: number, startTime: number) { +function logProgress( + aid: number, + processedAids: number, + totalAids: number, + startTime: number, +) { const elapsedTime = Date.now() - startTime; const elapsedSeconds = Math.floor(elapsedTime / 1000); const elapsedMinutes = Math.floor(elapsedSeconds / 60); diff --git a/src/db/raw/videoInfo.ts b/src/db/raw/videoInfo.ts index 10272b2..28299d3 100644 --- a/src/db/raw/videoInfo.ts +++ b/src/db/raw/videoInfo.ts @@ -1,4 +1,7 @@ -export async function getBiliBiliVideoInfo(bvidORaid?: string | number, region: string = "hangzhou") { +export async function getBiliBiliVideoInfo( + bvidORaid?: string | number, + region: string = "hangzhou", +) { const bvid = typeof bvidORaid === "string" ? bvidORaid : undefined; const aid = typeof bvidORaid === "number" ? bvidORaid : undefined; @@ -18,7 +21,10 @@ export async function getBiliBiliVideoInfo(bvidORaid?: string | number, region: } } -async function proxyRequestWithRegion(url: string, region: string): Promise { +async function proxyRequestWithRegion( + url: string, + region: string, +): Promise { const td = new TextDecoder(); // aliyun configure set --access-key-id $ALIYUN_AK --access-key-secret $ALIYUN_SK --region cn-shenzhen --profile CVSA-shenzhen --mode AK const p = await new Deno.Command("aliyun", { @@ -40,7 +46,9 @@ async function proxyRequestWithRegion(url: string, region: string): Promise Date: Sun, 30 Mar 2025 02:40:22 +0800 Subject: [PATCH 05/24] update: insert cover into bilibili_metadata --- packages/crawler/mq/task/getVideoDetails.ts | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/packages/crawler/mq/task/getVideoDetails.ts b/packages/crawler/mq/task/getVideoDetails.ts index 9b675e5..71e7566 100644 --- a/packages/crawler/mq/task/getVideoDetails.ts +++ b/packages/crawler/mq/task/getVideoDetails.ts @@ -24,10 +24,11 @@ export async function insertVideoInfo(client: Client, aid: number) { const title = data.View.title; const published_at = formatTimestampToPsql(data.View.pubdate * SECOND + 8 * HOUR); const duration = data.View.duration; + const cover = data.View.pic; await client.queryObject( - `INSERT INTO bilibili_metadata (aid, bvid, description, uid, tags, title, published_at, duration) + `INSERT INTO bilibili_metadata (aid, bvid, description, uid, tags, title, published_at, duration, cover_url) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)`, - [aid, bvid, desc, uid, tags, title, published_at, duration], + [aid, bvid, desc, uid, tags, title, published_at, duration, cover], ); const userExists = await userExistsInBiliUsers(client, aid); if (!userExists) { From f7806c6a39a2ec9e70f295652dd36704c9a70a73 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 03:23:38 +0800 Subject: [PATCH 06/24] add: tailwind dependency --- packages/frontend/package.json | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/packages/frontend/package.json b/packages/frontend/package.json index 8761ba2..ed2b48a 100644 --- a/packages/frontend/package.json +++ b/packages/frontend/package.json @@ -9,6 +9,10 @@ "astro": "astro" }, "dependencies": { - "astro": "^5.5.5" + "@astrojs/tailwind": "^6.0.2", + "astro": "^5.5.5", + "autoprefixer": "^10.4.21", + "postcss": "^8.5.3", + "tailwindcss": "^4.0.17" } } From 19d727628076acd868c0516b95c43fa173091dfe Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 03:24:11 +0800 Subject: [PATCH 07/24] fix: missing param in SQL for inserting into bilibili_metadata --- packages/crawler/mq/task/getVideoDetails.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/crawler/mq/task/getVideoDetails.ts b/packages/crawler/mq/task/getVideoDetails.ts index 71e7566..ee015fd 100644 --- a/packages/crawler/mq/task/getVideoDetails.ts +++ b/packages/crawler/mq/task/getVideoDetails.ts @@ -27,7 +27,7 @@ export async function insertVideoInfo(client: Client, aid: number) { const cover = data.View.pic; await client.queryObject( `INSERT INTO bilibili_metadata (aid, bvid, description, uid, tags, title, published_at, duration, cover_url) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8)`, + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`, [aid, bvid, desc, uid, tags, title, published_at, duration, cover], ); const userExists = await userExistsInBiliUsers(client, aid); From f585b49ee4dcce193ae884bba58d6cb54e3b7eb3 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 03:57:21 +0800 Subject: [PATCH 08/24] add: frontend with Astro --- packages/frontend/astro.config.mjs | 5 +++ packages/frontend/package.json | 3 +- packages/frontend/src/layouts/Layout.astro | 35 +++++++--------- packages/frontend/src/pages/index.astro | 7 +--- .../frontend/src/pages/song/[aid]/info.astro | 40 +++++++++++++++++++ packages/frontend/src/styles/global.css | 3 ++ packages/frontend/tailwind.config.js | 8 ++++ packages/frontend/tsconfig.json | 16 ++++++-- 8 files changed, 87 insertions(+), 30 deletions(-) create mode 100644 packages/frontend/src/pages/song/[aid]/info.astro create mode 100644 packages/frontend/src/styles/global.css create mode 100644 packages/frontend/tailwind.config.js diff --git a/packages/frontend/astro.config.mjs b/packages/frontend/astro.config.mjs index e6619b2..4baabd1 100644 --- a/packages/frontend/astro.config.mjs +++ b/packages/frontend/astro.config.mjs @@ -1,13 +1,18 @@ // @ts-check import { defineConfig } from "astro/config"; +import tailwind from "@astrojs/tailwind"; // https://astro.build/config +import tsconfigPaths from "vite-tsconfig-paths"; + export default defineConfig({ + integrations: [tailwind()], vite: { server: { fs: { allow: [".", "../../"], }, }, + plugins: [tsconfigPaths(),] }, }); diff --git a/packages/frontend/package.json b/packages/frontend/package.json index ed2b48a..78da34b 100644 --- a/packages/frontend/package.json +++ b/packages/frontend/package.json @@ -13,6 +13,7 @@ "astro": "^5.5.5", "autoprefixer": "^10.4.21", "postcss": "^8.5.3", - "tailwindcss": "^4.0.17" + "tailwindcss": "^3.0.24", + "vite-tsconfig-paths": "^5.1.4" } } diff --git a/packages/frontend/src/layouts/Layout.astro b/packages/frontend/src/layouts/Layout.astro index e455c61..9c3c39f 100644 --- a/packages/frontend/src/layouts/Layout.astro +++ b/packages/frontend/src/layouts/Layout.astro @@ -1,22 +1,15 @@ - - - - - - - - Astro Basics - - - - - +--- +import "../styles/global.css"; +--- - + + + + + + CVSA 前端 + + + + + diff --git a/packages/frontend/src/pages/index.astro b/packages/frontend/src/pages/index.astro index c04f360..2dffcbe 100644 --- a/packages/frontend/src/pages/index.astro +++ b/packages/frontend/src/pages/index.astro @@ -1,9 +1,6 @@ --- -import Welcome from '../components/Welcome.astro'; -import Layout from '../layouts/Layout.astro'; - -// Welcome to Astro! Wondering what to do next? Check out the Astro documentation at https://docs.astro.build -// Don't want to use any of this? Delete everything in this file, the `assets`, `components`, and `layouts` directories, and start fresh. +import Welcome from '@components/Welcome.astro'; +import Layout from '@layouts/Layout.astro'; --- diff --git a/packages/frontend/src/pages/song/[aid]/info.astro b/packages/frontend/src/pages/song/[aid]/info.astro new file mode 100644 index 0000000..063a644 --- /dev/null +++ b/packages/frontend/src/pages/song/[aid]/info.astro @@ -0,0 +1,40 @@ +--- +import Layout from "@layouts/Layout.astro"; + +// 路由参数 +const { aid } = Astro.params; +const videoAid = Number(aid); + +// 数据库查询函数 +async function getVideoMetadata(aid: number) { + // TODO: 实现bilibili_metadata表查询 + return {}; +} + +async function getVideoSnapshots(aid: number) { + // TODO: 实现video_snapshot表查询，按created_at排序，限制100条 + return []; +} + +// 获取数据 +const videoInfo = await getVideoMetadata(videoAid); +const snapshots = await getVideoSnapshots(videoAid); +--- + + +
+

视频信息: {videoAid}

+ + +
+

基本信息

+ 
{JSON.stringify(videoInfo, null, 2)}
+
+ + +
+

历史数据 (最新100条)

+ 
{JSON.stringify(snapshots, null, 2)}
+
+
+ diff --git a/packages/frontend/src/styles/global.css b/packages/frontend/src/styles/global.css new file mode 100644 index 0000000..b5c61c9 --- /dev/null +++ b/packages/frontend/src/styles/global.css @@ -0,0 +1,3 @@ +@tailwind base; +@tailwind components; +@tailwind utilities; diff --git a/packages/frontend/tailwind.config.js b/packages/frontend/tailwind.config.js new file mode 100644 index 0000000..b34cf45 --- /dev/null +++ b/packages/frontend/tailwind.config.js @@ -0,0 +1,8 @@ +/** @type {import('tailwindcss').Config} */ +module.exports = { + content: ["./src/**/*.{astro,html,js,jsx,md,mdx,svelte,ts,tsx,vue}"], + theme: { + extend: {}, + }, + plugins: [], +}; diff --git a/packages/frontend/tsconfig.json b/packages/frontend/tsconfig.json index a9210e6..9c3f979 100644 --- a/packages/frontend/tsconfig.json +++ b/packages/frontend/tsconfig.json @@ -1,5 +1,15 @@ { - "extends": "astro/tsconfigs/strict", - "include": [".astro/types.d.ts", "**/*"], - "exclude": ["dist"] + "extends": "astro/tsconfigs/strict", + "include": [".astro/types.d.ts", "**/*"], + "exclude": ["dist"], + "compilerOptions": { + "baseUrl": ".", + "paths": { + "@components/*": ["src/components/*"], + "@layouts/*": ["src/layouts/*"], + "@utils/*": ["src/utils/*"], + "@assets/*": ["src/assets/*"], + "@styles": ["src/styles/*"] + } + } } From 4fe266ce82e239e5e2afe74cdec75077e60e66a1 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 06:45:27 +0800 Subject: [PATCH 09/24] add: few pages --- .tokeignore | 3 +- deno.json | 35 ++- packages/frontend/astro.config.mjs | 10 +- .../src/assets/TitleBar Mobile Dark.svg | 7 + .../src/assets/TitleBar Mobile Light.svg | 7 + .../frontend/src/assets/TitleBar-Dark.svg | 15 ++ .../frontend/src/assets/TitleBar-Light.svg | 15 ++ packages/frontend/src/assets/astro.svg | 1 - packages/frontend/src/assets/background.svg | 1 - .../frontend/src/assets/header-logo-dark.svg | 15 ++ .../frontend/src/assets/header-logo-light.svg | 15 ++ packages/frontend/src/assets/标题-浅色.svg | 15 ++ packages/frontend/src/assets/标题-深色.svg | 15 ++ .../frontend/src/components/CloseIcon.svelte | 12 + .../src/components/DarkModeImage.svelte | 41 ++++ .../frontend/src/components/MenuIcon.svelte | 20 ++ .../frontend/src/components/SearchBox.svelte | 37 +++ .../frontend/src/components/SearchIcon.svelte | 12 + .../frontend/src/components/TitleBar.astro | 30 +++ .../src/components/TitleBarMobile.svelte | 42 ++++ .../frontend/src/components/Welcome.astro | 211 +----------------- packages/frontend/src/layouts/Layout.astro | 2 +- packages/frontend/src/pages/about.astro | 15 ++ .../frontend/src/pages/song/[aid]/info.astro | 10 +- 24 files changed, 351 insertions(+), 235 deletions(-) create mode 100644 packages/frontend/src/assets/TitleBar Mobile Dark.svg create mode 100644 packages/frontend/src/assets/TitleBar Mobile Light.svg create mode 100644 packages/frontend/src/assets/TitleBar-Dark.svg create mode 100644 packages/frontend/src/assets/TitleBar-Light.svg delete mode 100644 packages/frontend/src/assets/astro.svg delete mode 100644 packages/frontend/src/assets/background.svg create mode 100644 packages/frontend/src/assets/header-logo-dark.svg create mode 100644 packages/frontend/src/assets/header-logo-light.svg create mode 100644 packages/frontend/src/assets/标题-浅色.svg create mode 100644 packages/frontend/src/assets/标题-深色.svg create mode 100644 packages/frontend/src/components/CloseIcon.svelte create mode 100644 packages/frontend/src/components/DarkModeImage.svelte create mode 100644 packages/frontend/src/components/MenuIcon.svelte create mode 100644 packages/frontend/src/components/SearchBox.svelte create mode 100644 packages/frontend/src/components/SearchIcon.svelte create mode 100644 packages/frontend/src/components/TitleBar.astro create mode 100644 packages/frontend/src/components/TitleBarMobile.svelte create mode 100644 packages/frontend/src/pages/about.astro diff --git a/.tokeignore b/.tokeignore index 0c4d337..aafc28c 100644 --- a/.tokeignore +++ b/.tokeignore @@ -2,4 +2,5 @@ data *.json *.svg *.txt -*.md \ No newline at end of file +*.md +*config* \ No newline at end of file diff --git a/deno.json b/deno.json index 43d84de..947c793 100644 --- a/deno.json +++ b/deno.json @@ -1,20 +1,19 @@ { - "lock": false, - "workspace": [ - "./packages/crawler", - "./packages/frontend", - "./packages/backend", - "./packages/core" - ], - "nodeModulesDir": "auto", - "tasks": { - "crawler": "deno task --filter 'crawler' all" - }, - "fmt": { - "useTabs": true, - "lineWidth": 120, - "indentWidth": 4, - "semiColons": true, - "proseWrap": "always" - } + "lock": false, + "workspace": ["./packages/crawler", "./packages/frontend", "./packages/backend", "./packages/core"], + "nodeModulesDir": "auto", + "tasks": { + "crawler": "deno task --filter 'crawler' all" + }, + "fmt": { + "useTabs": true, + "lineWidth": 120, + "indentWidth": 4, + "semiColons": true, + "proseWrap": "always" + }, + "imports": { + "@astrojs/node": "npm:@astrojs/node@^9.1.3", + "@astrojs/svelte": "npm:@astrojs/svelte@^7.0.8" + } } diff --git a/packages/frontend/astro.config.mjs b/packages/frontend/astro.config.mjs index 4baabd1..79ca8e0 100644 --- a/packages/frontend/astro.config.mjs +++ b/packages/frontend/astro.config.mjs @@ -4,15 +4,21 @@ import tailwind from "@astrojs/tailwind"; // https://astro.build/config import tsconfigPaths from "vite-tsconfig-paths"; +import node from "@astrojs/node"; +import svelte from "@astrojs/svelte"; export default defineConfig({ - integrations: [tailwind()], + output: "server", + adapter: node({ + mode: "standalone", + }), + integrations: [tailwind(), svelte()], vite: { server: { fs: { allow: [".", "../../"], }, }, - plugins: [tsconfigPaths(),] + plugins: [tsconfigPaths()], }, }); diff --git a/packages/frontend/src/assets/TitleBar Mobile Dark.svg b/packages/frontend/src/assets/TitleBar Mobile Dark.svg new file mode 100644 index 0000000..3b1a7e5 --- /dev/null +++ b/packages/frontend/src/assets/TitleBar Mobile Dark.svg @@ -0,0 +1,7 @@ + + + Created with Pixso. + + + + diff --git a/packages/frontend/src/assets/TitleBar Mobile Light.svg b/packages/frontend/src/assets/TitleBar Mobile Light.svg new file mode 100644 index 0000000..365e6a4 --- /dev/null +++ b/packages/frontend/src/assets/TitleBar Mobile Light.svg @@ -0,0 +1,7 @@ + + + Created with Pixso. + + + + diff --git a/packages/frontend/src/assets/TitleBar-Dark.svg b/packages/frontend/src/assets/TitleBar-Dark.svg new file mode 100644 index 0000000..02f0398 --- /dev/null +++ b/packages/frontend/src/assets/TitleBar-Dark.svg @@ -0,0 +1,15 @@ + + + Created with Pixso. + + + + + + + + + + + + diff --git a/packages/frontend/src/assets/TitleBar-Light.svg b/packages/frontend/src/assets/TitleBar-Light.svg new file mode 100644 index 0000000..92482e0 --- /dev/null +++ b/packages/frontend/src/assets/TitleBar-Light.svg @@ -0,0 +1,15 @@ + + + Created with Pixso. + + + + + + + + + + + + diff --git a/packages/frontend/src/assets/astro.svg b/packages/frontend/src/assets/astro.svg deleted file mode 100644 index 8cf8fb0..0000000 --- a/packages/frontend/src/assets/astro.svg +++ /dev/null @@ -1 +0,0 @@ - diff --git a/packages/frontend/src/assets/background.svg b/packages/frontend/src/assets/background.svg deleted file mode 100644 index 4b2be0a..0000000 --- a/packages/frontend/src/assets/background.svg +++ /dev/null @@ -1 +0,0 @@ - diff --git a/packages/frontend/src/assets/header-logo-dark.svg b/packages/frontend/src/assets/header-logo-dark.svg new file mode 100644 index 0000000..002540f --- /dev/null +++ b/packages/frontend/src/assets/header-logo-dark.svg @@ -0,0 +1,15 @@ + + + Created with Pixso. + + + + + + + + + + + + diff --git a/packages/frontend/src/assets/header-logo-light.svg b/packages/frontend/src/assets/header-logo-light.svg new file mode 100644 index 0000000..acc0c15 --- /dev/null +++ b/packages/frontend/src/assets/header-logo-light.svg @@ -0,0 +1,15 @@ + + + Created with Pixso. + + + + + + + + + + + + diff --git a/packages/frontend/src/assets/标题-浅色.svg b/packages/frontend/src/assets/标题-浅色.svg new file mode 100644 index 0000000..77e3e1e --- /dev/null +++ b/packages/frontend/src/assets/标题-浅色.svg @@ -0,0 +1,15 @@ + + + Created with Pixso. + + + + + + + + + + + + diff --git a/packages/frontend/src/assets/标题-深色.svg b/packages/frontend/src/assets/标题-深色.svg new file mode 100644 index 0000000..a24984d --- /dev/null +++ b/packages/frontend/src/assets/标题-深色.svg @@ -0,0 +1,15 @@ + + + Created with Pixso. + + + + + + + + + + + + diff --git a/packages/frontend/src/components/CloseIcon.svelte b/packages/frontend/src/components/CloseIcon.svelte new file mode 100644 index 0000000..cb98947 --- /dev/null +++ b/packages/frontend/src/components/CloseIcon.svelte @@ -0,0 +1,12 @@ + + +
+ + + +
diff --git a/packages/frontend/src/components/DarkModeImage.svelte b/packages/frontend/src/components/DarkModeImage.svelte new file mode 100644 index 0000000..aa4caeb --- /dev/null +++ b/packages/frontend/src/components/DarkModeImage.svelte @@ -0,0 +1,41 @@ + + + \ No newline at end of file diff --git a/packages/frontend/src/components/MenuIcon.svelte b/packages/frontend/src/components/MenuIcon.svelte new file mode 100644 index 0000000..1c74e68 --- /dev/null +++ b/packages/frontend/src/components/MenuIcon.svelte @@ -0,0 +1,20 @@ + + +
+ + + Created with Pixso. + + + + + + + + + + + +
diff --git a/packages/frontend/src/components/SearchBox.svelte b/packages/frontend/src/components/SearchBox.svelte new file mode 100644 index 0000000..f851224 --- /dev/null +++ b/packages/frontend/src/components/SearchBox.svelte @@ -0,0 +1,37 @@ + + + +
+ +
+ + diff --git a/packages/frontend/src/components/SearchIcon.svelte b/packages/frontend/src/components/SearchIcon.svelte new file mode 100644 index 0000000..d23c8be --- /dev/null +++ b/packages/frontend/src/components/SearchIcon.svelte @@ -0,0 +1,12 @@ + + +
+ + + +
\ No newline at end of file diff --git a/packages/frontend/src/components/TitleBar.astro b/packages/frontend/src/components/TitleBar.astro new file mode 100644 index 0000000..af6d723 --- /dev/null +++ b/packages/frontend/src/components/TitleBar.astro @@ -0,0 +1,30 @@ +--- +import astroLogoLight from "@assets/标题-浅色.svg"; +import astroLogoDark from "@assets/标题-深色.svg"; +import DarkModeImage from "@components/DarkModeImage.svelte"; +import SearchBox from "@components/SearchBox.svelte"; +import TitleBarMobile from "@components/TitleBarMobile.svelte"; +--- + + + + diff --git a/packages/frontend/src/components/TitleBarMobile.svelte b/packages/frontend/src/components/TitleBarMobile.svelte new file mode 100644 index 0000000..bcdd28b --- /dev/null +++ b/packages/frontend/src/components/TitleBarMobile.svelte @@ -0,0 +1,42 @@ + + +
+ {#if !showSearchBox} +
+ +
+
+ + + +
+ {/if} + {#if showSearchBox} + + {/if} + +
diff --git a/packages/frontend/src/components/Welcome.astro b/packages/frontend/src/components/Welcome.astro index 52e0333..c55f130 100644 --- a/packages/frontend/src/components/Welcome.astro +++ b/packages/frontend/src/components/Welcome.astro @@ -1,210 +1,9 @@ --- -import astroLogo from '../assets/astro.svg'; -import background from '../assets/background.svg'; +import TitleBar from "@components/TitleBar.astro"; --- - - - +
+

正在施工中……

+
diff --git a/packages/frontend/src/layouts/Layout.astro b/packages/frontend/src/layouts/Layout.astro index 9c3c39f..e33cf4b 100644 --- a/packages/frontend/src/layouts/Layout.astro +++ b/packages/frontend/src/layouts/Layout.astro @@ -9,7 +9,7 @@ import "../styles/global.css"; CVSA 前端 - + diff --git a/packages/frontend/src/pages/about.astro b/packages/frontend/src/pages/about.astro new file mode 100644 index 0000000..9fe94a6 --- /dev/null +++ b/packages/frontend/src/pages/about.astro @@ -0,0 +1,15 @@ +--- +import TitleBar from "@components/TitleBar.astro"; +import Layout from '@layouts/Layout.astro'; +--- + + + + + + +
+

正在施工中……

+
+ + diff --git a/packages/frontend/src/pages/song/[aid]/info.astro b/packages/frontend/src/pages/song/[aid]/info.astro index 063a644..dd5778f 100644 --- a/packages/frontend/src/pages/song/[aid]/info.astro +++ b/packages/frontend/src/pages/song/[aid]/info.astro @@ -22,19 +22,19 @@ const snapshots = await getVideoSnapshots(videoAid); --- -
+

视频信息: {videoAid}

-
+

基本信息

- 
{JSON.stringify(videoInfo, null, 2)}
+ 
{JSON.stringify(videoInfo, null, 2)}
-
+

历史数据 (最新100条)

- 
{JSON.stringify(snapshots, null, 2)}
+ 
{JSON.stringify(snapshots, null, 2)}
From 8b17f8177c336c66cf9c4cdabc058aca87d84177 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 07:31:24 +0800 Subject: [PATCH 10/24] update: song page --- deno.json | 3 +- packages/{crawler => core}/db/pgConfig.ts | 9 ++ packages/crawler/db/init.ts | 2 +- packages/frontend/package.json | 38 ++++---- .../frontend/src/components/SearchBox.svelte | 15 +++- .../src/components/TitleBarMobile.svelte | 11 ++- packages/frontend/src/pages/404.astro | 10 +++ .../frontend/src/pages/song/[aid]/info.astro | 40 --------- .../frontend/src/pages/song/[id]/info.astro | 88 +++++++++++++++++++ packages/frontend/tsconfig.json | 3 +- 10 files changed, 153 insertions(+), 66 deletions(-) rename packages/{crawler => core}/db/pgConfig.ts (75%) create mode 100644 packages/frontend/src/pages/404.astro delete mode 100644 packages/frontend/src/pages/song/[aid]/info.astro create mode 100644 packages/frontend/src/pages/song/[id]/info.astro diff --git a/deno.json b/deno.json index 947c793..5e9eac4 100644 --- a/deno.json +++ b/deno.json @@ -14,6 +14,7 @@ }, "imports": { "@astrojs/node": "npm:@astrojs/node@^9.1.3", - "@astrojs/svelte": "npm:@astrojs/svelte@^7.0.8" + "@astrojs/svelte": "npm:@astrojs/svelte@^7.0.8", + "@core/db/": "./packages/core/db/" } } diff --git a/packages/crawler/db/pgConfig.ts b/packages/core/db/pgConfig.ts similarity index 75% rename from packages/crawler/db/pgConfig.ts rename to packages/core/db/pgConfig.ts index 5410760..adea921 100644 --- a/packages/crawler/db/pgConfig.ts +++ b/packages/core/db/pgConfig.ts @@ -8,6 +8,7 @@ if (unsetVars.length > 0) { const databaseHost = Deno.env.get("DB_HOST")!; const databaseName = Deno.env.get("DB_NAME")!; +const databaseNameCred = Deno.env.get("DB_NAME_CRED")!; const databaseUser = Deno.env.get("DB_USER")!; const databasePassword = Deno.env.get("DB_PASSWORD")!; const databasePort = Deno.env.get("DB_PORT")!; @@ -19,3 +20,11 @@ export const postgresConfig = { user: databaseUser, password: databasePassword, }; + +export const postgresConfigCred = { + hostname: databaseHost, + port: parseInt(databasePort), + database: databaseNameCred, + user: databaseUser, + password: databasePassword, +} \ No newline at end of file diff --git a/packages/crawler/db/init.ts b/packages/crawler/db/init.ts index a1835b0..33bc3c6 100644 --- a/packages/crawler/db/init.ts +++ b/packages/crawler/db/init.ts @@ -1,5 +1,5 @@ import { Pool } from "https://deno.land/x/postgres@v0.19.3/mod.ts"; -import { postgresConfig } from "db/pgConfig.ts"; +import { postgresConfig } from "@core/db/pgConfig.ts"; const pool = new Pool(postgresConfig, 12); diff --git a/packages/frontend/package.json b/packages/frontend/package.json index 78da34b..377d360 100644 --- a/packages/frontend/package.json +++ b/packages/frontend/package.json @@ -1,19 +1,23 @@ { - "name": "frontend", - "type": "module", - "version": "0.0.1", - "scripts": { - "dev": "astro dev", - "build": "astro build", - "preview": "astro preview", - "astro": "astro" - }, - "dependencies": { - "@astrojs/tailwind": "^6.0.2", - "astro": "^5.5.5", - "autoprefixer": "^10.4.21", - "postcss": "^8.5.3", - "tailwindcss": "^3.0.24", - "vite-tsconfig-paths": "^5.1.4" - } + "name": "frontend", + "type": "module", + "version": "0.0.1", + "scripts": { + "dev": "astro dev", + "build": "astro build", + "preview": "astro preview", + "astro": "astro" + }, + "dependencies": { + "@astrojs/tailwind": "^6.0.2", + "astro": "^5.5.5", + "autoprefixer": "^10.4.21", + "pg": "^8.11.11", + "postcss": "^8.5.3", + "tailwindcss": "^3.0.24", + "vite-tsconfig-paths": "^5.1.4" + }, + "devDependencies": { + "@types/pg": "^8.11.11" + } } diff --git a/packages/frontend/src/components/SearchBox.svelte b/packages/frontend/src/components/SearchBox.svelte index f851224..e23c47f 100644 --- a/packages/frontend/src/components/SearchBox.svelte +++ b/packages/frontend/src/components/SearchBox.svelte @@ -1,5 +1,14 @@
{#if !showSearchBox} -
+
+
{/if} {#if showSearchBox} - + {/if}
diff --git a/packages/frontend/src/pages/song/[id]/info.astro b/packages/frontend/src/pages/song/[id]/info.astro index 566a1fa..6651ca3 100644 --- a/packages/frontend/src/pages/song/[id]/info.astro +++ b/packages/frontend/src/pages/song/[id]/info.astro @@ -97,48 +97,48 @@ interface Snapshot { - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + +
ID{videoInfo?.id}ID{videoInfo?.id}
AID{videoInfo?.aid}AID{videoInfo?.aid}
BVID{videoInfo?.bvid}BVID{videoInfo?.bvid}
标题{videoInfo?.title}标题{videoInfo?.title}
描述{videoInfo?.description}描述{videoInfo?.description}
UID{videoInfo?.uid}UID{videoInfo?.uid}
标签{videoInfo?.tags}标签{videoInfo?.tags}
发布时间{videoInfo?.published_at ? format(new Date(videoInfo.published_at), 'yyyy-MM-dd HH:mm:ss', { locale: zhCN }) : '-'}发布时间{videoInfo?.published_at ? format(new Date(videoInfo.published_at), 'yyyy-MM-dd HH:mm:ss', { locale: zhCN }) : '-'}
时长 (秒){videoInfo?.duration}时长 (秒){videoInfo?.duration}
创建时间{videoInfo?.created_at ? format(new Date(videoInfo.created_at), 'yyyy-MM-dd HH:mm:ss', { locale: zhCN }) : '-'}创建时间{videoInfo?.created_at ? format(new Date(videoInfo.created_at), 'yyyy-MM-dd HH:mm:ss', { locale: zhCN }) : '-'}
封面{videoInfo?.cover_url ? videoInfo.cover_url : '-'}封面{videoInfo?.cover_url ? videoInfo.cover_url : '-'}
@@ -152,27 +152,27 @@ interface Snapshot { - - - - - - - - + + + + + + + + {snapshots.map((snapshot: Snapshot) => ( - - - - - - - - + + + + + + + + ))} From 7aa988f0fea56a0becbd5131fc35276dfc93a1b1 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 19:07:02 +0800 Subject: [PATCH 17/24] fix: remove deno from building & previewing --- packages/frontend/deno.json | 7 +++++++ packages/frontend/main.ts | 7 +++++++ packages/frontend/src/pages/song/[id]/info.astro | 15 ++++++++++++++- 3 files changed, 28 insertions(+), 1 deletion(-) create mode 100644 packages/frontend/deno.json create mode 100644 packages/frontend/main.ts diff --git a/packages/frontend/deno.json b/packages/frontend/deno.json new file mode 100644 index 0000000..acc30c9 --- /dev/null +++ b/packages/frontend/deno.json @@ -0,0 +1,7 @@ +{ + "name": "@cvsa/frontend", + "tasks": { + "preview": "bun run astro preview" + }, + "exports": "./main.ts" +} diff --git a/packages/frontend/main.ts b/packages/frontend/main.ts new file mode 100644 index 0000000..f31ba12 --- /dev/null +++ b/packages/frontend/main.ts @@ -0,0 +1,7 @@ +// DENO ASK ME TO EXPORT SOMETHING WHEN 'name' IS SPECIFIED +// AND IF I DON'T SPECIFY 'name', THE --filter FLAG IN `deno task` WON'T WORK. +// I DONT'T KNOW WHY +// SO HERE'S A PLACHOLDER EXPORT FOR DENO: +export const DENO = "FUCK YOU DENO"; +// Oh, maybe export the version is a good idea +export const VERSION = "1.0.13"; diff --git a/packages/frontend/src/pages/song/[id]/info.astro b/packages/frontend/src/pages/song/[id]/info.astro index 6651ca3..f6fd40b 100644 --- a/packages/frontend/src/pages/song/[id]/info.astro +++ b/packages/frontend/src/pages/song/[id]/info.astro @@ -2,10 +2,23 @@ import Layout from "@layouts/Layout.astro"; import TitleBar from "@components/TitleBar.astro"; import pg from "pg"; -import { postgresConfig } from "@core/db/pgConfig.ts"; import { format } from 'date-fns'; import { zhCN } from 'date-fns/locale'; +const databaseHost = process.env.DB_HOST +const databaseName = process.env.DB_NAME +const databaseUser = process.env.DB_USER +const databasePassword = process.env.DB_PASSWORD +const databasePort = process.env.DB_PORT + +const postgresConfig = { + hostname: databaseHost, + port: parseInt(databasePort!), + database: databaseName, + user: databaseUser, + password: databasePassword, +}; + // 路由参数 const { id } = Astro.params; const { Client } = pg; From 39acac09e7b24c261608f4b688fbc67eade989ba Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 19:10:23 +0800 Subject: [PATCH 18/24] version: frontend/1.2.6 --- packages/frontend/README.md | 50 ------------------------------------- packages/frontend/deno.json | 3 ++- packages/frontend/main.ts | 8 +----- 3 files changed, 3 insertions(+), 58 deletions(-) delete mode 100644 packages/frontend/README.md diff --git a/packages/frontend/README.md b/packages/frontend/README.md deleted file mode 100644 index 0eb9af9..0000000 --- a/packages/frontend/README.md +++ /dev/null @@ -1,50 +0,0 @@ -# Astro Starter Kit: Basics - -```sh -deno create astro@latest -- --template basics -``` - -[![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/github/withastro/astro/tree/latest/examples/basics) -[![Open with CodeSandbox](https://assets.codesandbox.io/github/button-edit-lime.svg)](https://codesandbox.io/p/sandbox/github/withastro/astro/tree/latest/examples/basics) -[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/withastro/astro?devcontainer_path=.devcontainer/basics/devcontainer.json) - -> 🧑‍🚀 **Seasoned astronaut?** Delete this file. Have fun! - -![just-the-basics](https://github.com/withastro/astro/assets/2244813/a0a5533c-a856-4198-8470-2d67b1d7c554) - -## 🚀 Project Structure - -Inside of your Astro project, you'll see the following folders and files: - -```text -/ -├── public/ -│ └── favicon.svg -├── src/ -│ ├── layouts/ -│ │ └── Layout.astro -│ └── pages/ -│ └── index.astro -└── package.json -``` - -To learn more about the folder structure of an Astro project, refer to -[our guide on project structure](https://docs.astro.build/en/basics/project-structure/). - -## 🧞 Commands - -All commands are run from the root of the project, from a terminal: - -| Command | Action | -| :--------------------- | :----------------------------------------------- | -| `deno install` | Installs dependencies | -| `deno dev` | Starts local dev server at `localhost:4321` | -| `deno build` | Build your production site to `./dist/` | -| `deno preview` | Preview your build locally, before deploying | -| `deno astro ...` | Run CLI commands like `astro add`, `astro check` | -| `deno astro -- --help` | Get help using the Astro CLI | - -## 👀 Want to learn more? - -Feel free to check [our documentation](https://docs.astro.build) or jump into our -[Discord server](https://astro.build/chat). diff --git a/packages/frontend/deno.json b/packages/frontend/deno.json index acc30c9..675c374 100644 --- a/packages/frontend/deno.json +++ b/packages/frontend/deno.json @@ -1,7 +1,8 @@ { "name": "@cvsa/frontend", "tasks": { - "preview": "bun run astro preview" + "preview": "bun run astro preview", + "build": "bun run astro build" }, "exports": "./main.ts" } diff --git a/packages/frontend/main.ts b/packages/frontend/main.ts index f31ba12..f6ef10d 100644 --- a/packages/frontend/main.ts +++ b/packages/frontend/main.ts @@ -1,7 +1 @@ -// DENO ASK ME TO EXPORT SOMETHING WHEN 'name' IS SPECIFIED -// AND IF I DON'T SPECIFY 'name', THE --filter FLAG IN `deno task` WON'T WORK. -// I DONT'T KNOW WHY -// SO HERE'S A PLACHOLDER EXPORT FOR DENO: -export const DENO = "FUCK YOU DENO"; -// Oh, maybe export the version is a good idea -export const VERSION = "1.0.13"; +export const VERSION = "1.2.6"; \ No newline at end of file From da8b2d3b4d4046dcbb9105f7cb7e671f0a3e6fa8 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 20:48:40 +0800 Subject: [PATCH 19/24] fix: timeout interva ltoo short in cleanup --- packages/crawler/mq/exec/snapshotTick.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/crawler/mq/exec/snapshotTick.ts b/packages/crawler/mq/exec/snapshotTick.ts index defe21a..abec4f6 100644 --- a/packages/crawler/mq/exec/snapshotTick.ts +++ b/packages/crawler/mq/exec/snapshotTick.ts @@ -371,8 +371,8 @@ export const scheduleCleanupWorker = async (_job: Job) => { const query = ` SELECT id, aid, type FROM snapshot_schedule - WHERE status IN ('pending', 'processing') - AND started_at < NOW() - INTERVAL '5 minutes' + WHERE status IN ('pending', 'processing') + AND started_at < NOW() - INTERVAL '30 minutes' `; const { rows } = await client.queryObject<{ id: bigint; aid: bigint; type: string }>(query); if (rows.length === 0) return; From b2edaf8fc42a6b808008746cf427785602a0fa05 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 21:16:05 +0800 Subject: [PATCH 20/24] improve: prevent Infinity for getAdjustedShortTermETA --- packages/crawler/mq/exec/snapshotTick.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/crawler/mq/exec/snapshotTick.ts b/packages/crawler/mq/exec/snapshotTick.ts index abec4f6..32e7b23 100644 --- a/packages/crawler/mq/exec/snapshotTick.ts +++ b/packages/crawler/mq/exec/snapshotTick.ts @@ -109,7 +109,7 @@ const log = (value: number, base: number = 10) => Math.log(value) / Math.log(bas * @param aid - aid of the video * @returns ETA in hours */ -const getAdjustedShortTermETA = async (client: Client, aid: number) => { +export const getAdjustedShortTermETA = async (client: Client, aid: number) => { const latestSnapshot = await getLatestSnapshot(client, aid); // Immediately dispatch a snapshot if there is no snapshot yet if (!latestSnapshot) return 0; @@ -117,7 +117,7 @@ const getAdjustedShortTermETA = async (client: Client, aid: number) => { if (!snapshotsEnough) return 0; const currentTimestamp = new Date().getTime(); - const timeIntervals = [3 * MINUTE, 20 * MINUTE, 1 * HOUR, 3 * HOUR, 6 * HOUR]; + const timeIntervals = [3 * MINUTE, 20 * MINUTE, 1 * HOUR, 3 * HOUR, 6 * HOUR, 72 * HOUR]; const DELTA = 0.00001; let minETAHours = Infinity; From 1322cc4671d960f35cceec650ff6f85e785fa0ba Mon Sep 17 00:00:00 2001 From: alikia2x Date: Mon, 31 Mar 2025 02:47:33 +0800 Subject: [PATCH 21/24] init: backend --- deno.json | 36 ++++++++++---------- packages/backend/deno.json | 13 ++++++++ packages/backend/main.ts | 9 +++++ packages/core/db/pgConfig.ts | 2 +- packages/frontend/main.ts | 2 +- packages/frontend/package.json | 42 ++++++++++++------------ packages/frontend/src/styles/content.css | 4 +-- packages/frontend/tsconfig.json | 26 +++++++-------- 8 files changed, 78 insertions(+), 56 deletions(-) create mode 100644 packages/backend/main.ts diff --git a/deno.json b/deno.json index fcee853..abfed7f 100644 --- a/deno.json +++ b/deno.json @@ -1,21 +1,21 @@ { - "lock": false, - "workspace": ["./packages/crawler", "./packages/frontend", "./packages/backend", "./packages/core"], - "nodeModulesDir": "auto", - "tasks": { - "crawler": "deno task --filter 'crawler' all" - }, - "fmt": { - "useTabs": true, - "lineWidth": 120, - "indentWidth": 4, - "semiColons": true, - "proseWrap": "always" - }, - "imports": { - "@astrojs/node": "npm:@astrojs/node@^9.1.3", - "@astrojs/svelte": "npm:@astrojs/svelte@^7.0.8", + "lock": false, + "workspace": ["./packages/crawler", "./packages/frontend", "./packages/backend", "./packages/core"], + "nodeModulesDir": "auto", + "tasks": { + "crawler": "deno task --filter 'crawler' all" + }, + "fmt": { + "useTabs": true, + "lineWidth": 120, + "indentWidth": 4, + "semiColons": true, + "proseWrap": "always" + }, + "imports": { + "@astrojs/node": "npm:@astrojs/node@^9.1.3", + "@astrojs/svelte": "npm:@astrojs/svelte@^7.0.8", "@core/db/": "./packages/core/db/", - "date-fns": "npm:date-fns@^4.1.0" - } + "date-fns": "npm:date-fns@^4.1.0" + } } diff --git a/packages/backend/deno.json b/packages/backend/deno.json index e69de29..bac3608 100644 --- a/packages/backend/deno.json +++ b/packages/backend/deno.json @@ -0,0 +1,13 @@ +{ + "name": "@cvsa/backend", + "imports": { + "hono": "jsr:@hono/hono@^4.7.5" + }, + "tasks": { + "start": "deno run --allow-net main.ts" + }, + "compilerOptions": { + "jsx": "precompile", + "jsxImportSource": "hono/jsx" + } +} diff --git a/packages/backend/main.ts b/packages/backend/main.ts new file mode 100644 index 0000000..daa7580 --- /dev/null +++ b/packages/backend/main.ts @@ -0,0 +1,9 @@ +import { Hono } from "hono"; + +const app = new Hono(); + +app.get("/", (c) => { + return c.text("Hello Hono!"); +}); + +Deno.serve(app.fetch); diff --git a/packages/core/db/pgConfig.ts b/packages/core/db/pgConfig.ts index adea921..cebba63 100644 --- a/packages/core/db/pgConfig.ts +++ b/packages/core/db/pgConfig.ts @@ -27,4 +27,4 @@ export const postgresConfigCred = { database: databaseNameCred, user: databaseUser, password: databasePassword, -} \ No newline at end of file +}; diff --git a/packages/frontend/main.ts b/packages/frontend/main.ts index f6ef10d..3f50c75 100644 --- a/packages/frontend/main.ts +++ b/packages/frontend/main.ts @@ -1 +1 @@ -export const VERSION = "1.2.6"; \ No newline at end of file +export const VERSION = "1.2.6"; diff --git a/packages/frontend/package.json b/packages/frontend/package.json index 377d360..d94c869 100644 --- a/packages/frontend/package.json +++ b/packages/frontend/package.json @@ -1,23 +1,23 @@ { - "name": "frontend", - "type": "module", - "version": "0.0.1", - "scripts": { - "dev": "astro dev", - "build": "astro build", - "preview": "astro preview", - "astro": "astro" - }, - "dependencies": { - "@astrojs/tailwind": "^6.0.2", - "astro": "^5.5.5", - "autoprefixer": "^10.4.21", - "pg": "^8.11.11", - "postcss": "^8.5.3", - "tailwindcss": "^3.0.24", - "vite-tsconfig-paths": "^5.1.4" - }, - "devDependencies": { - "@types/pg": "^8.11.11" - } + "name": "frontend", + "type": "module", + "version": "0.0.1", + "scripts": { + "dev": "astro dev", + "build": "astro build", + "preview": "astro preview", + "astro": "astro" + }, + "dependencies": { + "@astrojs/tailwind": "^6.0.2", + "astro": "^5.5.5", + "autoprefixer": "^10.4.21", + "pg": "^8.11.11", + "postcss": "^8.5.3", + "tailwindcss": "^3.0.24", + "vite-tsconfig-paths": "^5.1.4" + }, + "devDependencies": { + "@types/pg": "^8.11.11" + } } diff --git a/packages/frontend/src/styles/content.css b/packages/frontend/src/styles/content.css index 74c9bd1..6d23e15 100644 --- a/packages/frontend/src/styles/content.css +++ b/packages/frontend/src/styles/content.css @@ -3,7 +3,7 @@ @tailwind utilities; .content { - @apply text-gray-800 dark:text-zinc-100; + @apply text-gray-800 dark:text-zinc-100; h1, h2, @@ -73,4 +73,4 @@ ol li p { @apply inline; } -} \ No newline at end of file +} diff --git a/packages/frontend/tsconfig.json b/packages/frontend/tsconfig.json index bee52b7..b8575e7 100644 --- a/packages/frontend/tsconfig.json +++ b/packages/frontend/tsconfig.json @@ -1,16 +1,16 @@ { - "extends": "astro/tsconfigs/strict", - "include": [".astro/types.d.ts", "**/*"], - "exclude": ["dist"], - "compilerOptions": { - "baseUrl": ".", - "paths": { - "@components/*": ["src/components/*"], - "@layouts/*": ["src/layouts/*"], - "@utils/*": ["src/utils/*"], - "@assets/*": ["src/assets/*"], + "extends": "astro/tsconfigs/strict", + "include": [".astro/types.d.ts", "**/*"], + "exclude": ["dist"], + "compilerOptions": { + "baseUrl": ".", + "paths": { + "@components/*": ["src/components/*"], + "@layouts/*": ["src/layouts/*"], + "@utils/*": ["src/utils/*"], + "@assets/*": ["src/assets/*"], "@styles": ["src/styles/*"], - "@core/*": ["../core/*"] - } - } + "@core/*": ["../core/*"] + } + } } From a2b55d0900e45792a282ea5f6fbc6f20dfb85f09 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Mon, 31 Mar 2025 05:07:22 +0800 Subject: [PATCH 22/24] feat: root path for API improve: the GET video snapshots API --- deno.json | 3 +- packages/backend/database.ts | 20 ++++++ packages/backend/deno.json | 10 ++- packages/backend/main.ts | 21 ++++-- packages/backend/root.ts | 31 +++++++++ packages/backend/singers.ts | 103 ++++++++++++++++++++++++++++++ packages/backend/snapshots.ts | 89 ++++++++++++++++++++++++++ packages/backend/utils.ts | 5 ++ packages/core/db/schema.d.ts | 55 ++++++++++++++++ packages/core/db/videoSnapshot.ts | 33 ++++++++++ 10 files changed, 361 insertions(+), 9 deletions(-) create mode 100644 packages/backend/database.ts create mode 100644 packages/backend/root.ts create mode 100644 packages/backend/singers.ts create mode 100644 packages/backend/snapshots.ts create mode 100644 packages/backend/utils.ts create mode 100644 packages/core/db/schema.d.ts create mode 100644 packages/core/db/videoSnapshot.ts diff --git a/deno.json b/deno.json index abfed7f..a3864b1 100644 --- a/deno.json +++ b/deno.json @@ -3,7 +3,8 @@ "workspace": ["./packages/crawler", "./packages/frontend", "./packages/backend", "./packages/core"], "nodeModulesDir": "auto", "tasks": { - "crawler": "deno task --filter 'crawler' all" + "crawler": "deno task --filter 'crawler' all", + "backend": "deno task --filter 'backend' start" }, "fmt": { "useTabs": true, diff --git a/packages/backend/database.ts b/packages/backend/database.ts new file mode 100644 index 0000000..7eccea3 --- /dev/null +++ b/packages/backend/database.ts @@ -0,0 +1,20 @@ +import { type Client, Pool } from "https://deno.land/x/postgres@v0.19.3/mod.ts"; +import { postgresConfig } from "@core/db/pgConfig.ts"; +import { createMiddleware } from "hono/factory"; + +const pool = new Pool(postgresConfig, 4); + +export const db = pool; + +export const dbMiddleware = createMiddleware(async (c, next) => { + const connection = await pool.connect(); + c.set("db", connection); + await next(); + connection.release(); +}); + +declare module "hono" { + interface ContextVariableMap { + db: Client; + } +} diff --git a/packages/backend/deno.json b/packages/backend/deno.json index bac3608..54f0456 100644 --- a/packages/backend/deno.json +++ b/packages/backend/deno.json @@ -1,13 +1,17 @@ { "name": "@cvsa/backend", "imports": { - "hono": "jsr:@hono/hono@^4.7.5" + "hono": "jsr:@hono/hono@^4.7.5", + "zod": "npm:zod", + "yup": "npm:yup" }, "tasks": { - "start": "deno run --allow-net main.ts" + "dev": "deno serve --env-file=.env --allow-env --allow-net --watch main.ts", + "start": "deno serve --env-file=.env --allow-env --allow-net --host 127.0.0.1 main.ts" }, "compilerOptions": { "jsx": "precompile", "jsxImportSource": "hono/jsx" - } + }, + "exports": "./main.ts" } diff --git a/packages/backend/main.ts b/packages/backend/main.ts index daa7580..39fdd9b 100644 --- a/packages/backend/main.ts +++ b/packages/backend/main.ts @@ -1,9 +1,20 @@ import { Hono } from "hono"; +import { dbMiddleware } from "./database.ts"; +import { rootHandler } from "./root.ts"; +import { getSnapshotsHanlder } from "./snapshots.ts"; -const app = new Hono(); +export const app = new Hono(); -app.get("/", (c) => { - return c.text("Hello Hono!"); -}); +app.use('/video/*', dbMiddleware); -Deno.serve(app.fetch); +app.get("/", ...rootHandler); + +app.get('/video/:id/snapshots', ...getSnapshotsHanlder); + +const fetch = app.fetch; + +export default { + fetch, +} satisfies Deno.ServeDefaultExport; + +export const VERSION = "0.2.0"; \ No newline at end of file diff --git a/packages/backend/root.ts b/packages/backend/root.ts new file mode 100644 index 0000000..3cb15fd --- /dev/null +++ b/packages/backend/root.ts @@ -0,0 +1,31 @@ +import { getSingerForBirthday, pickSinger, pickSpecialSinger, type Singer } from "./singers.ts"; +import { VERSION } from "./main.ts"; +import { createHandlers } from "./utils.ts"; + +export const rootHandler = createHandlers((c) => { + let singer: Singer | Singer[] | null = null; + const shouldShowSpecialSinger = Math.random() < 0.016; + if (getSingerForBirthday().length !== 0){ + singer = getSingerForBirthday(); + for (const s of singer) { + delete s.birthday; + s.message = `祝${s.name}生日快乐~` + } + } + else if (shouldShowSpecialSinger) { + singer = pickSpecialSinger(); + } + else { + singer = pickSinger(); + } + return c.json({ + "project": { + "name": "中V档案馆", + "motto": "一起唱吧，心中的歌！" + }, + "status": 200, + "version": VERSION, + "time": Date.now(), + "singer": singer + }) +}) \ No newline at end of file diff --git a/packages/backend/singers.ts b/packages/backend/singers.ts new file mode 100644 index 0000000..47963bb --- /dev/null +++ b/packages/backend/singers.ts @@ -0,0 +1,103 @@ +export const singers = [ + { + "name": "洛天依", + "color": "#66CCFF", + "birthday": "0712", + }, + { + "name": "言和", + "color": "#00FFCC", + "birthday": "0711", + }, + { + "name": "乐正绫", + "color": "#EE0000", + "birthday": "0412", + }, + { + "name": "乐正龙牙", + "color": "#006666", + "birthday": "1002", + }, + { + "name": "徵羽摩柯", + "color": "#0080FF", + "birthday": "1210", + }, + { + "name": "墨清弦", + "color": "#FFFF00", + "birthday": "0520", + }, + { + "name": "星尘", + "color": "#9999FF", + "birthday": "0812", + }, + { + "name": "心华", + "color": "#EE82EE", + "birthday": "0210", + }, + { + "name": "海伊", + "color": "#3399FF", + "birthday": "0722", + }, + { + "name": "苍穹", + "color": "#8BC0B5", + "birthday": "0520", + }, + { + "name": "赤羽", + "color": "#FF4004", + "birthday": "1126", + }, + { + "name": "诗岸", + "color": "#F6BE72", + "birthday": "0119", + }, + { + "name": "牧心", + "color": "#2A2859", + "birthday": "0807", + }, +]; + +export interface Singer { + name: string; + color?: string; + birthday?: string; + message?: string; +} + +export const specialSingers = [ + { + "name": "雅音宫羽", + "message": "你是我最真模样，从来不曾遗忘。", + }, + { + "name": "初音未来", + "message": "初始之音，响彻未来!", + }, +]; + +export const pickSinger = () => { + const index = Math.floor(Math.random() * singers.length); + return singers[index]; +}; + +export const pickSpecialSinger = () => { + const index = Math.floor(Math.random() * specialSingers.length); + return specialSingers[index]; +}; + +export const getSingerForBirthday = (): Singer[] => { + const today = new Date(); + const month = String(today.getMonth() + 1).padStart(2, "0"); + const day = String(today.getDate()).padStart(2, "0"); + const datestring = `${month}${day}`; + return singers.filter((singer) => singer.birthday === datestring); +}; diff --git a/packages/backend/snapshots.ts b/packages/backend/snapshots.ts new file mode 100644 index 0000000..1ee3216 --- /dev/null +++ b/packages/backend/snapshots.ts @@ -0,0 +1,89 @@ +import type { Context } from "hono"; +import { createHandlers } from "./utils.ts"; +import type { BlankEnv, BlankInput } from "hono/types"; +import { getVideoSnapshots, getVideoSnapshotsByBV } from "@core/db/videoSnapshot.ts"; +import type { VideoSnapshotType } from "@core/db/schema.d.ts"; +import { boolean, mixed, number, object, ValidationError } from "yup"; + +const SnapshotQueryParamsSchema = object({ + ps: number().optional().positive(), + pn: number().optional().positive(), + offset: number().optional().positive(), + reverse: boolean().optional(), +}); + +const idSchema = mixed().test( + "is-valid-id", + 'id must be a string starting with "av" followed by digits, or "BV" followed by 10 alphanumeric characters, or a positive integer', + (value) => { + if (typeof value === "number") { + return Number.isInteger(value) && value > 0; + } + + if (typeof value === "string") { + if (value.startsWith("av")) { + const digitsOnly = value.substring(2); + return /^\d+$/.test(digitsOnly) && digitsOnly.length > 0; + } + + if (value.startsWith("BV")) { + const remainingChars = value.substring(2); + return /^[a-zA-Z0-9]{10}$/.test(remainingChars); + } + } + + return false; + }, +); + +type ContextType = Context; +export const getSnapshotsHanlder = createHandlers(async (c: ContextType) => { + const client = c.get("db"); + + try { + const idParam = await idSchema.validate(c.req.param("id")); + let videoId: number | string = idParam as string | number; + if (typeof videoId === "string" && videoId.startsWith("av")) { + videoId = videoId.slice(2); + } + const queryParams = await SnapshotQueryParamsSchema.validate(c.req.query()); + const { ps, pn, offset, reverse = false } = queryParams; + + let limit = 1000; + if (ps && ps > 1) { + limit = ps; + } + + let pageOrOffset = 1; + let mode: "page" | "offset" = "page"; + + if (pn && pn > 1) { + pageOrOffset = pn; + mode = "page"; + } else if (offset && offset > 1) { + pageOrOffset = offset; + mode = "offset"; + } + + let result: VideoSnapshotType[]; + + if (typeof videoId === "number") { + result = await getVideoSnapshots(client, videoId, limit, pageOrOffset, reverse, mode); + } else { + result = await getVideoSnapshotsByBV(client, videoId, limit, pageOrOffset, reverse, mode); + } + + const rows = result.map((row) => ({ + ...row, + aid: Number(row.aid), + })); + + return c.json(rows); + } catch (e) { + if (e instanceof ValidationError) { + return c.json({ message: "Invalid query parameters", errors: e.errors }, 400); + } else { + return c.json({ message: "Unhandled error", error: e }, 500); + } + } +}); diff --git a/packages/backend/utils.ts b/packages/backend/utils.ts new file mode 100644 index 0000000..9d06a7c --- /dev/null +++ b/packages/backend/utils.ts @@ -0,0 +1,5 @@ +import { createFactory } from 'hono/factory' + +const factory = createFactory(); + +export const createHandlers = factory.createHandlers; \ No newline at end of file diff --git a/packages/core/db/schema.d.ts b/packages/core/db/schema.d.ts new file mode 100644 index 0000000..a9a7296 --- /dev/null +++ b/packages/core/db/schema.d.ts @@ -0,0 +1,55 @@ +export interface AllDataType { + id: number; + aid: bigint; + bvid: string | null; + description: string | null; + uid: number | null; + tags: string | null; + title: string | null; + published_at: string | null; + duration: number; + created_at: string | null; +} + +export interface BiliUserType { + id: number; + uid: number; + username: string; + desc: string; + fans: number; +} + +export interface VideoSnapshotType { + id: number; + created_at: string; + views: number; + coins: number; + likes: number; + favorites: number; + shares: number; + danmakus: number; + aid: bigint; + replies: number; +} + +export interface LatestSnapshotType { + aid: bigint; + time: number; + views: number; + danmakus: number; + replies: number; + likes: number; + coins: number; + shares: number; + favorites: number; +} + +export interface SnapshotScheduleType { + id: number; + aid: bigint; + type?: string; + created_at: string; + started_at?: string; + finished_at?: string; + status: string; +} diff --git a/packages/core/db/videoSnapshot.ts b/packages/core/db/videoSnapshot.ts new file mode 100644 index 0000000..68f07ec --- /dev/null +++ b/packages/core/db/videoSnapshot.ts @@ -0,0 +1,33 @@ +import { Client } from "https://deno.land/x/postgres@v0.19.3/mod.ts"; +import { VideoSnapshotType } from "@core/db/schema.d.ts"; + +export async function getVideoSnapshots(client: Client, aid: number, limit: number, pageOrOffset: number, reverse: boolean, mode: 'page' | 'offset' = 'page') { + const offset = mode === 'page' ? (pageOrOffset - 1) * limit : pageOrOffset; + const order = reverse ? 'ASC' : 'DESC'; + const query = ` + SELECT * + FROM video_snapshot + WHERE aid = $1 + ORDER BY created_at ${order} + LIMIT $2 + OFFSET $3 + `; + const queryResult = await client.queryObject(query, [aid, limit, offset]); + return queryResult.rows; +} + +export async function getVideoSnapshotsByBV(client: Client, bv: string, limit: number, pageOrOffset: number, reverse: boolean, mode: 'page' | 'offset' = 'page') { + const offset = mode === 'page' ? (pageOrOffset - 1) * limit : pageOrOffset; + const order = reverse ? 'ASC' : 'DESC'; + const query = ` + SELECT vs.* + FROM video_snapshot vs + JOIN bilibili_metadata bm ON vs.aid = bm.aid + WHERE bm.bvid = $1 + ORDER BY vs.created_at ${order} + LIMIT $2 + OFFSET $3 + ` + const queryResult = await client.queryObject(query, [bv, limit, offset]); + return queryResult.rows; +} \ No newline at end of file From 9af7e52464b22d9f1e258da2d7821d9384ddd703 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Mon, 31 Mar 2025 05:12:18 +0800 Subject: [PATCH 23/24] version: crawler/1.0.17 --- packages/crawler/main.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/crawler/main.ts b/packages/crawler/main.ts index f31ba12..838dc21 100644 --- a/packages/crawler/main.ts +++ b/packages/crawler/main.ts @@ -4,4 +4,4 @@ // SO HERE'S A PLACHOLDER EXPORT FOR DENO: export const DENO = "FUCK YOU DENO"; // Oh, maybe export the version is a good idea -export const VERSION = "1.0.13"; +export const VERSION = "1.0.17"; From 834f81eff0b59339328cd5c83c504ec27ae45a54 Mon Sep 17 00:00:00 2001 From: alikia2x Date: Sun, 30 Mar 2025 21:29:02 +0000 Subject: [PATCH 24/24] doc: GitBook - No subject --- doc/zh/SUMMARY.md | 2 +- doc/zh/architecture/database-structure/README.md | 8 ++++++-- .../message-queue/latestvideosqueue-dui-lie.md | 2 ++ .../message-queue/video-tags-queue.md | 15 --------------- doc/zh/architecture/overview.md | 13 +++++++++++-- 5 files changed, 20 insertions(+), 20 deletions(-) create mode 100644 doc/zh/architecture/message-queue/latestvideosqueue-dui-lie.md delete mode 100644 doc/zh/architecture/message-queue/video-tags-queue.md diff --git a/doc/zh/SUMMARY.md b/doc/zh/SUMMARY.md index 981ee4b..b51d329 100644 --- a/doc/zh/SUMMARY.md +++ b/doc/zh/SUMMARY.md @@ -14,7 +14,7 @@ * [歌曲类型](architecture/database-structure/type-of-song.md) * [人工智能](architecture/artificial-intelligence.md) * [消息队列](architecture/message-queue/README.md) - * [VideoTagsQueue队列](architecture/message-queue/video-tags-queue.md) + * [LatestVideosQueue 队列](architecture/message-queue/latestvideosqueue-dui-lie.md) ## API 文档 diff --git a/doc/zh/architecture/database-structure/README.md b/doc/zh/architecture/database-structure/README.md index 15d0a59..33b5b2b 100644 --- a/doc/zh/architecture/database-structure/README.md +++ b/doc/zh/architecture/database-structure/README.md @@ -5,6 +5,10 @@ CVSA 使用 [PostgreSQL](https://www.postgresql.org/) 作为数据库。 CVSA 的所有公开数据（不包括用户的个人数据）都存储在名为 `cvsa_main` 的数据库中，该数据库包含以下表： * songs：存储歌曲的主要信息 -* bili\_user：存储 Bilibili 用户信息快照 -* all\_data：[分区 30](../../about/scope-of-inclusion.md#vocaloiduatu-fen-qu) 中所有视频的元数据。 +* bilibili\_user：存储 Bilibili 用户信息快照 +* bilibili\_metadata：[分区 30](../../about/scope-of-inclusion.md#vocaloiduatu-fen-qu) 中所有视频的元数据 * labelling\_result：包含由我们的 AI 系统 标记的 `all_data` 中视频的标签。 +* latest\_video\_snapshot：存储视频最新的快照 +* video\_snapshot：存储视频的快照，包括特定时间下视频的统计信息（播放量、点赞数等） +* snapshot\_schedule：视频快照的规划信息，为辅助表 + diff --git a/doc/zh/architecture/message-queue/latestvideosqueue-dui-lie.md b/doc/zh/architecture/message-queue/latestvideosqueue-dui-lie.md new file mode 100644 index 0000000..bc9564e --- /dev/null +++ b/doc/zh/architecture/message-queue/latestvideosqueue-dui-lie.md @@ -0,0 +1,2 @@ +# LatestVideosQueue 队列 + diff --git a/doc/zh/architecture/message-queue/video-tags-queue.md b/doc/zh/architecture/message-queue/video-tags-queue.md deleted file mode 100644 index a90540c..0000000 --- a/doc/zh/architecture/message-queue/video-tags-queue.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -description: 关于VideoTagsQueue队列的信息。 ---- - -# VideoTagsQueue队列 - -### 任务 - -视频标签队列包含两个任务：`getVideoTags`和`getVideosTags`。前者用于获取视频的标签，后者负责调度前者。 - -### 返回值 - -两个任务的返回值遵循以下表格： - -
创建时间观看硬币点赞收藏分享弹幕评论创建时间观看硬币点赞收藏分享弹幕评论
{format(new Date(snapshot.created_at), 'yyyy-MM-dd HH:mm:ss', { locale: zhCN })}{snapshot.views}{snapshot.coins}{snapshot.likes}{snapshot.favorites}{snapshot.shares}{snapshot.danmakus}{snapshot.replies}{format(new Date(snapshot.created_at), 'yyyy-MM-dd HH:mm:ss', { locale: zhCN })}{snapshot.views}{snapshot.coins}{snapshot.likes}{snapshot.favorites}{snapshot.shares}{snapshot.danmakus}{snapshot.replies}
返回值描述
0在 getVideoTags 中：标签成功获取
在 getVideosTags 中：所有无标签视频的相应任务已成功排队。
1在 getVideoTags 中：任务期间发生 fetch 错误
2在 getVideoTags 中：已达到 NetScheduler 设置的速率限制
3在 getVideoTags 中：未在任务数据中提供帮助
4在 getVideosTags 中：没有视频的 `tags` 为 NULL
1xx在 getVideosTags 中：队列中的任务数量超过了限制，因此 getVideosTags 停止添加任务。xx 是在执行期间添加到队列的任务数量。
diff --git a/doc/zh/architecture/overview.md b/doc/zh/architecture/overview.md index 33dbe0d..3e3312d 100644 --- a/doc/zh/architecture/overview.md +++ b/doc/zh/architecture/overview.md @@ -1,5 +1,4 @@ --- -icon: globe-pointer layout: title: visible: true @@ -15,4 +14,14 @@ layout: # 概览 -自动化是 CVSA 技术设计的最大亮点，为了实现自动化，我们使用BullMQ驱动的消息队列来并发处理数据采集生命周期中的各项任务。 +整个CVSA项目分为三个组件：**crawler**, **frontend** 和 **backend。** + +### **crawler** + +位于项目目录`packages/crawler` 下，它负责以下工作： + +* 抓取新的视频并收录作品 +* 持续监控视频的播放量等统计信息 + +整个 crawler 由 BullMQ 消息队列驱动，使用 Redis 和 PostgreSQL 管理状态。 +