alikia2x/OpenRewind
1
1
Fork 0
Code Issues Releases 1 Activity

fix: wrong field type in migrateToV3.ts

Browse Source 
improve: use single threaded ffmpeg to reduce peak CPU utilization
update: database-changelog.md & database-structure.md
This commit is contained in:
alikia2x (寒寒) 2024-12-26 18:12:20 +08:00
parent 1cd6d7b639
commit 8cf17838f4
Signed by: alikia2x
GPG Key ID: 56209E0CCD8420C6
4 changed files with 127 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

104
docs/database-changelog.md
View File

@ -1,11 +1,107 @@
# Database Schema Documentation
# Database Schema Changelog
This document outlines the changes made across different versions of
database structure used in the OpenRewind, including tables and fields.
## Version 2 Schema Changes
## Version 3 Schema Changes
Cooresponding version: Since 0.4.0
Corresponding version: Since 0.5.0
### Update `encoding_task` Table
#### Change `createAt` to `createdAt`
The column `createAt` was renamed to `createdAt` for consistency.
```sql
ALTER TABLE encoding_task RENAME COLUMN createAt TO createdAt;
```
#### Convert `createdAt` to Unix Timestamp
The `createdAt` column was updated to store Unix timestamps instead of formatted timestamps.
```typescript
const rows = db.prepare(`SELECT id, createdAt FROM encoding_task`).all() as { [x: string]: unknown; id: unknown; }[];
const updateStmt = db.prepare(`UPDATE encoding_task SET createdAt_new = ? WHERE id = ?`);
rows.forEach((row) => {
const unixTimestamp = convertTimestampToUnix(row.createdAt as string);
updateStmt.run(unixTimestamp, row.id);
});
```
### Update `frame` Table
#### Change `createAt` to `createdAt`
The column `createAt` was renamed to `createdAt` for consistency.
```sql
ALTER TABLE frame RENAME COLUMN createAt TO createdAt;
```
#### Convert `createdAt` to Unix Timestamp
The `createdAt` column was updated to store Unix timestamps instead of formatted timestamps.
```typescript
const rows = db.prepare(`SELECT id, createdAt FROM frame`).all() as { [x: string]: unknown; id: unknown; }[];
const updateStmt = db.prepare(`UPDATE frame SET createdAt_new = ? WHERE id = ?`);
rows.forEach((row) => {
const unixTimestamp = convertTimestampToUnix(row.createdAt as string);
updateStmt.run(unixTimestamp, row.id);
});
```
### Update `segments` Table
#### Rename Columns for Consistency
The columns `startAt` and `endAt` were renamed to `startedAt` and `endedAt` respectively.
```sql
ALTER TABLE segments RENAME COLUMN startAt TO startedAt;
ALTER TABLE segments RENAME COLUMN endAt TO endedAt;
```
#### Convert `startedAt` and `endedAt` to Unix Timestamps
The `startedAt` and `endedAt` columns were updated to store Unix timestamps instead of formatted timestamps.
```typescript
const rows = db.prepare(`SELECT id, startedAt, endedAt FROM segments`).all() as { [x: string]: unknown; id: unknown; }[];
const updateStart = db.prepare(`UPDATE segments SET startedAt_new = ? WHERE id = ?`);
const updateEnd = db.prepare(`UPDATE segments SET endedAt_new = ? WHERE id = ?`);
rows.forEach((row) => {
updateStart.run(convertTimestampToUnix(row.startedAt as string), row.id);
updateEnd.run(convertTimestampToUnix(row.endedAt as string), row.id);
});
```
### Drop Deprecated `encoded` Column
The deprecated `encoded` column was removed from the `frame` table.
```sql
ALTER TABLE frame DROP COLUMN encoded;
```
### Summary of Changes
- **Update `encoding_task` Table:**
- Renamed `createAt` to `createdAt`.
- Converted `createdAt` to store Unix timestamps.
- **Update `frame` Table:**
- Renamed `createAt` to `createdAt`.
- Converted `createdAt` to store Unix timestamps.
- Dropped the deprecated `encoded` column.
- **Update `segments` Table:**
- Renamed `startAt` and `endAt` to `startedAt` and `endedAt` respectively.
- Converted `startedAt` and `endedAt` to store Unix timestamps.
## Version 2 Schema
Corresponding version: 0.4.0
### New Table: `config`
@ -105,7 +201,7 @@ UPDATE frame SET encodeStatus = CASE WHEN encoded THEN 2 ELSE 0 END;
## Version 1 Schema
Cooresponding version: 0.3.x
Corresponding version: 0.3.x
### Table: `frame`

48
docs/database-structure.md
View File

@ -1,7 +1,6 @@
# Database Schema Documentation (Version 2)
# Database Schema Documentation (Version 3)
This document outlines the current structure of the database schema used in the application.
It includes tables, fields, and their descriptions.
This document outlines the current structure of the database schema used in the application. It includes tables, fields, and their descriptions.
## Table: `config`
@ -14,9 +13,7 @@ Stores configuration data.
### Key: version
The current database schema version, represented as a integer.
Since the `config` table does not exist in V1, the version must be at least 2.
The current database schema version, represented as an integer. Since the `config` table does not exist in V1, the version must be at least 2.
## Table: `frame`
@ -25,14 +22,13 @@ Stores information about individual frames.
| Column Name | Data Type | Constraints/Default | Description |
|-------------------|-----------|---------------------------------|-----------------------------------------------------------|
| `id` | INTEGER | PRIMARY KEY, AUTOINCREMENT | Unique identifier for each frame. |
| `createAt` | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | Timestamp when the frame was created. |
| `createdAt` | REAL | | Timestamp when the frame was created. |
| `imgFilename` | TEXT | | Filename of the image associated with the frame. |
| `segmentID` | INTEGER | NULL, FOREIGN KEY (segments.id) | ID of the segment to which the frame belongs. |
| `videoPath` | TEXT | NULL | Relative path to the video file if the frame was encoded. |
| `videoFrameIndex` | INTEGER | NULL | Index of the frame within the encoded video. |
| `collectionID` | INTEGER | NULL | ID of the collection to which the frame belongs. |
| `encodeStatus` | INTEGER | 0 | Indicates the encoding status of the frame. |
| `encodeStatus` | INTEGER | DEFAULT 0 | Indicates the encoding status of the frame. |
### Status Description
@ -53,15 +49,13 @@ Stores recognition data associated with frames.
## Table: `segments`
A segment is a period of time when a user uses a particular application.
While capturing the screen, OpenRewind detects the currently active window.
When it finds that the currently active window has changed to another application, a new segment will start.
A segment is a period of time when a user uses a particular application. While capturing the screen, OpenRewind detects the currently active window. When it finds that the currently active window has changed to another application, a new segment will start.
| Column Name | Data Type | Constraints/Default | Description |
|---------------|-----------|----------------------------|------------------------------------------------------|
| `id` | INTEGER | PRIMARY KEY, AUTOINCREMENT | Unique identifier for each segment. |
| `startAt` | TIMESTAMP | | Timestamp when the segment starts. |
| `endAt` | TIMESTAMP | | Timestamp when the segment ends. |
| `startedAt` | REAL | | Timestamp when the segment starts. |
| `endedAt` | REAL | | Timestamp when the segment ends. |
| `title` | TEXT | | Title of the segment. |
| `appName` | TEXT | | Name of the application associated with the segment. |
| `appPath` | TEXT | | Path to the application. |
@ -77,7 +71,7 @@ Stores encoding tasks that are queued for processing.
| Column Name | Data Type | Constraints/Default | Description |
|-------------|-----------|----------------------------|--------------------------------------|
| `id` | INTEGER | PRIMARY KEY, AUTOINCREMENT | Unique ID for the task. |
| `createAt` | TIMESTAMP | DEFAULT CURRENT_TIMESTAMP | Timestamp when the task was created. |
| `createdAt` | REAL | | Timestamp when the task was created. |
| `status` | INTEGER | DEFAULT 0 | Indicates the status of the task. |
### Task status Description
@ -85,17 +79,16 @@ Stores encoding tasks that are queued for processing.
- `0`: Pending
- `1`: In Progress
- `2`: Completed
- Once the task was set to this status, it will be imminently deleted by a trigger mentioned below.
- Once the task was set to this status, it will be imminently deleted by a trigger mentioned below.
## Table: `encoding_task_data`
Stores the frames that need to be encoded for the encoding task
Stores the frames that need to be encoded for the encoding task.
| Column Name | Data Type | Constraints/Default | Description |
|------------------|-----------|-------------------------------------|------------------------------------------------------|
| `encodingTaskID` | INTEGER | FOREIGN KEY (encoding_task.id) | ID for the encoding task associated with this frame. |
| `frame` | INTEGER | PRIMARY KEY, FOREIGN KEY (frame.id) | ID for the frame associated with the encoding task. |
| `encodingTaskID` | TIMESTAMP | FOREIGN KEY (encoding_task.id) | ID for the encoding task associated with this frame. |
## Virtual Table: `text_search`
@ -112,8 +105,7 @@ Used for full-text search on recognition data.
### `recognition_data_after_insert`
Triggered after inserting a new row into `recognition_data`.
Inserts a new row into `text_search` with the same data.
Triggered after inserting a new row into `recognition_data`. Inserts a new row into `text_search` with the same data.
```sql
CREATE TRIGGER IF NOT EXISTS recognition_data_after_insert AFTER INSERT ON recognition_data
@ -125,8 +117,7 @@ END;
### `recognition_data_after_update`
Triggered after updating a row in `recognition_data`.
Updates the associated `text_search` row.
Triggered after updating a row in `recognition_data`. Updates the associated `text_search` row.
```sql
CREATE TRIGGER IF NOT EXISTS recognition_data_after_update AFTER UPDATE ON recognition_data
@ -139,8 +130,7 @@ END;
### `recognition_data_after_delete`
Triggered after deleting a row from `recognition_data`.
Deletes the associated `text_search` row.
Triggered after deleting a row from `recognition_data`. Deletes the associated `text_search` row.
```sql
CREATE TRIGGER IF NOT EXISTS recognition_data_after_delete AFTER DELETE ON recognition_data
@ -151,11 +141,10 @@ END;
### `delete_encoding_task`
Triggered after updating the `status` of an encoding task to `2` (Completed).
Deletes the associated `encoding_task_data` and `encoding_task` rows.
Triggered after updating the `status` of an encoding task to `2` (Completed). Deletes the associated `encoding_task_data` and `encoding_task` rows.
```sql
CREATE TRIGGER delete_encoding_task
CREATE TRIGGER IF NOT EXISTS delete_encoding_task
AFTER UPDATE OF status
ON encoding_task
BEGIN
@ -165,5 +154,4 @@ BEGIN
DELETE FROM encoding_task
WHERE id = OLD.id AND NEW.status = 2;
END;
```
```

14
src/electron/backend/encoding.ts
View File

@ -1,5 +1,5 @@
import { Database } from "better-sqlite3";
import { exec, spawnSync } from "child_process";
import { exec } from "child_process";
import fs from "fs";
import path, { join } from "path";
import type { EncodingTask, Frame } from "./schema";
@ -7,10 +7,10 @@ import sizeOf from "image-size";
import { getEncodingTempDir, getRecordingsDir, getScreenshotsDir } from "../utils/backend.js";
import cache from "memory-cache";
const ENCODING_INTERVAL = 10000; // 10 sec
const CHECK_TASK_INTERVAL = 5000; // 5 sec
const MIN_FRAMES_TO_ENCODE = 60; // At least 10 mins (0.5fps)
const CONCURRENCY = 1; // Number of concurrent encoding tasks
const FRAME_RATE = 0.5;
const THREE_MINUTES = 180;
const MIN_FRAMES_TO_ENCODE = THREE_MINUTES * FRAME_RATE;
const CONCURRENCY = 1;
// Detect and insert encoding tasks
export function checkFramesForEncoding(db: Database) {
@ -114,9 +114,9 @@ export function processEncodingTasks(db: Database) {
cache.put("tasksPerforming", [...tasksPerforming, taskId.toString()]);
const videoPath = path.join(getRecordingsDir(), `${taskId}.mp4`);
const ffmpegCommand = `ffmpeg -f concat -safe 0 -i "${metaFilePath}" -c:v libx264 -r 30 "${videoPath}"`;
const ffmpegCommand = `ffmpeg -f concat -safe 0 -i "${metaFilePath}" -c:v libx264 -r 30 -threads 1 "${videoPath}"`;
console.log("FFMPEG", ffmpegCommand);
exec(ffmpegCommand, (error, stdout, stderr) => {
exec(ffmpegCommand, (error, _stdout, _stderr) => {
if (error) {
console.error(`FFmpeg error: ${error.message}`);
// Roll back transaction

4
src/electron/backend/migrate/migrateToV3.ts
View File

@ -12,7 +12,7 @@ function transformEncodingTask(db: Database) {
const createTableSql = `
CREATE TABLE IF NOT EXISTS encoding_task_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
createdAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
createdAt REAL,
status INT DEFAULT 0
);
@ -41,7 +41,7 @@ function transformFrame(db: Database) {
const createTableSql = `
CREATE TABLE frame_new(
id INTEGER PRIMARY KEY AUTOINCREMENT,
createdAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
createdAt REAL,
imgFilename TEXT,
segmentID INTEGER NULL,
videoPath TEXT NULL,