|
|
2c51c3c09c
|
add: log for getRegularSnapshotInterval
|
2025-03-26 23:04:08 +08:00 |
|
|
|
b286a9d7b1
|
update: use findSnapshotBefore instead of findClosetSnapshot in regular snapshot scheduling
|
2025-03-26 22:59:52 +08:00 |
|
|
|
cabb360a16
|
fix: missing entrance for schedule type 'new'
update: perf log text
|
2025-03-25 22:58:56 +08:00 |
|
|
|
2d8e990bc9
|
fix: prevent duplicate jobs added to queue
|
2025-03-25 21:33:58 +08:00 |
|
|
|
b33fd790d1
|
fix: snapshot process early returned incorrectly
|
2025-03-25 21:29:47 +08:00 |
|
|
|
86337e3802
|
add: set scheduled job status to 'processing' before adding to bullmq
|
2025-03-25 05:05:33 +08:00 |
|
|
|
76c9256662
|
test: looser rate limits for snapshot
|
2025-03-25 04:57:24 +08:00 |
|
|
|
a178b7fc16
|
update: force schedule for new songs
|
2025-03-25 04:52:19 +08:00 |
|
|
|
314beb54b5
|
add: upsertJob for scheduleCleanup
|
2025-03-24 23:39:29 +08:00 |
|
|
|
d8b47f8fc8
|
add: log for scheduleCleanupWorker
|
2025-03-24 23:37:46 +08:00 |
|
|
|
fa058b22fe
|
fix: job consumption rate too low, add outdated job cleanup
|
2025-03-24 23:36:01 +08:00 |
|
|
|
48b1130cba
|
feat: continuous monitoring of new songs
|
2025-03-24 04:53:43 +08:00 |
|
|
|
42db333d1a
|
temp: schedule for new songs
|
2025-03-24 04:40:10 +08:00 |
|
|
|
8be68248df
|
fix: ignored the case where snapshots are actually the same
|
2025-03-24 03:49:06 +08:00 |
|
|
|
6f4a26e8b3
|
test: dynamic interval for reglar snapshots
|
2025-03-24 03:43:26 +08:00 |
|
|
|
3028dc13c7
|
improve: performance when dispatching jobs
|
2025-03-24 02:35:55 +08:00 |
|
|
|
0455abce2e
|
fix: did not quit when job is already workingx
|
2025-03-24 01:10:27 +08:00 |
|
|
|
723b6090c4
|
fix: 1 dispatchRegularSnapshots working at a time
|
2025-03-24 01:06:48 +08:00 |
|
|
|
9060d28823
|
fix: prevent dispatchRegularSnapshots from running for too long
|
2025-03-24 00:57:48 +08:00 |
|
|
|
0be961e709
|
improve: target time finding
|
2025-03-23 23:31:16 +08:00 |
|
|
|
f9dd53c250
|
update: remove old jobScheduler
|
2025-03-23 21:48:42 +08:00 |
|
|
|
35d58be8fd
|
fix: may accessing a non-existent schedule
|
2025-03-23 21:42:05 +08:00 |
|
|
|
3a0dd26c68
|
fix: only 1 pending task at a time
|
2025-03-23 21:36:40 +08:00 |
|
|
|
9e764746fb
|
update: better task retrieval for snapshotTick
add: logging when snapshot the video
|
2025-03-23 20:49:41 +08:00 |
|
|
|
33c63fc29f
|
fix: NaN caused by not converting date string to timestamp
|
2025-03-23 20:41:13 +08:00 |
|
|
|
b654eb3643
|
fix: potential NaN as delay time when scheduling
|
2025-03-23 20:30:06 +08:00 |
|
|
|
7768a202b2
|
fix: failed to serialize bigint correctly
|
2025-03-23 20:23:00 +08:00 |
|
|
|
8652ac8fb7
|
fix: bugs of snapshot scheduling
|
2025-03-23 19:45:52 +08:00 |
|
|
|
18fc9752bb
|
fix: several bugs of snapshot scheduling
|
2025-03-23 18:46:25 +08:00 |
|
|
|
2c12310e8c
|
feat: snapshot based on persistent schedule
|
2025-03-23 17:44:59 +08:00 |
|
|
|
e38dc96275
|
update: insertion of snapshot schedule
|
2025-03-22 20:51:28 +08:00 |
|
|
|
9f9ef800d1
|
merge: hotfix into ref/snapshot
|
2025-03-22 01:48:03 +08:00 |
|
|
|
e5534cda24
|
fix: incorrect filter condition that causes empty tags
|
2025-03-22 00:58:36 +08:00 |
|
|
|
1895d601d9
|
update: dynamic delay factor for snapshotMilestoneVideo
|
2025-03-22 00:40:00 +08:00 |
|
|
|
8158ce10c0
|
fix: inserting videos into songs table regardless of classified label
|
2025-03-21 21:06:01 +08:00 |
|
|
|
00b52c01f7
|
fix: unexpected column bvid when inserting to songs table
|
2025-03-21 20:51:34 +08:00 |
|
|
|
2e8ed7ce70
|
add: ETA estimation for short-term snapshot
|
2025-03-20 01:57:33 +08:00 |
|
|
|
cd8aa826e1
|
fix: prevent videos from being crawled for too long
|
2025-03-17 00:33:28 +08:00 |
|
|
|
b07d0c18f9
|
update: preparation for snapshotSchedule
|
2025-03-17 00:25:31 +08:00 |
|
|
|
a9ac8de547
|
fix: unhandled timezone mismatch when inserting to database
|
2025-03-16 14:23:11 +08:00 |
|
|
|
0ff1c78dcc
|
fix: incorrect timestamp unit when inserting to database
|
2025-03-16 14:00:49 +08:00 |
|
|
|
a6c8fd7f3f
|
ref: code structure related to AI
|
2025-03-16 01:23:10 +08:00 |
|
|
|
7104a95af9
|
ref: rename table all_data, bili_user to bilibili_metadata, bilibili_user
|
2025-03-15 21:27:19 +08:00 |
|
|
|
5af2236109
|
temp: remove the scheduleSnapshotTick job
|
2025-03-15 21:25:26 +08:00 |
|
|
|
93bdddc21e
|
fix: unexpectedly commented code to upsert jobs
|
2025-03-15 16:51:07 +08:00 |
|
|
|
bce4161501
|
improve: better eta prediction
|
2025-03-11 23:40:15 +08:00 |
|
|
|
09f872b11a
|
improve: handling of invisible videos in snapshot
|
2025-03-09 23:01:10 +08:00 |
|
|
|
9a85da0532
|
update: handling for deleted videos
|
2025-03-09 22:57:26 +08:00 |
|
|
|
81b95c9569
|
update: minimum interval for snapshotMilestoneVideo
|
2025-03-09 22:52:12 +08:00 |
|
|
|
a138f6d572
|
add: auto-removal of jobs, aliyun-fc auto retry
|
2025-03-09 22:34:23 +08:00 |
|
