1
0
Fork 0
forked from mia/0x0

Add support for expiring files

SUPPLEMENTALLY:
- Add an `expiration` field to the `file` table of the database
- Produce a migration for the above change
- Overhaul the cleanup script, and integrate into fhost.py
  (now run using FLASK_APP=fhost flask prune)
- Replace the old cleanup script with a deprecation notice
- Add information about how to expire files to the index
- Update the README with information about the new script

Squashed commits:

Add a note explaining that expired files aren't immediately removed

Show correct times on the index page graph

Improve the migration script, removing the need for --legacy

Use automap in place of an explicit file map in migration

Remove vestigial `touch()`

Don't crash when upgrading a fresh database

Remove vestigial warning about legacy files

More efficiently filter to unexpired files when migrating

https://git.0x0.st/mia/0x0/pulls/72#issuecomment-224

Coalesce updates to the database during migration

https://git.0x0.st/mia/0x0/pulls/72#issuecomment-226

Remove vestigial database model

https://git.0x0.st/mia/0x0/pulls/72#issuecomment-261

prune:  Stream expired files from the database

(as opposed to collecting them all first)

config.example.py:  Add min & max expiration + description
This commit is contained in:
Emi Simpson 2022-11-22 16:15:50 -05:00 committed by Mia Herkt
parent 00dba0e189
commit af4b3b06c0
No known key found for this signature in database
GPG key ID: 72E154B8622EC191
6 changed files with 269 additions and 57 deletions

View file

@ -1,44 +1,8 @@
#!/usr/bin/env python3
"""
Copyright © 2020 Mia Herkt
Licensed under the EUPL, Version 1.2 or - as soon as approved
by the European Commission - subsequent versions of the EUPL
(the "License");
You may not use this work except in compliance with the License.
You may obtain a copy of the license at:
https://joinup.ec.europa.eu/software/page/eupl
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
either express or implied.
See the License for the specific language governing permissions
and limitations under the License.
"""
import os
import sys
import time
import datetime
from fhost import app
os.chdir(os.path.dirname(sys.argv[0]))
os.chdir(app.config["FHOST_STORAGE_PATH"])
files = [f for f in os.listdir(".")]
maxs = app.config["MAX_CONTENT_LENGTH"]
mind = 30
maxd = 365
for f in files:
stat = os.stat(f)
systime = time.time()
age = datetime.timedelta(seconds=(systime - stat.st_mtime)).days
maxage = mind + (-maxd + mind) * (stat.st_size / maxs - 1) ** 3
if age >= maxage:
os.remove(f)
print("This script has been replaced!!")
print("Instead, please run")
print("")
print(" $ FLASK_APP=fhost flask prune")
print("")
exit(1);