Add support for expiring files

SUPPLEMENTALLY:
- Add an `expiration` field to the `file` table of the database
- Produce a migration for the above change
- Overhaul the cleanup script, and integrate into fhost.py
  (now run using FLASK_APP=fhost flask prune)
- Replace the old cleanup script with a deprecation notice
- Add information about how to expire files to the index
- Update the README with information about the new script

Squashed commits:

Add a note explaining that expired files aren't immediately removed

Show correct times on the index page graph

Improve the migration script, removing the need for --legacy

Use automap in place of an explicit file map in migration

Remove vestigial `touch()`

Don't crash when upgrading a fresh database

Remove vestigial warning about legacy files

More efficiently filter to unexpired files when migrating

https://git.0x0.st/mia/0x0/pulls/72#issuecomment-224

Coalesce updates to the database during migration

https://git.0x0.st/mia/0x0/pulls/72#issuecomment-226

Remove vestigial database model

https://git.0x0.st/mia/0x0/pulls/72#issuecomment-261

prune:  Stream expired files from the database

(as opposed to collecting them all first)

config.example.py:  Add min & max expiration + description
This commit is contained in:
Emi Simpson 2022-11-22 16:15:50 -05:00 committed by Mia Herkt
parent 00dba0e189
commit af4b3b06c0
No known key found for this signature in database
GPG key ID: 72E154B8622EC191
6 changed files with 269 additions and 57 deletions

View file

@ -11,6 +11,15 @@ Or you can shorten URLs:
File URLs are valid for at least 30 days and up to a year (see below).
Shortened URLs do not expire.
Files can be set to expire sooner by adding an "expires" parameter (in hours)
curl -F'file=@yourfile.png' -F'expires=24' {{ fhost_url }}
OR by setting "expires" to a timestamp in epoch milliseconds
curl -F'file=@yourfile.png' -F'expires=1681996320000' {{ fhost_url }}
Expired files won't be removed immediately, but will be removed as part of
the next purge.
{% set max_size = config["MAX_CONTENT_LENGTH"]|filesizeformat(True) %}
Maximum file size: {{ max_size }}
Not allowed: {{ config["FHOST_MIME_BLACKLIST"]|join(", ") }}
@ -22,7 +31,7 @@ FILE RETENTION PERIOD
retention = min_age + (-max_age + min_age) * pow((file_size / max_size - 1), 3)
days
365 | \\
{{'{: 6}'.format(config.get("FHOST_MAX_EXPIRATION", 31536000000)//86400000)}} | \\
| \\
| \\
| \\
@ -30,7 +39,7 @@ retention = min_age + (-max_age + min_age) * pow((file_size / max_size - 1), 3)
| \\
| ..
| \\
197.5 | ----------..-------------------------------------------
{{'{: 6.1f}'.format((config.get("FHOST_MIN_EXPIRATION", 2592000000)/2 + config.get("FHOST_MAX_EXPIRATION", 31536000000)/2)/86400000)}} | ----------..-------------------------------------------
| ..
| \\
| ..
@ -39,7 +48,7 @@ retention = min_age + (-max_age + min_age) * pow((file_size / max_size - 1), 3)
| ...
| ....
| ......
30 | ....................
{{'{: 6}'.format(config.get("FHOST_MIN_EXPIRATION", 2592000000)//86400000)}} | ....................
0{{ ((config["MAX_CONTENT_LENGTH"]/2)|filesizeformat(True)).split(" ")[0].rjust(27) }}{{ max_size.split(" ")[0].rjust(27) }}
{{ max_size.split(" ")[1].rjust(54) }}
</pre>