aboutsummaryrefslogtreecommitdiffstats
path: root/util/datamanager.lua
Commit message (Collapse)AuthorAgeFilesLines
* util.datamanager: Always reset index after list shiftKim Alvefur2023-07-221-39/+4
| | | | | | Shifting the index does not work reliably yet, better to rebuild it from scratch. Since there is minimal parsing involved in that, it should be more efficient anyway.
* util.datamanager: Add way to close indexed list storeKim Alvefur2023-07-211-1/+9
|
* util.datamanager: Close file handle when done using itKim Alvefur2023-07-211-0/+1
| | | | | It gets closed eventually but at high load they could potentially lead to reaching FD limits faster.
* util.datamanager: Disable blockwise removalKim Alvefur2023-07-211-1/+4
| | | | In desperate need of tests
* util.datamanager: Disable block alignmentKim Alvefur2023-07-211-0/+2
| | | | | Until we have more test coverage. Somehow the index becomes incorrect after inserting padding, unclear why.
* util.datamanager: Pad list writes to avoid crossing block boundariesKim Alvefur2023-06-071-0/+7
| | | | | | | | | | | By padding items so that they do not cross block boundaries, it becomes eaiser to delete whole blocks with fallocate() without cutting items in half, improving efficiency of such operations. Since list stores are used for message archives, where the most common deletion operation would be of the oldest entires, at the top of the file. With this, all blocks that contain items to be removed could be deleted without needing to read, delete and write out the whole file.
* util.datamanager: Efficiently remove whole blocks to shift listsKim Alvefur2023-07-121-0/+40
| | | | | Using the new pposix.remove_blocks() it should be very performant to delete whole sections of a file, given a supporting file system.
* util.datamanager: Add way to efficiently remove first items in a listKim Alvefur2023-07-121-0/+92
| | | | | Copying data without parsing it should be more performant than parsing it serializing back.
* util.datamanager: Fix indexing first item if not at the very startKim Alvefur2023-07-101-2/+2
| | | | | | | | If the first item does not start at position 0 then the index function produces a phantom first entry covering position zero until where the real first item starts. When using the index, this would make it either appear as the first item was missing or cause an off-by-one issue with remaining items.
* util.datamanager: Reduce log level of left over debug messages to debug ↵Kim Alvefur2023-07-121-2/+2
| | | | | | (thanks Trung) These were mostly 'warn' to make them stand out from the debug noise
* util.datamanager: Fix missing separator in log lineKim Alvefur2023-07-101-1/+1
|
* util.datamanager: Halve size of list indexKim Alvefur2021-05-141-11/+15
| | | | | Instead of storing (start, length) tuples, store the offset to the end of items and derive length using the previous entry.
* util.datamanager: Add O(1) list indexing with on-disk indexKim Alvefur2021-05-111-0/+191
| | | | | | | | | | | | Index file contains offsets and lengths of each item() which allows seeking directly to each item and reading it without parsing the entire file. Also allows tricks like binary search, assuming items have some defined order. We take advantage of the 1-based indexing in tables to store a magic header in the 0 position, so that table index 1 ends up at file index 1.
* util: Prefix module imports with prosody namespaceKim Alvefur2023-03-171-4/+4
|
* spelling: non-existing mistakes (thanks timeless)0.12.0Kim Alvefur2022-03-071-2/+2
|
* Spelling: Fix various spelling mistakes (thanks timeless)Kim Alvefur2022-03-071-2/+2
| | | | | | Words, sometimes I wonder how they even work Maybe I missed something.
* util.datamanager: Support iterating over any file extensionKim Alvefur2021-02-071-4/+3
| | | | | | | | The 'typ' argument to all other functions is the actual file extension, but not here for some reason. May need this for iterating over the .bin files created by mod_http_file_share in the future.
* util.datamanager: Fix not passing variables as format string to logger ↵Kim Alvefur2021-01-161-1/+1
| | | | | | | (thanks semgrep) Passing error messages as format string could lead to errors, although not a problem anymore since util.format sanitizes this.
* Merge 0.11->trunkKim Alvefur2020-02-241-1/+1
|\
| * util.datamanager: Fix iterating over "users" (thanks marc0s)Kim Alvefur2020-02-241-1/+1
| | | | | | | | | | | | | | | | | | The 'store' path componend used to be unescaped until 756a2a00e7e7 added escaping to address issues with characters like '/' used in PEP, but with a special case for '_' which was already in common use in 'store' path components. Missed adding this escaping here.
* | util.datamanager: Ignore unused 'errno' variable [luacheck]Kim Alvefur2019-12-231-1/+2
| |
* | Fix various spelling mistakes [codespell]Kim Alvefur2019-05-031-1/+1
|/
* vairious: Add annotation when an empty environment is set [luacheck]Kim Alvefur2018-02-281-0/+1
|
* util.datamanager: Encode the 'store' path component, preserving underscoresKim Alvefur2017-10-081-1/+8
|
* util.datamanager: Use already known index instead of measuring length each ↵Kim Alvefur2017-04-171-2/+2
| | | | iteration
* util.datamanager: Log where the error came fromKim Alvefur2017-03-281-3/+4
|
* util.datamanager: Rearrange localsKim Alvefur2017-03-281-3/+2
|
* util.datamanager: Use pposix.atomic_appendKim Alvefur2017-02-281-23/+17
|
* util.datamanager: Remove check for file non-existence since ENOENT tells us ↵Kim Alvefur2017-03-221-18/+4
| | | | that already
* util.datamanager: Split long line [luacheck]Kim Alvefur2017-03-221-1/+3
|
* util.datamanager: Expose atomic_store as store_rawKim Alvefur2017-03-011-0/+1
|
* util.datamanager: Use atomic store function when writing the first entry of ↵Kim Alvefur2017-03-011-4/+1
| | | | a list
* util.datamanager: Import value of ENOENT from util.pposix (if available)Kim Alvefur2017-03-011-0/+1
|
* util.datamanager: Ignore ENOENT (no such file) when loading dataKim Alvefur2017-03-011-4/+13
|
* util.datamanager: Shorter patternKim Alvefur2016-11-201-1/+1
|
* util.datamanager: Remove extraneous decoding of + to space, which we never ↵Kim Alvefur2016-11-201-1/+1
| | | | never encode to
* util.datamanager: Use the 'base' argument to tonumber() to indicate hexadecimalKim Alvefur2016-11-201-1/+1
|
* util.datamanager: Add annotations to ignore name clashes [luacheck]Kim Alvefur2016-09-241-5/+5
|
* util.datamanager: Rename variables to avoid name clashes [luacheck]Kim Alvefur2016-09-241-12/+12
|
* util.datamanager: Import tostring and type (fix global access)Kim Alvefur2016-05-221-0/+2
|
* util.datamanager: Explicit handling of each error condition (see #632)Kim Alvefur2016-02-271-14/+16
|
* Merge 0.9->0.10Kim Alvefur2016-02-251-0/+1
|\
| * util.datamanager: Unreference file handle after closing it to prevent trying ↵Kim Alvefur2016-02-251-0/+1
| | | | | | | | to close it again (fixes #632)
* | util.datamanager: Add append to public apiKim Alvefur2015-12-111-0/+1
| |
* | util.datamanager: Return extra location infoKim Alvefur2015-12-111-3/+3
| |
* | util.datamanager: Make sure only strings are passed as data to append()Kim Alvefur2015-12-111-0/+1
| |
* | util.datamanager: Add some comments about the append functionKim Alvefur2015-12-111-0/+3
| |
* | util.datamanager: Handle potential error from :write() callKim Alvefur2015-12-111-3/+3
| |
* | util.datamanager: Handle potential issues from fallocateKim Alvefur2015-12-111-1/+10
| |
* | util.datamanager: No shadowing of variable [luacheck]Kim Alvefur2015-12-111-1/+3
| |
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-06 10:24:24 +0000