Compact/optimize events.db

[pe1uca]

Mostly a question but also a request if possible.

What would be the side effects of compacting the events' db?
By this I mean running a script to recreate the db with multiple lines for the same file into a single line.

From this

{"type":"userAction","id":"0717e48d-bfb1-458a-9518-7a87f8ad43c6","targetIds":["2fbf9989d3aca877183dbff5b38203299f38fb5d"],"actions":[{"action":"addTag","value":"phone wp"}],"date":"2025-05-26T13:27:10.905Z"}
{"type":"userAction","id":"aa6d1576-8e0c-4d0e-883e-d00d25238f9e","targetIds":["2fbf9989d3aca877183dbff5b38203299f38fb5d"],"actions":[{"action":"addTag","value":"black"},{"action":"addTag","value":"aztec"}],"date":"2025-05-26T13:31:07.145Z"}

To this

{"type":"userAction","id":"0717e48d-bfb1-458a-9518-7a87f8ad43c6","targetIds":["2fbf9989d3aca877183dbff5b38203299f38fb5d"],"actions":[{"action":"addTag","value":"phone wp"},{"action":"addTag","value":"black"},{"action":"addTag","value":"aztec"}],"date":"2025-05-26T13:27:10.905Z"}

It would be great to have a command like this gallery.js events compact for this process.

My use case is to use the data from home-gallery into another project.
I know I would extract the data into xmp files, but that also means reprocessing all the directories, so I'd like to leverage as much as possible the processing home-gallery has already done.

[xemle]

Hi @pe1uca

My use case is to use the data from home-gallery into another project.

Did I read correctly, you like to tag photos with HomeGallery and would like to use these tags in another project? If so, have you checked the gallery.js export meta command to dump all your tags to standard xmp sidecar files (see export meta docs)

I know I would extract the data into xmp files, but that also means reprocessing all the directories

At least for HomeGallery that should be not an issue. If these xmp files are written by another project the tags can be imported to HomeGallery, too.

If the xmp sidecar is not a solution for your, what would be your benefit to compact the event.db file? You could compact it, it should have not side effect so far as long as the format of ndjson is kept and the shape of event objects is not changed.

[pe1uca]

As mentioned, I'd like to leverage as much as possible the processing already done by HomeGallery.
If there's already an easy way to get all the tags of the files I don't see the reason of exporting each index and then to have another project processing all the files in each index.

Reading the DB files of HomeGallery seems very productive as I also won't have to reimplement a way to watch changes to each index plus configure each of them in the other project.
So I can just read database.db and get all the files present in my HomeGallery instance, then process events.db to get all the custom tags.

---

what would be your benefit to compact the event.db file?

I'm thinking it could speed up reading the file and applying the modifications to the tags.
As mentioned, instead of having multiple lines per event, could be just one, reducing the amount of lines to parse into JSON and reads from disk.
Also, the "removeTag"s which cancel any "addTag"s could be removed, again, speeding up the process of all the files.

[xemle]

I don't see the reason of exporting each

Exporting your tags (as your work) to a standard xmp sidecar file would unlock the "vendor lock" of HomeGallery. IMHO this strategy is better than understanding (and optimizing) implementation details of a "foreign" 3rd party software.

I am more than happy if you could provide a compact command for events as PR. No doubt. But I am not convinced yet of the need for such command.

On think would be great to export the tags as xmp sidecar file and than remove/purge these tags from events, so you can get rid of them - except you would like to get some meta info from the event, e.g. kind of history when which photo was edited...

I'm thinking it could speed up reading the file and applying the modifications to the tags

Do you have facts regarding files that the reading and appending is a major issue in your processing? Could you post times?

IMHO yes, by theory this particular issue could be improved. Considering the whole context of the gallery I guess this might not be a major issue in time and performance.

Feel free to add features to the gallery which you miss as PR. If requested I could give support getting into the coding part. And please do not hesitate to ask further questions.