Archive viewers and viewing solutions

[mootari]

I assume that slackdump is used for a large variety of cases, most of them being related to archival or public access.

I'd like to use this discussion to:

• get a better understanding of the use cases
• explore existing archive viewing solutions and their technical limitations
• explore workarounds, e.g. importing an archive into other chat solutions
• discuss data wrangling options (e.g. remove PII, reimport DMs as channels)

At some point some of these topics may need to be split out into separate discussions, but for the moment I'd like to keep them under one umbrella. Ultimately my hope is that I will be able to create a simple static viewer that can be bundled with an export, or a small and maintainable reference app.

[rusq]

Thanks @mootari, that's a great start. Right now everyone has to rely on the thirdparty solution, as obviously I lack frontend skills to produce something of value in this field. Over the time I have seen three viewers, I'll list them here for completeness:

1. Slack Export Viewer - Python/flask, most questions are related to this viewer. When I adjusted the export format in the past, I aimed for being compatible with it. Slackdump still has some compatibility issues that cause it to crash and burn
2. SlackLogView - C++/QT5. Extremely fast and neat viewer, very responsive author and contributors. It is written in C++, and currently does not support viewing of DMs, I was planning to open a couple of suggestions, namely - (a) support for DM/MPIM previews and mattermost file format support, so it would be able to preview the files straight away.
3. slack viewer - JS/node, learned about it recently from this issue. It does not seem to be recently maintained, last change was around 2 years ago, according to the master branch. Haven't tried it personally but it is reported to work with some minor changes.

[rusq]

@thayakawa-gh the issue with folder naming is fixed in v2.2.1. Thanks for flagging it!

I'm sorry I still haven't checked the SlackLogViewer, but I'll let you know when I do!

[rusq]

@thayakawa-gh playing around with SlackLogViewer — it works great! even shows the reaction emojis.

• Threads are working
• Files are expected not to work for now, but they are shown
• The DMs are displayed correctly, search within DMs works
• All UI interface elements are working on macOS without any adventures :)

Overall - great work! I think if you're planning to release it, it's good to go - I'll link to SlackLogViewer from the main README. btw, I really enjoy the "All channels" search function - it turns an export file into a searchable knowledge base. And also, the overall speed of the SlackLogViewer is amazing, comparing to the modern web-apps, which inevitably lagging due to their nature.

I still need to think about the approach which we discussed earlier, will share my thoughts, once I come up with something.

[thayakawa-gh]

Many thanks for fixing the issue and testing with SlackLogViewer.
I'd like to test v2.2.1 too, but I have no time to do so at all for a while. Sorry, but please wait a week or so. I'll also release the new version of SlackLogViewer at that time.

[thayakawa-gh]

I tested slackdump v2.2.2 with my workspace including some Katakana channel names and confirmed that they were correctly dumped.
Excellent! Thank you for fixing, this would be a great help for all Japanese users.

I also confirmed that files in exports dumped with -export-type mattermost and -export-token could be shown in SlackLogViewer.

I have released v1.2.Alpha-0 including a minor bug fix. It would be helpful if you could change the link from the actions to the top or release page.

[rusq]

Hi @thayakawa-gh,

I'm glad that it works and that it will make life easier for Japanese users! I have linked the releases page in the readme and docs, so that users would get the latest version (merged to master, see #161).

Thank you for testing and your feedback, greatly appreciated!

[mootari]

Data migration

• Rocket.Chat can import corporate exports that include DMs (more info here or in the PR referenced at the end).
• Mattermost can import corporate Slack exports directly as well (issue)
• Discord: There are contrib solutions (e.g. Slackord, slack2discord)

[rusq]

I did some digging on the mattermost in scope of #107, there's a bulk import format. As the structures are open source, and part of mattermost-server, which looks compatible with GPL 3, so at some point I was planning to use that structures to add the support for direct export to Mattermost bulk format. But I'm still undecided whether I should leave this part to the user, and just limit the scope of Slackdump to slack only, to avoid the burden of supporting another format in the future, especially that there's a mmetl tool that does the job, with some quirks though requiring manual effort of shuffling files around, which could be a quick fix to apply to the mmetl tool itself, but i got the impression that the support of that tool is not a priority for the mattermost team, looking at the size of the mattermost project, they have plenty other things to take care of :)

[mootari]

I'm not familiar with either format, but mmetl seems to take the right approach. I would probably recommend to focus on Slack's export format (which would likely be enough work in itself, given that more features might be added to Slack in the future), and contribute to mmetl if necessary.

It may also be worth asking who's going to benefit from a direct export to Mattermost from slackdump. I would expect such a conversion to be a one-time step in a migration from Slack to Mattermost, not a repeatedly performed action. Conversion guides and scripts could be documented in this repo's wiki and (hopefully) be maintained by the community.

[mootari]

... In the long run it would be awesome to have some automated testing capabilities:

1. A workspace with some test data (archived channels?) and a couple of test users
2. A reference export that is used to spot any changes in the export structure.
3. Puppeteer based export perhaps, and import into other tools like e.g. Mattermost.

This is admittedly very hand wavy and probably a lot (like, a lot) more complicated than I make it sound here.

[rusq]

@mootari very good point on testing! Right now there's no integrational tests, only unit, and coverage is less than I'd like it to be, but more than the threshold that was giving me the itch 😂

[mootari]

Browsing UIs

• linen exposes workspaces to Google. It has channel selection and thread displays (example).
• slack-archive exports messages and makes them accessible in a static UI (apparently? haven't verified yet). Seems we should be able to replace data-load.js with an adapter for slackdump exports (if the file format doesn't already match). Most promising solution so far.
• Block Kit is Slack's own style guide and component library for Slack apps.

Considerations:

• Display of DMs could emulate the Slack UI.
• In theory all data could be made available as static JSON, but that may require an insane amount of data to download and keep in memory. Alternative storage formats like Apache Arrow might be more suitable here.

[mootari]

I've blocked some time on Friday to look into slack-archive. Curious to see how "static" their static HTML is. At first glance it seems they use React at least in an intermediate step (maybe along with some SSR).

[mootari]

Block Kit may be a better fit to handle Slack's message storage format. I'll investigate that one first.

[mootari]

Sadly it seems that Block Kit is mostly a proprietary API for Slack apps that does not include all UI elements required to recreate the entire Slack UI.

[mootari]

I don't think I want to use any module from slack-archive as-is. The code is heavily interleaved with console/progress stuff. I'll probably extract or port their components instead. They reference interfaces from @slack/wep-api, so I expect compatibility with slackdump's export to be high.

[mazerj]

I was just playing around with slack-export-viewer (flask-based solution #1 above) and noticed that it mostly works, but looks for attachments in the wrong location. Or rather, slackdump puts them in a different directory than the viewer expects them to be. Not sure if this is a change in the format, but figured I'd ask before I tried to fix or patch a solution.
To be specific. slack-export-viewer is looking for attachments associated with DMs in: http://localhost:8080/dm/<ID>/attachments/FKW3PMKNH-mlab-bulb2.png
and (as far as I can tell) the correct URL should be: http://localhost:8080/<ID>/attachments/FKW3PMKNH-mlab-bulb2.png (that is w/o the dm subdir.
Not sure this whether this is a slack-export-viewer or slackdump issue. Any thoughts before I try to change something?

[rusq]

Slackdump has slackdump view command that allows to view archives, since v3.0.

[volker-fr]

There are still benefits from slack-export-viewer. It actually has two features: web server & export to a html. The export allows templating. As result I can export every week all new posts, including showing new messages in old threads.

[rusq]

There's no doubt, I quite like it myself. To me the benefit of having a built in viewer is to have a control over features, bugs... When someone creates an issue that X can't view images, I can always point to built-in viewer that can. None of the external viewers support the database format, and I doubt anyone would be interested or have time to add it. But for native exports these tools are indispensable.