Drupal Meets Matrix 03 - Better Conversations with Advanced Comment Threads

Commenting in Drupal hasn't noticeably changed in 15 years. That changes now.

Today, with the new Advanced Comment Threads (ACT) module, commenting in Drupal is as pleasant as any much more polished social media platform - and nicer than most.

It's a simple drop-in replacement for the built-in comment list field formatter. Install the module, go to manage display, select "Advanced Comment Threads" next to your comment field, and you instantly have a vastly improved comment authoring experience, no other configuration necessary.

But that's the end of the story - what has gone into this new module?

Why a new module?

In the previous video, we put together a basic polling system to make it so new comments appear on the page as they get posted into Drupal, and set up polling to make this happen every 10 seconds or so, with no custom code required. That in itself is pretty slick -- but the user experience left much to be desired:

• Polling needed to be paused while you write a new comment, or else you could lose your message.
• Posting a comment resulted in a full page reload.
• The new Drupal Canvas system had no support whatsoever for showing or using comments.
• There's basically no existing module I could find that substantially improves the commenting experience in Drupal.

As part of the Drupal Groups initiative, we're trying to sync live chat with comments on the new Drupal Groups website -- we need a way to make comments show up as they arrive, live on the page. And replacing the whole comment tree with every new comment just isn't going to cut it.

In this episode, I replace the polling recipe with ACT, show Matrix and Drupal sharing a conversation, and build a Canvas content template with bubble-style comments:

Performance challenges

At this writing, ACT has been through 4 alpha releases and 3 beta releases. The alphas largely focused on user experience. The betas have been focused on solidifying the internals to make sure this solution will scale.

Drupal is a tremendously complex application, and the fact that most Drupal sites respond quickly is a testament to its sophisticated caching strategies. When you visit a page, most elements of the page are built up from many objects that get calculated once and then stored into various caches for different users.

Different users have different views of the same content -- a moderator may see edit and delete links, authors of a comment may also have additional links that other users don't see. So a small site with a handful of users is no big deal here -- but that's not my only target. I'm building this for some internal project management tools with low traffic -- but also for Drupal.org, with tens of thousands of users who might be online at any one time. If thousands of users have browsers that are checking for new messages every 10 seconds, that's a bunch of new load that servers need to handle -- making pages suddenly have potentially hundreds of individual bits of content (comments) that have different rendering for different users, and with a different combination of comments already loaded in the browser -- we have a complex data structure that needs to get modeled to make this work. And if we push out a new comment on a page with thousands of open browsers, we want that to "just work".

So most of the work in the beta process has been making the data model and caching structures robust enough to work on sites with hundreds or thousands of users, on pages with hundreds or thousands of comments.

There is a page on this site with over 200 comments that has been a great test case I've used to sort out performance issues. I've built 3 different loading strategies into the module so we have options to test on real sites. I’ve kept the base module easy to deploy on small sites, while designing its shared data model, caching, and extension points with high-traffic sites in mind. Three submodules are where that scale gets help:

• _push - Notifies connected browsers when a discussion changes, through a running delivery service such as Mercure. Rather than checking every ten seconds for updates that usually aren’t there, browsers can react to a notification and fetch fresh content. Push spreads and coalesces those refresh requests, and retains periodic safety checks to recover missed updates. The current payloads contain lightweight metadata rather than complete comment bodies.
• _background - Prepares shared discussion snapshots outside web requests, helping readers avoid expensive cold-cache rebuilds. It provides a durable outbox, coalescing, and retry handling, with Drupal queues as the default execution backend. When Push is also enabled, notification preparation and publication can run through these workers too. This needs cron or a continuously running dispatcher and worker.
• _messenger - An alternative execution backend for Background, using Symfony Messenger and the site’s asynchronous transports. It allows separately supervised workers to process ACT jobs through infrastructure such as Redis or AMQP. It carries work references, not comment bodies, and Background remains usable without it.
• Content Push - (planned) - A further integration could deliver authorized comment content directly and update the page without an extra fetch for each supported change. HTTP reconciliation would remain available for recovery and updates that cannot be applied directly.

Much of this planning has come directly from the results of other performance work we've been doing, dealing with bot attacks on slow sites.