Aggregate Stats

A smithy's order book that every visitor writes to. The counts, the coin and the queue updates with each order whether by a visitor or a background job.

The smithy's order book

Waiting 7
At the forge 2
Done 3
Work in hand
5,533 gp
Ready to be paid for
465 gp
Dearest order open
Silvered Rapier, 2,100 gp

Next in line

  • Chain Hauberk at the forge 1,100 gp
  • Black Iron Axe at the forge 880 gp
  • Hatchet waiting 25 gp
  • Hunter's Spear waiting 520 gp
  • Steel Kiteshield waiting 700 gp
Order 

# frozen_string_literal: true

# One visitor's reading of the smithy's order book. The states are a number
# that goes up when anyone writes to the book, and the kind this tab looks
# at. Every figure is a derived that reads them and asks the database
# again. The actions that write an order say nothing about the page.
class Recipes::Stats::StatsChannel < ApplicationCable::Channel
  include Hibiki::Rails::Channel

  Order = Recipes::Stats::Order
  Book = Recipes::Stats::Book
  CaravanJob = Recipes::Stats::CaravanJob
  VIEWS = "app/recipes/stats"

  def build_graph
    @version = Hibiki::State.new(0)
    @kind = Hibiki::State.new(Book::ALL)
    @caravan = Hibiki::State.new("")

    @book = Hibiki::Derived.new { Book.new(kind: @kind.value) }

    # Each figure reads the version, so each broadcast makes it ask again, and
    # answers with plain values, so an answer equal to the last one stops
    # there. The kinds do not read the filter; the other three do.
    kinds = Hibiki::Derived.new { @version.value; Book.new.kinds }
    statuses = Hibiki::Derived.new { @version.value; @book.value.statuses }
    coin = Hibiki::Derived.new { @version.value; @book.value.coin }
    queue = Hibiki::Derived.new { @version.value; @book.value.queue }

    Hibiki::Effect.new do
      broadcast_morph target: "stats_kinds", partial: "#{VIEWS}/kinds",
                      locals: { kinds: kinds.value, kind: @book.value.kind }
    end

    Hibiki::Effect.new do
      broadcast_morph target: "stats_statuses", partial: "#{VIEWS}/statuses", locals: { statuses: statuses.value }
    end

    Hibiki::Effect.new do
      broadcast_morph target: "stats_coin", partial: "#{VIEWS}/coin", locals: { coin: coin.value }
    end

    Hibiki::Effect.new do
      broadcast_morph target: "stats_queue", partial: "#{VIEWS}/queue", locals: { queue: queue.value }
    end

    transmit_value(:stats_caravan) { @caravan.value }
  end

  def filter(data) = @kind.value = data["kind"].to_s

  def place(data) = Order.place(data["kind"])

  # Moves on the next order at a step, among the kind this tab looks at.
  def advance(data)
    @book.value.orders.advance!(data["from"]) if Order::STATUSES.first(2).include?(data["from"])
  end

  def collect = @book.value.orders.collect!

  def caravan
    @caravan.value = CaravanJob.dispatch ? "A caravan is on its way." : "A caravan is already on the road."
  end

  private

  def subscribed
    super
    return if subscription_rejected?

    # A write by anyone: this tab, another tab, a job. The block runs on a
    # cable thread, so it hops onto the graph thread before touching a
    # signal.
    stream_from Order::CHANGED do
      graph_actor&.post { @version.value += 1 }
    end
  end
end

The panel above is a smithy's order book. Place an order, such as "A weapon", and you can see a new order land in the book and every figure it touches update. Try opening this page in a second window and put the two side by side: a write in either window moves the figures in both. "Send a caravan" starts a background job that places five orders a second apart, and both windows count each one as it lands. Here the model says only that something changed, and each open page works its own figures out again.

The book is shared by every visitor, so the numbers may move while you watch. The orders are reset every night.

How it fits together

The book lives in a channel: the server side of an Action Cable connection, an object that stays alive while the page is open, one for each tab. A state holds a value and remembers who read it. Here @version is a state holding a number that goes up whenever anyone writes to the book, and @kind is a state holding the kind this tab is looking at. The channel's public methods are its actions, the methods the page may call. place, advance and collect write an order to the table and return; filter writes the chosen kind to @kind.

A derived is a value worked out from other values, and worked out again when one of them changes. Here each figure on the page is a derived. It reads @version first and then asks Book, a plain value object with a method for each figure. The book asks the database and answers with plain values: a hash of counts, a sum, a list of rows. The page renders its first figures from the same book, so the first render and every later one agree.

An effect is a block of code that runs once, and runs again whenever a value it read has changed. There are four, one for each figure. Each renders its part of the panel and sends it to the page, where it is morphed into place. This is what happens when you press "Fire one up":

  1. The button calls advance, which moves the oldest waiting order to the forge. After the commit the model broadcasts an empty message on one stream (announce).
  2. Every open page receives that broadcast and adds one to its @version.
  3. The four deriveds read @version, so each asks the book again. The step counts come back different, and so does the queue, where the order's badge now says "at the forge".
  4. The effects that read a changed answer run again, and the counts and the queue are morphed in.

We never wrote in the channel which write moves which figure: every figure works itself out through reactivity.

The announcement

The model's work is minimal. After each commit it broadcasts an empty message, which says only that something changed, and leaves each page to work out what that means for the figures it shows: 

# The model. A commit of any kind broadcasts an empty message.
CHANGED = "recipes:stats:orders"

after_commit { self.class.announce }

def self.announce = ActionCable.server.broadcast(CHANGED, {})

# The channel. Every open book listens, and each broadcast adds one to a number.
def subscribed
  super
  return if subscription_rejected?

  stream_from Order::CHANGED do
    graph_actor&.post { @version.value += 1 }
  end
end

Placing an order, moving one on and handing one over are a create, an update and a destroy; one after_commit covers all three.

Only changed figures are sent

After each broadcast, every derived asks the database again and compares the new answer with its last one, using ==. If the two are equal, the effect that reads it does not run, and nothing is sent for that figure. This works only when == compares what the figure shows. Hashes, numbers and Data rows do: two of them are equal when their contents are. A relation does not. Two relations are equal when their queries are, and the query is the same every time, so a derived holding one would never see a change. 

@version = Hibiki::State.new(0)
@kind = Hibiki::State.new(Book::ALL)

@book = Hibiki::Derived.new { Book.new(kind: @kind.value) }

# Reading the version is what makes a figure ask again after each broadcast.
kinds = Hibiki::Derived.new { @version.value; Book.new.kinds }
statuses = Hibiki::Derived.new { @version.value; @book.value.statuses }
coin = Hibiki::Derived.new { @version.value; @book.value.coin }
queue = Hibiki::Derived.new { @version.value; @book.value.queue }

# One of the four effects. It runs when `coin` gives a different answer,
# and at no other time.
Hibiki::Effect.new do
  broadcast_morph target: "stats_coin", partial: "#{VIEWS}/coin", locals: { coin: coin.value }
end

Each tab counts through its own filter

A Turbo broadcast sends the same HTML to everyone, so a visitor who wants to see one kind at a time needs figures of their own. Here the kind is a state in the tab's own channel, and three of the figures read it through @book. Choosing Weapons asks those three again, for weapons only, and leaves the counts on the kind buttons alone, since they count the whole book. Every tab receives the same announcement, and each works out its own figures from it. The buttons under the counts go through the filter too: with Weapons chosen, "Fire one up" takes the next weapon and is disabled when no weapon is waiting.

Writing from outside a channel

The channel's actions write to the table and return without telling the page anything. The caravan is a background job that knows nothing about the channel, and its orders still show up on every open page. This works because the model sends the broadcast after each commit, whoever made it. A console, a controller or a webhook could write orders too, and the page would follow: 

# The channel's actions. Each writes to the table and returns.
def place(data) = Order.place(data["kind"])
def collect = @book.value.orders.collect!

# The job. It has never heard of the channel.
def perform(pause: 1)
  LOAD.times do |n|
    sleep pause if n.positive?
    Order.place(Order::KINDS.sample)
  end
end

The broadcast is sent from an after_commit callback, so a write that skips callbacks sends none. The nightly reset clears the table with delete_all, which runs no callback, so it calls announce itself. Placing an order also deletes the oldest rows with delete_all, in the same transaction. Only the new order runs the callback, so the page hears one broadcast and the figures move once: 

# `delete_all` and `insert_all` run no callback, so `announce` is called
# by hand.
def self.reset!
  transaction do
    delete_all
    seed!
  end
  announce
end

# A new order and the rows it pushes out are one transaction, so they are
# one commit and one broadcast.
def self.place(kind)
  # ...
  transaction do
    create!(item: name, kind:, price:).tap { prune! }
  end
end

A job usually runs in another process. For its broadcast to reach the page, the app needs a cable adapter that works across processes, such as Solid Cable or Redis.

One more figure

Suppose the smith wants the average price of an open order on the page. That is a method on the book, a derived in the channel and a line on the page. The model, the actions and the job stay as they are: 

# Recipes::Stats::Book: the question.
def average = orders.unfinished.average(:price).to_i

# The channel, in build_graph: who asks it, and when.
average = Hibiki::Derived.new { @version.value; @book.value.average }
transmit_value(:stats_average) { Book.gp(average.value) }

# The page: where the answer goes.
#   <%= reactive :stats_average, Book.gp(book.average) %>

The same feature in stock Rails

broadcasts_to sends one record's partial, and a count is not in any record. So the usual stock Rails build has a job that works every figure out and broadcasts each one. Here is a sketch of it. It is not run here, and it leaves out Book, which would be the same in both versions, the partials and the caravan's job. 

class Order < ApplicationRecord
  STATUSES = %w[waiting forging done].freeze
  KINDS = %w[weapon armour tool].freeze

  # `broadcasts_to` would send this row's partial. No row holds a count,
  # so the counting is a job of our own, enqueued after every write.
  after_commit { StatsBroadcastJob.perform_later }

  scope :unfinished, -> { where.not(status: "done") }

  def self.of(kind) = KINDS.include?(kind) ? where(kind:) : all

  def self.place(kind)
    # ...as in the Hibiki version
  end

  # No callback runs after `delete_all` or `insert_all`, so the job is
  # enqueued by hand.
  def self.reset!
    transaction do
      delete_all
      seed!
    end
    StatsBroadcastJob.perform_later
  end
end

The counting is as short as before. What is new is a list, kept by hand in the job, of every figure on the page and the locals each one takes, and a second copy of it in the view. The table lists each behaviour of the demo above and where it comes from.

Stock RailsHibiki
A count follows a write by anyone A callback, a job, and a broadcast line in the job for each figure A callback that broadcasts an empty message, and one state that counts them
Which figures a write moves All of them are rendered and sent every time Each derived compares its answer; an equal one sends nothing
The first render and the later ones agree The locals are listed twice, in the view and in the job The page and the channel ask the same four methods
A filter for one visitor A stream for each filter, every figure rendered for each on every write, and the filter carried in the address and in each form One state in the tab's graph
A filter with free input, such as a price range Cannot be a stream; the page has to be refreshed instead The same one state
One more figure A method, a partial, a line in the view and a line in the job A method, a derived and a line in the view
A write that runs no callback Enqueue the job by hand Call announce by hand
A caravan of five orders, nobody looking Five jobs and eighty renders Five empty broadcasts that nobody receives
A write with a hundred tabs open Rendered once for each stream and shared Each tab asks the database its four questions

The last row is the honest cost. A broadcast renders once and everyone gets the copy; here every open tab counts for itself, which is what lets each one count something different. In your real-world app, you'd want to keep the queries cheap and indexed, and if the writes come in hundreds, announce once after the batch instead of once a row.

Stock Rails has a second answer with broadcasts_refreshes. The model says the page is stale, and every open page requests itself again and morphs in the difference. That is one line, it needs no list of figures, and the filter works because each page asks with its own address. For a small page it is a good answer. Its cost is in what a write sets off: a full request from every open tab, through the router, the controller and the layout, to render the whole page for the sake of three numbers, and whatever the visitor has chosen has to be in the address for the page to come back the same. Here a write sets off the four questions and sends the parts whose answers changed.