Infinite Scroll

The town crier's chronicle, loading more as it is scrolled. Entries other visitors add wait, so the list never moves while you read, and the address bar remembers how far down you are.

The town chronicle

Showing 120 of 300

  • Quests A novice mage swore to clear the mine before supper.
  • Trade A Sunmere fishwife raised the price of ale at the market.
  • Weather Thick snow lay along the Sunmere road.
  • Trade The smith raised the price of ale at the east gate.
  • Weather A hard frost lay on the castle roofs.
  • Quests A lone squire came back from the cellar with three rat tails without a word.
  • Weather Thick snow settled across the cabbage fields.
  • Trade A Sunmere fishwife sold forty cabbages by the mill.
  • Gossip The gravedigger reckons the bridge troll pays rent though nobody believes it.
  • Weather Thick snow crept in around the campfire.
  • Weather A light rain lay across the cabbage fields.
  • Sightings A goblin was seen digging on the castle wall.
  • Weather A hard frost crept in over the pond.
  • Quests A lone squire went looking for a lost letter by lantern light.
  • Gossip The baker claims a dragon was seen and will not be told otherwise.
  • Trade The smith sold forty cabbages at the market.
  • Quests A party of adventurers took the bounty on the troll in the pouring rain.
  • Quests An old ranger returned empty-handed before supper.
  • Quests A novice mage swore to clear the mine by lantern light.
  • Quests An old ranger swore to clear the mine before supper.
  • Sightings A white stag was seen digging beside the pond.
  • Quests The cook's apprentice came back from the cellar with three rat tails by lantern light.
  • Sightings A white stag was caught dancing under the old bridge.
  • Gossip A boy from Mossford swears the cellar is haunted as everyone already knew.
  • Quests A novice mage set out to find the cabbage thief by lantern light.
  • Trade The smith traded a bronze dagger for a pie at the east gate.
  • Sightings A white stag was spotted dozing in the cabbage patch.
  • Quests A novice mage came back from the cellar with three rat tails to much applause.
  • Weather Fog rattled across the cabbage fields.
  • Gossip The baker heard the well grants wishes though nobody believes it.
  • Quests An old ranger returned empty-handed to much applause.
  • Gossip A boy from Mossford heard the well grants wishes and will not be told otherwise.
  • Weather Hail blew along the Sunmere road.
  • Gossip The baker says the king eats nothing but cabbage or so the cook says.
  • Sightings The pond's old pike was spotted dozing beside the pond.
  • Sightings A goblin was seen digging under the old bridge.
  • Weather Thick snow crept in along the Sunmere road.
  • Trade The smith raised the price of ale at the docks.
  • Gossip The gravedigger heard the well grants wishes and half the town agrees.
  • Quests A novice mage took the bounty on the troll at dawn.
  • Quests A party of adventurers set out to find the cabbage thief by lantern light.
  • Quests A novice mage went looking for a lost letter without a word.
  • Weather Hail crept in on the market stalls.
  • Quests The ratcatcher went looking for a lost letter to much applause.
  • Quests A novice mage took the bounty on the troll at dawn.
  • Gossip A boy from Mossford says the king eats nothing but cabbage though nobody believes it.
  • Weather Thick snow rattled along the Sunmere road.
  • Sightings Somebody's goat was caught dancing in the cabbage patch.
  • Quests A lone squire returned empty-handed to much applause.
  • Quests An old ranger came back from the cellar with three rat tails by lantern light.
  • Weather A hard frost rattled on the castle roofs.
  • Quests An old ranger returned empty-handed before supper.
  • Quests The cook's apprentice set out to find the cabbage thief to much applause.
  • Sightings A gnome was seen digging by the campfire.
  • Trade A Sunmere fishwife traded a bronze dagger for a pie on the Sunmere road.
  • Sightings The pond's old pike was caught dancing on the castle wall.
  • Gossip The miller's wife swears the cellar is haunted though nobody believes it.
  • Trade The miller raised the price of ale on the Sunmere road.
  • Quests An old ranger swore to clear the mine by lantern light.
  • Gossip The baker says the king eats nothing but cabbage though nobody believes it.
  • Quests A party of adventurers swore to clear the mine in the pouring rain.
  • Gossip The baker insists the smith cheats at dice as everyone already knew.
  • Trade The innkeeper paid double for dry firewood at the market.
  • Quests The ratcatcher came back from the cellar with three rat tails before supper.
  • Weather A light rain lay on the market stalls.
  • Quests A lone squire returned empty-handed to much applause.
  • Quests A novice mage came back from the cellar with three rat tails without a word.
  • Gossip The innkeeper reckons the bridge troll pays rent or so the cook says.
  • Quests A party of adventurers took the bounty on the troll in the pouring rain.
  • Gossip The gravedigger claims a dragon was seen as everyone already knew.
  • Sightings A cloaked stranger was seen digging behind the inn.
  • Weather Hail settled on the castle roofs.
  • Weather A warm wind blew on the castle roofs.
  • Quests An old ranger swore to clear the mine in the pouring rain.
  • Trade A Sunmere fishwife raised the price of ale outside the general store.
  • Weather A hard frost crept in on the market stalls.
  • Quests An old ranger came back from the cellar with three rat tails to much applause.
  • Sightings Somebody's goat was seen mining by the campfire.
  • Quests A lone squire set out to find the cabbage thief to much applause.
  • Sightings A goblin was seen mining behind the inn.
  • Gossip The miller's wife says the king eats nothing but cabbage but only after two ales.
  • Trade The innkeeper paid double for dry firewood outside the general store.
  • Quests The ratcatcher took the bounty on the troll by lantern light.
  • Sightings A goblin was caught dancing in the cabbage patch.
  • Weather Fog lay across the cabbage fields.
  • Sightings Somebody's goat was seen digging behind the inn.
  • Weather A hard frost settled around the campfire.
  • Trade The miller bought a cart of iron ore on the Sunmere road.
  • Sightings A white stag was spotted dozing under the old bridge.
  • Gossip A boy from Mossford insists the smith cheats at dice as everyone already knew.
  • Quests A novice mage took the bounty on the troll before supper.
  • Trade A travelling merchant traded a bronze dagger for a pie at the docks.
  • Trade A travelling merchant raised the price of ale outside the general store.
  • Sightings The pond's old pike was spotted dozing beside the pond.
  • Sightings A white stag was caught dancing in the cabbage patch.
  • Quests An old ranger came back from the cellar with three rat tails at dawn.
  • Quests The ratcatcher set out to find the cabbage thief without a word.
  • Sightings A cloaked stranger was caught dancing on the castle wall.
  • Gossip The innkeeper claims a dragon was seen but only after two ales.
  • Trade A travelling merchant bought a cart of iron ore outside the general store.
  • Trade A Sunmere fishwife sold forty cabbages outside the general store.
  • Trade The smith bought a cart of iron ore by the mill.
  • Sightings A white stag was seen digging under the old bridge.
  • Sightings A cloaked stranger was spotted dozing in the cabbage patch.
  • Trade A Sunmere fishwife sold forty cabbages by the mill.
  • Trade A gnome pedlar bought a cart of iron ore by the mill.
  • Weather Hail came down along the Sunmere road.
  • Quests The cook's apprentice swore to clear the mine in the pouring rain.
  • Gossip The baker claims a dragon was seen but only after two ales.
  • Trade A gnome pedlar raised the price of ale at the docks.
  • Sightings A cloaked stranger was seen mining behind the inn.
  • Quests An old ranger took the bounty on the troll before supper.
  • Sightings The pond's old pike was heard arguing behind the inn.
  • Gossip The gravedigger reckons the bridge troll pays rent or so the cook says.
  • Sightings A gnome was found singing under the old bridge.
  • Gossip A guard at the gate claims a dragon was seen and half the town agrees.
  • Quests The ratcatcher set out to find the cabbage thief without a word.
  • Weather A light rain came down on the market stalls.
  • Weather A hard frost lay on the castle roofs.
  • Trade The miller sold forty cabbages outside the general store. 
# frozen_string_literal: true

# One visitor's reading of the town chronicle. The states are the kind, how
# many loads the box opened with, the newest entry it has caught up to, the
# last row shown when the sentinel last fired, and a number that goes up
# when anyone writes an entry. The box is rendered whole when the list
# starts over; a load renders only its twenty rows, in place of the
# sentinel, so the rows above it are never sent again.
class Recipes::Scroll::ScrollChannel < ApplicationCable::Channel
  include Hibiki::Rails::Channel

  Entry = Recipes::Scroll::Entry
  Chronicle = Recipes::Scroll::Chronicle
  VIEWS = "app/recipes/scroll"

  def build_graph
    # The tab subscribes with the reading its page was rendered from, head
    # included, so the two agree on which entry is the newest.
    start = Chronicle.from(params, head: params[:head])
    @version = Hibiki::State.new(0)
    @kind = Hibiki::State.new(start.kind)
    @page = Hibiki::State.new(start.page)
    @head = Hibiki::State.new(start.head)
    @before = Hibiki::State.new(nil)
    @jumps = Hibiki::State.new(0)

    @chronicle = Hibiki::Derived.new { Chronicle.new(kind: @kind.value, page: @page.value, head: @head.value) }

    # The rows the box opens with, and the twenty a load adds under them.
    # Neither reads the version: rows already sent stay as they were.
    first = Hibiki::Derived.new { @chronicle.value.lines }
    load = Hibiki::Derived.new { (before = @before.value) && @chronicle.value.below(before) }

    # The id of the last row on screen, which the next load starts below.
    @last = Hibiki::Derived.new { load.value&.last&.id || @before.value || first.value.last&.id }

    # These read the version, so a ping makes them ask again, and answer
    # with plain values, so an answer equal to the last one stops there.
    # An entry above the head changes `fresh` and nothing else.
    matched = Hibiki::Derived.new { @version.value; @chronicle.value.matched }
    shown = Hibiki::Derived.new { @version.value; @chronicle.value.shown(@last.value) }
    loads = Hibiki::Derived.new { @version.value; @chronicle.value.loads(@last.value) }
    fresh = Hibiki::Derived.new { @version.value; @chronicle.value.fresh }

    # The whole box, when the list starts over. Its first run also puts a
    # reconnected tab's box back to the reading the graph starts from.
    Hibiki::Effect.new do
      chronicle, lines = @chronicle.value, first.value
      broadcast_morph target: "scroll_box", partial: "#{VIEWS}/box",
                      locals: { lines:, kind: chronicle.kind, page: chronicle.window,
                                remaining: chronicle.matched - lines.size, jumps: @jumps.value }
    end

    # One load, in place of the sentinel it answers. Its rows and a new
    # sentinel go under the rows already shown. It must not read the
    # version: run again for a ping, it would add the same rows twice.
    Hibiki::Effect.new do
      next unless (lines = load.value)

      chronicle, last = @chronicle.value, @last.value
      broadcast_replace target: "scroll_more", partial: "#{VIEWS}/load",
                        locals: { lines:, kind: chronicle.kind, page: chronicle.loads(last),
                                  remaining: chronicle.matched - chronicle.shown(last) }
    end

    Hibiki::Effect.new do
      broadcast_morph target: "scroll_fresh", partial: "#{VIEWS}/fresh", locals: { fresh: fresh.value }
    end

    transmit_value(:scroll_count) { "#{shown.value} of #{matched.value}" }

    transmit_url do
      urls.app_recipe_path("scroll", **@chronicle.value.with(page: loads.value).to_query)
    end
  end

  # The sentinel names the last row above it. A load is taken only when
  # that row is still the last: a sentinel that comes into view twice
  # before the first reply lands, or one left over from a list that has
  # since started over, names a row that no longer is.
  def more(data)
    before = data["before"].to_i
    @before.value = before if before == @last.value
  end

  def set_kind(data)
    @kind.value = data["kind"].to_s
    start_over
  end

  # The bell. The tab that rang it catches up, so it sees its own entry;
  # every other tab hears the ping and counts one more waiting.
  def ring
    Entry.ring!
    catch_up
  end

  # Moves the head to the newest entry and goes back to the top.
  def catch_up
    @head.value = Entry.head
    start_over
  end

  private

  # The first load again, with the box scrolled to its top.
  def start_over
    @page.value = 1
    @before.value = nil
    @jumps.value += 1
  end

  def subscribed
    super
    return if subscription_rejected?

    # An entry written by anyone. The block runs on a cable thread, so it
    # hops onto the graph thread before touching a signal.
    stream_from Entry::CHANGED do
      graph_actor&.post { @version.value += 1 }
    end
  end

  def urls = Rails.application.routes.url_helpers
end

The panel above is the town chronicle, a few hundred entries that auto-load as you scroll to the bottom. The count in the corner and the address bar update as you scroll, so a reload opens the chronicle as far down as you had read.

Try opening this page in a second window and ring the bell there. The list in this window does not move. A count of new entries appears above the box, and they come in when you press it. We wrote no JavaScript for any of this: everything works from reactivity defined in the channel. The chronicle is shared by every visitor and resets each night.

How it fits together

An island is the part of the page Hibiki looks after, here the whole panel. It belongs to a channel: the server side of an Action Cable connection, an object that stays alive while the page is open, one for each tab. The channel's public methods are its actions, the methods the page may call, and its states are values that remember who read them. Here the states are the kind; the newest entry this tab has caught up to, which we call the head; how many loads the box opened with; and the last row on screen when the box asked for more.

Chronicle is a plain value object built from the first three. It asks the database for the rows, the count and the number of entries waiting, and the page renders from it first, the channel after, so the two agree. In the channel it is a derived: a value computed from the states, computed again when one of them changes. Three effects read from it, one rendering the whole box, one a load of twenty rows, and one the count of new entries. This is what happens when you scroll to the foot:

  1. The foot of the list comes into view. The client calls more with the id of the last row above it.
  2. The action writes that id into the @before state, if that row is still the last one on screen. It renders nothing.
  3. A derived asks the database for the twenty rows below that id. The effect that read them renders those rows with a new sentinel under them, and sends them in place of the sentinel that asked. They land under the rows already shown, which are left as they are.
  4. The count and the address bar read the last row shown and follow, by transmit_value and transmit_url.

The sentinel

<%# The wrapper is the sentinel. It names the last row above it. %>
<%= tag.div id: "scroll_more", **on(:more, event: :visible, with: { before: last.id }) do %>
  <%# The link is the click, and its href is the way with no socket. %>
  <%= link_to "Load more", href, data: { turbo: false, **on(:more, with: { before: last.id }, fallback: true)[:data] } %>
<% end %>

Something has to notice that the visitor has reached the foot. Here it is the wrapper around the "Load more" link, which sends its action on visible. That is not a DOM event: the client watches the element with an IntersectionObserver and performs the action when it scrolls into view. It fires once, and the load that answers brings a new sentinel under its rows, so if twenty more rows do not push that one out of view, it fires too, and a tall box fills itself.

A sentinel can come into view twice before the first answer lands, which is why it names the last row above it. The action takes a load only while that row is still the last one on screen. By the time the second request arrives, the first has put twenty rows under that row, so the request is ignored. A sentinel left over from a list that has since started over is turned away the same way.

The link inside is a second control for the same action. A click sends it over the socket. With no socket, the browser follows the link instead, and its address is this page with one more load, opened at the last row already read. Only the link carries fallback: true, because on the wrapper it would make the browser follow the link and leave the page whenever the visitor scrolled to the foot while offline.

Appending a load

# The value object. One load is the twenty rows below the last one shown.
def below(id) = entries.where(id: ...id).newest.limit(Entry::PER).map(&:line)

def entries = Entry.of(kind).where(id: ..head)

# The channel. The action takes a load only while the row it names is
# still the last on screen.
def more(data)
  before = data["before"].to_i
  @before.value = before if before == @last.value
end

load = Hibiki::Derived.new { (before = @before.value) && @chronicle.value.below(before) }

# The rows and a new sentinel replace the sentinel that asked for them.
Hibiki::Effect.new do
  next unless (lines = load.value)

  broadcast_replace target: "scroll_more", partial: "#{VIEWS}/load", locals: { lines:, ... }
end

A load is the twenty rows below the last one shown. Chronicle#below finds them by the id the sentinel sent, and the effect that reads them renders them with a new sentinel and puts the lot where the old sentinel stood, by broadcast_replace. (There is no append helper; replacing the sentinel is how a load adds rows, as a lazy frame does in stock Rails.) The rows above are never sent again, so a load far down the chronicle is the same size as the first.

The rows are found by id, which serves as the cursor. Counting rows from the top would also work here, since the head keeps new entries out of the list, but the id does two more things. The database reaches a load deep in the chronicle as quickly as the first, and the action can tell a request for the current foot from a stale one, as the sentinel section showed.

The load effect leaves alone the @version state that goes up when anyone writes an entry. If it read that too, each entry anyone wrote would run it again and add the same twenty rows a second time. The whole box is rendered only when the list starts over, on a new kind or a catch-up: the first load is morphed into the box, and later loads begin from its last row.

New entries wait

# The value object. Entries above the head are counted, not shown.
def fresh = Entry.of(kind).where(id: (head + 1)..).count

# The channel. A ping from anyone's bell makes the count ask again.
fresh = Hibiki::Derived.new { @version.value; @chronicle.value.fresh }

Hibiki::Effect.new do
  broadcast_morph target: "scroll_fresh", partial: "#{VIEWS}/fresh", locals: { fresh: fresh.value }
end

# The pill's action. The rows follow, since they read the head.
def catch_up
  @head.value = Entry.head
  start_over
end

A row added at the top of a list pushes down whatever is being read. So each tab holds a head, the id of the newest entry it had when its page was rendered, and its rows are the ones at or below it. Ids only go up, so the head marks a place in the chronicle that later entries do not move. (It is not in the address bar, so a reload starts from the newest entry.)

When anyone writes an entry, the model's after_commit broadcasts an empty message, and each channel that hears it adds one to a @version state. The counts read that state too, so they ask the database again; the rows never do. For a tab reading below the new entry only fresh has changed, so the pill is rendered and the box is left alone. Pressing it moves the head to the newest entry and starts the list over. The tab that rang the bell does the same on its own, so it sees its entry at once.

Back to the top

<%# `jumps` goes up when the kind changes or the tab catches up. The client
    sets scrollTop when this helper's output changes, and at no other time. %>
<%= tag.div id: "scroll_box", class: "h-96 overflow-y-auto",
      **props(scroll_top: (0 if jumps.positive?), key: jumps) do %>
  ...
<% end %>

A new kind and a catch-up both start the list over, and the box should show its top then. A scroll position is a DOM property rather than an attribute, so a rendered fragment on its own cannot set it; the props helper can. The client assigns scrollTop when the helper's output changes and leaves the visitor's own scrolling alone otherwise. Here jumps goes up on a new kind or a catch-up. A load replaces the sentinel and never renders the box, so the box stays where it was.

The same feature in stock Rails

Plain infinite scroll needs no JavaScript in stock Rails either. A turbo-frame with loading="lazy" at the foot of each page fetches the next one when it scrolls into view, and it works with no socket at all. What costs more is everything around it in this example. Here is a sketch. It is not run here, and it leaves out the cap on the table. 

class EntriesController < ApplicationController
  def index
    # The head and the cursor travel in every URL. Paging by offset would
    # show a row twice each time an entry is added above it.
    @kind = params[:kind].presence_in(Entry::KINDS)
    @head = (params[:head] || Entry.maximum(:id)).to_i
    reach = Entry.of(@kind).where(id: ..@head)

    @entries = reach.then { params[:before] ? it.where(id: ...params[:before].to_i) : it }
                    .newest.limit(Entry::PER).to_a
    @matched = reach.count
    @shown = @entries.any? ? reach.where(id: @entries.last.id..).count : @matched

    # A lazy frame asks for the next page only.
    render partial: "page" if turbo_frame_request?
  end

  def create
    Entry.create!(**Entry.compose)
    redirect_to entries_path(kind: params[:kind]), status: :see_other
  end
end

The table lists each behaviour of the demo above and where it comes from.

Stock RailsHibiki
Load when the foot comes into view A lazy turbo-frame in each page event: :visible on a wrapper
No row twice when an entry is added A head and a cursor, carried in every frame URL A head and a cursor, kept as states in the channel
"3 new entries" A broadcast of a hidden mark for each entry, and a Stimulus controller that counts the ones of this tab's kind A derived count and a partial
"Showing 40 of 300" A stream inside each frame response; nothing moves it when a row is removed transmit_value
Changing the kind A form and a visit An action that writes the kind and starts the list over
The address bar after scrolling Not kept. A frame can advance the URL, but to its own cursor, which reloads as one page transmit_url
With no socket Scrolling still loads; only the new-entries count stops Scrolling stops; the link loads one more by a visit
A load far down the list One query for twenty rows, and twenty rows sent The same: one query for twenty rows, sent in place of the sentinel

Both versions append twenty rows a load, and both find them from a head and a cursor. What differs is where those two live. A lazy frame is plain HTTP, so it carries them in every URL, works out the count of new entries in the browser, and keeps loading with the socket down. Here they are states in the channel, and the count, the address bar and the pill are values derived from them.