Paginate work package activities tab at the database level

The activity tab paginator previously loaded every journal and
changeset for a work package on each request, merged and sorted them
in Ruby, then sliced for the requested page. The eager-loading
wrapper ran four queries sized to the full pre-pagination set
(capped at limit × MAX_PAGES = 2000 rows by a Ruby-side ceiling).
Anchor jumps materialised the same array to call `find_index`.

Pagination now happens in Postgres. A `UNION ALL` of journals and
changesets, ordered by `(activity_at, id) DESC`, is paginated by
pagy at the database level. The eager-loading wrapper runs against
the page slice only — typically 20 rows instead of up to 2000.
Anchor jumps resolve to a `(created_at, id)` tuple and count rows
ahead in one query, so the path is independent of history size and
the MAX_PAGES cap goes away.

The `internal_visible` predicate is also exposed as a `Journal`
class scope (`internal_visible_for(project:, user:)`) so the
activities feed and anchor lookups share one visibility definition.
Existing fluent callers (`work_package.journals.internal_visible`)
keep working via a delegating shim on the has_many extension.
Anchor lookups go through the same scope so unviewable internal
journals fall back to page 1 rather than leaking through URL
behaviour.

A concurrent index on `journals (journable_type, journable_id,
created_at DESC, id DESC)` serves the new `WHERE … ORDER BY` plan
and supports the row-constructor predicate
`(created_at, id) > (?, ?)` used by anchor counting.

The `activity-N` anchor format (resolves a journal's
`sequence_version` via a LATERAL `ROW_NUMBER`) is preserved for
old bookmarks.

Author: akabiru

app/models/journal.rb

@@ -122,7 +122,8 @@ class Journal < ApplicationRecord
 
   include ::Scopes::Scoped
 
-  scopes :with_sequence_version
+  scopes :with_sequence_version,
+         :internal_visible_for
 
   # Scopes to all journals excluding the initial journal - useful for change
   # logs like the history on issue#show

app/models/journals/scopes/internal_visible_for.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+# -- copyright
+# OpenProject is an open source project management software.
+# Copyright (C) the OpenProject GmbH
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License version 3.
+#
+# OpenProject is a fork of ChiliProject, which is a fork of Redmine. The copyright follows:
+# Copyright (C) 2006-2013 Jean-Philippe Lang
+# Copyright (C) 2010-2013 the ChiliProject Team
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+#
+# See COPYRIGHT and LICENSE files for more details.
+# ++
+
+module Journals::Scopes
+  # Restricts journals to those the given user is allowed to see in the given
+  # project. Internal journals are hidden unless the project has internal
+  # comments enabled, the Enterprise add-on is active, and the user holds
+  # `view_internal_comments`.
+  #
+  # This is a class scope (rather than a `has_many` extension) so it composes
+  # inside a subquery — for example, the journals leg of a UNION.
+  #
+  # The `user:` default of `User.current` assumes request context. Pass `user:`
+  # explicitly from background jobs where `User.current` is not set.
+  module InternalVisibleFor
+    extend ActiveSupport::Concern
+
+    class_methods do
+      def internal_visible_for(project:, user: User.current)
+        if EnterpriseToken.allows_to?(:internal_comments) &&
+            project.enabled_internal_comments &&
+            user.allowed_in_project?(:view_internal_comments, project)
+          all
+        else
+          where(internal: false)
+        end
+      end
+    end
+  end
+end

app/models/work_package/journalized.rb

@@ -34,13 +34,7 @@ module WorkPackage::Journalized
   included do
     acts_as_journalized journals_association_extension: proc {
       def internal_visible
-        if EnterpriseToken.allows_to?(:internal_comments) &&
-            proxy_association.owner.project.enabled_internal_comments &&
-            User.current.allowed_in_project?(:view_internal_comments, proxy_association.owner.project)
-          all
-        else
-          where(internal: false)
-        end
+        merge(Journal.internal_visible_for(project: proxy_association.owner.project))
       end
     }
 

app/services/work_packages/activities_tab/paginator.rb

@@ -28,7 +28,8 @@
 # See COPYRIGHT and LICENSE files for more details.
 #++
 
-# Paginates work package activities (journals and changesets) with support for filtering and anchor navigation.
+# Paginates work package activities (journals and changesets) with support for
+# filtering and anchor navigation.
 #
 # Filter modes:
 # - :all - Shows all activities (default)
@@ -39,6 +40,11 @@
 # - "comment-{journal_id}" - Navigate to specific journal by ID
 # - "activity-{sequence_version}" - Navigate to journal by sequence version
 #
+# Internally, the activities feed is materialised as a single UNION ALL
+# relation of journals and changesets (see {ActivitiesQuery}) and paginated
+# by pagy at the database level. Only the page slice is hydrated and wrapped
+# for eager-loading, keeping per-request cost independent of total history size.
+#
 # @param work_package [WorkPackage] The work package to paginate activities for
 # @param params [Hash] Pagination and filtering parameters
 #
@@ -52,8 +58,6 @@ class WorkPackages::ActivitiesTab::Paginator
   include Pagy::Method
   include WorkPackages::ActivitiesTab::JournalSortingInquirable
 
-  MAX_PAGES = 100
-
   def self.paginate(work_package, params = {})
     new(work_package, params).call
   end
@@ -67,124 +71,138 @@ def initialize(work_package, params = {})
   end
 
   def call
-    anchor_type, target_record_id = extract_target_record_id
+    anchor_type, target_record_id = parse_anchor
 
-    pagy, records =
+    pagy_obj, page_relation =
       if anchor_type && target_record_id
-        @filter = :all # Ignore filter when jumping to specific journal
-        pagy_array_for_target_journal(anchor_type, target_record_id)
+        @filter = :all
+        pagy_at_anchor(anchor_type, target_record_id)
       else
-        pagy(:offset, capped_journals, **pagy_options)
+        pagy(:offset, activities_scope, **pagy_options)
       end
 
-    # For UI display: if user wants "oldest first" UI, reverse the array
-    records = records.reverse if journal_sorting.asc?
+    activities = load_activities(page_relation)
+    activities = activities.reverse if journal_sorting.asc?
 
-    [pagy, records]
+    [pagy_obj, activities]
   end
 
   private
 
+  def activities_query
+    @activities_query ||= ActivitiesQuery.new(work_package, filter:)
+  end
+
+  def activities_scope
+    @activities_scope ||= activities_query.call
+  end
+
+  def limit
+    params[:limit] || Pagy::DEFAULT[:limit]
+  end
+
   def pagy_options
     {
       page: params[:page] || 1,
-      limit: params[:limit] || Pagy::DEFAULT[:limit],
+      limit:,
       request: { params: }
     }.compact
   end
 
-  def extract_target_record_id
+  def parse_anchor
     anchor = params[:anchor] # e.g., "comment-78758" (without #)
-    return nil unless anchor
+    return unless anchor
 
     match = anchor.match(/^(comment|activity)-(\d+)$/)
-    match && match.length == 3 ? [match[1].inquiry, match[2].to_i] : []
-  end
-
-  def pagy_array_for_target_journal(anchor_type, target_record_id)
-    journals = base_journals
-
-    target_index = journals.find_index do |record|
-      if anchor_type.comment?
-        record.id == target_record_id
-      elsif anchor_type.activity?
-        record.sequence_version == target_record_id
-      else
-        false
-      end
-    end
+    return unless match
 
-    if target_index
-      limit = pagy_options[:limit]
-      target_page = (target_index / limit) + 1
-      pagy(:offset, journals, **pagy_options, page: target_page)
-    else
-      # Journal might be filtered out or deleted - fallback to page 1
-      pagy(:offset, journals, **pagy_options, page: 1)
-    end
+    [match[1].inquiry, match[2].to_i]
   end
 
-  def base_journals
-    combine_and_sort_records(fetch_journals, fetch_revisions)
+  # An unresolvable anchor (deleted, never existed, not visible to the user)
+  # falls back to the default page from `pagy_options`.
+  def pagy_at_anchor(anchor_type, target_record_id)
+    options = pagy_options
+    options[:page] = page_for_anchor(anchor_type, target_record_id) || options[:page]
+    pagy(:offset, activities_scope, **options)
   end
 
-  def capped_journals
-    max_records = (params[:limit] || Pagy::DEFAULT[:limit]) * MAX_PAGES
-    base_journals.first(max_records)
+  # Resolves an anchor to its target page by counting records ahead of it.
+  # Returns nil when the anchor is unresolvable (e.g. deleted) so the caller
+  # falls back to page 1.
+  def page_for_anchor(anchor_type, target_record_id)
+    activity_at, anchor_id = locate_anchor(anchor_type, target_record_id)
+    return nil unless activity_at && anchor_id
+
+    rows_ahead = activities_scope
+      .where("(activities.activity_at, activities.id) > (?, ?)", activity_at, anchor_id)
+      .count(:all)
+
+    (rows_ahead / limit) + 1
   end
 
-  def fetch_journals
-    API::V3::Activities::ActivityEagerLoadingWrapper.wrap(fetch_ar_journals)
+  # Anchors must observe the same visibility rules as the activities feed.
+  # Otherwise the count-ahead would route an unviewable journal to a page
+  # number and leak the existence of internal journals through the URL.
+  def locate_anchor(anchor_type, target_record_id)
+    if anchor_type.comment?
+      activities_query.visible_journals
+        .where(id: target_record_id)
+        .pick(:created_at, :id)
+    elsif anchor_type.activity?
+      locate_anchor_by_sequence_version(target_record_id)
+    end
   end
 
-  def fetch_ar_journals
-    journals = work_package
-      .journals
-      .internal_visible
-      .includes(
-        :user,
-        :customizable_journals,
-        :attachable_journals,
-        :storable_journals,
-        :notifications
-      )
-      .reorder(version: :desc) # Always fetch newest first for pagination
+  def locate_anchor_by_sequence_version(sequence_version)
+    activities_query.visible_journals
       .with_sequence_version
-
-    case filter
-    when :only_comments then apply_comments_only_filter(journals)
-    when :only_changes then apply_changes_only_filter(journals)
-    else
-      journals
-    end
+      .where(ranked: { sequence_version: sequence_version })
+      .pick(:created_at, :id)
   end
 
-  def fetch_revisions
-    return Changeset.none if filter == :only_comments
+  def load_activities(page_relation)
+    activity_refs = page_relation.pluck(Arel.sql("activities.kind"), Arel.sql("activities.id"))
+    activities_by_kind = load_page_activities_by_kind(activity_refs)
 
-    work_package.changesets.includes(:user, :repository)
+    ordered_activities = activity_refs.filter_map { |kind, id| activities_by_kind[kind][id] }
+    eager_load_journals(ordered_activities)
   end
 
-  def combine_and_sort_records(journals, revisions)
-    (journals + revisions).sort_by do |record|
-      timestamp = record_timestamp(record)
-      [-timestamp, -record.id] # Always sort DESC (newest first)
-    end
+  def load_page_activities_by_kind(activity_refs)
+    ids_by_kind = activity_refs.group_by(&:first).transform_values { it.map(&:last) }
+    {
+      ActivitiesQuery::KIND_JOURNAL => load_page_journals(ids_by_kind[ActivitiesQuery::KIND_JOURNAL] || []),
+      ActivitiesQuery::KIND_CHANGESET => load_page_changesets(ids_by_kind[ActivitiesQuery::KIND_CHANGESET] || [])
+    }
   end
 
-  def record_timestamp(record)
-    if record.is_a?(API::V3::Activities::ActivityEagerLoadingWrapper)
-      record.created_at&.to_i
-    elsif record.is_a?(Changeset)
-      record.committed_on.to_i
-    end
+  def load_page_journals(ids)
+    return {} if ids.empty?
+
+    Journal
+      .where(id: ids)
+      .with_sequence_version
+      .includes(:user, :customizable_journals, :attachable_journals, :storable_journals, :notifications)
+      .index_by(&:id)
   end
 
-  def apply_comments_only_filter(scope)
-    scope.where.not(notes: [nil, ""])
+  def load_page_changesets(ids)
+    return {} if ids.empty?
+
+    Changeset
+      .where(id: ids)
+      .includes(:user, :repository)
+      .index_by(&:id)
   end
 
-  def apply_changes_only_filter(scope)
-    JournalChangesFilter.apply(scope)
+  # Substitutes journals with their eager-loading wrappers so the wrapper's
+  # batch queries (journable, predecessor, data, notifications) run against
+  # the page slice only. Order from the input is preserved.
+  def eager_load_journals(activities)
+    journals = activities.grep(Journal)
+    wrapped_by_id = API::V3::Activities::ActivityEagerLoadingWrapper.wrap(journals).index_by(&:id)
+
+    activities.map { it.is_a?(Journal) ? wrapped_by_id[it.id] : it }
   end
 end