Add References, every reference between packs that pks lists

.github/workflows/ci.yml

@@ -12,3 +12,6 @@ jobs:
       contents: read
     uses: rubyatscale/shared-config/.github/workflows/ci.yml@main # zizmor: ignore[unpinned-uses,secrets-inherit] internal reusable workflow tracked at @main by convention so shared-config updates propagate automatically; its ci.yml reads SLACK_WEBHOOK_URL but declares no workflow_call secrets, so inherit is the only way to pass it
     secrets: inherit
+    with:
+      # With pks, spec/query_packwerk/pks_spec.rb checks the references that the specs use against what pks lists
+      test-command: bin/install-pks && PATH="$PWD/tmp/pks:$PATH" bundle exec rspec

README.md

@@ -1,7 +1,7 @@
 # QueryPackwerk
 
 QueryPackwerk is a Ruby gem for querying and analyzing [Packwerk](https://github.com/Shopify/packwerk) violations in Ruby applications.
-It provides a friendly API for exploring `package.yml` and `package_todo.yml` files, making it easier to manage module boundaries and dependencies in your codebase.
+It provides a friendly API for exploring `package.yml` and `package_todo.yml` files, and the references between packs that [pks](https://github.com/rubyatscale/pks) lists, making it easier to manage module boundaries and dependencies in your codebase.
 
 ## Installation
 
@@ -80,10 +80,33 @@ query_packwerk:001:0> Violations.all
 
 ```
 
-If you change any of the package.yml or package_todo.yml files, you can reload the running console:
+Explore references, whether or not they break a rule:
+
+package_todo.yml files record only the references that break a pack's rules. To list every reference from one pack to another pack's constants, including those that a declared dependency or a public constant allows, QueryPackwerk runs [pks](https://github.com/rubyatscale/pks) 0.7.0 or later. It runs `bin/pks` if your app has one, or else `pks` on your `PATH`. See pks's [installation instructions](https://github.com/rubyatscale/pks/blob/main/INSTALLATION.md).
+
+```ruby
+# Get all references to a pack's constants
+query_packwerk:001:0> references_for("pack_name")
+
+# Get the 'shape' of the references to a pack that break no rule
+query_packwerk:001:0> References.where(producing_pack: "packs/pack_name", allowed: true).anonymous_source_counts
+
+# Get the references to a pack that break a rule no package_todo.yml records, which `pks check` reports
+query_packwerk:001:0> References.where(producing_pack: "packs/pack_name", unrecorded: true)
+
+# Get all references from a pack to other packs' constants
+query_packwerk:001:0> references_from("pack_name")
+
+# Get all references
+query_packwerk:001:0> References.all
+```
+
+References have the same methods as violations, like `sources_with_locations` and `anonymous_source_counts`. Each one is where pks finds it, so its source is found even where the code names a constant relatively, as `Bar` inside `module Foo`. A reference that pks infers from an association, as `has_many :tacos` does for `Taco`, has the association as its source. pks gives no positions in ERB templates, so there a reference's sources are wherever the template names the constant. Finding sources parses each file they're in, which for a pack that thousands of files reference takes a while, and keeps each one in memory, about 200 KB for a file of 100 lines, until `reload!`.
+
+If you change your code or any of the package.yml or package_todo.yml files, you can reload the running console:
 
 ```ruby
-# Reset the cache
+# Reset the cache, so the next query of references runs pks again
 query_packwerk:001:0> reload!
 ```
 
@@ -102,6 +125,9 @@ consumers = QueryPackwerk.consumers("my_pack")
 
 # Get a count of all violation types
 counts = QueryPackwerk.violation_counts_for("my_pack")
+
+# Get how other packs use your pack, whether or not they break a rule
+QueryPackwerk.references_for("my_pack").anonymous_source_counts
 ```
 
 ## Examples
@@ -124,6 +150,8 @@ QueryPackwerk.anonymous_violation_counts_for("my_pack", threshold: 3)
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
+The specs for references use what pks lists for an app they write, and one spec checks that against pks itself. It runs where pks is on your `PATH`, as in CI. To install the version it checks against, run `bin/install-pks` and add `tmp/pks` to your `PATH`.
+
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, bump `VERSION` in `lib/query_packwerk/version.rb` in your PR and run `bundle install` so `Gemfile.lock` matches. When the PR is merged and CI passes on `main`, the gem is tagged and published to RubyGems automatically. Merges that don't change the version don't publish anything.
 
 ## Contributing

bin/install-pks

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Installs pks in tmp/pks, as CI does, so pks_spec.rb checks the specs' listing against pks.
+# Keep version at QueryPackwerk::Pks::MINIMUM_VERSION, with each archive's sha256.
+set -euo pipefail
+
+version=0.7.0
+case "$(uname -sm)" in
+  'Linux x86_64')
+    target=x86_64-unknown-linux-gnu
+    sha256=51b8fb321a463769d59b99a1be2832fca035c92729e0bc26ecaca88c0dc9baa7
+    ;;
+  'Darwin arm64')
+    target=aarch64-apple-darwin
+    sha256=0c360f17e1d4e617711d9dce5869e2034841fa584a911b1d7b831ce536ed500e
+    ;;
+  *)
+    echo "bin/install-pks has no pks $version for $(uname -sm)" >&2
+    exit 1
+    ;;
+esac
+
+archive="tmp/pks/pks-$target.tar.xz"
+mkdir -p tmp/pks
+curl --proto '=https' --tlsv1.2 -fsSL -o "$archive" \
+  "https://github.com/rubyatscale/pks/releases/download/v$version/pks-$target.tar.xz"
+
+actual=$(shasum -a 256 "$archive" 2> /dev/null || sha256sum "$archive")
+if [ "${actual%% *}" != "$sha256" ]; then
+  echo "$archive's sha256 is ${actual%% *}, not $sha256" >&2
+  exit 1
+fi
+
+tar -xJf "$archive" -C tmp/pks --strip-components 1 "pks-$target/pks"
+tmp/pks/pks --version

lib/query_packwerk.rb

@@ -9,15 +9,21 @@
 # QueryPackwerk is a tool for querying Packwerk violations.
 #
 # It is built on top of ParsePackwerk, and provides a Ruby-friendly API
-# for querying package.yml and package_todo.yml files.
+# for querying package.yml and package_todo.yml files. It also queries every
+# reference between packages, allowed or not, which pks lists.
 #
 # See also: https://github.com/rubyatscale/parse_packwerk
 module QueryPackwerk
   autoload :Console, 'query_packwerk/console'
   autoload :ConsoleHelpers, 'query_packwerk/console_helpers'
   autoload :QueryInterface, 'query_packwerk/query_interface'
+  autoload :ConstantUse, 'query_packwerk/constant_use'
+  autoload :ConstantUses, 'query_packwerk/constant_uses'
   autoload :Violations, 'query_packwerk/violations'
   autoload :Violation, 'query_packwerk/violation'
+  autoload :References, 'query_packwerk/references'
+  autoload :Reference, 'query_packwerk/reference'
+  autoload :Pks, 'query_packwerk/pks'
   autoload :Packages, 'query_packwerk/packages'
   autoload :Package, 'query_packwerk/package'
   autoload :RuleRewriter, 'query_packwerk/rule_rewriter'
@@ -30,7 +36,7 @@ module QueryPackwerk
 
   sig { params(name: String).returns(T.nilable(QueryPackwerk::Package)) }
   def package(name)
-    Packages.where(name: name).first
+    Packages.where(name: full_name(name)).first
   end
 
   # Get all violations where other packages access code from this package
@@ -48,7 +54,7 @@ def todos_for(pack_name)
   end
 
   # Get where the violations occurred (where other packages access this package)
-  sig { params(pack_name: String).returns(T::Hash[String, T::Array[String]]) }
+  sig { params(pack_name: String).returns(T::Hash[String, T::Array[[String, String]]]) }
   def violation_sources_for(pack_name)
     violations_for(pack_name).sources_with_locations
   end
@@ -77,17 +83,38 @@ def consumers(pack_name, threshold: 0)
     violations_for(pack_name).consumers(threshold: threshold)
   end
 
+  # Get all references from other packages to this package's constants, whether
+  # or not they break the packages' rules (i.e., how others use this package).
+  # These come from pks, which must be installed.
+  sig { params(pack_name: String).returns(QueryPackwerk::References) }
+  def references_for(pack_name)
+    QueryPackwerk::References.where(producing_pack: full_name(pack_name))
+  end
+
+  # Get all references from this package to other packages' constants, whether
+  # or not they break the packages' rules (i.e., how this package uses others).
+  # These come from pks too.
+  sig { params(pack_name: String).returns(QueryPackwerk::References) }
+  def references_from(pack_name)
+    QueryPackwerk::References.where(consuming_pack: full_name(pack_name))
+  end
+
   # In case anyone is still using shorthand for pack names
   sig { params(pack_name: String).returns(String) }
   def full_name(pack_name)
-    return pack_name if pack_name.match?(%r{\A(packs|components)/})
+    return pack_name if pack_name.match?(%r{\A(packs|components)/}) || ParsePackwerk.find(pack_name)
 
     "packs/#{pack_name}"
   end
 
-  sig { void }
+  # nil rather than void, which the console shows as T::Private::Types::Void::VOID
+  sig { returns(NilClass) }
   def reload!
+    # ParsePackwerk keeps the packs it reads, along with their package_todo.yml files
+    ParsePackwerk.bust_cache!
     Packages.reload!
     Violations.reload!
+    References.reload!
+    nil
   end
 end

lib/query_packwerk/console_helpers.rb

@@ -25,6 +25,10 @@ def help(type = nil)
         help_violation
       when 'violations'
         help_violations
+      when 'reference'
+        help_reference
+      when 'references'
+        help_references
       when 'package'
         help_package
       when 'packages'
@@ -44,6 +48,8 @@ def help_help
           help            # This help
           help_violation  # Help for QueryPackwerk::Violation
           help_violations # Help for QueryPackwerk::Violations
+          help_reference  # Help for QueryPackwerk::Reference
+          help_references # Help for QueryPackwerk::References
           help_package    # Help for QueryPackwerk::Package
           help_packages   # Help for QueryPackwerk::Packages
       HELP
@@ -63,7 +69,7 @@ def help_violation
         .sources_with_locations # Returns array of [file:line, source] pairs for each violation
         .source_counts          # Returns hash of how often each violation source occurs
         .anonymous_sources      # Returns array of violation sources with arguments anonymized
-        .anonymous_sources_with_locations # Returns array of [file:line, source] pairs for each violation with arguments anonymized
+        .anonymous_sources_with_locations # Returns hash of anonymized source => [file:line]
         .count                  # Returns total number of violation occurrences
       HELP
     end
@@ -78,16 +84,66 @@ def help_violations
         .sources                # Returns hash of constant => source strings for all violations
         .sources_with_locations # Returns hash of constant => [file:line, source] pairs
         .source_counts          # Returns hash of constant => {source => count} with occurrence counts
-        .sources_with_contexts  # Returns hash of constant => [file:line, source] pairs for each violation with context
+        .sources_with_contexts  # Returns hash of constant => {anonymized source => [file:line with the lines around it]}
         .anonymous_sources      # Returns hash of constant => anonymized sources
-        .anonymous_sources_with_locations # Returns array of [file:line, source] pairs for each violation with arguments anonymized
-        .consumers(threshold)   # Returns hash of consuming package names and violation counts
-        .producers(threshold)   # Returns hash of producing package names and violation counts
+        .anonymous_sources_with_locations # Returns hash of constant => {anonymized source => [file:line]}
+        .consumers(threshold: 0) # Returns hash of consuming package names and violation counts
+        .producers(threshold: 0) # Returns hash of producing package names and violation counts
         .including_files(globs) # Returns new Violations filtered to include specific files
         .excluding_files(globs) # Returns new Violations filtered to exclude specific files
       HELP
     end
 
+    sig { returns(NilClass) }
+    def help_reference
+      puts <<~HELP
+        QueryPackwerk::Reference (lib/query_packwerk/reference.rb)
+        ------------------------------------------------
+        .constant_name          # Returns the referenced constant's name
+        .file                   # Returns the file that makes the reference
+        .line                   # Returns the line of the reference, or nil in an ERB template
+        .column                 # Returns the column of the reference in bytes from 0, or nil in an ERB template
+        .location               # Returns file:line, or the file for an ERB template
+        .producing_pack         # Returns the Package that defines the constant
+        .consuming_pack         # Returns the Package whose file makes the reference
+        .violations             # Returns the rules the reference breaks, with each one's type and whether it's strict and recorded
+        .violation_types        # Returns the types of the rules the reference breaks
+        .unrecorded_violation_types # Returns the types of those rules that no package_todo.yml records, which pks check reports
+        .allowed?               # Returns whether the reference breaks none of the rules
+        .sources                # Returns array of AST nodes representing the reference
+        .sources_with_locations # Returns array of [file:line, source] pairs
+        .source_counts          # Returns hash of how often each source occurs
+        .anonymous_sources      # Returns array of sources with arguments anonymized
+        .anonymous_sources_with_locations # Returns hash of anonymized source => [file:line]
+        .count                  # Returns how many sources make the reference, which is more than one only in an ERB template
+      HELP
+    end
+
+    sig { returns(NilClass) }
+    def help_references
+      puts <<~HELP
+        QueryPackwerk::References (lib/query_packwerk/references.rb)
+        --------------------------------------------------
+        Every reference from a package to another package's constants, whether or not it breaks their rules.
+        These come from pks 0.7.0 or later (https://github.com/rubyatscale/pks), in bin/pks or on your PATH.
+
+        References.all          # Returns all references
+        References.where(conditions) # Returns new References filtered by conditions, as allowed: true, violation_types: 'privacy' or unrecorded: true
+        .raw_sources            # Returns hash of constant => AST nodes for all references
+        .sources                # Returns hash of constant => source strings for all references
+        .sources_with_locations # Returns hash of constant => [file:line, source] pairs
+        .source_counts          # Returns hash of constant => {source => count} with occurrence counts
+        .sources_with_contexts  # Returns hash of constant => {anonymized source => [file:line with the lines around it]}
+        .anonymous_sources      # Returns hash of constant => anonymized sources
+        .anonymous_sources_with_locations # Returns hash of constant => {anonymized source => [file:line]}
+        .anonymous_source_counts # Returns hash of constant => {anonymized source => count}
+        .consumers(threshold: 0) # Returns hash of consuming package names and reference counts
+        .producers(threshold: 0) # Returns hash of producing package names and reference counts
+        .including_files(globs) # Returns new References filtered to include specific files
+        .excluding_files(globs) # Returns new References filtered to exclude specific files
+      HELP
+    end
+
     sig { returns(NilClass) }
     def help_package
       puts <<~HELP
@@ -104,6 +160,8 @@ def help_package
         .directory            # Returns the package directory path
         .todos                # Returns Violations where this package is the consumer
         .violations           # Returns Violations where this package is the producer
+        .incoming_references  # Returns References from other packages to this package (needs pks)
+        .outgoing_references  # Returns References from this package to other packages (needs pks)
         .consumers            # Returns Packages that consume this package
         .consumer_names       # Returns array of names of consuming packages
         .consumer_counts      # Returns hash of consumer package names and counts
@@ -141,7 +199,12 @@ def help_query_packwerk
         Package Dependencies (how this package uses others):
           todos_for(pack_name)                       # Get all todos where this package accesses others
 
+        Package References, whether or not they break a rule (needs pks):
+          references_for(pack_name)                  # Get all references from others to this package
+          references_from(pack_name)                 # Get all references from this package to others
+
         Violations.all                               # Get all violations
+        References.all                               # Get all references (needs pks)
       HELP
     end
   end