Split out `singletons-base` test suite into its own package

[RyanGlScott]

Before I explain why I want to do this, let me first make some observations about singletons-base's minimum GHC requirements.

Historically speaking, singletons-base has always been tied to a particular GHC version, as its implementation almost always requires the latest version of GHC to build. Or, at the very least, to build without needing heavy amounts of CPP. While this has been true of every singletons release that I can recall, GHC 9.4 is, remarkably, not that different from 9.2 in terms of its impact on singletons-base. In fact, one can compile the current state of the master branch with both 9.2 and 9.4, requiring no other changes than some minor changes to version bounds.

This is pretty remarkable, and I think it would be worth considering making the next release of singletons-base build with both GHC 9.2 and 9.4, given how straightforward it would be to do so. There's only one sticking point with this plan that I am aware of: the singletons-base test suite. The output of this test suite is intimately tied to one particular GHC version, and indeed, the output of the test suite changes depending on whether you use 9.2 or 9.4. Granted, it's mostly whitespace and formatting changes (e.g., rendering Type instead of Data.Kind.Type), but these are changes nonetheless.

I think it would be a shame to have the test suite be the sole reason why we can't widen the support window for singletons-base a little. In light of this, I propose that we split out the singletons-base test suite into its own singletons-base-tests package. The GHC support window for singletons-base-tests would be even tighter than it would be for singletons-base. We could continue to run the test suite in CI, and we could even upload singletons-base-tests to Hackage so that those who want to run it themselves can continue to do so.

Aside from the GHC support window benefits, another benefit to doing this is that singletons-base would no longer require a custom Setup.hs script; only singletons-base-tests would. Custom Setup.hs scripts often pose issues for things like cross-compilation, so this seems like a natural way to remove that roadblock.

[goldfirere]

I don't yet find this plan compelling. Instead, would it be possible just to have two (or more) different sets of golden files, keyed on the GHC version? That seems like it has the same amount of duplication as your plan (still need to maintain multiple golden files), but less user-facing complexity.

This might be different if simplifying Setup.hs was a key motivator, but that is presented more like a "hey, look, extra bonus!" than a reason to go down this route in the first place.

[RyanGlScott]

My main motivator is making it easier to support multiple versions of GHC as is reasonable. I disagree that maintaining multiple sets of golden files would involve the same amount of duplication. In fact, it would involve substantially more duplication, since now we have to double the amount of files we keep track of in version control. (There are a lot of golden files.) I'm not excited about that prospect.

[goldfirere]

Ah -- I think I had missed the point that the testsuite would cover only some GHC versions, so there would be no duplication. But then I think we can achieve that goal without a separate package: just conditionally have the testsuite fail on the wrong GHC version. (Or maybe succeed vacuously?) In any case, it worries me a bit that this plan would mean that we advertise that singletons works with GHC-x, but it would be completely untested there.

[RyanGlScott]

I think we can achieve that goal without a separate package: just conditionally have the testsuite fail on the wrong GHC version. (Or maybe succeed vacuously?)

Ah, I hadn't considered that. Indeed, making the test suite conditional sounds like substantially less work, so I like that idea better.

In any case, it worries me a bit that this plan would mean that we advertise that singletons works with GHC-x, but it would be completely untested there.

I think "completely untested" is a bit of an exaggeration. After all, most of the interesting work that singletons does is at compile time, so being able to compile singletons-base at all is already providing quite a bit of test coverage. It's certainly not giving you all of the test coverage that the test suite provides, but it's quite a bit more than nothing.

Relatedly, I've never been happy with making all of the test suites be golden tests, as it's coupled far too tightly with the output of GHC's -ddump-splices flag. I would favor moving some of these tests over to simple unit tests, at which point you could actually run them on multiple versions of GHC without remorse.

[goldfirere]

I agree about making the tests more well targeted. And I also agree about "completely untested"..... but the testsuite does exist for a reason. Maybe the tested-with bit in a .cabal file is sufficient to label this fact, though.

[RyanGlScott]

To make this a bit more concrete, 47 out of the 122 tests in the singletons-base test suite fail with GHC 9.4:

Details

Test suite singletons-base-test-suite: RUNNING...
Testsuite
  Singletons
    Nat:                  FAIL (5.34s)
      198c198
      <     data SNat :: Nat -> GHC.Types.Type
      ---
      >     data SNat :: Nat -> Type
      215c215
      <                                                                               -> GHC.Types.Type) t1) t2)
      ---
      >                                                                               -> Type) t1) t2)
      233c233
      <                                                                                               -> GHC.Types.Type) t1) t2) t3)
      ---
      >                                                                                               -> Type) t1) t2) t3)
      266c266
      <                                                                                  -> GHC.Types.Type) t1) t2)
      ---
      >                                                                                  -> Type) t1) t2)
      301,302c301
      <              Data.Type.Equality.TestEquality (SNat :: Nat
      <                                                       -> GHC.Types.Type) where
      ---
      >              Data.Type.Equality.TestEquality (SNat :: Nat -<truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '$0=="Testsuite.Singletons.Nat"' to rerun this test only.
    Empty:                FAIL (3.52s)
      5c5
      <     data SEmpty :: Empty -> GHC.Types.Type
      ---
      >     data SEmpty :: Empty -> Type

      Use -p '$0=="Testsuite.Singletons.Empty"' to rerun this test only.
    Maybe:                FAIL (5.07s)
      94c94
      <     data SMaybe :: forall a. Maybe a -> GHC.Types.Type
      ---
      >     data SMaybe :: forall a. Maybe a -> Type
      111c111
      <                                                                                       -> GHC.Types.Type) t1) t2)
      ---
      >                                                                                       -> Type) t1) t2)
      129c129
      <                                                                                               -> GHC.Types.Type) t1) t2) t3)
      ---
      >                                                                                               -> Type) t1) t2) t3)
      167,168c167
      <              Data.Type.Equality.TestEquality (SMaybe :: Maybe a
      <                                                         -> GHC.Types.Type) where
      ---
      >              Data.Type.Equality.TestEquality (SMaybe :: Maybe a -> Type) where
      172,173c171
      <              Data.Type.Coercion.TestCoercion (SMaybe :: Maybe a
      <                                                         -> GHC.Types<truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/Maybe/' to rerun this test only.
    BoxUnBox:             FAIL (3.64s)
      41c41
      <     data SBox :: forall a. Box a -> GHC.Types.Type
      ---
      >     data SBox :: forall a. Box a -> Type

      Use -p '/BoxUnBox/' to rerun this test only.
    Operators:            SKIP

      Use -p '/Operators/' to rerun this test only.
    HigherOrder:          SKIP

      Use -p '/HigherOrder/' to rerun this test only.
    Contains:             OK (4.99s)
    AsPattern:            SKIP

      Use -p '/AsPattern/' to rerun this test only.
    DataValues:           SKIP

      Use -p '/DataValues/' to rerun this test only.
    EqInstances:          SKIP

      Use -p '/EqInstances/' to rerun this test only.
    CaseExpressions:      OK (5.08s)
    Star:                 SKIP

      Use -p '/Star/' to rerun this test only.
    ReturnFunc:           SKIP

      Use -p '/ReturnFunc/' to rerun this test only.
    Lambdas:              FAIL (5.42s)
      755c755
      <     data SFoo :: forall a b. Foo a b -> GHC.Types.Type
      ---
      >     data SFoo :: forall a b. Foo a b -> Type

      Use -p '$0=="Testsuite.Singletons.Lambdas"' to rerun this test only.
    LambdasComprehensive: SKIP

      Use -p '/LambdasComprehensive/' to rerun this test only.
    Error:                OK (4.98s)
    TopLevelPatterns:     FAIL (5.17s)
      35c35
      <     data SBool :: Bool -> GHC.Types.Type
      ---
      >     data SBool :: Bool -> Type
      46c46
      <     data SFoo :: Foo -> GHC.Types.Type
      ---
      >     data SFoo :: Foo -> Type

      Use -p '/TopLevelPatterns/' to rerun this test only.
    LetStatements:        SKIP

      Use -p '/LetStatements/' to rerun this test only.
    LambdaCase:           OK (5.26s)
    Sections:             SKIP

      Use -p '/Sections/' to rerun this test only.
    PatternMatching:      SKIP

      Use -p '/PatternMatching/' to rerun this test only.
    Records:              FAIL (5.12s)
      64c64
      <     data SRecord :: forall a. Record a -> GHC.Types.Type
      ---
      >     data SRecord :: forall a. Record a -> Type

      Use -p '/Records/' to rerun this test only.
    T29:                  OK (4.95s)
    T33:                  FAIL (4.68s)
      30c30
      <     Lazy pattern converted into regular pattern in promotion
      ---
      >     Lazy pattern converted into regular pattern during singleton generation.
      36c36
      <     Lazy pattern converted into regular pattern during singleton generation.
      ---
      >     Lazy pattern converted into regular pattern in promotion

      Use -p '$0=="Testsuite.Singletons.T33"' to rerun this test only.
    T54:                  OK (4.68s)
    Classes:              SKIP

      Use -p '$0=="Testsuite.Singletons.Classes"' to rerun this test only.
    Classes2:             SKIP

      Use -p '/Classes2/' to rerun this test only.
    FunDeps:              OK (4.55s)
    T78:                  OK (4.63s)
    OrdDeriving:          FAIL (6.61s)
      466c466
      <     data SNat :: Nat -> GHC.Types.Type
      ---
      >     data SNat :: Nat -> Type
      478c478
      <     data SFoo :: forall a b c d. Foo a b c d -> GHC.Types.Type
      ---
      >     data SFoo :: forall a b c d. Foo a b c d -> Type
      481,485c481,485
      <               (Sing n)
      <               -> (Sing n)
      <               -> (Sing n)
      <               -> (Sing n)
      <               -> SFoo (A n n n n :: Foo a b c d)
      ---
      >               (Sing n) ->
      >               (Sing n) ->
      >               (Sing n) ->
      >               (Sing n) ->
      >               SFoo (A n n n n :: Foo a b c d)
      487,491c487,491
      <               (Sing n)
      <               -> (Sing n)
      <               -> (Sing n)
      <               -> (Sing n)
      <               -> SFoo (B n n n n :: Foo a b c d)
      ---
      >               (Sing n) ->
      >               (Sing n) ->
      >               (Sing n) ->
      >               (Sing n) ->
      >               SFoo (B n n n n :: Foo a b c d)
      493,497c493,497
      <               (Sing n)
      <               -> (Sing n)
      <               -> (Sin<truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/OrdDeriving/' to rerun this test only.
    BoundedDeriving:      OK (4.81s)
    BadBoundedDeriving:   OK (2.54s)
    EnumDeriving:         FAIL (4.77s)
      73c73
      <     data SFoo :: Foo -> GHC.Types.Type
      ---
      >     data SFoo :: Foo -> Type
      87c87
      <     data SQuux :: Quux -> GHC.Types.Type
      ---
      >     data SQuux :: Quux -> Type
      102c102
      <                                                                                                        -> GHC.Types.Type) t)
      ---
      >                                                                                                        -> Type) t)
      106c106
      <                                                                                      -> GHC.Types.Type) t)
      ---
      >                                                                                      -> Type) t)
      195c195
      <                                                                                                        -> GHC.Types.Type) t)
      ---
      >                                                                                                        -> Type) t)
      199c199
      <                                                                                      <truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/Singletons.EnumDeriving/' to rerun this test only.
    BadEnumDeriving:      OK (2.69s)
    Fixity:               OK (4.48s)
    Undef:                OK (4.50s)
    T124:                 OK (4.33s)
    T136:                 FAIL (4.36s)
      118,119c118
      <                               -> Sing (Apply (SuccSym0 :: TyFun [Bool] [Bool]
      <                                                           -> GHC.Types.Type) t)
      ---
      >                               -> Sing (Apply (SuccSym0 :: TyFun [Bool] [Bool] -> Type) t)
      122,123c121
      <                               -> Sing (Apply (PredSym0 :: TyFun [Bool] [Bool]
      <                                                           -> GHC.Types.Type) t)
      ---
      >                               -> Sing (Apply (PredSym0 :: TyFun [Bool] [Bool] -> Type) t)
      127c125
      <                                                                              -> GHC.Types.Type) t)
      ---
      >                                                                              -> Type) t)
      131c129
      <                                                               -> GHC.Types.Type) t)
      ---
      >                                                               -> Type) t)

      Use -p '$0=="Testsuite.Singletons.T136"' to rerun this test only.
    T136b:                FAIL (3.36s)
      52c52
      <                             -> Sing (Apply (MethSym0 :: TyFun Bool Bool -> GHC.Types.Type) t)
      ---
      >                             -> Sing (Apply (MethSym0 :: TyFun Bool Bool -> Type) t)

      Use -p '/T136b/' to rerun this test only.
    T153:                 OK (1.76s)
    T157:                 OK (1.78s)
    T159:                 FAIL (3.27s)
      22,23c22,23
      <     type ST0 :: T0 -> GHC.Types.Type
      <     data ST0 :: T0 -> GHC.Types.Type
      ---
      >     type ST0 :: T0 -> Type
      >     data ST0 :: T0 -> Type
      105,106c105,106
      <     type ST1 :: T1 -> GHC.Types.Type
      <     data ST1 :: T1 -> GHC.Types.Type
      ---
      >     type ST1 :: T1 -> Type
      >     data ST1 :: T1 -> Type
      218c218
      <     data ST2 :: T2 -> GHC.Types.Type
      ---
      >     data ST2 :: T2 -> Type

      Use -p '/T159/' to rerun this test only.
    T167:                 OK (4.42s)
    T145:                 OK (3.38s)
    PolyKinds:            OK (3.32s)
    PolyKindsApp:         OK (3.09s)
    T150:                 SKIP

      Use -p '/T150/' to rerun this test only.
    T160:                 OK (3.86s)
    T163:                 FAIL (3.30s)
      27c27
      <     data (%+) :: forall a b. (+) a b -> GHC.Types.Type
      ---
      >     data (%+) :: forall a b. (+) a b -> Type

      Use -p '/T163/' to rerun this test only.
    T166:                 OK (2.76s)
    T172:                 OK (4.10s)
    T175:                 OK (3.34s)
    T176:                 OK (4.48s)
    T178:                 FAIL (4.52s)
      145c145
      <     data SOcc :: Occ -> GHC.Types.Type
      ---
      >     data SOcc :: Occ -> Type
      164c164
      <                                                                               -> GHC.Types.Type) t1) t2)
      ---
      >                                                                               -> Type) t1) t2)
      179c179
      <                                                                                  -> GHC.Types.Type) t1) t2)
      ---
      >                                                                                  -> Type) t1) t2)
      213c213
      <                                                                                                                 -> GHC.Types.Type) t1) t2) t3)
      ---
      >                                                                                                                 -> Type) t1) t2) t3)
      249c249
      <                                                       -> GHC.Types.Type) where
      ---
      >                                                       -> Type) where
      253c253
      <        <truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/T178/' to rerun this test only.
    T183:                 OK (4.77s)
    T184:                 OK (4.80s)
    T187:                 FAIL (4.18s)
      64c64
      <     data SEmpty :: Empty -> GHC.Types.Type
      ---
      >     data SEmpty :: Empty -> Type
      75c75
      <                                                                                   -> GHC.Types.Type) t1) t2)
      ---
      >                                                                                   -> Type) t1) t2)
      82c82
      <                                                                                      -> GHC.Types.Type) t1) t2)
      ---
      >                                                                                      -> Type) t1) t2)
      87c87
      <                                                         -> GHC.Types.Type) where
      ---
      >                                                         -> Type) where
      91c91
      <                                                         -> GHC.Types.Type) where
      ---
      >                                                         -> Type) where

      Use -p '/T187/' to rerun this test only.
    T190:                 FAIL (4.55s)
      156c156
      <     data ST :: T -> GHC.Types.Type where ST :: ST (T :: T)
      ---
      >     data ST :: T -> Type where ST :: ST (T :: T)
      167c167
      <                                                                           -> GHC.Types.Type) t1) t2)
      ---
      >                                                                           -> Type) t1) t2)
      174c174
      <                                                                              -> GHC.Types.Type) t1) t2)
      ---
      >                                                                              -> Type) t1) t2)
      186c186
      <                                                                                                        -> GHC.Types.Type) t)
      ---
      >                                                                                                        -> Type) t)
      190c190
      <                                                                                    -> GHC.Types.Type) t)
      ---
      >                                                                          <truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/T190/' to rerun this test only.
    ShowDeriving:         FAIL (5.12s)
      295,296c295
      <     data SFoo1 :: Foo1 -> GHC.Types.Type
      <       where SMkFoo1 :: SFoo1 (MkFoo1 :: Foo1)
      ---
      >     data SFoo1 :: Foo1 -> Type where SMkFoo1 :: SFoo1 (MkFoo1 :: Foo1)
      302c301
      <     data SFoo2 :: forall a. Foo2 a -> GHC.Types.Type
      ---
      >     data SFoo2 :: forall a. Foo2 a -> Type
      331c330
      <     data SFoo3 :: Foo3 -> GHC.Types.Type
      ---
      >     data SFoo3 :: Foo3 -> Type
      352c351
      <                                                                                     -> GHC.Types.Type) t1) t2) t3)
      ---
      >                                                                                     -> Type) t1) t2) t3)
      369c368
      <                                                                                     -> GHC.Types.Type) t1) t2) t3)
      ---
      >                                                                                     -> Type) t1) t2) t3)
      488c487
      <                                                                                     -> GHC.Types.Type) t1) t2) t3)
      ---
      >             <truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/Singletons.ShowDeriving/' to rerun this test only.
    EmptyShowDeriving:    FAIL (4.29s)
      49c49
      <     data SFoo :: Foo -> GHC.Types.Type
      ---
      >     data SFoo :: Foo -> Type
      63c63
      <                                                                                               -> GHC.Types.Type) t1) t2) t3)
      ---
      >                                                                                               -> Type) t1) t2) t3)

      Use -p '/EmptyShowDeriving/' to rerun this test only.
    StandaloneDeriving:   FAIL (4.88s)
      305c305
      <     data ST :: forall a b. T a b -> GHC.Types.Type
      ---
      >     data ST :: forall a b. T a b -> Type
      316c316
      <     data SS :: S -> GHC.Types.Type
      ---
      >     data SS :: S -> Type
      332c332
      <                                                                                     -> GHC.Types.Type) t1) t2)
      ---
      >                                                                                     -> Type) t1) t2)
      351c351
      <                                                                                        -> GHC.Types.Type) t1) t2)
      ---
      >                                                                                        -> Type) t1) t2)
      383c383
      <                                                                                     -> GHC.Types.Type) t1) t2) t3)
      ---
      >                                                                                     -> Type) t1) t2) t3)
      415c415
      <                                                                           -> GHC.Types.Type) t1) t2)
      --<truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/StandaloneDeriving/' to rerun this test only.
    T197:                 OK (4.15s)
    T197b:                FAIL (3.29s)
      54c54
      <     data (%:*:) :: forall a b. (:*:) a b -> GHC.Types.Type
      ---
      >     data (%:*:) :: forall a b. (:*:) a b -> Type
      65c65
      <     data SPair :: forall a b. Pair a b -> GHC.Types.Type
      ---
      >     data SPair :: forall a b. Pair a b -> Type

      Use -p '/T197b/' to rerun this test only.
    T200:                 FAIL (4.27s)
      146c146
      <     data SErrorMessage :: ErrorMessage -> GHC.Types.Type
      ---
      >     data SErrorMessage :: ErrorMessage -> Type

      Use -p '/T200/' to rerun this test only.
    T204:                 OK (3.24s)
    T206:                 OK (1.79s)
    T209:                 FAIL (3.01s)
      80c80
      <     data SHm :: Hm -> GHC.Types.Type where SHm :: SHm (Hm :: Hm)
      ---
      >     data SHm :: Hm -> Type where SHm :: SHm (Hm :: Hm)

      Use -p '/T209/' to rerun this test only.
    T216:                 OK (3.20s)
    T226:                 OK (2.81s)
    T229:                 OK (4.17s)
    T249:                 OK (3.24s)
    OverloadedStrings:    OK (4.19s)
    T271:                 FAIL (4.87s)
      160,161c160,161
      <                      (Sing n)
      <                      -> SConstant (Constant n :: Constant (a :: Type) (b :: Type))
      ---
      >                      (Sing n) ->
      >                      SConstant (Constant n :: Constant (a :: Type) (b :: Type))

      Use -p '/T271/' to rerun this test only.
    T287:                 OK (3.38s)
    TypeRepTYPE:          OK (2.56s)
    T296:                 OK (3.24s)
    T297:                 OK (3.34s)
    T312:                 OK (3.55s)
    T313:                 OK (3.32s)
    T316:                 OK (4.51s)
    T322:                 OK (4.40s)
    T326:                 FAIL (3.27s)
      26,27c26,27
      <     type PC1 :: GHC.Types.Type -> Constraint
      <     class PC1 (a :: GHC.Types.Type) where
      ---
      >     type PC1 :: Type -> Constraint
      >     class PC1 (a :: Type) where
      55,56c55,56
      <     type PC2 :: GHC.Types.Type -> Constraint
      <     class PC2 (a :: GHC.Types.Type) where
      ---
      >     type PC2 :: Type -> Constraint
      >     class PC2 (a :: Type) where
      59c59
      <     class SC2 (a :: GHC.Types.Type) where
      ---
      >     class SC2 (a :: Type) where
      63c63
      <     type SC2 :: GHC.Types.Type -> Constraint
      ---
      >     type SC2 :: Type -> Constraint

      Use -p '/T326/' to rerun this test only.
    NatSymbolReflexive:   OK (1.99s)
    T323:                 OK (1.84s)
    T332:                 FAIL (4.26s)
      59,60c59
      <     data SBar :: Bar -> GHC.Types.Type
      <       where SMkBar :: SBar (MkBar :: Bar)
      ---
      >     data SBar :: Bar -> Type where SMkBar :: SBar (MkBar :: Bar)

      Use -p '/T332/' to rerun this test only.
    T342:                 OK (3.38s)
    FunctorLikeDeriving:  FAIL (7.25s)
      957,961c957,961
      <                  (Sing n)
      <                  -> (Sing n)
      <                  -> (Sing n)
      <                  -> (Sing n)
      <                  -> ST (MkT1 n n n n :: T x a)
      ---
      >                  (Sing n) ->
      >                  (Sing n) ->
      >                  (Sing n) ->
      >                  (Sing n) ->
      >                  ST (MkT1 n n n n :: T x a)

      Use -p '/FunctorLikeDeriving/' to rerun this test only.
    T353:                 OK (3.36s)
    T358:                 OK (4.37s)
    T367:                 OK (4.43s)
    T371:                 OK (4.55s)
    T376:                 OK (3.35s)
    T378a:                OK (4.47s)
    T378b:                SKIP

      Use -p '/T378b/' to rerun this test only.
    T401:                 OK (2.48s)
    T401b:                OK (2.71s)
    T402:                 OK (3.50s)
    T410:                 OK (3.39s)
    T412:                 FAIL (3.66s)
      163c163
      <     data SD1 :: forall a b. D1 a b -> GHC.Types.Type
      ---
      >     data SD1 :: forall a b. D1 a b -> Type
      226,227c226,227
      <     type PC2 :: GHC.Types.Type -> GHC.Types.Type -> Constraint
      <     class PC2 (a :: GHC.Types.Type) (b :: GHC.Types.Type) where
      ---
      >     type PC2 :: Type -> Type -> Constraint
      >     class PC2 (a :: Type) (b :: Type) where
      231c231
      <     class SC2 (a :: GHC.Types.Type) (b :: GHC.Types.Type) where
      ---
      >     class SC2 (a :: Type) (b :: Type) where
      235c235
      <     type SC2 :: GHC.Types.Type -> GHC.Types.Type -> Constraint
      ---
      >     type SC2 :: Type -> Type -> Constraint
      246,247c246,247
      <     type T2aSym0 :: (~>) GHC.Types.Type ((~>) GHC.Types.Type GHC.Types.Type)
      <     data T2aSym0 :: (~>) GHC.Types.Type ((~>) GHC.Types.Type GHC.Types.Type)
      ---
      >     type T2aSym0 :: (~>) Type ((~>) Type Type)
      >     data T2aSym0 :: (~>) Type ((~>) Type Type)
      255,257c255,256
      <     type T2aSym1 :: GHC.Types.Type
      <                     -> (~>) GHC.Types.Type GHC.Types.Type
      <   <truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/T412/' to rerun this test only.
    T414:                 OK (3.40s)
    T443:                 OK (3.40s)
    T445:                 SKIP

      Use -p '/T445/' to rerun this test only.
    T450:                 FAIL (4.55s)
      50,51c50,51
      <     type SMessage :: PMessage -> GHC.Types.Type
      <     data SMessage :: PMessage -> GHC.Types.Type
      ---
      >     type SMessage :: PMessage -> Type
      >     data SMessage :: PMessage -> Type
      119,121c119,121
      <     type PMkFunctionSym0 :: forall (a :: GHC.Types.Type)
      <                                    (b :: GHC.Types.Type). (~>) ((~>) a b) (PFunction (a :: GHC.Types.Type) (b :: GHC.Types.Type))
      <     data PMkFunctionSym0 :: (~>) ((~>) a b) (PFunction (a :: GHC.Types.Type) (b :: GHC.Types.Type))
      ---
      >     type PMkFunctionSym0 :: forall (a :: Type)
      >                                    (b :: Type). (~>) ((~>) a b) (PFunction (a :: Type) (b :: Type))
      >     data PMkFunctionSym0 :: (~>) ((~>) a b) (PFunction (a :: Type) (b :: Type))
      129,132c129,131
      <     type PMkFunctionSym1 :: forall (a :: GHC.Types.Type)
      <                                    (b :: GHC.Types.Type). (~>) a b
      <                                                           -> PFunction (a :: GHC.Types.Type) (b :: GHC.Type<truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/T450/' to rerun this test only.
    T453:                 OK (3.32s)
    NegativeLiterals:     OK (3.31s)
    T470:                 OK (4.41s)
    T480:                 OK (2.66s)
    T487:                 OK (4.31s)
    T489:                 OK (4.21s)
    T492:                 OK (2.17s)
    Natural:              FAIL (4.24s)
      63c63
      <     data SAge :: Age -> GHC.Types.Type
      ---
      >     data SAge :: Age -> Type

      Use -p '/Natural/' to rerun this test only.
    T511:                 OK (2.81s)
  Promote
    Constructors:         OK (3.25s)
    GenDefunSymbols:      OK (3.08s)
    Newtypes:             SKIP

      Use -p '/Newtypes/' to rerun this test only.
    Pragmas:              OK (3.26s)
    Prelude:              OK (3.88s)
    T180:                 OK (3.76s)
    T361:                 OK (2.96s)
  Database client
    Database:             OK (5.43s)
    Main:                 OK (1.97s)
  InsertionSort
    InsertionSortImp:     OK (3.23s)

47 out of 122 tests failed (38.75s)

So maintaining two sets of golden files (one set for 9.2 and another for 9.4) wouldn't quite double the amount of files, but it would involve enough duplication to where I wouldn't be excited to maintain it.

Maybe the tested-with bit in a .cabal file is sufficient to label this fact, though.

The tested-with stanza in a .cabal file is primarily used by CI tooling to determine which versions of GHC it's built against, not as a way to indicate the testing coverage.

[RyanGlScott]

In #531, I've decided to keep the status quo and require singletons-{th,base} to use GHC 9.4. I'm still not quite happy with this state of affairs, but it has become evident that we'll need to re-engineer the test suite in non-trivial ways to accomplish what I want, which is not something that should hold up a GHC 9.4–compatible release. I'll keep this issue open as a reminder to perform this engineering.

[RyanGlScott]

Note that the custom Setup.hs script (touted as one of the justifications for re-engineering the test suite as described above) will be removed once we switch to a code-generators–based test suite. See #532. The rest of the reasons cited above for refactoring the test suite still apply.