Skip to content

Compute score bounds based on grading conf and warn about discrepancies #439

Closed

Description

@Matistjati
  • Check that no test case gets score.txt and a verdict other than accepted
  • Warn if some test data node has infinite max, but we can prove from children's ranges with aggregation that a finite bound can be found
  • Warn if any test data node has negative range lower bound. In particular, this will give a warning for every node without an explicitly set range, which I think is good?
  • In the new "Kattis" mode, error on objective: min

Activity

  1. gkreitz commented on Jul 15, 2026

    @gkreitz
    Contributor

    Check that score > max_score never happens at any node in the tree

    What check do you think is missing? There is already a check on max (and min) score for every group (which is, AFAIK, the only level which has max scores in legacy).

  2. Matistjati commented on Jul 15, 2026

    @Matistjati
    ContributorAuthor

    I had 3 bullet points and deleted the wrong one, oops. Updated

  3. gkreitz commented on Jul 15, 2026

    @gkreitz
    Contributor

    Warn if some test data node has infinite max, but we can prove from children's ranges with aggregation that a finite bound can be found

    Why should this be a warning? It's a reasonable thing to compute and make use of (i.e., if a problem has no limit on the root group, but obviously can't go above 100 due to childrens' max scores and aggregation, we would treat it as having 100 on the root when installing), but why do we want to force a problem setter to manually write files? The default is also +/-inf, so this can get very trigger happy.

    I'd be more inclined to give a warning if there is a finite bound we can prove from aggregation rules cannot be reached.

    Warn if any test data node has negative range lower bound. In particular, this will give a warning for every node without an explicitly set range, which I think is good?
    In the new "Kattis" mode, error on objective: min

    If we want to drop minimization and negative scores, why don't we just update the spec? Do we have any reason to believe there are problems out there which people care about that use these? I particularly don't want to add a warning which would be triggered by the default value for a field.

  4. Matistjati commented on Jul 15, 2026

    @Matistjati
    ContributorAuthor

    Why should this be a warning? It's a reasonable thing to compute and make use of (i.e., if a problem has no limit on the root group, but obviously can't go above 100 due to childrens' max scores and aggregation, we would treat it as having 100 on the root when installing), but why do we want to force a problem setter to manually write files? The default is also +/-inf, so this can get very trigger happy.

    I hold a perhaps extreme position on this. Every single time I've seen a problem where range was omitted, it was unintentional and buggy. In the edge-case where you do want infinite max_score, it's trivial to simply specify this explicitly in the group.

    IMO, one should therefore always specify the range in every node. testdata_tools already does this, and given the ambiguities in the spec (that I tigtened maybe 6 months ago), any sane generator ought to do the same. If you are hand-editing testdata.yaml files, it is very easy to get it wrong, so I prefer forcing authors to spend 2 minutes deciding this explicitly.

    If we want to drop minimization and negative scores, why don't we just update the spec? Do we have any reason to believe there are problems out there which people care about that use these? I particularly don't want to add a warning which would be triggered by the default value for a field.

    There are mixed opinions on backporting. Perhaps I should make an attempt. But Kattis has in fact killed off support for minimization and negative scores IIRC, and I would therefore be very happy to explicitly forbid it, so authors discover it using problemtools and not during a running contest.

  5. gkreitz commented on Jul 15, 2026

    @gkreitz
    Contributor

    Check that no test case gets score.txt and a verdict other than accepted

    Lol, this actually gets a bit annoying with the mulitpass test problem I built. I was accumulating score between passes in score.txt. Now I suddenly need to rm that file on a WA. I guess the recommended approach would be to accumulate score in a different file and rename to score.txt on the final verdict (if AC).

  6. Matistjati commented on Jul 15, 2026

    @Matistjati
    ContributorAuthor

    Well, score.txt and non-AC verdict is broken in Kattis as far as I'm aware 🙂

  7. gkreitz commented on Jul 15, 2026

    @gkreitz
    Contributor

    There are mixed opinions on backporting. Perhaps I should make an attempt. But Kattis has in fact killed off support for minimization and negative scores IIRC, and I would therefore be very happy to explicitly forbid it, so authors discover it using problemtools and not during a running contest.

    Well, "legacy" is the current version. I strongly dislike that we are in a position where making changes to the current version of the spec is painful (cf. Kattis/problem-package-format#585)

  8. gkreitz commented on Jul 15, 2026

    @gkreitz
    Contributor

    Well, score.txt and non-AC verdict is broken in Kattis as far as I'm aware 🙂

    I'll be happy to fix that once all installed problems behave properly. :) Adding a fix that causes Judge Errors feels bad.

  9. gkreitz commented on Jul 15, 2026

    @gkreitz
    Contributor

    I added a check for score.txt in #441. Good suggestion, thanks!

    IMO, one should therefore always specify the range in every node.

    I don't want to require that. Your feature request points in a good direction for computing bounds based on given max scores and aggregation rules. I'll think a bit about it a bit more, but my initial take is that I want to:

    • Propagate given score constraints based on aggregation (for non-custom graders)
    • Warn if an unreachable bound has been set
    • Warn if the root group (after propagation) has unbounded score, and is not explicitly setting range

    I think that solves several issues while still being user friendly for problem setters who do not use tools.

    Regarding minimization aspects of this issue, I feel like those are sort of wontfix until we've tried dropping min from the spec. If not, they are duplicate of your earlier #345.

  10. changed the title [-]Sanity checks[/-] [+]Compute score bounds based on grading conf and warn about discrepancies[/+] on Jul 15, 2026
  11. gkreitz commented on Jul 15, 2026

    @gkreitz
    Contributor

    I'll take the liberty to edit your issue description to something more descriptive for what remains.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions