infer-subhogs silently drops proteins (3% of all proteins on fungi-30) #113

Problem

fastoma-infer-subhogs loses proteins: they are in the input rootHOG FASTA but end up in no final HOG, not even as a singleton. They are then missing from FastOMA_HOGs.orthoxml, RootHOGs.tsv and the orthologous groups, and nothing warns about it except the "placed X out of Y proteins. Z proteins in original roothog" log line (Z > Y).

On the fungi-30 dataset (FastOMA 0.5.1, dessimozlab/fastoma:dev), 414 of 22,878 rootHOGs dropped 8,113 of 265,464 proteins, e.g. F0412257 lost 367 of 496 and F0489395 lost 143 of 193. Rate of rootHOGs dropping proteins, by what happened in them:

How many proteins are dropped changes with small MSA differences (F0412257: 367 with mafft 7.505, 11 with mafft 7.526), so the problem is the bookkeeping, not a few unlucky families.

Cause

A HOG's membership is held in four parallel structures (_members, representatives, MSA rows, gene-tree leaves) that are kept in sync by hand. The tests found these places where members get lost:

• split_hog keeps only the first-level subHOGs that a partition's representatives lead to; the others disappear (_hog_class.py:636-643).
• LevelHOGProcessor.process() returns early when MSA filtering leaves no rows and forgets the frozen HOGs (_infer_subhog.py:690).
• the column filter in HOG.__init__ can drop all representative rows of a merged HOG; the HOG is then lost at the next level (_hog_class.py:333 -> _utils_subhog.py:485).
• rows dropped by the column filter are not reported as removed (_utils_subhog.py:456,485), so their representatives are never frozen.

Tests (strict xfail on PR #112, they pass once fixed)

• tests/test_hog_class.py::test_split_hog_keeps_all_members
• tests/test_hog_class.py::test_merged_hog_column_filter_keeps_representatives
• tests/test_level_processor.py::test_process_when_filter_empties_msa_keeps_frozen_hogs
• tests/test_msa_filter.py::test_filter_msa_all_columns_gappy_reports_rows
• tests/test_subhog_stage.py::test_all_input_proteins_in_final_hogs[D0625237]
• tests/test_fungi30.py::test_no_protein_dropped for the 10 drops rootHOGs of tests/data/fungi30/families.tsv

Proposed fix

1. Fix the four places above.
2. Check the invariant in the code, not only in tests: at the end of LevelHOGProcessor.process() the members of the output HOGs must equal the members of the input HOGs. If not, keep the missing proteins as frozen singleton HOGs and log a warning with their ids, instead of dropping them.

When fixed, move the drops rootHOGs of the fungi-30 fixture to golden (python tests/fungi30.py regenerate).