Draft: work in progress. This recipe is not finished yet; sections may be incomplete and results may change.

Status: draft (2026-11-14), sections 1-10. Haplotyping runs with migrate-n 7.0.113 (6.1.89 gives the same numbers); the runs without haplotyping (true and arbitrary phase) with 7.0.112. Use 7.0.113 / 6.1.89 or later for haplotyping: earlier versions counted the burn-in in the phase report, compared the haplotype move with the likelihood of the previous locus in the burn-in of every locus after the first (serial runs), lost or mixed up the phase records of MPI workers that ran several loci, and ignored the masks of section 8. Still to do: a run without heating, replicate data sets.

Sequencing a diploid individual gives its genotype: at every heterozygous site two bases, but not which bases sit together on one chromosome. Migrate needs haplotypes. This recipe simulates data with known haplotypes, turns them into what a sequencing experiment delivers, and compares three ways to analyse them: the true haplotypes, an arbitrary phase taken as known, and migrate’s haplotyping, which samples the phase together with the genealogy. Read 00-basics and 01-migration first.

Contents

  1. The problem
  2. Simulated data with known haplotypes
  3. How migrate resolves the phase
  4. Preparing the infile: tools/haplotype_blocks.py
  5. The parmfile
  6. Results: true, arbitrary and resolved phase
  7. The phase report and how well the phase mixes
  8. Known phase: masks per locus
  9. Writing the phased infile
  10. What next

1. The problem

An individual heterozygous at sites 14 and 27 of a locus carries either the haplotypes A..G and C..T, or A..T and C..G. With h heterozygous sites there are 2^(h-1) phases. Phasing programs (PHASE, SHAPEIT, BEAGLE) pick one, usually from linkage disequilibrium in large samples, and the analysis then treats it as known. With few individuals and few loci, as in most migrate data sets, that guess is poor, and a wrong phase creates recombinant-looking haplotypes that need extra mutations: Theta goes up, and since the false haplotypes are spread over both populations, M goes down (section 6).

2. Simulated data with known haplotypes

python3 scripts/simulate_unphased.py data --seed 7005

Two populations (North, South), 10 diploid individuals each, 5 unlinked loci of 500 bp without recombination inside a locus, Theta = 0.01 per site in both, M = 100 in both directions (M = m/mu), JC69, simulated with msprime (through source/migrate-codex-7/tests/sim/simulate.py). The script writes four files in the new infile syntax:

FileContent
data/phased.txtthe true haplotypes, two lines per individual (N01:1, N01:2)
data/unphased.iupacone line per individual, IUPAC code (R Y S W K M) at every heterozygous site: the genotype call
data/unphased.twothe IUPAC line split into two lines, the alphabetically first base always on copy 1: an arbitrary phase
data/truth.txtper individual and locus the heterozygous sites and the true phase

unphased.iupac is what a sequencing run delivers; unphased.two is what an analysis gets that ignores the phase. Of the 100 individual-loci, 85 are heterozygous (10.4 heterozygous sites on average), 76 at two or more sites, so that their phase is unknown.

Seed 7005 was chosen over 7001, whose data had one individual-locus with about 70 heterozygous sites.

3. How migrate resolves the phase

Migrate’s unit of known phase is the sublocus. A locus can be cut into linked subloci ((s26 s25 s1 ...) in the second infile line); inside a sublocus the two copies of an individual are taken as given, between subloci their order is unknown. With haplotyping=YES, migrate adds a move to the MCMC that swaps heterozygous subloci between the two copies of an individual: one sublocus, or (in half of the moves when the individual has more than two) all of its heterozygous subloci from a random one onward. The move is accepted with the likelihood ratio, so the chain samples the phase together with the genealogy and the parameters, and the estimates integrate over the phase.

Individuals must be named name:1 and name:2 (any order of the copies), and the individuals to phase are listed in the infile comments:

#% N01: * 0 * * 0

one mask per locus (section 8). Individuals with fewer than two heterozygous subloci in a locus have nothing to resolve there and are left out.

Two settings matter:

  • freqs-from-data=YES:locus pools the base frequencies of a locus over its subloci. With YES every sublocus, some of them a single site, gets its own frequencies, and cutting a locus would change the model. (JC69, as here, has no frequencies, but the recipe keeps the setting so that it carries over to F84 or HKY.)
  • Heating. The phase is coupled to the genealogy, probably so that a phase that fits only a different genealogy is rejected in the cold chain, while heated chains reach it and swapping chains carries it over. In an earlier test four heated chains found the true phase more often than one chain (118 instead of 96 of 192 individual-loci, 8 seeds; docs/haplotyping.md). Keep the default heating; with heating=NO the outfile warns.

4. Preparing the infile: tools/haplotype_blocks.py

python3 ../../tools/haplotype_blocks.py data/unphased.iupac data/unphased.blocks

The tool reads one line per individual with IUPAC codes (or two lines name:1, name:2 in any phase), and cuts every locus into the fewest blocks such that no block holds two heterozygous sites of an individual to be phased: one scan from left to right, cutting just before a site that would give an individual a second heterozygous site in the current block. Stretches without heterozygous sites stay with their neighbours. Here it made 17, 28, 28, 32 and 28 blocks (133 subloci). It writes the blocks as subloci, the IUPAC lines as name:1/name:2 (arbitrary phase), the #% lines, and prints the heterozygous sites and blocks per individual and locus. Options: --individuals, --exclude, --max-het H (do not phase an individual with more than H heterozygous sites in a locus), --keep-loci (do not cut, and keep the phase as given there), --phased-subloci (cut only at existing sublocus boundaries), --known (parts of the phase that are known, section 8).

Without recombination inside a locus, any cut that separates the heterozygous sites of every individual gives the same model. The finest such cut starts a sublocus at every variable site:

python3 scripts/cut_every_site.py data/unphased.blocks data/unphased.allsites

which gives 24, 44, 42, 44 and 44 subloci (196). Section 6 runs both; they should agree.

A pitfall found on the way: haplotype_blocks.py writes a blank between the blocks, and migrate ignores blanks inside a sequence. A script that counts sites in these lines must remove the blanks first; the first version of cut_every_site.py did not, its cuts drifted, and subloci held two heterozygous sites of one individual, whose relative phase was then fixed at the arbitrary input. Theta came out 1.6 times too high. Checking that no heterozygous sublocus of any individual holds more than one heterozygous site (884 heterozygous subloci, none with two) found it.

5. The parmfile

scripts/make_parmfiles.sh

writes five parmfiles from migrate’s default parmfile, changing only:

datatype=SequenceData
datamodel=JC69
infile=unphased.blocks              (phased.txt, unphased.two, unphased.allsites, infile.phased)
haplotyping=YES:reporting           (NO for phased.txt, unphased.two and infile.phased)
phased-infile=YES:infile.phased     (NO without haplotyping; section 9)
freqs-from-data=YES:locus
heating=YES:1:POWER:5
random-seed=OWN:20261113

haplotyping=YES:reporting adds the phase report (section 7); YES:not-reporting resolves the phase without it. The run length is the default (burn-in 10,000 x 100, 10,000 samples x 100).

6. Results: true, arbitrary and resolved phase

Medians of the “All” rows (95% HPD in brackets), true values Theta = 0.01, M = 100:

DataTheta_NorthTheta_SouthM_South->NorthM_North->SouthGenealogy acceptance
true haplotypes0.0081 [0.0051-0.0115]0.0079 [0.0039-0.0121]80 [0-225]126 [0-338]0.155
arbitrary phase as known0.0196 [0.0037-0.0510]0.0188 [0.0084-0.0335]13 [0-58]14 [0-66]0.077
blocks, haplotyping0.0076 [0.0051-0.0115]0.0067 [0.0031-0.0122]28 [0-172]69 [0-326]0.156
every site, haplotyping0.0082 [0.0051-0.0114]0.0094 [0.0045-0.0140]38 [1-199]81 [1-205]0.155

M_South->North is migration into North from South, forward in time.

Taking the arbitrary phase as known more than doubles Theta and cuts M to a seventh; the true values are outside the 95% HPD of M. The false haplotypes need more mutations (higher Theta) and look unrelated to the haplotypes of the other population (lower M). The genealogy acceptance halves, a sign that the data fit no genealogy well. Haplotyping recovers the estimates of the true haplotypes, and the acceptance returns to the level of the true phase. Both cuts give the same answer, as they should.

Four seeds (20261113, 11, 22, 33) of each cut:

CutTheta_NorthTheta_SouthM_South->NorthM_North->South
blocks0.0072-0.00880.0063-0.008220-11269-146
every site0.0082-0.00860.0081-0.009838-10081-114

The M medians vary between seeds; the 95% HPDs (0 to 170-420) include all of them and the true value. Theta is stable.

All runs used 6 MPI processes (one master, five workers, one locus each) on an M-series Mac, two runs at a time: true or arbitrary phase about 3 minutes, blocks about 4.5 minutes, every site about 5.5 minutes.

7. The phase report and how well the phase mixes

With haplotyping=YES:reporting the outfile lists, per locus and individual, the phase states the chain visited and their frequencies:

Locus: 1
1:N01        0100011100:0.000 0101010100:0.005 0101011100:0.995 ...  (19445)
1:N02        010001010:0.001 ... 010001110:0.462 ... 010011110:0.531 ...  (19504)
1:N03        00:0.005 01:0.995    (19333)

A state has one character per heterozygous sublocus of the individual: 0 keeps the input copy, 1 swaps it. A state and its complement are the same phase (the copies are exchangeable) and are merged; the form starting with 0 is shown. The number in brackets counts the recorded states (the haplotype moves of the cold chain after the burn-in). N01 and N03 are resolved; N02 has two phases that differ in one block, each with about half of the probability: the data cannot tell which.

scripts/score_phase.py compares the report with the true haplotypes:

python3 scripts/score_phase.py data/unphased.blocks outfile data/phased.txt

per locus and in total: individual-loci phased, how often the posterior mode is the true phase (all heterozygous sites right), the mean posterior probability of the true phase, and the switch accuracy (the fraction of neighbouring heterozygous sites whose relative phase the mode gets right).

RunMode = true phase (of 76)P(true)Switch accuracy
blocks, 4 seeds67-680.77-0.810.98
every site, 4 seeds60-690.72-0.810.94-0.98
blocks, started at the true phase680.840.98
every site, started at the true phase680.820.98

Started at the arbitrary phase, the chain finds the true phase about as often as when it starts at the true phase: the burn-in is long enough. The 8-9 individual-loci whose mode is not the true phase are mostly cases like N02 above, where two phases fit about equally well; P(true) of 0.8 says that some of the probability sits on such alternatives. A wrong phase of a few individuals matters less for Theta and M than a wrong phase of all of them (section 6). To check the phase itself, run several chains (seeds or replicates) and compare the reports; where they disagree, the phase is not resolved.

Earlier versions (before 7.0.113 / 6.1.89) counted the burn-in, which starts at the arbitrary phase, in the report: with these runs the mode was the true phase for only 47-64 of 76 individual-loci.

8. Known phase: masks per locus

Sometimes part of the phase is known: long reads phase a stretch of a locus, a family gives the phase of a child, or another program phased some loci reliably. The #% line takes a mask per locus (needs 7.0.113 / 6.1.89; older versions read the masks but ignored them):

#% N01: * 0 * * 0
#% rid1324: * 00*** 0

(the second line for an infile whose locus 2 has five subloci, the first two phased by long reads).

  • * resolves the phase of the locus.
  • 0 keeps the phase as given; the individual is not phased there. - (no data at this locus) is the same.
  • One character per sublocus: the * subloci are resolved; the 0 subloci keep their phase as given, relative to each other, and anchor the * subloci. The anchored subloci appear in the phase report but never change.

A name alone (#% N01:) resolves every locus. The masks need not be written by hand: haplotype_blocks.py writes one per locus (0 at loci kept with --keep-loci and at loci where the individual has fewer than two heterozygous blocks, * elsewhere), and with --known FILE it writes the per-sublocus masks from a list of known sites:

# individual  locus  sites (1-based within the locus)
N01  2  all
N02  1  1-250
N03  *  all
N04  4  30-200,400-500
S01  3  1-100
python3 ../../tools/haplotype_blocks.py data/unphased.iupac unphased.known --known known.txt

It cuts so that no block mixes known and unknown heterozygous sites of an individual (known sites may share a block, so locus 3 needs 26 instead of 28 blocks here), and writes for example

#% N02: 000*0***00******* * * * 0

An individual whose phase is fully known is not listed; one with unknown names, loci or site ranges stops the tool with an error. cut_every_site.py spreads a per-sublocus mask over the new subloci.

Two tests on the blocks file (same settings as section 6):

RunTheta_NorthTheta_SouthM_South->NorthM_North->SouthMode = true phaseP(true)Switch accuracy
no known phase (section 7)0.00760.0067286967 of 760.800.98
every other heterozygous block anchored0.00800.007411210670 of 760.900.99
--known list above0.00830.00688310963 of 710.820.98

In the first test (data/unphased.anchors) every other heterozygous block of every individual was set to its true phase and masked 0, the rest left at the arbitrary phase and *; in the second (data/unphased.known) the sites of the list above (data/known.txt) were set to their true phase (data/known.two) and haplotype_blocks.py --known wrote the masks. scripts/known_phase_tests.py makes all four files. Anchoring half of the blocks raises P(true) from 0.80 to 0.90: the known part removes some of the alternative phases. In the second test N03 is fully known and drops out, and the anchored parts are a small share of the data, so the scores change little. With the masks * 0 * - * for every individual, loci 2 and 4 were not phased and the other three were.

The known phase must be right: a 0 block in the wrong phase stays wrong, and the * blocks are then resolved relative to it.

9. Writing the phased infile

phased-infile=YES:infile.phased (7.0.114 / 6.1.90) writes, at the end of a haplotyping run, the infile with the most probable phase of every individual and locus: the posterior mode of the phase report. At every heterozygous sublocus whose mode state is 1 the two copies are swapped; everything else is copied from the input. The blocks and every-site parmfiles of this recipe have the line; it does not change the run (same seed, same outfile). The file starts with one comment line per individual and locus:

# N01 locus 1: mode 0101011100 probability 0.995
# N02 locus 1: mode 010011110 probability 0.531
# N04 locus 1: mode 0100110001 probability 0.692

The mode is taken for each individual and locus on its own, not as the best joint configuration of all individuals. The #% lines are kept. data/infile.phased is the file of the blocks run of section 6; scripts/score_phased_infile.py compares it with the true haplotypes:

python3 scripts/score_phased_infile.py data/infile.phased data/phased.txt

67 of 76 individual-loci have the true phase (switch accuracy 0.98), as the report said (section 7). The probability of the mode tells which ones to trust:

Probability of the modeIndividual-lociTrue phase
0.95 or more3535
0.8-0.951212
0.5-0.82418
below 0.552

Analysed with haplotyping=NO (parmfiles/parmfile.modephase), the phased file gives estimates close to those of the true haplotypes, unlike the arbitrary phase:

DataTheta_NorthTheta_SouthM_South->NorthM_North->SouthGenealogy acceptance
true haplotypes0.00810.0079801260.155
phased infile (mode), haplotyping=NO0.00790.0076681020.157
arbitrary phase as known0.01960.018813140.077

Still, the phased file treats a phase that is right with probability 0.5 as known. For migrate itself keep haplotyping on, which integrates over the phase; the phased file is for programs that need haplotypes, for plots and haplotype networks, or as the start of a further haplotyping run (it keeps the #% lines). Report the uncertain individual-loci with it.

10. What next

  • A run without heating and longer runs, to see how much of the slow phase mixing heating removes.
  • Replicate data sets (as in 05-model-selection) for the coverage of Theta and M under haplotyping.
  • 06-assignment: assigning individuals of unknown origin; the two can be combined (an unphased individual of unknown origin).
  • Real data: tools/haplotype_blocks.py works on any new-syntax sequence infile with IUPAC codes; check its summary for individuals with very many heterozygous sites (--max-het).

References

  • Stephens M, Smith NJ, Donnelly P (2001) A new statistical method for haplotype reconstruction from population data. Am J Hum Genet 68:978-989.

Available Materials