McGill ChIP-seq peak detection benchmark

We manually annotated several ChIP-seq data sets from the McGill Epigenomes Portal by visually inspecting them using UCSC genome browser software. When we saw peaks, we created some annotated regions:

When we saw regions without peaks, we created some noPeaks annotated regions (1 or more overlapping peaks is a false positive).

We saved the 7 data sets of annotated regions to a database that can be viewed and downloaded. The original annotation files can be found under the annotations/ subdirectory.

To download the signal, annotated regions, and peak calls, use this R script or this list of data files (all genome positions are relative to hg19).

Each annotation data set is named like H3K4me3_PGP_immune:

These data are useful for benchmarking peak detection algorithms, as was done in the paper Visual annotations and a supervised learning approach for evaluating and calibrating ChIP-seq peak detectors. Some examples of benchmarks that can be computed using these data:

Train error on each data set

We applied several peak detectors to these data sets. Each data set consists of several chunks, which are groups of nearby annotations on the same chromosome.

The tables below show the train error for each algorithm and for each data set:

H3K36me3_AM_immune

Algorithm error.percent FPR.percent FNR.percent
hmcan.broad.trained 5.11 2.69 5.35
hmcan.trained 9.24 5.82 8.38
rseg.trained 10.44 9.93 4.94
rseg.default 10.50 10.00 4.94
sicer.trained 11.76 14.10 1.61
macs.broad.trained 17.21 18.13 5.75
hmcan.broad.default 17.90 21.27 2.72
macs.broad.default 22.26 27.91 1.41
homer.trained 25.93 30.37 4.54
homer.default 25.99 30.60 4.34
macs.trained 26.16 21.04 17.56
hmcan.default 29.09 36.57 1.72
macs.default 34.37 43.43 1.72
sicer.default 51.12 3.28 85.47

H3K36me3_TDH_immune

Algorithm error.percent FPR.percent FNR.percent
hmcan.broad.trained 16.51 11.27 8.25
hmcan.trained 26.03 13.97 19.00
rseg.default 30.79 27.30 5.50
rseg.trained 30.79 27.30 5.50
sicer.trained 42.06 32.86 14.50
macs.broad.trained 44.60 31.59 20.50
hmcan.broad.default 46.83 46.83 0.00
macs.trained 51.43 28.41 36.25
macs.broad.default 54.13 53.65 0.75
homer.trained 56.19 28.73 43.25
homer.default 56.67 54.13 4.00
sicer.default 57.14 19.68 59.00
hmcan.default 64.92 64.92 0.00
macs.default 66.19 65.71 0.75

H3K36me3_TDH_other

Algorithm error.percent FPR.percent FNR.percent
hmcan.broad.trained 4.00 2.00 3.12
hmcan.trained 11.00 5.50 8.59
sicer.trained 19.00 19.00 0.00
rseg.default 22.50 5.00 27.34
rseg.trained 22.50 5.00 27.34
hmcan.broad.default 26.00 25.50 0.78
macs.broad.trained 35.00 32.00 4.69
macs.broad.default 38.00 36.50 2.34
macs.trained 47.50 39.50 12.50
hmcan.default 49.00 49.00 0.00
homer.default 49.00 39.50 14.84
homer.trained 49.00 39.50 14.84
macs.default 57.50 55.50 3.12
sicer.default 63.00 1.00 96.88

H3K4me3_PGP_immune

Algorithm error.percent FPR.percent FNR.percent
macs.trained 17.21 4.32 23.93
macs.broad.trained 19.41 2.00 31.18
macs.default 20.03 4.76 28.24
macs.broad.default 22.48 2.10 36.45
hmcan.trained 24.67 4.32 37.05
homer.default 25.25 2.16 41.22
homer.trained 25.25 2.16 41.22
hmcan.default 28.14 6.70 39.66
hmcan.broad.trained 31.01 4.88 47.36
hmcan.broad.default 33.91 7.20 49.06
sicer.trained 37.90 8.23 54.56
rseg.trained 42.12 10.48 58.69
rseg.default 43.56 13.92 56.17
sicer.default 56.73 2.53 96.01

H3K4me3_TDH_immune

Algorithm error.percent FPR.percent FNR.percent
macs.broad.trained 13.75 3.19 19.69
macs.trained 14.43 6.50 15.38
macs.broad.default 16.80 4.52 23.02
macs.default 17.67 9.48 16.27
hmcan.trained 18.27 5.47 24.11
homer.default 21.46 4.67 31.25
homer.trained 21.46 4.67 31.25
hmcan.broad.trained 21.92 4.31 32.69
hmcan.default 23.75 10.69 25.30
hmcan.broad.default 26.14 9.31 31.99
sicer.trained 31.13 9.34 41.02
rseg.trained 37.99 12.70 47.87
rseg.default 38.67 14.36 46.33
sicer.default 51.42 2.81 88.94

H3K4me3_TDH_other

Algorithm error.percent FPR.percent FNR.percent
macs.broad.trained 6.50 0.95 9.35
macs.trained 6.79 3.39 6.47
macs.default 17.37 12.50 11.27
macs.broad.default 19.49 7.99 20.98
homer.default 21.90 9.46 22.90
homer.trained 21.90 9.46 22.90
hmcan.trained 24.45 3.30 35.61
hmcan.broad.trained 30.80 2.86 46.64
hmcan.default 30.88 12.76 33.09
rseg.trained 30.95 10.85 35.85
rseg.default 31.53 11.89 35.37
sicer.trained 37.30 8.59 49.40
hmcan.broad.default 37.66 13.19 43.65
sicer.default 60.29 1.30 97.24

H3K4me3_XJ_immune

Algorithm error.percent FPR.percent FNR.percent
macs.trained 15.54 4.06 24.89
macs.broad.trained 19.03 3.62 32.74
macs.default 20.26 6.97 29.63
macs.broad.default 28.47 10.14 41.04
homer.trained 32.68 8.47 52.44
homer.default 33.41 11.02 49.63
hmcan.trained 33.99 17.46 40.00
hmcan.default 37.55 18.43 45.63
hmcan.broad.trained 42.12 10.67 68.00
sicer.trained 44.01 10.58 72.00
rseg.trained 46.84 10.58 77.78
sicer.default 47.93 1.85 94.67
hmcan.broad.default 50.33 26.98 57.33
rseg.default 56.21 31.04 62.52