Difference between revisions of "Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus"

(Study outcomes)
 
(8 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
__NUMBEREDHEADINGS__
 
__NUMBEREDHEADINGS__
== Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus ==
+
=== Citation ===
 
David E. Condon, Phu V. Tran, Yu-Chin Lien, Jonathan Schug, Michael K. Georgieff, Rebecca A. Simmons and Kyoung-Jae Won, Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus, 2018, BMC Bioinformatics, 19:31.
 
David E. Condon, Phu V. Tran, Yu-Chin Lien, Jonathan Schug, Michael K. Georgieff, Rebecca A. Simmons and Kyoung-Jae Won, Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus, 2018, BMC Bioinformatics, 19:31.
  
Line 6: Line 6:
  
 
=== Summary ===
 
=== Summary ===
The paper considers identification of differentially methylated regions (DMRs) from bisulfite sequencing data (BSSEQ). A new package (defiant) is introduced. The paper claims that shows superior performance to other approaches as shown in analyses of a series of benchmarking tests on artificial and real data.
+
The paper considers identification of differentially methylated regions (DMRs) from bisulfite sequencing data (BSSEQ). A new package (defiant) is introduced. The paper claims that Defiant shows superior performance to other approaches as shown in analyses of a series of benchmarking tests on artificial and real data.
  
 
=== Study outcomes ===
 
=== Study outcomes ===
Line 12: Line 12:
 
The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of precision, recall and false negatives. The following result is obtained:
 
The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of precision, recall and false negatives. The following result is obtained:
 
* Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads
 
* Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads
* Sometimes precision and recall equal to zero is obtained which might indicate switched class labels.
+
* Sometimes precision and recall equal to zero is obtained which which is a strange outcome because the inverse predictor (i.e. predict DMRs by non-DMR calls of the algorithm) would have better performance.
  
 
Outcome O1 is presented as Figure 3 in the original publication.  
 
Outcome O1 is presented as Figure 3 in the original publication.  
Line 32: Line 32:
 
* The paper presents a new approach (defiant) and at the same times provides several analyses for comparing the performance of the new approach with existing algorithms. Such a study setting is very frequently found in the literature but has a high risk for biased outcomes. One reason for such a bias might be that typically application examples are selected to nicely demonstrate performance benefits. Moreover, new approaches are often established if existing methods have minor performance in a new application setup. For such a setup, a new approach then has good chances to outperform and it remains rather unclear how performance comparisons translates to new application settings.
 
* The paper presents a new approach (defiant) and at the same times provides several analyses for comparing the performance of the new approach with existing algorithms. Such a study setting is very frequently found in the literature but has a high risk for biased outcomes. One reason for such a bias might be that typically application examples are selected to nicely demonstrate performance benefits. Moreover, new approaches are often established if existing methods have minor performance in a new application setup. For such a setup, a new approach then has good chances to outperform and it remains rather unclear how performance comparisons translates to new application settings.
 
* The same simulation data was used as in [27]. Therefore, the outcomes of both studies are comparable, at least with respect to the analyzed data.
 
* The same simulation data was used as in [27]. Therefore, the outcomes of both studies are comparable, at least with respect to the analyzed data.
* There
+
* On top of the presented benchmark results, experimental data is investigated in the paper. However, only qualitative and descriptive comparisons were feasible for this real data sets.
  
 
==== Design for Outcome O1 and O2 ====
 
==== Design for Outcome O1 and O2 ====
 
* 16 "benchmark" data sets were analyzed taken from [27]
 
* 16 "benchmark" data sets were analyzed taken from [27]
* The outcome of O1 might indicate switched class labels. The outcome O2 would also be in agreement with switched class labels because also F1-score equals to zero occurs.
+
* The outcomes O1 and O2 shows analyses, where precision and recall are zero. In such a case, the inverse predictor has better performance. It might also indicate switched class labels. If this guess holds true, then the performance of MethylKit, MethylSig, RADMeth and RnBeads could be underestimated.
* It is not specified, how wonfiguration parameters of the individual methods were chosen.
+
* It is not specified in the paper, how configuration parameters of the individual methods were chosen.
 +
* No dependency on configuration parameters is investigated.
  
 
=== Further comments and aspects ===
 
=== Further comments and aspects ===

Latest revision as of 11:38, 25 February 2020

1 Citation

David E. Condon, Phu V. Tran, Yu-Chin Lien, Jonathan Schug, Michael K. Georgieff, Rebecca A. Simmons and Kyoung-Jae Won, Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus, 2018, BMC Bioinformatics, 19:31.

https://doi.org/10.1186/s12859-018-2037-1

2 Summary

The paper considers identification of differentially methylated regions (DMRs) from bisulfite sequencing data (BSSEQ). A new package (defiant) is introduced. The paper claims that Defiant shows superior performance to other approaches as shown in analyses of a series of benchmarking tests on artificial and real data.

3 Study outcomes

3.1 Outcome O1

The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of precision, recall and false negatives. The following result is obtained:

  • Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads
  • Sometimes precision and recall equal to zero is obtained which which is a strange outcome because the inverse predictor (i.e. predict DMRs by non-DMR calls of the algorithm) would have better performance.

Outcome O1 is presented as Figure 3 in the original publication. Since "Defiant" is occuring twice in the figure legend and RnBeads is missing, we guess that the magenta markers belong to RnBead (i.e. like the colors chosen in Fig. 4).

3.2 Outcome O2

The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of the F1-score which is the harmonic mean of precision and recall. The following outcome is observed:

  • Metilene is superior
  • Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads

Outcome O2 is presented as Figure 4 in the original publication.

3.3 Further outcomes

  • Defiant was fastest, followed by Metilene. Therefore, both methods which showed best performance are also fastest.
  • Defiant, Metilene and RADMeth required less memory (RAM) than MethylKit, MethylSig and RnBeads

4 Study design and evidence level

4.1 General aspects

  • The paper presents a new approach (defiant) and at the same times provides several analyses for comparing the performance of the new approach with existing algorithms. Such a study setting is very frequently found in the literature but has a high risk for biased outcomes. One reason for such a bias might be that typically application examples are selected to nicely demonstrate performance benefits. Moreover, new approaches are often established if existing methods have minor performance in a new application setup. For such a setup, a new approach then has good chances to outperform and it remains rather unclear how performance comparisons translates to new application settings.
  • The same simulation data was used as in [27]. Therefore, the outcomes of both studies are comparable, at least with respect to the analyzed data.
  • On top of the presented benchmark results, experimental data is investigated in the paper. However, only qualitative and descriptive comparisons were feasible for this real data sets.

4.2 Design for Outcome O1 and O2

  • 16 "benchmark" data sets were analyzed taken from [27]
  • The outcomes O1 and O2 shows analyses, where precision and recall are zero. In such a case, the inverse predictor has better performance. It might also indicate switched class labels. If this guess holds true, then the performance of MethylKit, MethylSig, RADMeth and RnBeads could be underestimated.
  • It is not specified in the paper, how configuration parameters of the individual methods were chosen.
  • No dependency on configuration parameters is investigated.

5 Further comments and aspects

None.

6 References

[27] Jühling F, Kretzmer H, Bernhart SH, Otto C, Stadler PF, Hoffmann S. Metilene: fast and sensitive calling of differentially methylated regions from bisulfite sequencing data. Genome Res. 2015; https://doi.org/10.1101/gr.196394.11