Difference between revisions of "Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus"

(Created page with "__NUMBEREDHEADINGS__ == Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus == David E....")
 
 
(13 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
__NUMBEREDHEADINGS__
 
__NUMBEREDHEADINGS__
== Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus ==
+
=== Citation ===
 
David E. Condon, Phu V. Tran, Yu-Chin Lien, Jonathan Schug, Michael K. Georgieff, Rebecca A. Simmons and Kyoung-Jae Won, Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus, 2018, BMC Bioinformatics, 19:31.
 
David E. Condon, Phu V. Tran, Yu-Chin Lien, Jonathan Schug, Michael K. Georgieff, Rebecca A. Simmons and Kyoung-Jae Won, Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus, 2018, BMC Bioinformatics, 19:31.
  
 
[https://doi.org/10.1186/s12859-018-2037-1 https://doi.org/10.1186/s12859-018-2037-1]
 
[https://doi.org/10.1186/s12859-018-2037-1 https://doi.org/10.1186/s12859-018-2037-1]
 
  
 
=== Summary ===
 
=== Summary ===
The paper considers identification of differentially methylated regions (DMRs) from bisulfite sequencing data (BSSEQ). A new package (defiant) is introduced. The paper claims that shows superior performance to other approaches as shown in analyses of a series of benchmarking tests on artificial and real data.
+
The paper considers identification of differentially methylated regions (DMRs) from bisulfite sequencing data (BSSEQ). A new package (defiant) is introduced. The paper claims that Defiant shows superior performance to other approaches as shown in analyses of a series of benchmarking tests on artificial and real data.
  
 
=== Study outcomes ===
 
=== Study outcomes ===
List the paper results concerning method comparison and benchmarking:
 
 
==== Outcome O1 ====
 
==== Outcome O1 ====
The performance of ...
+
The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of precision, recall and false negatives. The following result is obtained:
 +
* Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads
 +
* Sometimes precision and recall equal to zero is obtained which which is a strange outcome because the inverse predictor (i.e. predict DMRs by non-DMR calls of the algorithm) would have better performance.
  
Outcome O1 is presented as Figure X in the original publication.  
+
Outcome O1 is presented as Figure 3 in the original publication.
 +
Since "Defiant" is occuring twice in the figure legend and RnBeads is missing, we guess that the magenta markers belong to RnBead (i.e. like the colors chosen in Fig. 4).
  
 
==== Outcome O2 ====
 
==== Outcome O2 ====
...
+
The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of the F1-score which is the harmonic mean of precision and recall. The following outcome is observed:
 +
* Metilene is superior
 +
* Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads
  
Outcome O2 is presented as Figure X in the original publication.  
+
Outcome O2 is presented as Figure 4 in the original publication.  
 
   
 
   
==== Outcome On ====
 
...
 
 
Outcome On is presented as Figure X in the original publication.
 
 
 
==== Further outcomes ====
 
==== Further outcomes ====
If intended, you can add further outcomes here.
+
* Defiant was fastest, followed by Metilene. Therefore, both methods which showed best performance are also fastest.
 
+
* Defiant, Metilene and RADMeth required less memory (RAM) than MethylKit, MethylSig and RnBeads
  
 
=== Study design and evidence level ===
 
=== Study design and evidence level ===
 
==== General aspects ====
 
==== General aspects ====
You can describe general design aspects here.
+
* The paper presents a new approach (defiant) and at the same times provides several analyses for comparing the performance of the new approach with existing algorithms. Such a study setting is very frequently found in the literature but has a high risk for biased outcomes. One reason for such a bias might be that typically application examples are selected to nicely demonstrate performance benefits. Moreover, new approaches are often established if existing methods have minor performance in a new application setup. For such a setup, a new approach then has good chances to outperform and it remains rather unclear how performance comparisons translates to new application settings.
The study designs for describing specific outcomes are listed in the following subsections:
+
* The same simulation data was used as in [27]. Therefore, the outcomes of both studies are comparable, at least with respect to the analyzed data.
 +
* On top of the presented benchmark results, experimental data is investigated in the paper. However, only qualitative and descriptive comparisons were feasible for this real data sets.
  
==== Design for Outcome O1 ====
+
==== Design for Outcome O1 and O2 ====
 
* 16 "benchmark" data sets were analyzed taken from [27]
 
* 16 "benchmark" data sets were analyzed taken from [27]
* Configuration parameters were chosen ...
+
* The outcomes O1 and O2 shows analyses, where precision and recall are zero. In such a case, the inverse predictor has better performance. It might also indicate switched class labels. If this guess holds true, then the performance of MethylKit, MethylSig, RADMeth and RnBeads could be underestimated.
* ...
+
* It is not specified in the paper, how configuration parameters of the individual methods were chosen.
==== Design for Outcome O2 ====
+
* No dependency on configuration parameters is investigated.
* The outcome was generated for ...
 
* Configuration parameters were chosen ...
 
* ...
 
 
 
...
 
 
 
==== Design for Outcome O ====
 
* The outcome was generated for ...
 
* Configuration parameters were chosen ...
 
* ...
 
  
 
=== Further comments and aspects ===
 
=== Further comments and aspects ===
 +
None.
  
 
=== References ===
 
=== References ===
 
[27]  Jühling F, Kretzmer H, Bernhart SH, Otto C, Stadler PF, Hoffmann S. Metilene: fast and sensitive calling of differentially methylated regions from bisulfite sequencing data. Genome Res. 2015; https://doi.org/10.1101/gr.196394.11
 
[27]  Jühling F, Kretzmer H, Bernhart SH, Otto C, Stadler PF, Hoffmann S. Metilene: fast and sensitive calling of differentially methylated regions from bisulfite sequencing data. Genome Res. 2015; https://doi.org/10.1101/gr.196394.11

Latest revision as of 11:38, 25 February 2020

1 Citation

David E. Condon, Phu V. Tran, Yu-Chin Lien, Jonathan Schug, Michael K. Georgieff, Rebecca A. Simmons and Kyoung-Jae Won, Defiant: (DMRs: easy, fast, identification and ANnoTation) identifies differentially Methylated regions from iron-deficient rat hippocampus, 2018, BMC Bioinformatics, 19:31.

https://doi.org/10.1186/s12859-018-2037-1

2 Summary

The paper considers identification of differentially methylated regions (DMRs) from bisulfite sequencing data (BSSEQ). A new package (defiant) is introduced. The paper claims that Defiant shows superior performance to other approaches as shown in analyses of a series of benchmarking tests on artificial and real data.

3 Study outcomes

3.1 Outcome O1

The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of precision, recall and false negatives. The following result is obtained:

  • Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads
  • Sometimes precision and recall equal to zero is obtained which which is a strange outcome because the inverse predictor (i.e. predict DMRs by non-DMR calls of the algorithm) would have better performance.

Outcome O1 is presented as Figure 3 in the original publication. Since "Defiant" is occuring twice in the figure legend and RnBeads is missing, we guess that the magenta markers belong to RnBead (i.e. like the colors chosen in Fig. 4).

3.2 Outcome O2

The performance of Defiant, MethylKit, MethylSig, Metilene, RADmeth and RnBeads was compared in terms of the F1-score which is the harmonic mean of precision and recall. The following outcome is observed:

  • Metilene is superior
  • Metilene and Defiant clearly outperform MethylKit, MethylSig, RADmeth and RnBeads

Outcome O2 is presented as Figure 4 in the original publication.

3.3 Further outcomes

  • Defiant was fastest, followed by Metilene. Therefore, both methods which showed best performance are also fastest.
  • Defiant, Metilene and RADMeth required less memory (RAM) than MethylKit, MethylSig and RnBeads

4 Study design and evidence level

4.1 General aspects

  • The paper presents a new approach (defiant) and at the same times provides several analyses for comparing the performance of the new approach with existing algorithms. Such a study setting is very frequently found in the literature but has a high risk for biased outcomes. One reason for such a bias might be that typically application examples are selected to nicely demonstrate performance benefits. Moreover, new approaches are often established if existing methods have minor performance in a new application setup. For such a setup, a new approach then has good chances to outperform and it remains rather unclear how performance comparisons translates to new application settings.
  • The same simulation data was used as in [27]. Therefore, the outcomes of both studies are comparable, at least with respect to the analyzed data.
  • On top of the presented benchmark results, experimental data is investigated in the paper. However, only qualitative and descriptive comparisons were feasible for this real data sets.

4.2 Design for Outcome O1 and O2

  • 16 "benchmark" data sets were analyzed taken from [27]
  • The outcomes O1 and O2 shows analyses, where precision and recall are zero. In such a case, the inverse predictor has better performance. It might also indicate switched class labels. If this guess holds true, then the performance of MethylKit, MethylSig, RADMeth and RnBeads could be underestimated.
  • It is not specified in the paper, how configuration parameters of the individual methods were chosen.
  • No dependency on configuration parameters is investigated.

5 Further comments and aspects

None.

6 References

[27] Jühling F, Kretzmer H, Bernhart SH, Otto C, Stadler PF, Hoffmann S. Metilene: fast and sensitive calling of differentially methylated regions from bisulfite sequencing data. Genome Res. 2015; https://doi.org/10.1101/gr.196394.11