Difference between revisions of "Gene set analysis methods: a systematic comparison"

(First version)
 
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
__NUMBEREDHEADINGS__
 
__NUMBEREDHEADINGS__
== Gene set analysis methods: a systematic comparison ==
+
=== Citation ===
 
Mathur, R., Rotroff, D., Ma, J., Shojaie, A., & Motsinger-Reif, A. , Gene set analysis methods: a systematic comparison, 2018, BioData mining, 11(1), 8.
 
Mathur, R., Rotroff, D., Ma, J., Shojaie, A., & Motsinger-Reif, A. , Gene set analysis methods: a systematic comparison, 2018, BioData mining, 11(1), 8.
  
[https://doi.org/10.1186/s13040-018-0166-8: Permanent link to the paper]
+
[https://doi.org/10.1186/s13040-018-0166-8 Permanent link to the paper]
 
 
  
 
=== Summary ===
 
=== Summary ===
 
Approaches for gene set analyses were assessed by using simulated data that were generated based on a real experimental data set.
 
Approaches for gene set analyses were assessed by using simulated data that were generated based on a real experimental data set.
 +
 +
There are competitive tests (COMP) that uses the distribution of a reference gene set (e.g. all gene that are not in the gene set) as reference and self-contained (SELF) approaches that do not rely on a reference.
 +
 +
* The authors compared four different methods:
 +
** Gene Set Enrichment Analysis (GSEA-SELF and GSEA-COMP)
 +
** Significance Analysis of Function and Expression (SAFE) based on the t-test as gene-wise test and offers Wilcoxon rank sum, Fisher’s Exact Test, Pearson’s Chi-squared type statistic and a t-statistic as global (gene set wide) tests
 +
** sigPathway
 +
** Correlation Adjusted Mean Rank (CAMERA)
 +
  
 
=== Study outcomes ===
 
=== Study outcomes ===
 +
==== Outcome O1: False positives under null distribution ====
 +
The frequency of false-positives was assessed by using a significance level alpha=0.05.
 +
Consequently all approaches (except FET-1k) showed around 5% false-positive or less.
 +
FET-1k ("SAFE with Fisher's Exact test as global statistic") had around than 20%.
  
==== Outcome O1 ====
+
Outcome O1 is presented as Figure 2 in the original publication for the prostate data template and in the "Additional File 1" for the other templates.  
The performance of ...
 
  
Outcome O1 is presented as Figure X in the original publication.  
+
Baseline of this outcome is that all approaches excep FET-1k perform similarly well in terms of false-positives.
  
==== Outcome O2 ====
+
==== Outcome O2: GSEA vs. SAFE vs. sigPathway vs. CAMERA ====
...
+
* sigPathway showed superior performance
 +
* The order in terms of performance seems to be sigPathway > SELF-GSEA-FDR > SELF-GSEA-Q > COMP-GSEA-FDR > SELF-GSEA-Q > CAMERA > SAFE-Wilcoxon
 +
* SAFE-Wilcoxon could NOT detect any differentially regulated pathway(s)
 +
* In general, the performance increases with increasing fraction of regulated genes (parameter \pi in the paper). Similarly, it also increases with increasing effect size \tau. However, "COMP GSEA Q" shows counterintuitive dependency of performance on effect size.
  
Outcome O2 is presented as Figure X in the original publication.  
+
Outcome O2 is presented as Figure 3 in the original publication and in supplemental figures. The numbers are also provided in the supplement.
 
   
 
   
==== Outcome On ====
+
==== Outcome O3: Power of SAFE for different configurations ====
...
+
* SAFE again performs weak for most configurations
 +
* Only "aveDiff-boot" seems to have a good power that also improves with increasing magnitudes \tau of the regulation
 +
* FET-1k, FET-10k (SAFE with Fisher's Exact Test) work partly: both approaches could identify the regulated pathway but the show counterintuitive performance (i.e. decreasing performances for increasing magnitudes of regulation)
 +
 
 +
Outcome O3 is presented as Figure 4 in the original publication.  
  
Outcome On is presented as Figure X in the original publication.
+
==== Outcome O4: Power of GSEA ====
 +
* COMP-GSEA-FDR and Self-GSEA-FDR showed superior performance
 +
* Comp-GSEA-Q and SELF-GSEA-Q showed counterintuitive performance, i.e. the performance deceases with increasing effect size \tau
 +
* Overall, the performance seems to be: SELF-GSEA-FDR > COMP-GSEA-FDR > SELF-GSEA-Q > COMP-GSEA-Q
  
==== Further outcomes ====
+
==== Outcome O5: Overlapping gene sets ====
If intended, you can add further outcomes here.
+
Some genes may occur in several gene sets. For gene sets that overlap with the target gene sets, the power only behaves most reasonable for COMP-GSEA-FDR.
 +
 
 +
Here, reasonable means that and increasing effect size and an increasing proportion of regulated genes should lead to and increasing power.
  
  
 
=== Study design and evidence level ===
 
=== Study design and evidence level ===
 
==== General aspects ====
 
==== General aspects ====
 +
* The authors consider different sizes of the gene sets
 +
* The authors consider different proportions of regulated genes in the gene sets
 +
* The authors consider different magnitudes of the underlying effect size (i.e. log-fold-changes)
 +
* The authors consider three null simulations (without regulation) as reference for outcome O1
 
* In this publication, the authors published a novel simulation approach termed (FANGS)
 
* In this publication, the authors published a novel simulation approach termed (FANGS)
* The authors compared four different methods:
+
* The simulation approach is available in this R package (FANGS) offers the opportunity to reproduce the simulations and repeat the analysis for other gene set methods.
** Gene Set Enrichment Analysis (GSEA)
+
* The authors provide a comprehensive list of the used configuration parameters
** Significance Analysis of Function and Expression (SAFE)
+
* The authors evaluated the following alternative configurations
** sigPathway, and
+
** For GSEA one alternative
** Correlation Adjusted Mean RAnk (CAMERA).
+
** For SAFE five alternative setups
 +
** For sigPathway and CAMERA no other configurations were considered
 +
* Three experimental data sets were used as foundations for simulating data
 +
** prostate cancer (264 cases, 160 controls)
 +
** ischemic stroke (20 cases, 20 controls)
 +
** normal brain tissue (21 cases, 20 controls)
  
 
==== Design for Outcome O1 ====
 
==== Design for Outcome O1 ====
* The outcome was generated for ...
+
* The authors consider three null simulations (without regulation) as reference:
* Configuration parameters were chosen ...
+
** permutation of class labels
* ...
+
** independently sampled expression of all features (=genes)
 +
** centering the simulated data, i.e. set effect size to zero
 +
* Default configuration parameters and the alternative parameters described above were evaluated
 +
* Only the prostat cancer data set was considered as template for simulations
  
==== Design for Outcome O2 ====
+
==== Design for Outcome O2 and O4 ====
* The outcome was generated for ...
+
For GSEA, only GSEA-Q is shown in the main figure 2 (prostate cancer). In contrast, the supplemental figures also show the results for GSEA-FDR (stroke and brain data sets). For outcome O4 also both, GSEA-Q and GSEA-FDR were considered (and GSEA-FDR outperformed GSEA-Q). Figure 4 contains the missing comparison of GSEA-Q and GSEA-FDR
* Configuration parameters were chosen ...
 
* ...
 
  
...
+
* In total, six analyses were performed (3 data sets x 2 regulated pathways):
 +
** One simulated data set is based on simulating differential expression of one pathway
 +
** The analysis was repeated for all three data sets as template
 +
** For each of the three data sets the analysis was repeated by selecting two different pathways as differentially regulated.  
 +
* Default configuration parameters were chosen
  
==== Design for Outcome O ====
+
==== Design for Outcome O3 ====
* The outcome was generated for ...
+
* The weak performance of SAFE for the default configuration in O2 seems to be the motivation for investigation  of other configurations for SAFE
* Configuration parameters were chosen ...
+
* The outcome O3 was only generated for one data set (prostate cancer) and two regulated pathways
* ...
+
 
 +
==== Design for Outcome O5 ====
 +
At least to my understanding, the discussion and interpretation of this outcome is not very clear in the paper.
  
 
=== Further comments and aspects ===
 
=== Further comments and aspects ===
 
+
* Gene sets from MSigDB were used
=== References ===
+
* The authors are aware of the fact that different null hypotheses are tested by the different approaches
The list of cited or related literature is placed here.
+
* sigPathway and CAMERA offers other options that are discussed in the article but not evaluated. For SAGE, multiple setups were investigated.

Latest revision as of 13:54, 25 February 2020

1 Citation

Mathur, R., Rotroff, D., Ma, J., Shojaie, A., & Motsinger-Reif, A. , Gene set analysis methods: a systematic comparison, 2018, BioData mining, 11(1), 8.

Permanent link to the paper

2 Summary

Approaches for gene set analyses were assessed by using simulated data that were generated based on a real experimental data set.

There are competitive tests (COMP) that uses the distribution of a reference gene set (e.g. all gene that are not in the gene set) as reference and self-contained (SELF) approaches that do not rely on a reference.

  • The authors compared four different methods:
    • Gene Set Enrichment Analysis (GSEA-SELF and GSEA-COMP)
    • Significance Analysis of Function and Expression (SAFE) based on the t-test as gene-wise test and offers Wilcoxon rank sum, Fisher’s Exact Test, Pearson’s Chi-squared type statistic and a t-statistic as global (gene set wide) tests
    • sigPathway
    • Correlation Adjusted Mean Rank (CAMERA)


3 Study outcomes

3.1 Outcome O1: False positives under null distribution

The frequency of false-positives was assessed by using a significance level alpha=0.05. Consequently all approaches (except FET-1k) showed around 5% false-positive or less. FET-1k ("SAFE with Fisher's Exact test as global statistic") had around than 20%.

Outcome O1 is presented as Figure 2 in the original publication for the prostate data template and in the "Additional File 1" for the other templates.

Baseline of this outcome is that all approaches excep FET-1k perform similarly well in terms of false-positives.

3.2 Outcome O2: GSEA vs. SAFE vs. sigPathway vs. CAMERA

  • sigPathway showed superior performance
  • The order in terms of performance seems to be sigPathway > SELF-GSEA-FDR > SELF-GSEA-Q > COMP-GSEA-FDR > SELF-GSEA-Q > CAMERA > SAFE-Wilcoxon
  • SAFE-Wilcoxon could NOT detect any differentially regulated pathway(s)
  • In general, the performance increases with increasing fraction of regulated genes (parameter \pi in the paper). Similarly, it also increases with increasing effect size \tau. However, "COMP GSEA Q" shows counterintuitive dependency of performance on effect size.

Outcome O2 is presented as Figure 3 in the original publication and in supplemental figures. The numbers are also provided in the supplement.

3.3 Outcome O3: Power of SAFE for different configurations

  • SAFE again performs weak for most configurations
  • Only "aveDiff-boot" seems to have a good power that also improves with increasing magnitudes \tau of the regulation
  • FET-1k, FET-10k (SAFE with Fisher's Exact Test) work partly: both approaches could identify the regulated pathway but the show counterintuitive performance (i.e. decreasing performances for increasing magnitudes of regulation)

Outcome O3 is presented as Figure 4 in the original publication.

3.4 Outcome O4: Power of GSEA

  • COMP-GSEA-FDR and Self-GSEA-FDR showed superior performance
  • Comp-GSEA-Q and SELF-GSEA-Q showed counterintuitive performance, i.e. the performance deceases with increasing effect size \tau
  • Overall, the performance seems to be: SELF-GSEA-FDR > COMP-GSEA-FDR > SELF-GSEA-Q > COMP-GSEA-Q

3.5 Outcome O5: Overlapping gene sets

Some genes may occur in several gene sets. For gene sets that overlap with the target gene sets, the power only behaves most reasonable for COMP-GSEA-FDR.

Here, reasonable means that and increasing effect size and an increasing proportion of regulated genes should lead to and increasing power.


4 Study design and evidence level

4.1 General aspects

  • The authors consider different sizes of the gene sets
  • The authors consider different proportions of regulated genes in the gene sets
  • The authors consider different magnitudes of the underlying effect size (i.e. log-fold-changes)
  • The authors consider three null simulations (without regulation) as reference for outcome O1
  • In this publication, the authors published a novel simulation approach termed (FANGS)
  • The simulation approach is available in this R package (FANGS) offers the opportunity to reproduce the simulations and repeat the analysis for other gene set methods.
  • The authors provide a comprehensive list of the used configuration parameters
  • The authors evaluated the following alternative configurations
    • For GSEA one alternative
    • For SAFE five alternative setups
    • For sigPathway and CAMERA no other configurations were considered
  • Three experimental data sets were used as foundations for simulating data
    • prostate cancer (264 cases, 160 controls)
    • ischemic stroke (20 cases, 20 controls)
    • normal brain tissue (21 cases, 20 controls)

4.2 Design for Outcome O1

  • The authors consider three null simulations (without regulation) as reference:
    • permutation of class labels
    • independently sampled expression of all features (=genes)
    • centering the simulated data, i.e. set effect size to zero
  • Default configuration parameters and the alternative parameters described above were evaluated
  • Only the prostat cancer data set was considered as template for simulations

4.3 Design for Outcome O2 and O4

For GSEA, only GSEA-Q is shown in the main figure 2 (prostate cancer). In contrast, the supplemental figures also show the results for GSEA-FDR (stroke and brain data sets). For outcome O4 also both, GSEA-Q and GSEA-FDR were considered (and GSEA-FDR outperformed GSEA-Q). Figure 4 contains the missing comparison of GSEA-Q and GSEA-FDR

  • In total, six analyses were performed (3 data sets x 2 regulated pathways):
    • One simulated data set is based on simulating differential expression of one pathway
    • The analysis was repeated for all three data sets as template
    • For each of the three data sets the analysis was repeated by selecting two different pathways as differentially regulated.
  • Default configuration parameters were chosen

4.4 Design for Outcome O3

  • The weak performance of SAFE for the default configuration in O2 seems to be the motivation for investigation of other configurations for SAFE
  • The outcome O3 was only generated for one data set (prostate cancer) and two regulated pathways

4.5 Design for Outcome O5

At least to my understanding, the discussion and interpretation of this outcome is not very clear in the paper.

5 Further comments and aspects

  • Gene sets from MSigDB were used
  • The authors are aware of the fact that different null hypotheses are tested by the different approaches
  • sigPathway and CAMERA offers other options that are discussed in the article but not evaluated. For SAGE, multiple setups were investigated.