Below is an official ranking table generated by the CASP12 contact prediction assessors in their paper. As shown here, our method RaptorX-Contact has overall the best rank. RaptorX-Contact was not fully implemented when participating in CASP12 (May 1, 2016 -- July 20, 2016). A more mature implementation of our method has much better performance than that used in CASP12. See our PLoS CB and CASP12 papers for details. In this table, Full List means that only the submitted contacts with probability score > 0.5 are considered, but this is logically flawed since you can scale up the predicted probability scores (without changing the predicted contact ranking order) so that all your predicted contacts would be considered. The CASP12 assessors also proposed a logically flawed metric F1(prob). Here is the explanation why F1(prob) and Full List are flawed.

We also generated a ranking list in terms of the total F1 score when top L/5 long- and medium-range contacts are evaluated. Ranking does not change much when top L/2 contacts are evaluated. However, it does not necessarily mean that F1 is the best metric to rank contact predictors since a predicted contact map with a higher F1 score may not lead to a better 3D modeling. Ultimately, we need to evaluate how much a predicted contact map can help with 3D modeling, but this is very challenging since there is no a single criterion to choose top predicted contacts to assist folding.

 

Group Name

Rank

F1

Precision

Recall

RaptorX-Contact   

1

12.386

55.831

7.029

MetaPSICOV        

2

10.919

51.307

6.155

iFold_1            

3

10.917

50.916

6.153

MULTICOM-CONSTRUCT

4

10.847

50.449

6.121

Pcons-net         

5

10.810

49.536

6.113

RBO-Epsilon       

6

10.736

48.601

6.081

FALCON_COLORS     

7

10.387

47.253

5.880

Yang-Server       

8

10.186

46.460

5.763

Deepfold-Contact  

9

10.003

46.442

5.644

PconsC31          

10

 9.734

45.728

5.483

IGBteam           

11

 9.428

45.596

5.288

MULTICOM-CLUSTER  

12

 9.114

42.661

5.138

naive             

13

 9.016

42.009

5.085

raghavagps        

14

 9.005

40.269

5.110

Shen-Group        

15

 8.984

40.938

5.085

AkbAR             

16

 8.901

41.417

5.021

MULTICOM-NOVEL    

17

 8.235

36.630

4.680

Zhang_Contact     

18

 8.111

38.259

4.570

PLCT              

19

 7.690

35.133

4.356

PconsC2           

20

 7.347

34.976

4.132

Distill           

21

 5.889

27.762

3.315

FLOUDAS_SERVER    

22

 5.371

24.255

3.041

BG2               

23

 5.073

25.475

2.831

BAKER_GREMLIN     

24

 5.072

25.451

2.831

myprotein-me      

25

 4.442

21.699

2.485

ZHOU-SPARKS-X     

26

 3.982

19.030

2.236

Wang4             

27

 3.547

17.193

1.988

Wang2             

28

 2.664

11.725

1.517

FLOUDAS           

29

 2.566

11.617

1.452

RRCpred           

30

 2.196

10.051

1.243

Wang3             

31

 2.163

 9.359

1.235

Wang1              

32

 1.825

 7.733

1.046

KScons            

33

 0.505

 2.521

0.281

FONT

34

 0.160

 0.857

0.088