Incorrect evaluation result

[ewinge]

I get strange results when evaluating this model:
http://vectors.nlpl.eu/repository/10/11.zip

Serial              Dataset       Num Pairs       Not found             Rho

 1         EN-MC-30.txt              30               0         -0.1146
 2     EN-MEN-TR-3k.txt            3000               5          0.2697
 3     EN-MTurk-287.txt             287              10          0.3212
 4     EN-MTurk-771.txt             771               3          0.1994
 5         EN-RG-65.txt              65               0         -0.0006
 6   EN-RW-STANFORD.txt            2034             947          0.3325
 7    EN-SIMLEX-999.txt             999               2         -0.0299
 8      EN-VERB-143.txt             144              58          0.0305
 9    EN-WS-353-ALL.txt             353               0          0.2661
10    EN-WS-353-REL.txt             252               0          0.2858
11    EN-WS-353-SIM.txt             203               0          0.2514
12        EN-YP-130.txt             130               0          0.1524

The SimLex-999 score is negative, while gensim evaluation of the same model yields a score of 0.4189 with SimLex-999. Am I doing something wrong, or is there a bug somewhere?

[sanjanaramprasad]

Was this resolved?

I just tried it out with glove vectors and these are the results, which don't seem right. Or am I missing something?

Serial              Dataset       Num Pairs       Not found             Rho

     1     EN-MEN-TR-3k.txt            3000               0          0.2485
     2         EN-MC-30.txt              30               0          0.0142
     3     EN-MTurk-771.txt             771               0          0.2482
     4    EN-SIMLEX-999.txt             999               0          0.1687
     5      EN-VERB-143.txt             144               0          0.1959
     6        EN-YP-130.txt             130               0          0.1269
     7   EN-RW-STANFORD.txt            2034              29          0.2241
     8         EN-RG-65.txt              65               0          0.0148
     9    EN-WS-353-ALL.txt             353               0          0.1375
    10    EN-WS-353-SIM.txt             203               0          0.1802
    11    EN-WS-353-REL.txt             252               0          0.0994
    12     EN-MTurk-287.txt             287               0          0.2995
    13  EN-SimVerb-3500.txt            3500               0          0.0798```