The Probabilistic Automata learning Competition (PAutomaC) was the first on-line challenge about learning non-deterministic probabilistic finite state machines (HMM, PFA, …). It happened in the context of ICGI 2012 and its official website was here.
As this site is sometime down, you can find most of the information on this page.
Article
The paper describing the competition, its context, the data, and its results can be found here (the corresponding bibtex to cite it).
Data
The data, now a highly referred benchmark, is available here.
Results
Each submission was evaluated using the perplexity over a test set :
where 
To decide the final overall rank of each participant, we attributed points for each data set: the leader of a problem at the end of the competition scored 5 points, the second 3, the third 2 and the fourth 1. In case of equality on a problem (we considered the 10 first digits of the perplexity score), the earliest submission wins. The winner is the participant whose score was the highest.
Here is the final results:
| Rank | Team name | Overall score |
| 1 | Shibata Yoshinaka | 212 |
| 2 | Mans Hulden | 124 |
| 3 | David Llorens | 122 |
| 4 | Raphael Bailly | 75 |
| 5 | Fabio Kepler | 14 |
And here is the details for each dataset:
| Problem 1 | Minimal perplexity (solution): 29.8978935527 First: Shibata Yoshinaka (29.9942067897) Second: Mans Hulden (30.1311274219) Third: Raphael Bailly (30.1470533439) Fourth: David Llorens (30.3945552209) |
Problem 2 | Minimal perplexity (solution): 168.330805339 First: David Llorens (168.420077615) Second: Raphael Bailly (168.429251783) Third: Shibata Yoshinaka (168.430021868) Fourth: Mans Hulden (168.455164033) |
||
| Problem 3 | Minimal perplexity (solution): 49.956082986 First: Shibata Yoshinaka (50.0418507948) Second: Mans Hulden (50.0436956232) Third: Raphael Bailly (50.1738154718) Fourth: David Llorens (50.6745782951) |
Problem 4 | Minimal perplexity (solution): 80.8184226132 First: Shibata Yoshinaka (80.8273603669) Second: Mans Hulden (80.8366169447) Third: David Llorens (80.8428256389) Fourth: Raphael Bailly (80.8444904746) |
||
| Problem 5 | Minimal perplexity (solution): 33.2352988504 First: Shibata Yoshinaka (33.2372712605) Second: Raphael Bailly (33.2374213986) Third: David Llorens (33.2375906271) Fourth: Mans Hulden (33.2407049655) |
Problem 6 | Minimal perplexity (solution): 66.9849579244 First: David Llorens (66.999870186) Second: Shibata Yoshinaka (67.0071283713) Third: Mans Hulden (67.0435371541) Fourth: Raphael Bailly (67.0591279193) |
||
| Problem 7 | Minimal perplexity (solution): 51.2242694583 First: Shibata Yoshinaka (51.2490257063) Second: David Llorens (51.259252935) Third: Raphael Bailly (51.2638591454) Fourth: Mans Hulden (51.2652735083) |
Problem 8 | Minimal perplexity (solution): 81.3750634047 First: Shibata Yoshinaka (81.4030796991) Second: Mans Hulden (81.7101718832) Third: David Llorens (81.7106926455) Fourth: Raphael Bailly (81.7993032504) |
||
| Problem 9 | Minimal perplexity (solution): 20.8395901703 First: David Llorens (20.8504042224) Second: Shibata Yoshinaka (20.8556889232) Third: Mans Hulden (20.8890396115) Fourth: Raphael Bailly (25.2291177014) |
Problem 10 | Minimal perplexity (solution): 33.3030058501 First: Shibata Yoshinaka (33.3340448672) Second: Mans Hulden (33.4013200502) Third: Raphael Bailly (33.7240544494) Fourth: David Llorens (34.039402651) |
||
| Problem 11 | Minimal perplexity (solution): 31.8113642161 First: Shibata Yoshinaka (31.8527663573) Second: Raphael Bailly (32.1377252373) Third: Mans Hulden (32.1380609252) Fourth: David Llorens (32.5464384875) |
Problem 12 | Minimal perplexity (solution): 21.655287002 First: Shibata Yoshinaka (21.6634964464) Second: Raphael Bailly (21.6709158854) Third: Mans Hulden (21.6710198077) Fourth: David Llorens (21.7690034354) |
||
| Problem 13 | Minimal perplexity (solution): 62.8058396015 First: David Llorens (62.8162371307) Second: Shibata Yoshinaka (62.8201155862) Third: Mans Hulden (63.0727402033) Fourth: Raphael Bailly (100.680861723) |
Problem 14 | Minimal perplexity (solution): 116.791881846 First: Shibata Yoshinaka (116.83624651) Second: David Llorens (116.839012269) Third: Mans Hulden (116.840857711) Fourth: Raphael Bailly (116.913796957) |
||
| Problem 15 | Minimal perplexity (solution): 44.2420495474 First: Shibata Yoshinaka (44.2742933327) Second: Mans Hulden (44.2845628737) Third: David Llorens (44.7006913136) Fourth: Fabio Kepler (45.2081312742) |
Problem 16 | Minimal perplexity (solution): 30.7110624887 First: David Llorens (30.7185779881) Second: Shibata Yoshinaka (30.718658311) Third: Mans Hulden (30.8442547844) Fourth: Fabio Kepler (31.8085930297) |
||
| Problem 17 | Minimal perplexity (solution): 47.3112160937 First: Shibata Yoshinaka (47.3523770757) Second: Mans Hulden (47.3543279913) Third: David Llorens (47.9215129705) Fourth: Fabio Kepler (48.1088694268) |
Problem 18 | Minimal perplexity (solution): 57.3288608287 First: Shibata Yoshinaka (57.3315532127) Second: David Llorens (57.3319679745) Third: Mans Hulden (57.3391495076) Fourth: Fabio Kepler (57.53358215) |
||
| Problem 19 | Minimal perplexity (solution): 17.8768660563 First: Shibata Yoshinaka (17.879926413) Second: David Llorens (17.9194921096) Third: Mans Hulden (17.9298641387) Fourth: Fabio Kepler (18.816377518) |
Problem 20 | Minimal perplexity (solution): 90.9717263176 First: Shibata Yoshinaka (90.9992603081) Second: Mans Hulden (91.016087013) Third: Raphael Bailly (91.351157888) Fourth: David Llorens (93.5036207506) |
||
| Problem 21 | Minimal perplexity (solution): 30.518860165 First: Shibata Yoshinaka (30.5683255996) Second: Mans Hulden (30.6048696431) Third: Raphael Bailly (30.7140208919) Fourth: David Llorens (32.2169198508) |
Problem 22 | Minimal perplexity (solution): 25.9815361778 First: Shibata Yoshinaka (25.988088295) Second: Raphael Bailly (26.0103173257) Third: Mans Hulden (26.0775047173) Fourth: David Llorens (26.080426128) |
||
| Problem 23 | Minimal perplexity (solution): 18.4081615041 First: Shibata Yoshinaka (18.4131290179) Second: Mans Hulden (18.4176370219) Third: David Llorens (18.4473506829) Fourth: Raphael Bailly (18.5473422635) |
Problem 24 | Minimal perplexity (solution): 38.7287795405 First: Shibata Yoshinaka (38.7317292927) Second: David Llorens (38.7321511272) Third: Mans Hulden (38.7367779951) Fourth: Raphael Bailly (38.7529827589) |
||
| Problem 25 | Minimal perplexity (solution): 65.7350539501 First: Shibata Yoshinaka (65.7831272154) Second: Mans Hulden (65.9783424361) Third: Raphael Bailly (66.0692698647) Fourth: David Llorens (67.2655744007) |
Problem 26 | Minimal perplexity (solution): 80.7427626831 First: Shibata Yoshinaka (80.83304279) Second: David Llorens (80.8368259964) Third: Mans Hulden (82.6574456624) Fourth: Fabio Kepler (111.502198108) |
||
| Problem 27 | Minimal perplexity (solution): 42.427078513 First: David Llorens (42.4558720951) Second: Shibata Yoshinaka (42.4643259335) Third: Mans Hulden (42.4731370395) Fourth: Raphael Bailly (42.7120658143) |
Problem 28 | Minimal perplexity (solution): 52.7435104626 First: Shibata Yoshinaka (52.8408850963) Second: Mans Hulden (52.8554872243) Third: Raphael Bailly (53.0842735756) Fourth: David Llorens (53.1975964081) |
||
| Problem 29 | Minimal perplexity (solution): 24.0308339109 First: Shibata Yoshinaka (24.0415850092) Second: David Llorens (24.1059251738) Third: Mans Hulden (24.1993622441) Fourth: Raphael Bailly (24.817229785) |
Problem 30 | Minimal perplexity (solution): 22.925985377 First: Mans Hulden (22.9324399616) Second: Shibata Yoshinaka (22.9342971336) Third: Raphael Bailly (22.959864738) Fourth: David Llorens (23.2114632503) |
||
| Problem 31 | Minimal perplexity (solution): 41.2136431636 First: Shibata Yoshinaka (41.2334334527) Second: Mans Hulden (41.242875789) Third: Raphael Bailly (41.4169157107) Fourth: David Llorens (41.6228893864) |
Problem 32 | Minimal perplexity (solution): 32.6134162732 First: David Llorens (32.6193653586) Second: Shibata Yoshinaka (32.6215722582) Third: Mans Hulden (32.7427172024) Fourth: Raphael Bailly (38.2998438203) |
||
| Problem 33 | Minimal perplexity (solution): 31.8650289444 First: Shibata Yoshinaka (31.8708456878) Second: Mans Hulden (31.8720393486) Third: Raphael Bailly (31.9204127484) Fourth: David Llorens (32.0299349334) |
Problem 34 | Minimal perplexity (solution): 19.9546848395 First: Shibata Yoshinaka (19.9685424755) Second: Mans Hulden (20.4280156428) Third: Raphael Bailly (20.4758085653) Fourth: David Llorens (20.5422570638) |
||
| Problem 35 | Minimal perplexity (solution): 33.776935538 First: Shibata Yoshinaka (33.8003689017) Second: Raphael Bailly (33.835031553) Third: David Llorens (34.2954696648) Fourth: Mans Hulden (34.3258837259) |
Problem 36 | Minimal perplexity (solution): 37.985692906 First: Shibata Yoshinaka (38.0178533505) Second: Raphael Bailly (38.175541431) Third: Mans Hulden (38.2025866842) Fourth: Fabio Kepler (38.2061985438) |
||
| Problem 37 | Minimal perplexity (solution): 20.9797622037 First: Shibata Yoshinaka (21.0009797709) Second: Mans Hulden (21.0159257337) Third: David Llorens (21.0159990401) Fourth: Fabio Kepler (21.0254753292) |
Problem 38 | Minimal perplexity (solution): 21.4457989928 First: Shibata Yoshinaka (21.4586574169) Second: Mans Hulden (21.4938249892) Third: Raphael Bailly (21.5143686533) Fourth: David Llorens (21.5957107834) |
||
| Problem 39 | Minimal perplexity (solution): 10.0020442634 First: Mans Hulden (10.0029067359) Second: Shibata Yoshinaka (10.0033975406) Third: David Llorens (10.0036814974) Fourth: Raphael Bailly (10.0047386563) |
Problem 40 | Minimal perplexity (solution): 8.2009545433 First: David Llorens (8.2056273558) Second: Shibata Yoshinaka (8.2065172057) Third: Mans Hulden (8.2552841778) Fourth: Fabio Kepler (8.3657560064) |
||
| Problem 41 | Minimal perplexity (solution): 13.9124713717 First: Shibata Yoshinaka (13.9209608465) Second: Raphael Bailly (13.9322712597) Third: David Llorens (13.9401938505) Fourth: Mans Hulden (13.9406884683) |
Problem 42 | Minimal perplexity (solution): 16.0037636643 First: David Llorens (16.005128979) Second: Shibata Yoshinaka (16.0065476587) Third: Raphael Bailly (16.0080509451) Fourth: Mans Hulden (16.0080772707) |
||
| Problem 43 | Minimal perplexity (solution): 32.6370243149 First: Shibata Yoshinaka (32.7230067617) Second: Mans Hulden (32.7467889092) Third: David Llorens (32.7765507902) Fourth: Raphael Bailly (32.8174850182) |
Problem 44 | Minimal perplexity (solution): 11.7089059654 First: Shibata Yoshinaka (11.7252949184) Second: Raphael Bailly (11.7780498805) Third: Mans Hulden (11.7981937504) Fourth: Fabio Kepler (11.9197914258) |
||
| Problem 45 | Minimal perplexity (solution): 24.0422109361 First: David Llorens (24.0447630792) Second: Mans Hulden (24.0481772965) Third: Shibata Yoshinaka (24.0499661118) Fourth: Raphael Bailly (24.0837116426) |
Problem 46 | Minimal perplexity (solution): 11.9819819343 First: Shibata Yoshinaka (11.987760725) Second: Mans Hulden (11.9985351577) Third: Raphael Bailly (12.0815197404) Fourth: David Llorens (12.1060872896) |
||
| Problem 47 | Minimal perplexity (solution): 4.1189756456 First: David Llorens (4.1191471501) Second: Shibata Yoshinaka (4.1191548104) Third: Raphael Bailly (4.1195914266) Fourth: Mans Hulden (4.1244230576) |
Problem 48 | Minimal perplexity (solution): 8.0362199917 First: Shibata Yoshinaka (8.0386767334) Second: Mans Hulden (8.0415999503) Third: Raphael Bailly (8.0449144221) Fourth: Fabio Kepler (8.1827678063) |
Real data ranking
Here are the leading participants for the real data problem.
| Problem 1 | First: Mans Hulden (70.5808700051) Second: David Llorens (70.5992773102) Third: 3-gram Baseline (70.6452538406) Fourth: Fabio Kepler (70.6588545428) |
Problem 2 | First: David Llorens (5.1242718819) Second: Mans Hulden (5.1263087907) Third: Fabio Kepler (5.1281949353) Fourth: 3-gram Baseline (5.1281949436) |
