A submission that simply uploads the supplied predictions file has errors as follows:
0.39273 0.21997 0.0288 0.05341 0.17435 0.20224
Which, today, put me in the top 50 (!).
If we look at what is in first place across the leaderboard, then a simple %improvement shows the following 'progress' over these benchmarks.
GPA 6.73%
Grit 3.73%
Material hardship 15.59%
Eviction 6.74%
Layoff 2.70%
Job training 4.16%
Early days but predicting layoffs already appears to be quite hard.
Posted by: the_Brit @ April 7, 2017, 6:47 p.m.Compared with three weeks ago, this is the extent of progress in the top leaderboard score compared with the mean prediction.
GPA 6.73% -> 7.21%
Grit 3.73% -> 5.51%
Material hardship 15.59% -> NO CHANGE
Eviction 6.74% -> 12.98%
Layoff 2.70% -2.91%
Job training 4.16% -> NO CHANGE
So, incremental progress with GPA and layoffs; big improvement on evictions, good progress with grit.
With quite a few people seeming to join in, those hardship and training scores continue to look tough to beat and haven't moved for at least 3 weeks (!)..
With a couple of weeks to go, this is progress compared to 28-April and 7-April.
Leaderboard top score compared with the mean prediction.
GPA 6.73% -> 7.21% -> NO CHANGE
Grit 3.73% -> 5.51% -> 5.86%
Material hardship 15.59% -> NO CHANGE -> 16.15%
Eviction 6.74% -> 12.98% -> 14.30%
Layoff 2.70% -> 2.91% -> 3.77%
Job training 4.16% -> NO CHANGE -> 4.43%
No change to GPA scores for the last (nearly) 3 months. My suspicion is that this is an outcome that is heavily 'machine-learned' with lots of input features. Incremental progress for other outcomes. The improvement over the mean is lowest for job layoffs and job training, and greatest for material hardship and for evictions. Of course, this could be reflecting some 'overfitting' to repeated dips into the test data, and scores for the held-out data could be very different and rather worse.
Posted by: the_Brit @ July 18, 2017, 7:05 a.m.Now that the public leaderboard is presumably final, these were the overall improvements over the mean prediction by the end of the competition.
GPA -7.2%
Grit -5.9%
Material hardship -19.7%
Eviction -14.3%
Layoff -4.5%
Job training -4.4%
The GPA top scores have been there for some time, elsewhere scores on material hardship saw some fairly stunning improvements in the last few days.
Interestingly, 21% of the GPA scores are the mean prediction, compared with 39% for job training and layoff. One might, therefore, conclude that more than average modellers' effort went into the GPA scores, and less on some other outcomes? And, generally, fewer took on the classification tasks than the regression/continuous tasks.
Posted by: the_Brit @ Aug. 3, 2017, 2:12 p.m.