Evaluating submissions

We will evaluate submissions based on predictive validity, measured in the held-out test data by mean squared error loss for continuous outcomes and Brier loss for binary outcomes. This means it is in your interest to submit probability predictions for the binary outcomes, rather than classification predictions.

A leaderboard will rank submissions according to these criteria, using a set of held-out data. After the challenge closes, we will produce a finalized ranking of submissions based on a separate set of withheld true outcome data.

For more information, see our post about evaluating submissions.

Terms and conditions

To access the data under the Participate tab, you will complete a consent form with the following statements.

Confidentiality Agreement

1. I understand that each member of my research team must request their own copy of the data files. I understand that each member of my research team must request their own copy of the data files. Files cannot be transferred. I will not at any time give, sell, show, disclose or otherwise disseminate the FFCWS data to anyone.

2. I agree to store and process the data in a secure manner. I will take steps to ensure that the data is safe-guarded, using protections such as password-protected access to all computers storing the data.

3. I agree to destroy all copies of the FFCWS after the conclusion of this project. If I plan to continue research with this data, I will download a new version of the FFCWS data through the Princeton University Office of Population Research (OPR) Data Archive at: http://opr.princeton.edu/archive/.

4. I agree not to attempt to identify individuals, families, households, or hospitals. I also agree that in the event the identity of an individual, family, household, or hospital is discovered inadvertently, I will (a) make no use of this knowledge, (b) inform the PIs, and (c) not inform any other persons of the discovered identity.

5. I agree to report any disclosure of participants or errors in data/documentation to FFCWS staff through ffdata@princeton.edu.

Submission Consent Form

1. I consent for the Fragile Families Challenge to publish my submitted data, code, and narrative explanation publicly.

2. The Fragile Families Challenge may use my predictions and code to produce a collaborative model.

3. If at any point you wish to remove your data, code, or narrative explanation from the project, please e-mail the project coordinator at ilundberg@princeton.edu.

4. I understand that I may submit up to ten times per day.