Hello! I am a bit confused about submitting predictions for the test set (for columnar, not image data). Here is a bit of the code I use for the training/validation set (I add the relevant lines):
df, y, nas, mapper = proc_df(trn_x, 'Survived', do_scale=True)
md = ColumnarModelData.from_data_frame(PATH, val_idx, df, y.astype(np.float32), cat_flds=cat_vars, bs=24)
m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars),0.04, 1, [1000,500], [0.001,0.01])
m.fit(lr, 3, metrics=[accuracy_thresh(.5)])
Now I am not sure how to get the predictions for the test data. First of all, I tried to bring the test data in the same format as the validation/training data, so I did this:
df_test, _, nas, mapper = proc_df(test_df, 'Survived', do_scale=True, mapper=mapper, na_dict=nas)
However, there is no ‘Survived’ column in the test data. Should I add one by hand (with all the entires 0 for example)? And what should I do after? I was thinking of doing this
md = ColumnarModelData.from_data_frame(PATH, val_idx, df, y.astype(np.float32), cat_flds=cat_vars, bs=24, test_df=df_test)
m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars),0.04, 1, [1000,500], [0.001,0.01])
But I am not sure. Do I still need val_idx in this second call? Also, how do I get the actual predictions for the test set? Is m.predict() the right thing to call?
Thank you!