Compute worker reference data folder

[ObadaS]

Description

Add a folder to put reference data in on the compute worker, similar to /codabench/data. This folder will only be mounted during the scoring phase.

Checklist

• Code review by me
• Hand tested by me
• I'm proud of my work
• Code review by reviewer
• Hand tested by reviewer
• CircleCi tests are passing
• Ready to merge

[Didayolo]

Note: test that the folder is not accessible during ingestion

[Didayolo]

Change (fix I made)

Line 526

        self.data_dir = os.path.join(Settings.HOST_DIRECTORY, "data")  # absolute path to data in the host
        self.reference_dir = os.path.join(Settings.BASE_DIR, "reference")  # path inside the worker container

Instead of:

        self.data_dir = os.path.join(Settings.HOST_DIRECTORY, "data")  # absolute path to data in the host
        self.reference_dir = os.path.join(Settings.HOST_DIRECTORY, "reference")

Why

self.reference_dir is used with the _get_host_path convertion:

Line 1283

            # Reference data, only for the scoring phase.
            volumes_host.extend([self._get_host_path(self.reference_dir)])

While self.data_dir is not:

Line 1262

        volumes_host = [
            self._get_host_path(program_dir),
            self._get_host_path(self.output_dir),
            self.data_dir,
            self._get_host_path(self.root_dir, "submission")
        ]

By the way, we could update the code here (line 1262) to convert self.data_dir in the same way we convert the other paths.