Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ limitations under the License.
# Contributing

Welcome to CLAIMED! If you are interested in contributing to the [CLAIMED code repo](README.md)
then checkout the [Contribution Process](https://github.com/claimed-framework/community/blob/main/contribution_process.md) and
the [Code of Conduct](https://github.com/claimed-framework/component-library/blob/master/CODE_OF_CONDUCT.md)).
then check out the [Contribution Process](https://github.com/claimed-framework/claimed/blob/main/contribution_process.md) and
the [Code of Conduct](https://github.com/claimed-framework/component-library/blob/master/CODE_OF_CONDUCT.md).

The [CLAIMED community repo]([https://github.com/elyra-ai/community](https://github.com/claimed-framework/community)) contains information on how the community
The [CLAIMED community repo](https://github.com/elyra-ai/community) contains information on how the community
is organized and other information that is pertinent to contributing.

### Getting Started

It's encouraged that you look under the [Issues]([https://github.com/IBM/claimed/issues](https://github.com/claimed-framework/component-library/issues)) tab for contribution opportunites.
It's encouraged that you look under the [Issues](https://github.com/claimed-framework/component-library/issues) tab for contribution opportunities.
6 changes: 3 additions & 3 deletions GettingStarted.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,7 @@ Otherwise, you can use absolute paths in your scripts/variables `/<folder>/...`.

You can use key-value secrets in KubeFlow as well to avoid publishing sensible information in pod configs and logs.
You can add the secrets in the Kubernetes project that is running KubeFlow.
Then, you can add secrets to a specfic step in the pipeline with the following code:
Then, you can add secrets to a specific step in the pipeline with the following code:

```python
from kubernetes.client import V1EnvVar, V1EnvVarSource, V1SecretKeySelector
Expand Down Expand Up @@ -413,7 +413,7 @@ Your operator script has to follow certain requirements to be processed by C3. C

- The operator name is the python file: `my_operator_name.py` -> `claimed-my-operator-name`
- The operator description is the first doc string in the script: `"""Operator description"""`
- The required pip packages are listed in comments starting with pip install: `# pip install <package1> <package2>` or `# pip install -r ~/requierments.txt`
- The required pip packages are listed in comments starting with pip install: `# pip install <package1> <package2>` or `# pip install -r ~/requirements.txt`
- The interface is defined by environment variables `my_parameter = os.getenv('my_parameter')`.
- You can cast a specific type by wrapping `os.getenv()` with `int()`, `float()`, `bool()`. The default type is string. Only these four types are currently supported. You can use `None` as a default value but not pass the `NoneType` via the `job.yaml`.
- Output paths for KubeFlow can be defined with `os.environ['my_output_parameter'] = ...'`. Note that operators cannot return values but always have to save outputs in files.
Expand Down Expand Up @@ -543,7 +543,7 @@ Therefore, the code gets wrapped by a coordinator script: The grid wrapper.

### 5.1 C3 grid computing requirements

You can use the same code for the grid wrapper as for an operator by adding an extra functon which is passed to C3.
You can use the same code for the grid wrapper as for an operator by adding an extra function which is passed to C3.
The grid wrapper executes this function in each batch and passes specific parameters to the function:
The first parameter is the batch id, followed by all variables defined in the operator interface.
You need to adapt the variables based on the batch, e.g., by adding the batch id to input and output paths.
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ To do this it relies on a configuration file where the benchmark is defined. Thi

- `run_name`: Name of the parent (top-level) run under the experiment. NOTE: This item should not be included in the config if you wish to use the parameters extraction function in `mlfow_utils` to compile results.

- `defaults`: Defaults that are set for all tasks. Can be overriden under each task.
- `defaults`: Defaults that are set for all tasks. Can be overridden under each task.

- `tasks`: List of tasks to perform. Tasks specify parameters for the decoder, datamodule to be used and training parameters.

Expand Down Expand Up @@ -258,7 +258,7 @@ For instance:
iterate --summarize --config configs/summarize_results.yaml
```

The results and hyperparameters are extracted into a csv file. For example, if `storage_uri` is `/opt/benchmark_experiments/hpo`, then sumarized results will be saved in last file as shown below:
The results and hyperparameters are extracted into a csv file. For example, if `storage_uri` is `/opt/benchmark_experiments/hpo`, then summarized results will be saved in last file as shown below:
```
/opt/benchmark_experiments/
└──hpo_results
Expand Down
2 changes: 1 addition & 1 deletion docs/handover.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ optimization_space:

### Creating terratorch configs

Benchmark does not actually assemble a yaml config file for each `train()`. Instead, it uses the programatic interface of terratorch.
Benchmark does not actually assemble a yaml config file for each `train()`. Instead, it uses the programmatic interface of terratorch.

See e.g. `model_fitting.py:fit_model` and `model_fitting.py:launch_training`

Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ To do this it relies on a configuration file where the benchmark is defined. Thi

- `description`: Description added to top level mlflow run.

- `defaults`: Defaults that are set for all tasks. Can be overriden under each task.
- `defaults`: Defaults that are set for all tasks. Can be overridden under each task.

- `tasks`: List of tasks to perform. Tasks specify parameters for the decoder, datamodule to be used and training parameters.

Expand Down
2 changes: 1 addition & 1 deletion docs/new_backbone.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ The `PrithviModelFactory`, despite its name, is actually quite versatile and mak

As such, instead of passing a string with the name of the backbone, you can pass it an instance of the actual backbone.

There are, however, two conditions that must be fulfiled:
There are, however, two conditions that must be fulfilled:

### Feature info

Expand Down
2 changes: 1 addition & 1 deletion docs/ray.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Instructions for CCC follow.
Please make sure your benchmark works for the single node case first,

!!! warning
This tool uses bayesian optimization, and will parallelize it over the number of GPUs provided. However, bayesian parallelization does not parallelize super well, so using a large number of GPUs is not reccomended.
This tool uses bayesian optimization, and will parallelize it over the number of GPUs provided. However, bayesian parallelization does not parallelize super well, so using a large number of GPUs is not recommended.

With no HPO, this will parallelize over tasks.

Expand Down
2 changes: 1 addition & 1 deletion examples/operator_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
def main(num_values, *args, **kwargs):
# TODO: Add your code.
random_values = np.random.rand(num_values)
# C3 adds setup code to your script which initalize the logging.
# C3 adds setup code to your script which initializes the logging.
# You can just use logging.debug(), logging.info(), logging.warning() in your code.
logging.info(f'Random values: {random_values}')

Expand Down
2 changes: 1 addition & 1 deletion src/claimed/c3/create_containerless_operator.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ def create_containerless_operator(
filename, file_extension = os.path.splitext(file_path)

if file_extension != '.py':
raise NotImplementedError('Containerless operators currenly only support python scripts')
raise NotImplementedError('Containerless operators currently only support python scripts')

all_pip_packages_found = ''
with open(file_path, 'r') as file:
Expand Down
2 changes: 1 addition & 1 deletion src/claimed/c3/create_gridwrapper.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ def main():
parser.add_argument('--keep-generated-files', action='store_true',
help='Do not delete temporary generated files.')
parser.add_argument('--platform', type=str, default='linux/amd64',
help='Select image platform, default is linux/amd64. Alternativly, select linux/arm64".')
help='Select image platform, default is linux/amd64. Alternatively, select linux/arm64".')
parser.add_argument('--image_version', type=str, default='python3.12',
help='Select python or R version (defaults to python3.12).')

Expand Down
2 changes: 1 addition & 1 deletion src/claimed/c3/create_operator.py
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,7 @@ def main():
parser.add_argument('--keep-generated-files', action='store_true',
help='Do not delete temporary generated files.')
parser.add_argument('--platform', type=str, default='linux/amd64',
help='Select image platform, default is linux/amd64. Alternativly, select linux/arm64".')
help='Select image platform, default is linux/amd64. Alternatively, select linux/arm64".')
parser.add_argument('--image_version', type=str, default='python3.12',
help='Select python or R version (defaults to python3.12).')
parser.add_argument('--skip-docker-build', action='store_true',
Expand Down
2 changes: 1 addition & 1 deletion src/claimed/components/analyze/spark-ts-trends.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: spark-ts-trends
description: Computes trends of time series by fitting a low order polinomial
description: Computes trends of time series by fitting a low order polynomial


inputs:
Expand Down
2 changes: 1 addition & 1 deletion src/claimed/components/deploy/condition-blessing.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: model-blessing
description: Assesses metrics of an asset (ML model, dashboard, report, ...) and decides on readyness for production deployment.
description: Assesses metrics of an asset (ML model, dashboard, report, ...) and decides on readiness for production deployment.


inputs:
Expand Down
4 changes: 2 additions & 2 deletions src/claimed/components/input/input-Xview-download.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ inputs:
- {name: password, type: String, description: "password for the Xview webpage to authorize login"}
- {name: move_to_dir, type: String, description: "move_to_dir the directory where the dataset should be saved"}
- {name: chromedriver_path, type: String, description: "chromedriver_path the directory where the local copy of chromedriver is saved"}
- {name: max_download_time, type: String, description: "max_download_time before timeout, must be ajusted acording to the file size and internet speed"}
- {name: label, type: String, description: "standing for TI=Traning Images, TL=Training Lables, VI=Validation Images"}
- {name: max_download_time, type: String, description: "max_download_time before timeout, must be adjusted according to the file size and internet speed"}
- {name: label, type: String, description: "standing for TI=Training Images, TL=Training Labels, VI=Validation Images"}


outputs:
Expand Down
2 changes: 1 addition & 1 deletion src/claimed/components/input/input-codenet-LangClass.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: Pulls Codenet classification data from the ml-exchange.org
description: Pulls Codenet classification data.zip from the ml-exchange.org in a form ready for text classification in the folowing format:
description: Pulls Codenet classification data.zip from the ml-exchange.org in a form ready for text classification in the following format:


inputs:
Expand Down
2 changes: 1 addition & 1 deletion src/claimed/components/transform/spark-sql.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: SparkSQL
description: Execute arbitrary SQL queries againts CSV and PARQUET files
description: Execute arbitrary SQL queries against CSV and PARQUET files

inputs:
- {name: data_file, type: String, description: 'file name for CSV or PARQUET file - must end with .csv or .parquet (default: data.csv)'}
Expand Down
4 changes: 2 additions & 2 deletions src/claimed/iterate/plot_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ def average_seeds(


def extract_1x_data(df_all):
"""Extract only resutls trained on 100% of the data"""
"""Extract only results trained on 100% of the data"""
return df_all[
(df_all["partition_name"] == "1.00x train")
| (df_all["partition_name"] == "default")
Expand Down Expand Up @@ -119,7 +119,7 @@ def normalize_bootstrap_and_plot(


class Normalizer:
"""Class used to normalize results beween min and max for each dataset."""
"""Class used to normalize results between min and max for each dataset."""

def __init__(self, range_dict):
"""Initialize a new instance of Normalizer class."""
Expand Down
2 changes: 1 addition & 1 deletion tests/test_compiler.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ def test_convert_notebook(
inputs = py.get_inputs()
assert isinstance(inputs, dict), "Inputs is not a dict."
outputs = py.get_outputs()
assert isinstance(outputs, dict), "Ouputs is not a dict."
assert isinstance(outputs, dict), "Outputs is not a dict."
requirements = py.get_requirements()
assert isinstance(requirements, list), "Requirements is not a list."

Expand Down