MlflowTracker

pydantic model mlopus.kedro.MlflowTracker

Bases: MlflowRunMixin, HookWithFactory

Hook to track pipeline information in MLflow.

Find here a fully commented example covering all settings that can be customized in this hook.

field logs: Logs = Logs(enabled=True, path=logs, files=[])

Configure how to track logs.

field report: Report = Report(enabled=True, path=kedro-session.yml)

Configure the session report.

field tags: Tags = Tags(report=False, values={}, mlflow=TagsMlflow(prefix=[], suffix=[], enabled=True))

Configure extra tags.

field metrics: Metrics = Metrics(report=False, datasets=[], mlflow=MetricsMlflow(prefix=[], suffix=[], enabled=True, prepend_dataset=True))

Configure the collection of MLflow metrics.

field params: Params = Params(report=True, mappings={}, mlflow=ParamsMlflow(prefix=[], suffix=[], enabled=True))

Configure the collection of MLflow params.

field overrides: Overrides = Overrides(rules=[], scopes=[], report=False, mlflow=ConfigMlflow(rules=[], scopes=[], prefix=[], suffix=[], enabled=False))

Configure how to track Kedro config overrides.

field config: Config = Config(rules=[], scopes=[], report=False, mlflow=ConfigMlflow(rules=[], scopes=[], prefix=[], suffix=[], enabled=False))

Configure how to track the Kedro config.

field nodes: Nodes = Nodes(rules=[], report=True, mlflow=NodesMlflow(rules=[], prefix=[], suffix=[], prepend_pipeline=True, enabled=False))

Configure how to track Kedro node settings.

field datasets: Datasets = Datasets(rules=[], report=True, include_non_pydantic=False, mlflow=DatasetsMlflow(rules=[], prefix=[], suffix=[], enabled=False))

Configure how to track Kedro dataset settings.

class Config

Bases: object

Pydantic class config.

field run_manager: MlflowRunManager | None [Required] (alias 'mlflow')

Instance or dict to be parsed into instance of MlflowRunManager

pydantic model mlopus.kedro.hooks.mlflow_tracker.Report

Bases: BaseModel

Configure the session report.

field enabled: bool = True

Add session report file to MLflow run artifacts.

field path: str = 'kedro-session.yml'

Session report file path inside run artifacts.

pydantic model mlopus.kedro.hooks.mlflow_tracker.Logs

Bases: BaseModel

Configure how to track logs.

field enabled: bool = True

Upload logs to MLflow run.

field path: str = 'logs'

Path to logs dir inside run artifacts.

field files: List[LogFile] [Optional]

Local log files to upload.

pydantic model mlopus.kedro.hooks.mlflow_tracker.LogFile

Bases: BaseModel

Log file settings.

field path: str [Required]

Path to local log file.

field alias: str = None

File alias when uploading.

field cleanup: bool = True

Clear file contents before the first pipeline runs.

pydantic model mlopus.kedro.hooks.mlflow_tracker.Tags

Bases: BaseModel

Configure extra tags.

field report: bool = False

Include extra tags in session report file.

field values: dict [Optional]

Extra tags dict. See mlopus.mlflow.schema.Run.tags

field mlflow: TagsMlflow = TagsMlflow(prefix=[], suffix=[], enabled=True)

Configure how extra tags are set in the MLflow run.

pydantic model mlopus.kedro.hooks.mlflow_tracker.TagsMlflow

Bases: _PrefixSuffix

Configure how extra tags are set in the MLflow run.

field enabled: bool = True

Enable setting extra tags in MLflow run.

class Config

Bases: object

Pydantic class config.

field prefix: List[str] [Optional]

Prefix for every key.

field suffix: List[str] [Optional]

Suffix for every key.

pydantic model mlopus.kedro.hooks.mlflow_tracker.Metrics

Bases: BaseModel

Configure the collection of MLflow metrics.

field report: bool = True

Include metrics in session report file.

field datasets: List[str] [Optional]

If a node outputs to any of these datasets, collect the output data as metrics (must be a dict). See mlopus.mlflow.schema.Run.metrics

field mlflow: MetricsMlflow = MetricsMlflow(prefix=[], suffix=[], enabled=True, prepend_dataset=True)

Configure how to log metrics in the MLflow run.

pydantic model mlopus.kedro.hooks.mlflow_tracker.MetricsMlflow

Bases: _PrefixSuffix

Configure how to log metrics in the MLflow run.

field enabled: bool = True

Log metrics in the MLflow run.

field prepend_dataset: bool = True

Prepend dataset name to metric keys.

class Config

Bases: object

Pydantic class config.

field prefix: List[str] [Optional]

Prefix for every key.

field suffix: List[str] [Optional]

Suffix for every key.

pydantic model mlopus.kedro.hooks.mlflow_tracker.Params

Bases: BaseModel

Configure the collection of MLflow params.

field report: bool = True

Include collected params in session report file.

field mappings: Dict[str, List[_ParamMapping]] [Optional]

A mapping of target metric keys to arbitrary paths inside the Kedro configuration from where the params for those keys will be obtained. Example: sampling_ratio: parameters.sampler.ratio

field mlflow: ParamsMlflow = ParamsMlflow(prefix=[], suffix=[], enabled=True)

Configure how to log collected params in the MLflow run.

pydantic model mlopus.kedro.hooks.mlflow_tracker.ParamsMlflow

Bases: _PrefixSuffix

Configure how to log collected params in the MLflow run.

field enabled: bool = True

Log collected params in MLflow run.

class Config

Bases: object

Pydantic class config.

field prefix: List[str] [Optional]

Prefix for every key.

field suffix: List[str] [Optional]

Suffix for every key.

pydantic model mlopus.kedro.hooks.mlflow_tracker.Overrides

Bases: Config

Configure how to track Kedro config overrides.

class Config

Bases: object

Pydantic class config.

field report: bool = True

Include in session report file.

field mlflow: ConfigMlflow = ConfigMlflow(rules=[], scopes=[], prefix=[], suffix=[], enabled=False)

Configure how to store in MLflow run params. The rules and scopes defined in the mlflow section will further narrow down the rules and scopes already defined here.

field scopes: List[str] [Optional]

List of Kedro config scopes to include.

field rules: List[Rule] [Optional]

List of Rule s for including/excluding keys.

pydantic model mlopus.kedro.hooks.mlflow_tracker.Config

Bases: _ScopedRuleSet

Configure how to track the Kedro config.

field report: bool = True

Include in session report file.

field mlflow: ConfigMlflow = ConfigMlflow(rules=[], scopes=[], prefix=[], suffix=[], enabled=False)

Configure how to store in MLflow run params. The rules and scopes defined in the mlflow section will further narrow down the rules and scopes already defined here.

class Config

Bases: object

Pydantic class config.

field scopes: List[str] [Optional]

List of Kedro config scopes to include.

field rules: List[Rule] [Optional]

List of Rule s for including/excluding keys.

pydantic model mlopus.kedro.hooks.mlflow_tracker.ConfigMlflow

Bases: _ScopedPrefixSuffixRuleSet

Configure how to store in MLflow run params.

field enabled: bool = True

Store in MLflow run params.

class Config

Bases: object

Pydantic class config.

field prefix: List[str] [Optional]

Prefix for every key.

field suffix: List[str] [Optional]

Suffix for every key.

field scopes: List[str] [Optional]

List of Kedro config scopes to include.

field rules: List[Rule] [Optional]

List of Rule s for including/excluding keys.

pydantic model mlopus.kedro.hooks.mlflow_tracker.Datasets

Bases: _RuleSet

Configure how to track Kedro dataset settings.

Important

• The tracking of a dataset is only triggered when a node attempts to save/load to/from it.

field report: bool = True

Include in session report file.

field include_non_pydantic: bool = False

Use the __dict__ attribute to describe settings of non-pydantic datasets.

field mlflow: DatasetsMlflow = DatasetsMlflow(rules=[], prefix=[], suffix=[], enabled=False)

Configure how to store dataset settings in MLFlow run params.

class Config

Bases: object

Pydantic class config.

field rules: List[Rule] [Optional]

List of Rule s for including/excluding keys.

pydantic model mlopus.kedro.hooks.mlflow_tracker.DatasetsMlflow

Bases: _PrefixSuffixRuleSet

Configure how to store dataset settings in MLFlow run params.

field enabled: bool = True

Store dataset settings in MLflow run params.

class Config

Bases: object

Pydantic class config.

field prefix: List[str] [Optional]

Prefix for every key.

field suffix: List[str] [Optional]

Suffix for every key.

field rules: List[Rule] [Optional]

List of Rule s for including/excluding keys.

pydantic model mlopus.kedro.hooks.mlflow_tracker.Nodes

Bases: _NodeRuleSet

Configure how to track Kedro node settings.

field report: bool = True

Include in session report file.

field mlflow: NodesMlflow = NodesMlflow(rules=[], prefix=[], suffix=[], prepend_pipeline=True, enabled=False)

Configure how to store in MLflow run params. The rules defined in the mlflow section will further narrow down the rules already defined here.

class Config

Bases: object

Pydantic class config.

field rules: List[NodeRule] [Optional]

List of NodeRule s for including/excluding nodes.

pydantic model mlopus.kedro.hooks.mlflow_tracker.NodesMlflow

Bases: _PipelinesRuleSet

Configure how to store in MLflow run params.

field enabled: bool = True

Store in MLflow run params.

class Config

Bases: object

Pydantic class config.

field prepend_pipeline: bool = True

Prepend pipeline name to keys.

field prefix: List[str] [Optional]

Prefix for every key.

field suffix: List[str] [Optional]

Suffix for every key.

field rules: List[NodeRule] [Optional]

List of NodeRule s for including/excluding nodes.

pydantic model mlopus.kedro.hooks.mlflow_tracker.Rule

Bases: BaseRule[Tuple[str, dict]]

class Config

Bases: object

Pydantic class config.

field effect: Effect [Required]

Rule effect: exclude-all matching keys or include-only matching keys.

field keys: List[str] [Optional]

Match rule if key is in list.

field expr: List[re.Pattern] [Optional]

Match rule if key matches any of these patterns.

pydantic model mlopus.kedro.hooks.mlflow_tracker.NodeRule

Bases: BaseRule[_Node]

field keys: List[str] [Optional] (alias 'names')

Match rule if node name is in list.

field tags: List[str] [Optional]

Match rule if node has any of these tags.

class Config

Bases: object

Pydantic class config.

field effect: Effect [Required]

Rule effect: exclude-all matching keys or include-only matching keys.

field expr: List[re.Pattern] [Optional]

Match rule if key matches any of these patterns.