echofilter.optim package#

Optimization, criterions and metrics.

Submodules#

echofilter.optim.criterions module#

Evaluation criterions.

echofilter.optim.criterions.mask_accuracy(input, target, threshold=0.5, ndim=None, reduction='mean')[source]#

Measure accuracy of input compared to binary targets.

Parameters

  • input (torch.Tensor) – Input tensor.

  • target (torch.Tensor) – Target tensor, the same shape as input.

  • threshold (float, optional) – Threshold which entries in input and target must exceed to be binarised as the positive class. Default is 0.5.

  • ndim (int or None) – Number of dimensions to keep. If None, only the first (batch) dimension is kept and the rest are flattened. Default is None.

  • reduction ("none" or "mean" or "sum", optional) – Specifies the reduction to apply to the output: "none" | "mean" | "sum". "none": no reduction will be applied, "mean": the sum of the output will be divided by the number of elements in the output, "sum": the output will be summed. Default: "mean".

Returns

The fraction of input which has the same class as target after thresholding.

Return type

torch.Tensor

echofilter.optim.criterions.mask_accuracy_with_logits(input, *args, **kwargs)[source]#

Measure accuracy with logit inputs.

Pass through a sigmoid, binarize, then measure accuracy of predictions compared to ground truth target.

See also

mask_accuracy

echofilter.optim.criterions.mask_active_fraction(input, threshold=0.5, ndim=None, reduction='mean')[source]#

Measure the fraction of input which exceeds a threshold.

Parameters

  • input (torch.Tensor) – Input tensor.

  • threshold (float, optional) – Threshold which entries in input must exceed. Default is 0.5.

  • ndim (int or None) – Number of dimensions to keep. If None, only the first (batch) dimension is kept and the rest are flattened. Default is None.

  • reduction ("none" or "mean" or "sum", optional) – Specifies the reduction to apply to the output: "none" | "mean" | "sum". "none": no reduction will be applied, "mean": the sum of the output will be divided by the number of elements in the output, "sum": the output will be summed. Default: "mean".

Returns

The fraction of input which exceeds threshold, with shaped corresponding to reduction.

Return type

torch.Tensor

echofilter.optim.criterions.mask_active_fraction_with_logits(input, *args, **kwargs)[source]#

Convert logits to probabilities, and measure what fraction exceed threshold.

See also

mask_active_fraction

echofilter.optim.criterions.mask_f1_score(input, target, reduction='mean', **kwargs)[source]#

Measure F1-score of probability input.

Binarize, then measure the F1-score of the input vs ground truth target,

Parameters

  • input (torch.Tensor) – Input tensor.

  • target (torch.Tensor) – Target tensor, the same shape as input.

  • threshold (float, optional) – Threshold which entries in input and target must exceed to be binarised as the positive class. Default is 0.5.

  • ndim (int or None) – Number of dimensions to keep. If None, only the first (batch) dimension is kept and the rest are flattened. Default is None.

  • reduction ("none" or "mean" or "sum", optional) – Specifies the reduction to apply to the output: "none" | "mean" | "sum". "none": no reduction will be applied, "mean": the sum of the output will be divided by the number of elements in the output, "sum": the output will be summed. Default: "mean".

Returns

The F1-score of input as compared to target after thresholding. The F1-score is the harmonic mean of precision and recall.

Return type

torch.Tensor

See also

mask_precision, mask_recall

echofilter.optim.criterions.mask_f1_score_with_logits(input, *args, **kwargs)[source]#

Measure F1-score of logit input.

Convert logits to probabilities with sigmoid, apply a threshold, then measure the F1-score of the tensor as compared to ground truth.

See also

mask_f1_score

echofilter.optim.criterions.mask_jaccard_index(input, target, threshold=0.5, ndim=None, reduction='mean')[source]#

Measure Jaccard Index from probabilities.

Measure the Jaccard Index (intersection over union) of the input as compared to a ground truth target, after binarising with a threshold.

Parameters

  • input (torch.Tensor) – Input tensor.

  • target (torch.Tensor) – Target tensor, the same shape as input.

  • threshold (float, optional) – Threshold which entries in input and target must exceed to be binarised as the positive class. Default is 0.5.

  • ndim (int or None) – Number of dimensions to keep. If None, only the first (batch) dimension is kept and the rest are flattened. Default is None.

  • reduction ("none" or "mean" or "sum", optional) – Specifies the reduction to apply to the output: "none" | "mean" | "sum". "none": no reduction will be applied, "mean": the sum of the output will be divided by the number of elements in the output, "sum": the output will be summed. Default: "mean".

Returns

The Jaccard Index of input as compared to target. The Jaccard Index is the number of elements where both input and target exceed threshold, divided by the number of elements where at least one of input and target exceeds threshold.

Return type

torch.Tensor

echofilter.optim.criterions.mask_jaccard_index_with_logits(input, *args, **kwargs)[source]#

Measure Jaccard Index from logits.

Convert logits to probabilities with sigmoid, apply a threshold, then measure the Jaccard Index (intersection over union) of the tensor as compared to ground truth.

See also

mask_jaccard_index

echofilter.optim.criterions.mask_precision(input, target, threshold=0.5, ndim=None, reduction='mean')[source]#

Measure precision of probability input.

Binarize with a threshold, then measure precision compared to a ground truth target.

Parameters

  • input (torch.Tensor) – Input tensor.

  • target (torch.Tensor) – Target tensor, the same shape as input.

  • threshold (float, optional) – Threshold which entries in input and target must exceed to be binarised as the positive class. Default is 0.5.

  • ndim (int or None) – Number of dimensions to keep. If None, only the first (batch) dimension is kept and the rest are flattened. Default is None.

  • reduction ("none" or "mean" or "sum", optional) – Specifies the reduction to apply to the output: "none" | "mean" | "sum". "none": no reduction will be applied, "mean": the sum of the output will be divided by the number of elements in the output, "sum": the output will be summed. Default: "mean".

Returns

The precision of input as compared to target after thresholding. The fraction of predicted positive cases, input > 0.5, which are true positive cases (input > 0.5 and `target > 0.5). If there are no predicted positives, the output is 0 if there are any positives to predict and 1 if there are none.

Return type

torch.Tensor

echofilter.optim.criterions.mask_precision_with_logits(input, *args, **kwargs)[source]#

Measure precision of logit input.

Pass through sigmoid, threshold, then measure precision.

See also

mask_precision

echofilter.optim.criterions.mask_recall(input, target, threshold=0.5, ndim=None, reduction='mean')[source]#

Measure recall of probability input.

Binarize with a threshold, then measure the recall compared to a ground truth target.

Parameters

  • input (torch.Tensor) – Input tensor.

  • target (torch.Tensor) – Target tensor, the same shape as input.

  • threshold (float, optional) – Threshold which entries in input and target must exceed to be binarised as the positive class. Default is 0.5.

  • ndim (int or None) – Number of dimensions to keep. If None, only the first (batch) dimension is kept and the rest are flattened. Default is None.

  • reduction ("none" or "mean" or "sum", optional) – Specifies the reduction to apply to the output: "none" | "mean" | "sum". "none": no reduction will be applied, "mean": the sum of the output will be divided by the number of elements in the output, "sum": the output will be summed. Default: "mean".

Returns

The recall of input as compared to target after thresholding. The fraction of true positive cases, target > 0.5, which are true positive cases (input > 0.5 and `target > 0.5). If there are no true positives, the output is 1.

Return type

torch.Tensor

echofilter.optim.criterions.mask_recall_with_logits(input, *args, **kwargs)[source]#

Measure recall of logit input.

Pass through sigmoid, binarize, then measure recall of ground truth target.

See also

mask_recall

echofilter.optim.meters module#

Meters for tracking measurements during training.

class echofilter.optim.meters.AverageMeter(name, fmt=':f')[source]#

Bases: object

Compute and store the average and current value.

reset()[source]#
update(val, n=None)[source]#
class echofilter.optim.meters.ProgressMeter(num_batches, meters, prefix='')[source]#

Bases: object

display(batch)[source]#

echofilter.optim.schedulers module#

class echofilter.optim.schedulers.MesaOneCycleLR(optimizer, max_lr, total_steps=None, pct_start=0.25, pct_end=0.75, **kwargs)[source]#

Bases: echofilter.optim.torch_backports.OneCycleLR

A 1-cycle learning rate schedule with a flat region at maximum learning rate.

Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.

The 1cycle learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training.

This scheduler is not chainable.

Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence):

  1. A value for total_steps is explicitly provided.

  2. A number of epochs (epochs) and a number of steps per epoch (steps_per_epoch) are provided. In this case, the number of total steps is inferred by total_steps = epochs * steps_per_epoch

You must either provide a value for total_steps or provide a value for both epochs and steps_per_epoch.

Parameters

  • optimizer (torch.optim.optimizer.Optimizer) – Wrapped optimizer.

  • max_lr (float or list) – Upper learning rate boundaries in the cycle for each parameter group.

  • total_steps (int) – The total number of steps in the cycle. Note that if a value is provided here, then it must be inferred by providing a value for epochs and steps_per_epoch. Default: None

  • epochs (int) – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None

  • steps_per_epoch (int) – The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None

  • pct_start (float) – The percentage of the cycle (in number of steps) spent increasing the learning rate. Default: 0.25

  • pct_end (float) – The percentage of the cycle (in number of steps) spent before decreasing the learning rate. Default: 0.75

  • anneal_strategy ({"cos", "linear"}) – Specifies the annealing strategy: “cos” for cosine annealing, “linear” for linear annealing. Default: “cos”.

  • cycle_momentum (bool, default=True) – If True, momentum is cycled inversely to learning rate between “base_momentum” and “max_momentum”. Default: True

  • base_momentum (float or list) – Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is “base_momentum” and learning rate is “max_lr”. Default: 0.85

  • max_momentum (float or list) – Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is “max_momentum” and learning rate is “base_lr” Default: 0.95

  • div_factor (float) – Determines the initial learning rate via initial_lr = max_lr/div_factor Default: 25

  • final_div_factor (float) – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor Default: 1e4

  • last_epoch (int) – The index of the last batch. This parameter is used when resuming a training job. Since step() should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. Default: -1

Example

>>> data_loader = torch.utils.data.DataLoader(...)
>>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
>>> scheduler = MesaOneCycleLR(optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10)
>>> for epoch in range(10):
>>>     for batch in data_loader:
>>>         train_batch(...)
>>>         scheduler.step()
get_lr()[source]#

echofilter.optim.torch_backports module#

Pytorch classes backported from later versions.

This contains functions copied from newer versions of pytorch than v1.2.0, which is the latest version currently available from IBM compiled for ppc64 architectures.

From PyTorch:

Copyright (c) 2016- Facebook, Inc (Adam Paszke) Copyright (c) 2014- Facebook, Inc (Soumith Chintala) Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu) Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) Copyright (c) 2011-2013 NYU (Clement Farabet) Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) Copyright (c) 2006 Idiap Research Institute (Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)

From Caffe2:

Copyright (c) 2016-present, Facebook Inc. All rights reserved.

All contributions by Facebook: Copyright (c) 2016 Facebook Inc.

All contributions by Google: Copyright (c) 2015 Google Inc. All rights reserved.

All contributions by Yangqing Jia: Copyright (c) 2015 Yangqing Jia All rights reserved.

All contributions from Caffe: Copyright(c) 2013, 2014, 2015, the respective contributors All rights reserved.

All other contributions: Copyright(c) 2015, 2016 the respective contributors All rights reserved.

Caffe2 uses a copyright model similar to Caffe: each contributor holds copyright over their contributions to Caffe2. The project versioning records all such contribution and copyright details. If a contributor wants to further mark their specific copyright on a particular contribution, they should indicate their copyright solely in the commit message of the change when it is committed.

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  3. Neither the names of Facebook, Deepmind Technologies, NYU, NEC Laboratories America and IDIAP Research Institute nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

class echofilter.optim.torch_backports.OneCycleLR(optimizer, max_lr, total_steps=None, epochs=None, steps_per_epoch=None, pct_start=0.3, anneal_strategy='cos', cycle_momentum=True, base_momentum=0.85, max_momentum=0.95, div_factor=25.0, final_div_factor=10000.0, last_epoch=- 1)[source]#

Bases: echofilter.optim.torch_backports._LRScheduler

Backported from pytorch 1.4.0.

Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.

The 1cycle learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training.

This scheduler is not chainable.

Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence):

  1. A value for total_steps is explicitly provided.

  2. A number of epochs (epochs) and a number of steps per epoch (steps_per_epoch) are provided. In this case, the number of total steps is inferred by total_steps = epochs * steps_per_epoch

You must either provide a value for total_steps or provide a value for both epochs and steps_per_epoch.

Parameters

  • optimizer (torch.optim.optimizer.Optimizer) – Wrapped optimizer.

  • max_lr (float or list) – Upper learning rate boundaries in the cycle for each parameter group.

  • total_steps (int) – The total number of steps in the cycle. Note that if a value is provided here, then it must be inferred by providing a value for epochs and steps_per_epoch. Default: None

  • epochs (int) – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None

  • steps_per_epoch (int) – The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None

  • pct_start (float) – The percentage of the cycle (in number of steps) spent increasing the learning rate. Default: 0.3

  • anneal_strategy ({'cos', 'linear'}) – Specifies the annealing strategy: “cos” for cosine annealing, “linear” for linear annealing. Default: ‘cos’

  • cycle_momentum (bool) – If True, momentum is cycled inversely to learning rate between ‘base_momentum’ and ‘max_momentum’. Default: True

  • base_momentum (float or list) – Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is ‘base_momentum’ and learning rate is ‘max_lr’. Default: 0.85

  • max_momentum (float or list) – Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is ‘max_momentum’ and learning rate is ‘base_lr’ Default: 0.95

  • div_factor (float) – Determines the initial learning rate via initial_lr = max_lr/div_factor Default: 25

  • final_div_factor (float) – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor Default: 1e4

  • last_epoch (int) – The index of the last batch. This parameter is used when resuming a training job. Since step() should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. Default: -1

Example

>>> data_loader = torch.utils.data.DataLoader(...)
>>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
>>> scheduler = torch.optim.lr_scheduler.OneCycleLR(
>>>     optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10
>>> )
>>> for epoch in range(10):
>>>     for batch in data_loader:
>>>         train_batch(...)
>>>         scheduler.step()
get_lr()[source]#

echofilter.optim.utils module#

Utility functions for interacting with optimizers.

echofilter.optim.utils.get_current_lr(optimizer)[source]#

Get the learning rate of an optimizer.

Parameters

optimizer (torch.optim.Optimizer) – An optimizer, with a learning rate common to all parameter groups.

Returns

The learning rate of the first parameter group.

Return type

float

echofilter.optim.utils.get_current_momentum(optimizer)[source]#

Get the momentum of an optimizer.

Parameters

optimizer (torch.optim.Optimizer) – An optimizer which implements momentum or betas (where momentum is the first beta, c.f. torch.optim.Adam) with a momentum common to all parameter groups.

Returns

The momentum of the first parameter group.

Return type

float