echofilter.optim package
Contents
echofilter.optim package#
Optimization, criterions and metrics.
Submodules#
echofilter.optim.criterions module#
Evaluation criterions.
- echofilter.optim.criterions.mask_accuracy(input, target, threshold=0.5, ndim=None, reduction='mean')[source]#
Measure accuracy of input compared to binary targets.
- Parameters
input (torch.Tensor) – Input tensor.
target (torch.Tensor) – Target tensor, the same shape as
input
.threshold (float, optional) – Threshold which entries in
input
andtarget
must exceed to be binarised as the positive class. Default is0.5
.ndim (int or None) – Number of dimensions to keep. If
None
, only the first (batch) dimension is kept and the rest are flattened. Default isNone
.reduction (
"none"
or"mean"
or"sum"
, optional) – Specifies the reduction to apply to the output:"none"
|"mean"
|"sum"
."none"
: no reduction will be applied,"mean"
: the sum of the output will be divided by the number of elements in the output,"sum"
: the output will be summed. Default:"mean"
.
- Returns
The fraction of
input
which has the same class astarget
after thresholding.- Return type
- echofilter.optim.criterions.mask_accuracy_with_logits(input, *args, **kwargs)[source]#
Measure accuracy with logit inputs.
Pass through a sigmoid, binarize, then measure accuracy of predictions compared to ground truth target.
See also
- echofilter.optim.criterions.mask_active_fraction(input, threshold=0.5, ndim=None, reduction='mean')[source]#
Measure the fraction of input which exceeds a threshold.
- Parameters
input (torch.Tensor) – Input tensor.
threshold (float, optional) – Threshold which entries in
input
must exceed. Default is0.5
.ndim (int or None) – Number of dimensions to keep. If
None
, only the first (batch) dimension is kept and the rest are flattened. Default isNone
.reduction (
"none"
or"mean"
or"sum"
, optional) – Specifies the reduction to apply to the output:"none"
|"mean"
|"sum"
."none"
: no reduction will be applied,"mean"
: the sum of the output will be divided by the number of elements in the output,"sum"
: the output will be summed. Default:"mean"
.
- Returns
The fraction of
input
which exceedsthreshold
, with shaped corresponding toreduction
.- Return type
- echofilter.optim.criterions.mask_active_fraction_with_logits(input, *args, **kwargs)[source]#
Convert logits to probabilities, and measure what fraction exceed threshold.
See also
- echofilter.optim.criterions.mask_f1_score(input, target, reduction='mean', **kwargs)[source]#
Measure F1-score of probability input.
Binarize, then measure the F1-score of the input vs ground truth target,
- Parameters
input (torch.Tensor) – Input tensor.
target (torch.Tensor) – Target tensor, the same shape as
input
.threshold (float, optional) – Threshold which entries in
input
andtarget
must exceed to be binarised as the positive class. Default is0.5
.ndim (int or None) – Number of dimensions to keep. If
None
, only the first (batch) dimension is kept and the rest are flattened. Default isNone
.reduction (
"none"
or"mean"
or"sum"
, optional) – Specifies the reduction to apply to the output:"none"
|"mean"
|"sum"
."none"
: no reduction will be applied,"mean"
: the sum of the output will be divided by the number of elements in the output,"sum"
: the output will be summed. Default:"mean"
.
- Returns
The F1-score of
input
as compared totarget
after thresholding. The F1-score is the harmonic mean of precision and recall.- Return type
See also
- echofilter.optim.criterions.mask_f1_score_with_logits(input, *args, **kwargs)[source]#
Measure F1-score of logit input.
Convert logits to probabilities with sigmoid, apply a threshold, then measure the F1-score of the tensor as compared to ground truth.
See also
- echofilter.optim.criterions.mask_jaccard_index(input, target, threshold=0.5, ndim=None, reduction='mean')[source]#
Measure Jaccard Index from probabilities.
Measure the Jaccard Index (intersection over union) of the input as compared to a ground truth target, after binarising with a threshold.
- Parameters
input (torch.Tensor) – Input tensor.
target (torch.Tensor) – Target tensor, the same shape as
input
.threshold (float, optional) – Threshold which entries in
input
andtarget
must exceed to be binarised as the positive class. Default is0.5
.ndim (int or None) – Number of dimensions to keep. If
None
, only the first (batch) dimension is kept and the rest are flattened. Default isNone
.reduction (
"none"
or"mean"
or"sum"
, optional) – Specifies the reduction to apply to the output:"none"
|"mean"
|"sum"
."none"
: no reduction will be applied,"mean"
: the sum of the output will be divided by the number of elements in the output,"sum"
: the output will be summed. Default:"mean"
.
- Returns
The Jaccard Index of
input
as compared totarget
. The Jaccard Index is the number of elements where bothinput
andtarget
exceedthreshold
, divided by the number of elements where at least one ofinput
andtarget
exceedsthreshold
.- Return type
- echofilter.optim.criterions.mask_jaccard_index_with_logits(input, *args, **kwargs)[source]#
Measure Jaccard Index from logits.
Convert logits to probabilities with sigmoid, apply a threshold, then measure the Jaccard Index (intersection over union) of the tensor as compared to ground truth.
See also
- echofilter.optim.criterions.mask_precision(input, target, threshold=0.5, ndim=None, reduction='mean')[source]#
Measure precision of probability input.
Binarize with a threshold, then measure precision compared to a ground truth target.
- Parameters
input (torch.Tensor) – Input tensor.
target (torch.Tensor) – Target tensor, the same shape as
input
.threshold (float, optional) – Threshold which entries in
input
andtarget
must exceed to be binarised as the positive class. Default is0.5
.ndim (int or None) – Number of dimensions to keep. If
None
, only the first (batch) dimension is kept and the rest are flattened. Default isNone
.reduction (
"none"
or"mean"
or"sum"
, optional) – Specifies the reduction to apply to the output:"none"
|"mean"
|"sum"
."none"
: no reduction will be applied,"mean"
: the sum of the output will be divided by the number of elements in the output,"sum"
: the output will be summed. Default:"mean"
.
- Returns
The precision of
input
as compared totarget
after thresholding. The fraction of predicted positive cases, input > 0.5, which are true positive cases (input > 0.5 and `target > 0.5). If there are no predicted positives, the output is0
if there are any positives to predict and1
if there are none.- Return type
- echofilter.optim.criterions.mask_precision_with_logits(input, *args, **kwargs)[source]#
Measure precision of logit input.
Pass through sigmoid, threshold, then measure precision.
See also
- echofilter.optim.criterions.mask_recall(input, target, threshold=0.5, ndim=None, reduction='mean')[source]#
Measure recall of probability input.
Binarize with a threshold, then measure the recall compared to a ground truth target.
- Parameters
input (torch.Tensor) – Input tensor.
target (torch.Tensor) – Target tensor, the same shape as
input
.threshold (float, optional) – Threshold which entries in
input
andtarget
must exceed to be binarised as the positive class. Default is0.5
.ndim (int or None) – Number of dimensions to keep. If
None
, only the first (batch) dimension is kept and the rest are flattened. Default isNone
.reduction (
"none"
or"mean"
or"sum"
, optional) – Specifies the reduction to apply to the output:"none"
|"mean"
|"sum"
."none"
: no reduction will be applied,"mean"
: the sum of the output will be divided by the number of elements in the output,"sum"
: the output will be summed. Default:"mean"
.
- Returns
The recall of
input
as compared totarget
after thresholding. The fraction of true positive cases, target > 0.5, which are true positive cases (input > 0.5 and `target > 0.5). If there are no true positives, the output is1
.- Return type
echofilter.optim.meters module#
Meters for tracking measurements during training.
echofilter.optim.schedulers module#
- class echofilter.optim.schedulers.MesaOneCycleLR(optimizer, max_lr, total_steps=None, pct_start=0.25, pct_end=0.75, **kwargs)[source]#
Bases:
echofilter.optim.torch_backports.OneCycleLR
A 1-cycle learning rate schedule with a flat region at maximum learning rate.
Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.
The 1cycle learning rate policy changes the learning rate after every batch.
step
should be called after a batch has been used for training.This scheduler is not chainable.
Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence):
A value for total_steps is explicitly provided.
A number of epochs (epochs) and a number of steps per epoch (steps_per_epoch) are provided. In this case, the number of total steps is inferred by total_steps = epochs * steps_per_epoch
You must either provide a value for total_steps or provide a value for both epochs and steps_per_epoch.
- Parameters
optimizer (
torch.optim.optimizer.Optimizer
) – Wrapped optimizer.max_lr (float or list) – Upper learning rate boundaries in the cycle for each parameter group.
total_steps (int) – The total number of steps in the cycle. Note that if a value is provided here, then it must be inferred by providing a value for epochs and steps_per_epoch. Default: None
epochs (int) – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None
steps_per_epoch (int) – The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None
pct_start (float) – The percentage of the cycle (in number of steps) spent increasing the learning rate. Default: 0.25
pct_end (float) – The percentage of the cycle (in number of steps) spent before decreasing the learning rate. Default: 0.75
anneal_strategy ({"cos", "linear"}) – Specifies the annealing strategy: “cos” for cosine annealing, “linear” for linear annealing. Default: “cos”.
cycle_momentum (bool, default=True) – If
True
, momentum is cycled inversely to learning rate between “base_momentum” and “max_momentum”. Default: Truebase_momentum (float or list) – Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is “base_momentum” and learning rate is “max_lr”. Default: 0.85
max_momentum (float or list) – Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is “max_momentum” and learning rate is “base_lr” Default: 0.95
div_factor (float) – Determines the initial learning rate via initial_lr = max_lr/div_factor Default: 25
final_div_factor (float) – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor Default: 1e4
last_epoch (int) – The index of the last batch. This parameter is used when resuming a training job. Since
step()
should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. Default: -1
Example
>>> data_loader = torch.utils.data.DataLoader(...) >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> scheduler = MesaOneCycleLR(optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10) >>> for epoch in range(10): >>> for batch in data_loader: >>> train_batch(...) >>> scheduler.step()
echofilter.optim.torch_backports module#
Pytorch classes backported from later versions.
This contains functions copied from newer versions of pytorch than v1.2.0, which is the latest version currently available from IBM compiled for ppc64 architectures.
From PyTorch:
Copyright (c) 2016- Facebook, Inc (Adam Paszke) Copyright (c) 2014- Facebook, Inc (Soumith Chintala) Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu) Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) Copyright (c) 2011-2013 NYU (Clement Farabet) Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) Copyright (c) 2006 Idiap Research Institute (Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
From Caffe2:
Copyright (c) 2016-present, Facebook Inc. All rights reserved.
All contributions by Facebook: Copyright (c) 2016 Facebook Inc.
All contributions by Google: Copyright (c) 2015 Google Inc. All rights reserved.
All contributions by Yangqing Jia: Copyright (c) 2015 Yangqing Jia All rights reserved.
All contributions from Caffe: Copyright(c) 2013, 2014, 2015, the respective contributors All rights reserved.
All other contributions: Copyright(c) 2015, 2016 the respective contributors All rights reserved.
Caffe2 uses a copyright model similar to Caffe: each contributor holds copyright over their contributions to Caffe2. The project versioning records all such contribution and copyright details. If a contributor wants to further mark their specific copyright on a particular contribution, they should indicate their copyright solely in the commit message of the change when it is committed.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the names of Facebook, Deepmind Technologies, NYU, NEC Laboratories America and IDIAP Research Institute nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- class echofilter.optim.torch_backports.OneCycleLR(optimizer, max_lr, total_steps=None, epochs=None, steps_per_epoch=None, pct_start=0.3, anneal_strategy='cos', cycle_momentum=True, base_momentum=0.85, max_momentum=0.95, div_factor=25.0, final_div_factor=10000.0, last_epoch=- 1)[source]#
Bases:
echofilter.optim.torch_backports._LRScheduler
Backported from pytorch 1.4.0.
Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate. This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.
The 1cycle learning rate policy changes the learning rate after every batch.
step
should be called after a batch has been used for training.This scheduler is not chainable.
Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence):
A value for total_steps is explicitly provided.
A number of epochs (epochs) and a number of steps per epoch (steps_per_epoch) are provided. In this case, the number of total steps is inferred by total_steps = epochs * steps_per_epoch
You must either provide a value for total_steps or provide a value for both epochs and steps_per_epoch.
- Parameters
optimizer (
torch.optim.optimizer.Optimizer
) – Wrapped optimizer.max_lr (float or list) – Upper learning rate boundaries in the cycle for each parameter group.
total_steps (int) – The total number of steps in the cycle. Note that if a value is provided here, then it must be inferred by providing a value for epochs and steps_per_epoch. Default: None
epochs (int) – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None
steps_per_epoch (int) – The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None
pct_start (float) – The percentage of the cycle (in number of steps) spent increasing the learning rate. Default: 0.3
anneal_strategy ({'cos', 'linear'}) – Specifies the annealing strategy: “cos” for cosine annealing, “linear” for linear annealing. Default: ‘cos’
cycle_momentum (bool) – If
True
, momentum is cycled inversely to learning rate between ‘base_momentum’ and ‘max_momentum’. Default: Truebase_momentum (float or list) – Lower momentum boundaries in the cycle for each parameter group. Note that momentum is cycled inversely to learning rate; at the peak of a cycle, momentum is ‘base_momentum’ and learning rate is ‘max_lr’. Default: 0.85
max_momentum (float or list) – Upper momentum boundaries in the cycle for each parameter group. Functionally, it defines the cycle amplitude (max_momentum - base_momentum). Note that momentum is cycled inversely to learning rate; at the start of a cycle, momentum is ‘max_momentum’ and learning rate is ‘base_lr’ Default: 0.95
div_factor (float) – Determines the initial learning rate via initial_lr = max_lr/div_factor Default: 25
final_div_factor (float) – Determines the minimum learning rate via min_lr = initial_lr/final_div_factor Default: 1e4
last_epoch (int) – The index of the last batch. This parameter is used when resuming a training job. Since
step()
should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. Default: -1
Example
>>> data_loader = torch.utils.data.DataLoader(...) >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> scheduler = torch.optim.lr_scheduler.OneCycleLR( >>> optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10 >>> ) >>> for epoch in range(10): >>> for batch in data_loader: >>> train_batch(...) >>> scheduler.step()
echofilter.optim.utils module#
Utility functions for interacting with optimizers.
- echofilter.optim.utils.get_current_lr(optimizer)[source]#
Get the learning rate of an optimizer.
- Parameters
optimizer (torch.optim.Optimizer) – An optimizer, with a learning rate common to all parameter groups.
- Returns
The learning rate of the first parameter group.
- Return type
- echofilter.optim.utils.get_current_momentum(optimizer)[source]#
Get the momentum of an optimizer.
- Parameters
optimizer (torch.optim.Optimizer) – An optimizer which implements momentum or betas (where momentum is the first beta, c.f.
torch.optim.Adam
) with a momentum common to all parameter groups.- Returns
The momentum of the first parameter group.
- Return type