Changelog#

All notable changes to echofilter will be documented here.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Categories for changes are: Added, Changed, Deprecated, Removed, Fixed, Security.

Version 1.2.0#

Release date: 2022-11-18. Full commit changelog.

Changed#

Inference#

  • The checkpoints have been renamed to a simpler structure: echofilter-v{MAJOR}_{ORIENTATION}_ep{EPOCHS}. For example, the current default checkpoint is now named as "echofilter-v1_bifacing_700ep". This is in keeping with the format used in the manuscript. The previous names for the checkpoints will still work (as an alias to the new names). (#330)

  • The ordering of the colour palette listing produced by --show-colors has changed. By default, the colours are now ordered by hue (subsorted by value) instead of sorted alphabetically. The previous default behaviour can be obtained by using the new “alphabetic” argument: --show-colors "alphabetic". (#322)

Fixed#

Inference#

  • Fix a bug with Echoview interaction which was causing the code to fail when overwriting an existing line in the EV file. (#332)

  • Fix hiding the Echoview window while echofilter is running and interacting with Echoview. This bug was first seen when using Echoview 13, and was not affecting interactions with Echoview 10 or 11. (#328)

Metadata#

  • Add missing requests package to requirements. (#320)

Added#

Inference#

  • Add API option --show-cache-dir, which shows the path to the directory in which downloaded checkpoints will be cached. (#321)

  • Add "alphabetic" and "full-alphabetic" options to --show-colors which let the colours be sorted by alphabetically, instead of by hue. (#322)

Version 1.1.1#

Release date: 2022-11-16. Full commit changelog.

Fixed#

Inference#

  • EVL final value pad was for a timestamp in between the preceding two, not extending forward in time by half a timepoint. (#300)

Metadata#

  • Declare python_requires<3.11 requirement. (#302)

  • Declare torch<1.12.0 requirement. (#302)

Version 1.1.0#

Release date: 2022-11-12. Full commit changelog.

Changed#

Inference#

  • Disable logit smoothing by default. The previous behaviour can be restored by setting --logit-smoothing-sigma=1 at the CLI. (#293)

Fixed#

Inference#

  • Fix bug where joined segments of data would have their first ping dropped. (#272)

Training#

  • Make the number of channels in the first block respect the initial_channels argument. (#271)

Miscellaneous#

  • Fix unseen internal bugs, including in generate_shards. (#283)

Added#

Inference#

  • Add support for using a config file to provide arguments to the CLI. (#294)

  • Add --continue-on-error argument to inference routine, which will capture an error when processing an individual file and continue running the rest. (#245)

  • Break up large files into more manageable chunks of at most 1280 pings, to reduce out-of-memory errors. (#245)

  • Reduce GPU memory consumption during inference by moving outputs to CPU memory sooner. (#245)

  • Fill in missing values in the input file through 2d linear interpolation. (#246)

  • Pad Sv data in timestamp dimension during inference to ensure the data is fully within the network’s effective receptive field. (#277)

  • Add --prenorm-nan-value and --postnorm-nan-value options to control what value NaN values in the input are mapped to. (#274)

  • Add support for providing a single path as a string to the run_inference API. (Note that the CLI already supported this and so is unchanged). (#288)

  • Add more verbosity messages. (#276, #278, #292)

ev2csv#

  • Add --keep-thresholds option which allow for exporting Sv data with thresholds and exclusions enabled (set as they currently are in the EV file). The default behaviour is still to export raw Sv data (disabling all thresholds). The default file name for the CSV file depends on whether the export is of raw or thresholded data. (#275)

  • Add --keep-ext argument to ev2csv, which allows the existing extension on the input path to be kept preceding the new file extension. (#242)

Tests#

  • Add tests which check that inference commands run, whether checking their outputs. (#289)

Internal#

  • Add EVR reader echofilter.raw.loader.evr_reader. (#280)

Version 1.0.3#

Release date: 2022-11-15. Full commit changelog.

This minor patch fix addresses package metadata.

Fixed#

Metadata#

  • Declare python_requires>=3.6,<3.11 requirement. (#264, #302)

  • Declare torch<1.12.0 requirement. (#302)

Version 1.0.2#

Release date: 2022-11-06. Full commit changelog.

This minor patch fix addresses github dependencies so the package can be pushed to PyPI.

Changed#

Requirements#

  • Change torch_lr_finder train requirement from a specific github commit ref to >=0.2.0. (#260)

  • Remove ranger from train requirements. (#261)

Training#

  • Default optimizer changed from "rangerva" to "adam". If you have manually installed ranger you can still use the "rangerva" optimizer if you specify it. (#261)

Version 1.0.1#

Release date: 2022-11-06. Full commit changelog.

This patch fix addresses requirement inconsistencies and documentation building. This release is provided under the AGPLv3 license.

Changed#

Requirements#

  • Add a vendorized copy of functions from torchutils and remove it from the requirements. (#249)

Fixed#

Release#

  • Added checkpoints.yaml file to package_data. (#255)

  • Added appdirs package, required for caching model checkpoints. (#240)

  • Support for pytorch>=1.11 by dropping import of torch._six.container_abcs. (#250)

Version 1.0.0#

Release date: 2020-10-18. Full commit changelog.

This is the first major release of echofilter.

Added#

Inference#

  • Add support for loading checkpoints shipped as part of the package. (#228)

  • More detailed error messages when unable to download or load a model i.e. due to a problem with the Internet connection, a 404 error, or because the hard disk is out of space. (#228)

Documentation#

  • Add Usage Guide source and sphinx documentation PDF generation routines (#232, #233, #234, #235)

Version 1.0.0rc3#

Release date: 2020-09-23. Full commit changelog.

This is the third release candidate for the forthcoming v1.0.0 major release.

Fixed#

Inference#

  • Include extension in temporary EVL file, fixing issue importing it into Echoview. (#224)

Version 1.0.0rc2#

Release date: 2020-09-23. Full commit changelog.

This is the second release candidate for the forthcoming v1.0.0 major release.

Fixed#

Inference#

  • Fix reference to echofilter.raw.loader.evl_loader when loading EVL files into Echoview. (#222)

Version 1.0.0rc1#

Release date: 2020-09-23. Full commit changelog.

This is a release candidate for the forthcoming v1.0.0 major release.

Changed#

Inference#

  • Import lines into Echoview twice, once with and once without offset. (#218)

  • EVL outputs now indicate raw depths, before any offset or clipping is applied. (#218)

  • Change default --lines-during-passive value from "predict" to "interpolate-time". (#216)

  • Disable all bad data region outputs by default. (#217)

  • Change default nearfield cut-off behaviour to only clip the bottom line (upfacing data) and not the turbulence line (downfacing data). (#219)

Training#

  • Reduce minimum distance by which surface line must be above turbulence line from 0.25m to 0m. (#212)

  • Reduce minimum distance by which bottom line must be above surface line from 0.5m to 0.02m. (#212)

Fixed#

Inference#

  • Change nearfield line for downfacing recordings to be nearfield distance below the shallowest recording depth, not at a depth equal to the nearfield distance. (#214)

Added#

Inference#

  • Add new checkpoints: v2.0, v2.1 for stationary model; v2.0, v2.1, v2.2 for conditional hybrid model. (#213)

  • Add notes to lines imported into Echoview. (#215)

  • Add arguments controlling color and thickness of offset lines (--color-surface-offset, etc). (#218)

  • Add argument --cutoff-at-nearfield which re-enables clipping of the turbulence line at nearfield depth with downfacing data. (#219)

Version 1.0.0b4#

Release date: 2020-07-05. Full commit changelog.

This is a beta pre-release of v1.0.0.

Changed#

Inference#

  • Arguments relating to top are renamed to turbulence, and “top” outputs are renamed “turbulence”. (#190)

  • Change default checkpoint from conditional_mobile-stationary2_effunet6x2-1_lc32_v1.0 to conditional_mobile-stationary2_effunet6x2-1_lc32_v2.0. (#208)

  • Status value in EVL outputs extends to final sample (as per specification, not observed EVL files). (#201)

  • Rename --nearfield-cutoff argument to --nearfield, add --no-cutoff-at-nearfield argument to control whether the turbulence/bottom line can extend closer to the echosounder that the nearfield line. (#203)

  • Improved UI help and verbosity messages. (#187, #188, #203, #204, #207)

Training#

  • Use 0m as target for surface line for downfacing, not the top of the echogram. (#191)

  • Don’t include periods where the surface line is below the bottom line in the training loss. (#191)

  • Bottom line target during nearfield is now the bottom of the echogram, not 0.5m above the bottom. (#191)

  • Normalise training samples separately, based on their own Sv intensity distribution after augmentation. (#192)

  • Record echofilter version number in checkpoint file. (#193)

  • Change “optimal” depth zoom augmentation, used for validation, to cover a slightly wider depth range past the deepest bottom and shallowest surface line. (#194)

  • Don’t record fraction of image which is active during training. (#206)

Miscellaneous#

  • Rename top->turbulence, bot->bottom surf->surface, throughout all code. (#190)

  • Convert undefined value -10000.99 to NaN when loading lines from EVL files. (#191)

  • Include surface line in transect plots. (#191)

  • Move argparser and colour styling into ui subpackage. (#198)

  • Move inference command line interface to its own module to increase responsiveness for non-processing actions (--help, --version, --list-checkpoints, --list-colors). (#199)

Fixed#

Inference#

  • Fix depth extent of region boxes. (#186)

  • EVL and EVR outputs extend half a timestamp interval so it is clear what is inside their extent. (#200)

Training#

  • Labels for passive collection times in Minas Passage and Grand Passage datasets are manually set for samples where automatic labeling failed. (#191)

  • Interpolate surface depths during passive periods. (#191)

  •  Smooth out anomalies in the surface line, and exclude the smoothed version from the training loss. (#191)

  •  Use a looser nearfield removal process when removing the nearfield zone from the bottom line targets, so nearfield is removed from all samples where it needs to be. (#191)

  • When reshaping samples, don’t use higher order interpolation than first for the bottom line with upfacing data, as the boundaries are rectangular (#191)

  • The precision criterion’s measurement value when there are no predicted positives equals 1 and if there are no true positives and 0 otherwise (previously 0.5 regardless of target). (#195)

Added#

Inference#

  • Add nearfield line to EV file when importing lines, and add --no-nearfield-line argument to disable this. (#203)

  • Add arguments to control display of nearfield line, --color-nearfield and --thickness-nearfield. (#203)

  • Add -r and -R short-hand arguments for recursive and non-recursive directory search. (#189)

  • Add -s short-hand argument for --skip (#189)

  • Add two new model checkpoints to list of available checkpoints, conditional_mobile-stationary2_effunet6x2-1_lc32_v1.1 and conditional_mobile-stationary2_effunet6x2-1_lc32_v2.0. (#208)

  • Use YAML file to define list of available checkpoints. (#208, #209)

  • Default checkpoint is shown with an asterisk in checkpoint list. (#202)

Training#

  • Add cold/warm restart option, for training a model with initial weights from the output of a previously trained model. (#196)

  • Add option to manually specify training and validation partitions. (#205)

Version 1.0.0b3#

Release date: 2020-06-25. Full commit changelog.

This is a beta pre-release of v1.0.0.

Changed#

Inference#

  • Rename --crop-depth-min argument to --crop-min-depth, and --crop-depth-max argument to --crop-max-depth. (#174)

  • Rename --force_unconditioned argument to --force-unconditioned. (#166)

  • Default offset of surface line is now 1m. (#168)

  • Change default --checkpoint so it is always the same (the conditional model), independent of the --facing argument. (#177)

  • Change default --lines-during-passive from "redact" to "predict". (#176)

  • Change --sufix-csv behaviour so it should no longer include ".csv" extension, matching how --suffix-file is handled. (#171, #175)

  • Change handling of --suffix-var and --sufix-csv to prepend with "-" as a delimiter if none is included in the string, as was already the case for --sufix-file. (#170, #171)

  • Include --suffix-var string in region names. (#173)

  • Improved UI help and verbosity messages. (#166, #167, #170, #179, #180, #182)

  • Increase default verbosity level from 1 to 2. (#179)

Fixed#

Inference#

  • Autocrop with upward facing was running with reflected data as its input, resulting in the data being processed upside down and by the wrong conditional model. (#172)

  • Remove duplicate leading byte order mark character from evr file output, which was preventing the file from importing into Echoview. (#178)

  • Fix \r\n line endings being mapped to \r\r\n on Windows in evl and evr output files. (#178)

  • Show error message when importing the evr file into the ev file fails. (#169)

  • Fix duplicated Segments tqdm progress bar. (#180)

Added#

Inference#

  • Add --offset-surface argument, which allows the surface line to be adjusted by a fixed distance. (#168)

Version 1.0.0b2#

Release date: 2020-06-18. Full commit changelog.

This is a beta pre-release of v1.0.0.

Changed#

Inference#

  • Change default value of --offset to 1m. (#159)

  • Use a default --nearfield-cutoff of 1.7m. (#159, #161)

  • Show total run time when inference is finished. (#156)

  • Only ever report number of skipped regions if there were some which were skipped. (#156)

Fixed#

Inference#

  • When using the “redact” method for --lines-during-passive (the default option), depths were redacted but the timestamps were not, resulting in a temporal offset which accumulated with each passive region. (#155)

  • Fix behaviour with --suffix-file, so files are written to the filename with the suffix. (#160)

  • Fix type of --offset-top and --offset-bottom arguments from int to float. (#159)

  • Documentation for --overwrite-ev-lines argument. (#157)

Added#

Inference#

  • Add ability to specify whether to use recursive search through subdirectory tree, or just files in the specified directory, to both inference.py and ev2csv.py. Add --no-recursive-dir-search argument to enable the non-recursive mode. (#158)

  • Add option to cap the top or bottom line (depending on orientation) so it cannot go too close to the echosounder, with --nearfield-cutoff argument. (#159)

  • Add option to skip outputting individual evl lines, with --no-top-line, --no-bottom-line, --no-surface-line arguments. (#162)

Version 1.0.0b1#

Release date: 2020-06-17. Full commit changelog.

This is a beta pre-release of v1.0.0.

Changed#

Training#

  • Built-in line offsets and nearfield line are removed from training targets. (#82)

  • Training validation is now against data which is cropped by depth to zoom in on only the “optimal” range of depths (from the shallowest ground truth surface line to the deepest bottom line), using echofilter.data.transforms.OptimalCropDepth. (#83, #109)

  • Training augmentation stack. (#79, #83, #106, #124)

  • Train using normalisation based on the 10th percentile as the zero point and standard deviation robustly estimated from the interdecile range. (#80)

  • Use log-avg-exp for logit_is_passive and logit_is_removed. (#97)

  • Exclude data during removed blocks from top and bottom line targets. (#92, #110, #136)

  • Seeding of workers and random state during training. (#93, #126)

  • Change names of saved checkpoints and log. (#122, #132)

  • Save UNet state to checkpoint, not the wrapped model. (#133)

  • Change and reduce number of images generated when training. (#95, #98, #99, #101, #108, #112, #114, #127)

Inference#

  • Change checkpoints available to be used for inference. (#147)

  • Change default checkpoint to be dependent on the --facing argument. (#147)

  • Default line status of output lines changed from 1 to 3. (#135)

  • Default handling of lines during passive data collection changed from implicit "predict" to "redact". (#138)

  • By default, output logits are smoothed using a Gaussian with width of 1 pixel (relative to the model’s latent output space) before being converted into output probibilities. (#144)

  • By default, automatically cropping to zoom in on the depth range of interest if the fraction of the depth which could be removed is at least 35% of the original depth. (#149)

  • Change default normalisation behaviour to be based on the current input’s distribution of Sv values instead of the statistics used for training. (#80)

  • Output surface line as an evl file. (f829cb7)

  • Output regions as an evr file. (#141, #142, #143)

  • By default, when running on a .ev file, the generated lines and regions are imported into the file. (#152)

  • Renamed --csv-suffix argument to --suffix-csv. (#152)

  • Improved UI help and verbosity messages. (#81, #129, #137, #145)

Miscellaneous#

  • Set Sv values outside the range (-1e37, 1e37) to be NaN (previously values lower than -1e6 were set to NaN). (#140)

  • Move modules into subpackages. (#104, #130)

  • General code tidy up and refactoring. (#85, #88, #89, #94, #96, #146)

  • Change code to use the black style. (#86, #87)

Fixed#

Training#

  • Edge-cases when resizing data such as lines crossing; surface lines marked as undefined with value -10000.99. (#90)

  • Seeding numpy random state for dataloader workers during training. (#93)

  • Resume train schedule when resuming training from existing checkpoint. (#120)

  • Setting state for RangerVA when resuming training from existing checkpoint. (#121)

  • Running LRFinder after everything else is set up for the model. (#131)

Inference#

  • Exporting raw data in ev2csv required more Echoview parameters to be disabled, such as the minimum value threshold. (#100)

Miscellaneous#

  • Fixed behaviour when loading data from CSVs with different number of depth samples and range of depths for different rows in the CSV file. (#102, #103)

Added#

Training#

  • New augmentations: RandomCropDepth, RandomGrid, ElasticGrid, (#83, #105, #124)

  • Add outputs and loss terms for auxiliary targets: original top and bottom line, variants of the patches mask. (#91)

  • Add option to exclude passive and removed blocks from line targets. (#92)

  • Interpolation method option added to Rescale, randomly selected for training. (#79)

  • More input scaling options. (#80)

  • Add option to specify pooling operation for logit_is_passive and logit_is_removed. (#97)

  • Support training on Grand Passage dataset. (#101)

  • Support training on multiple datasets. (#111, #113)

  • Add stationary2 dataset which contains both MinasPassage and two copies of GrandPassage with different augmentations, and mobile+stationary2 dataset. (#111, #113)

  • Add conditional model architecture training wrapper. (#116)

  • Add outputs for conditional targets to tensorboard. (#125, #134)

  • Add stratified data sampler, which preserves the balance between datasets in each training batch. (#117)

  • Training process error catching. (#119)

  • Training on multiple GPUs on the same node for a single model. (#123, #133)

Inference#

  • Add --line-status argument, which controls the status to use in the evl output for the lines. (#135)

  • Add multiple methods of how to handle lines during passive data, and argument --lines-during-passive to control which method to use. (#138, #148)

  • Add --offset, --offset-top, --offset-bottom arguments, which allows the top and bottom lines to be adjusted by a fixed distance. (#139)

  • Write regions to evr file. (#141, #142, #143)

  • Add --logit-smoothing-sigma argument, which controls the kernel width for Gaussian smoothing applied to the logits before converting to predictions. (#144)

  • Generating outputs from conditional models, adding --unconditioned argument to disable usage of conditional probability outputs. (#147)

  • Add automatic cropping to zoom in on the depth range of interest. Add --auto-crop-threshold argument, which controls the threshold for when this occurs. (#149)

  • Add --list-checkpoints action, which lists the available checkpoints. (#150)

  • Fast fail if outputs already exist before processing already begins (and overwrite mode is not enabled). (#151)

  • Import generated line and region predictions from the .evl and .evr files into the .ev file and save it with the new lines and regions included. The --no-ev-import argument prevents this behaviour. (#152)

  • Add customisation of imported lines. The --suffix-var argument controls the suffix append to the name of the line variable. The --overwrite-ev-lines argument controls whether lines are overwritten if lines already exist with the same name. Also add arguments to customise the colour and thickness of the lines. (#152)

  • Add --suffix-file argument, will allows a suffix common to all the output files to be set. (#152)

Miscellaneous#

  • Add -V alias for --version to all command line interfaces. (#84)

  • Loading data from CSV files which contain invalid characters outside the UTF-8 set (seen in the Grand Passage dataset’s csv files). (#101)

  • Handle raw and masked CSV data of different sizes (occuring in Grand Passage’s csv files due to dropped rows containing invalid chararcters). (#101)

  • Add seed argument to separation script. (#56)

  • Add sample script to extract raw training data from ev files. (#55)

Version 0.1.4#

Release date: 2020-05-19. Full commit changelog.

Added#

  • Add ability to set orientation of echosounder with --facing argument (#77) The orientation is shown to the user if it was automatically detected as upward-facing (#76)

Version 0.1.3#

Release date: 2020-05-16. Full commit changelog.

Fixed#

  • EVL writer needs to output time to nearest 0.1ms. (#72)

Added#

  • Add --suffix argument to the command line interface of ev2csv. (#71)

  • Add --variable-name argument to inference.py (the main command line interface). (#74)

Version 0.1.2#

Release date: 2020-05-14. Full commit changelog.

Fixed#

  • In ev2csv, the files generator needed to be cast as a list to measure the number of files. (#66)

  • Echoview is no longer opened during dry-run mode. (#66)

  • In parse_files_in_folders (affecting ev2csv), string inputs were not being handled correctly. (#66)

  • Relative paths need to be converted to absolute paths before using them in Echoview. (#68, #69)

Added#

  • Support hiding or minimizing Echoview while the script is running. The default behaviour is now to hide the window if it was created by the script. The same Echoview window is used throughout the the processing. (#67)

Version 0.1.1#

Release date: 2020-05-12. Full commit changelog.

Fixed#

  • Padding in echofilter.modules.pathing.FlexibleConcat2d when only one dim size doesn’t match. (#64)

Version 0.1.0#

Release date: 2020-05-12. Initial release.