clustpy.deep.neural_networks package

Submodules

clustpy.deep.neural_networks.convolutional_autoencoder module

class clustpy.deep.neural_networks.convolutional_autoencoder.ConvolutionalAutoencoder(input_height: int, fc_layers: list, conv_encoder_name: str = 'resnet18', conv_decoder_name: str = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.ReLU'>, fc_decoder_layers: list = None, decoder_output_fn: ~torch.nn.modules.module.Module = None, pretrained_encoder_weights: ~torchvision.models._api.Weights = None, pretrained_decoder_weights: ~torchvision.models._api.Weights = None, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None, **fc_kwargs)[source]

Bases: _AbstractAutoencoder

A convolutional autoencoder based on the ResNet architecture.

Parameters:

input_height (int) – height of the images for the decoder (assume square images)
fc_layers (list) – list of the different layer sizes from flattened convolutional layer input to embedding. First input needs to be 512 if conv_encoder_name=”resnet18” and 2048 if conv_encoder_name=”resnet50”. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.
conv_encoder_name (str) – name of convolutional resnet encoder part of the autoencoder. Can be ‘resnet18’ or ‘resnet50’ (default: ‘resnet18’)
conv_decoder_name (str) – name of convolutional resnet decoder part of the autoencoder. Can be ‘resnet18’ or ‘resnet50’. If None it will be the same as conv_encoder_name (default: None)
activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
fc_decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)
pretrained_encoder_weights (torchvision.models._api.Weights) – weights from torch.vision.models, indicates whether pretrained resnet weights should be used for the encoder. (default: None)
pretrained_decoder_weights (torchvision.models._api.Weights) – weights from torch.vision.models, indicates whether pretrained resnet weights should be used for the decoder (not implemented yet). (default: None)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)
fc_kwargs (dict) – additional parameters for FullyConnectedBlock

conv_encoder

convolutional resnet encoder part of the autoencoder

Type:: ResNetEncoder

conv_decoder

convolutional resnet decoder part of the autoencoder

Type:: ResNetEncoder

fc_encoder

fully connected encoder part of the convolutional autoencoder, together with conv_encoder is responsible for embedding data points

Type:: FullyConnectedBlock

fc_decoder

fully connected decoder part of the convolutional autoencoder, together with conv_decoder is responsible for reconstructing data points from the embedding

Type:: FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

References

He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

and

LeCun, Yann, et al. “Backpropagation applied to handwritten zip code recognition.” Neural computation 1.4 (1989): 541-551.

decode(embedded: Tensor) → Tensor[source]

Apply the decoder function to embedded. Runs x through the conv_decoder and then the fc_decoder.

Parameters:: embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points
Returns:: decoded – returns the reconstruction of embedded
Return type:: torch.Tensor

encode(x: Tensor) → Tensor[source]

Apply the encoder function to x. Runs x through the conv_encoder and then the fc_encoder.

Parameters:: x (torch.Tensor) – input data point, can also be a mini-batch of points
Returns:: embedded – the embedded data point with dimensionality embedding_size
Return type:: torch.Tensor

clustpy.deep.neural_networks.feedforward_autoencoder module

@authors: Lukas Miklautz

class clustpy.deep.neural_networks.feedforward_autoencoder.FeedforwardAutoencoder(layers: list, batch_norm: bool = False, dropout: float = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list = None, decoder_output_fn: ~torch.nn.modules.module.Module = None, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None)[source]

Bases: _AbstractAutoencoder

A flexible feedforward autoencoder.

Parameters:

layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.
batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)
dropout (float) – Set the amount of dropout you want to use (default: None)
activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)
decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

References

E.g.: Ballard, Dana H. “Modular learning in neural networks.” Aaai. Vol. 647. 1987.

or

Kramer, Mark A. “Nonlinear principal component analysis using autoassociative neural networks.” AIChE journal 37.2 (1991): 233-243.

decode(embedded: Tensor) → Tensor[source]

Apply the decoder function to embedded.

Parameters:: embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points
Returns:: decoded – returns the reconstruction of embedded
Return type:: torch.Tensor

encode(x: Tensor) → Tensor[source]

Apply the encoder function to x.

Parameters:: x (torch.Tensor) – input data point, can also be a mini-batch of points
Returns:: embedded – the embedded data point with dimensionality embedding_size
Return type:: torch.Tensor

clustpy.deep.neural_networks.neighbor_encoder module

@authors: Collin Leiber

class clustpy.deep.neural_networks.neighbor_encoder.NeighborEncoder(layers: list, n_neighbors: int, decode_self: bool = False, batch_norm: bool = False, dropout: float = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list = None, decoder_output_fn: ~torch.nn.modules.module.Module = None, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None)[source]

Bases: FeedforwardAutoencoder

A NeighborEncoder. Does not compare the reconstruction of an object to itself but to its nearest neighbors. For more information see the stated reference. If n_neighbors is 0 and decode_self is true, the NeighborEncoder will work as a regular FeedforwardAutoencoder.

Parameters:

layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.
n_neighbors (int) – the number of nearest neighbors to be considered
decode_self (bool) – specifies whether a point itself should also be decoded (default: False)
batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)
dropout (float) – Set the amount of dropout you want to use (default: None)
activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)
decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing the data point itself (class is FullyConnectedBlock). Only used if decode_self is true.

Type:: FullyConnectedBlock

neighbor_decoders

list containing one decoder network (class is FullyConnectedBlock) for each nearest neighbor

Type:: list

fitted

boolean value indicating whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

Examples

>>> from clustpy.data import create_subspace_data
>>> from clustpy.deep import get_dataloader
>>> from scipy.spatial.distance import pdist, squareform

>>> X, L = create_subspace_data(1500, subspace_features=(3, 50), random_state=1)
>>> n_neighbors = 3

>>> dist_matrix = squareform(pdist(X))
>>> neighbor_ids = np.argsort(dist_matrix, axis=1)
>>> neighbors = [X[neighbor_ids[:, 1 + i]] for i in range(n_neighbors)]
>>> # Alternatively: neighbors = get_neighbors_batchwise(X, n_neighbors)

>>> dataloader = get_dataloader(X, 256, True, additional_inputs=neighbors)
>>> neighbor_encoder = NeighborEncoder(layers=[X.shape[1], 512, 256, 10], n_neighbors=n_neighbors, decode_self=False)
>>> neighbor_encoder.fit(dataloader=dataloader, n_epochs=5, lr=1e-3)

References

Yeh, Chin-Chia Michael, et al. “Representation Learning by Reconstructing Neighborhoods.” arXiv preprint arXiv:1811.01557 (2018).

decode(embedded: Tensor) → Tensor[source]

Apply the decoder function of each neighbor network to embedded. Returns a (n_neighbors x batch_size x dimensionality) tensor if decode_self is false, else a (n_neighbors + 1 x batch_size x dimensionality) tensor

Parameters:: embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points
Returns:: decoded_neighbors – returns the reconstructions of the embedded sample concerning its neighbor decoders
Return type:: torch.Tensor

fit(n_epochs: int, optimizer_params: dict, batch_size: int = 128, dataloader: ~torch.utils.data.dataloader.DataLoader = None, evalloader: ~torch.utils.data.dataloader.DataLoader = None, optimizer_class: ~torch.optim.optimizer.Optimizer = <class 'torch.optim.adam.Adam'>, ssl_loss_fn: ~torch.nn.modules.loss._Loss = MSELoss(), patience: int = 5, scheduler: <module 'torch.optim.lr_scheduler' from '/home/docs/checkouts/readthedocs.org/user_builds/clustpy/envs/v0.0.2/lib/python3.12/site-packages/torch/optim/lr_scheduler.py'> = None, scheduler_params: dict = None, corruption_fn: ~collections.abc.Callable = None, model_path: str = None) → NeighborEncoder[source]

Trains the NeighborEncoder in place. Equal to fit function of the FeedforwardAutoencoder but does only work with a dataloader (not with a regular data array). This is because the dataloader must contain the nearest neighbors of each point at the positions 2, 3, ….

Parameters:

n_epochs (int) – number of epochs for training
optimizer_params (dict) – parameters of the optimizer, includes the learning rate
batch_size (int) – size of the data batches (default: 128)
dataloader (torch.utils.data.DataLoader) – dataloader to be used for training (default: default=None)
evalloader (torch.utils.data.DataLoader) – dataloader to be used for evaluation, early stopping and learning rate scheduling if scheduler=torch.optim.lr_scheduler.ReduceLROnPlateau (default: None)
optimizer_class (torch.optim.Optimizer) – optimizer to be used (default: torch.optim.Adam)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss (default: torch.nn.MSELoss())
patience (int) – patience parameter for EarlyStopping (default: 5)
scheduler (torch.optim.lr_scheduler) – learning rate scheduler that should be used. If torch.optim.lr_scheduler.ReduceLROnPlateau is used then the behaviour is matched by providing the validation_loss calculated based on samples from evalloader (default: None)
scheduler_params (dict) – dictionary of the parameters of the scheduler object (default: None)
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)
model_path (str) – if specified will save the trained model to the location. If evalloader is used, then only the best model w.r.t. evaluation loss is saved (default: None)

Returns:

self – this instance of the NeighborEncoder

Return type:

NeighborEncoder

loss(batch: list, ssl_loss_fn: ~torch.nn.modules.loss._Loss, device: ~torch.device, corruption_fn: ~collections.abc.Callable = None) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)[source]

Calculate the loss of a single batch of data. Corresponds to the sum of losses concerning each neighbor. batch must contain the data object at the first position and the neighbors at the following positions.

Parameters:

batch (list) – the different parts of a dataloader (id, samples, 1-nearest-neighbor, 2-nearest-neighbor, …)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss
device (torch.device) – device to be trained on
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)

Returns:

loss – the sum of the reconstruction losses of the input sample, the embedded input sample, the reconstructions of the embedded sample concerning its neighbor decoders

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor)

clustpy.deep.neural_networks.neighbor_encoder.get_neighbors_batchwise(X: ndarray, n_neighbors: int, metric: str = 'sqeuclidean', batch_size: int = 10000) → list[source]

For large datasets it is often not possible to determine the nearest neighbors in a trivial manner. Therefore, here is an implementation that calculates the nearest neighbors in batches. Ignores the objects themselves (with distance of 0) as nearest neighbors. It reduces the memory consumption of a trivial nearest neighbor implementation from (data_size x data_size) to (batch_size x data_size). A list is returned, which can be given as additional input into a DataLoader and is therefore directly compatible with the NeighborEncoder. Due to runtime concerns it is still recommended to use a more complex nearest neighbor retrieval implementation (e.g. from sklearn.neighbor)!

Parameters:

X (np.ndarray) – The given data set
n_neighbors (int) – The number of nearest neighbors to identify
metric (str) – The distance metric to be used. See scipy.spatial.distance.cdist for more information (default: sqeuclidean)
batch_size (int) – The size of the batches (default: 10000)

Returns:

nearest_neigbors – A list containing the nearest neighbors as torch.Tensors, i.e. [1-nearest-neighbor tensor, 2-nearest-neighbor tensor, …]

Return type:

list

Examples

>>> from clustpy.data import create_subspace_data
>>> from clustpy.deep import get_dataloader
>>> X, L = create_subspace_data(1500, subspace_features=(3, 50), random_state=1)
>>> n_neighbors = 3
>>> neighbors = get_neighbors_batchwise(X, n_neighbors)
>>> dataloader = get_dataloader(X, 256, True, additional_inputs=neighbors)
>>> neighbor_encoder = NeighborEncoder(layers=[X.shape[1], 512, 256, 10], n_neighbors=n_neighbors)
>>> neighbor_encoder.fit(dataloader=dataloader, n_epochs=5, lr=1e-3)

clustpy.deep.neural_networks.stacked_autoencoder module

@authors: Collin Leiber

class clustpy.deep.neural_networks.stacked_autoencoder.StackedAutoencoder(layers: list, batch_norm: bool = False, dropout: float = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_output_fn: ~torch.nn.modules.module.Module = None, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None)[source]

Bases: FeedforwardAutoencoder

A stacked autoencoder. Regarding its architecture, it corresponds to a standard FeedforwardAutoencoder but uses a different training strategy. First, each layer is trained separately in a greedy manner, referred to as layer-wise training. Afterward, all layers are trained at once to finetune the autoencoder.

Parameters:

layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. Note that in case of a StackedAutoencoder the decoder requires the reversed structure of the encoder.
batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)
dropout (float) – Set the amount of dropout you want to use (default: None)
activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

References

E.g.: Bengio, Yoshua, et al. “Greedy layer-wise training of deep networks.” Advances in neural information processing systems 19 (2006).

or

Vincent, Pascal, et al. “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.” Journal of machine learning research 11.12 (2010).

fit(n_epochs_per_layer: int = 20, n_epochs: int = 100, optimizer_params: dict = None, batch_size: int = 128, data: ~numpy.ndarray | ~torch.Tensor = None, data_eval: ~numpy.ndarray | ~torch.Tensor = None, dataloader: ~torch.utils.data.dataloader.DataLoader = None, evalloader: ~torch.utils.data.dataloader.DataLoader = None, optimizer_class: ~torch.optim.optimizer.Optimizer = <class 'torch.optim.adam.Adam'>, ssl_loss_fn: ~torch.nn.modules.loss._Loss = MSELoss(), patience: int = 5, scheduler: <module 'torch.optim.lr_scheduler' from '/home/docs/checkouts/readthedocs.org/user_builds/clustpy/envs/v0.0.2/lib/python3.12/site-packages/torch/optim/lr_scheduler.py'> = None, scheduler_params: dict = {}, corruption_fn: ~collections.abc.Callable = None, model_path: str = None) → StackedAutoencoder[source]

Trains the autoencoder in place. First, a greedy layer-wise training is performed. Afterward, the weights are finetuned by training all layer simultaneously.

Parameters:

n_epochs_per_layer (int) – number of epochs for training each layer separately (default: 20)
n_epochs (int) – number of epochs for the final finetuning (default: 100)
optimizer_params (dict) – parameters of the optimizer, includes the learning rate (default: {“lr”: 1e-3})
batch_size (int) – size of the data batches (default: 128)
data (np.ndarray | torch.Tensor) – train data set. If data is passed then dataloader can remain empty (default: None)
data_eval (np.ndarray | torch.Tensor) – evaluation data set. If data_eval is passed then evalloader can remain empty. Only used for finetuning (default: None)
dataloader (torch.utils.data.DataLoader) – dataloader to be used for training (default: default=None)
evalloader (torch.utils.data.DataLoader) – dataloader to be used for evaluation, early stopping and learning rate scheduling if scheduler=torch.optim.lr_scheduler.ReduceLROnPlateau. Only used for finetuning (default: None)
optimizer_class (torch.optim.Optimizer) – optimizer to be used (default: torch.optim.Adam)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss (default: torch.nn.MSELoss())
patience (int) – patience parameter for EarlyStopping. Only used for finetuning (default: 5)
scheduler (torch.optim.lr_scheduler) – learning rate scheduler that should be used. If torch.optim.lr_scheduler.ReduceLROnPlateau is used then the behaviour is matched by providing the validation_loss calculated based on samples from evalloader. Only used for finetuning (default: None)
scheduler_params (dict) – dictionary of the parameters of the scheduler object. Only used for finetuning (default: {})
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)
model_path (str) – if specified will save the trained model to the location. If evalloader is used, then only the best model w.r.t. evaluation loss is saved (default: None)

Returns:

self – this instance of the autoencoder

Return type:

StackedAutoencoder

Raises:

ValueError – data cannot be None if dataloader is None:
ValueError – evalloader cannot be None if scheduler=torch.optim.lr_scheduler.ReduceLROnPlateau:

layerwise_training(n_epochs_per_layer: int = 20, optimizer_params: dict = None, batch_size: int = 128, data: ~numpy.ndarray | ~torch.Tensor = None, dataloader: ~torch.utils.data.dataloader.DataLoader = None, optimizer_class: ~torch.optim.optimizer.Optimizer = <class 'torch.optim.adam.Adam'>, ssl_loss_fn: ~torch.nn.modules.loss._Loss = MSELoss(), corruption_fn: ~collections.abc.Callable = None) → StackedAutoencoder[source]

Trains the autoencoder in a greedy layer-wise fashion.

Parameters:

n_epochs_per_layer (int) – number of epochs for training each layer separately (default: 20)
optimizer_params (dict) – parameters of the optimizer, includes the learning rate (default: {“lr”: 1e-3})
batch_size (int) – size of the data batches (default: 128)
data (np.ndarray | torch.Tensor) – train data set. If data is passed then dataloader can remain empty (default: None)
dataloader (torch.utils.data.DataLoader) – dataloader to be used for training (default: default=None)
optimizer_class (torch.optim.Optimizer) – optimizer to be used (default: torch.optim.Adam)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss (default: torch.nn.MSELoss())
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)

Returns:

self – this instance of the autoencoder

Return type:

StackedAutoencoder

Raises:

ValueError – data cannot be None if dataloader is None:

clustpy.deep.neural_networks.variational_autoencoder module

@authors: Lukas Miklautz, Donatella Novakovic, Collin Leiber

class clustpy.deep.neural_networks.variational_autoencoder.VariationalAutoencoder(layers: list, batch_norm: bool = False, dropout: float = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list = None, decoder_output_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.Sigmoid'>, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None)[source]

Bases: FeedforwardAutoencoder

A variational autoencoder (VAE).

Parameters:

layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the dimension of the mean and variance value in the central layer. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.
batch_norm (bool) – set True if you want to use torch.nn.BatchNorm1d (default: False)
dropout (float) – set the amount of dropout you want to use (default: None)
activation (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)
decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: torch.nn.Sigmoid)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

mean

mean value of the central layer

Type:: torch.nn.Linear

log_variance

logarithmic variance of the central layer (use logarithm of variance - numerical purposes)

Type:: torch.nn.Linear

fitted

boolean value indicating whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

References

Kingma, Diederik P., and Max Welling. “Auto-encoding variational Bayes.” Int. Conf. on Learning Representations.

encode(x: ~torch.Tensor) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>)[source]

Apply the encoder function to x. Overwrites function from FeedforwardAutoencoder.

Parameters:: x (torch.Tensor) – input data point, can also be a mini-batch of points
Returns:: tuple – mean value of the central VAE layer, logarithmic variance value of the central VAE layer (use logarithm of variance - numerical purposes)
Return type:: (torch.Tensor, torch.Tensor)

forward(x: ~torch.Tensor) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)[source]

Applies both the encode and decode function. The forward function is automatically called if we call self(x). Overwrites function from FeedforwardAutoencoder.

Parameters:: x (torch.Tensor) – input data point, can also be a mini-batch of embedded points
Returns:: tuple – sampling using q_mean and q_logvar, mean value of the central VAE layer, logarithmic variance value of the central VAE layer (use logarithm of variance - numerical purposes), the reconstruction of the data point
Return type:: (torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor)

loss(batch: list, ssl_loss_fn: ~torch.nn.modules.loss._Loss, device: ~torch.device, corruption_fn: ~collections.abc.Callable = None, beta: float = 1) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)[source]

Calculate the loss of a single batch of data.

Parameters:

batch (list) – the different parts of a dataloader (id, samples, …)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss
device (torch.device) – device to be trained on
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)
beta (float) – weighting of the KL loss (default: 1)

Returns:

total_loss – the ssl loss of the input sample, the sampling, the reconstruction of the data point

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor)

Module contents

class clustpy.deep.neural_networks.ConvolutionalAutoencoder(input_height: int, fc_layers: list, conv_encoder_name: str = 'resnet18', conv_decoder_name: str = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.ReLU'>, fc_decoder_layers: list = None, decoder_output_fn: ~torch.nn.modules.module.Module = None, pretrained_encoder_weights: ~torchvision.models._api.Weights = None, pretrained_decoder_weights: ~torchvision.models._api.Weights = None, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None, **fc_kwargs)[source]

Bases: _AbstractAutoencoder

A convolutional autoencoder based on the ResNet architecture.

Parameters:

input_height (int) – height of the images for the decoder (assume square images)
fc_layers (list) – list of the different layer sizes from flattened convolutional layer input to embedding. First input needs to be 512 if conv_encoder_name=”resnet18” and 2048 if conv_encoder_name=”resnet50”. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.
conv_encoder_name (str) – name of convolutional resnet encoder part of the autoencoder. Can be ‘resnet18’ or ‘resnet50’ (default: ‘resnet18’)
conv_decoder_name (str) – name of convolutional resnet decoder part of the autoencoder. Can be ‘resnet18’ or ‘resnet50’. If None it will be the same as conv_encoder_name (default: None)
activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
fc_decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)
pretrained_encoder_weights (torchvision.models._api.Weights) – weights from torch.vision.models, indicates whether pretrained resnet weights should be used for the encoder. (default: None)
pretrained_decoder_weights (torchvision.models._api.Weights) – weights from torch.vision.models, indicates whether pretrained resnet weights should be used for the decoder (not implemented yet). (default: None)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)
fc_kwargs (dict) – additional parameters for FullyConnectedBlock

conv_encoder

convolutional resnet encoder part of the autoencoder

Type:: ResNetEncoder

conv_decoder

convolutional resnet decoder part of the autoencoder

Type:: ResNetEncoder

fc_encoder

fully connected encoder part of the convolutional autoencoder, together with conv_encoder is responsible for embedding data points

Type:: FullyConnectedBlock

fc_decoder

fully connected decoder part of the convolutional autoencoder, together with conv_decoder is responsible for reconstructing data points from the embedding

Type:: FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

References

He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

and

LeCun, Yann, et al. “Backpropagation applied to handwritten zip code recognition.” Neural computation 1.4 (1989): 541-551.

decode(embedded: Tensor) → Tensor[source]

Apply the decoder function to embedded. Runs x through the conv_decoder and then the fc_decoder.

Parameters:: embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points
Returns:: decoded – returns the reconstruction of embedded
Return type:: torch.Tensor

encode(x: Tensor) → Tensor[source]

Apply the encoder function to x. Runs x through the conv_encoder and then the fc_encoder.

Parameters:: x (torch.Tensor) – input data point, can also be a mini-batch of points
Returns:: embedded – the embedded data point with dimensionality embedding_size
Return type:: torch.Tensor

class clustpy.deep.neural_networks.FeedforwardAutoencoder(layers: list, batch_norm: bool = False, dropout: float = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list = None, decoder_output_fn: ~torch.nn.modules.module.Module = None, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None)[source]

Bases: _AbstractAutoencoder

A flexible feedforward autoencoder.

Parameters:

layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.
batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)
dropout (float) – Set the amount of dropout you want to use (default: None)
activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)
decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

References

E.g.: Ballard, Dana H. “Modular learning in neural networks.” Aaai. Vol. 647. 1987.

or

Kramer, Mark A. “Nonlinear principal component analysis using autoassociative neural networks.” AIChE journal 37.2 (1991): 233-243.

decode(embedded: Tensor) → Tensor[source]

Apply the decoder function to embedded.

Parameters:: embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points
Returns:: decoded – returns the reconstruction of embedded
Return type:: torch.Tensor

encode(x: Tensor) → Tensor[source]

Apply the encoder function to x.

Parameters:: x (torch.Tensor) – input data point, can also be a mini-batch of points
Returns:: embedded – the embedded data point with dimensionality embedding_size
Return type:: torch.Tensor

class clustpy.deep.neural_networks.NeighborEncoder(layers: list, n_neighbors: int, decode_self: bool = False, batch_norm: bool = False, dropout: float = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list = None, decoder_output_fn: ~torch.nn.modules.module.Module = None, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None)[source]

Bases: FeedforwardAutoencoder

A NeighborEncoder. Does not compare the reconstruction of an object to itself but to its nearest neighbors. For more information see the stated reference. If n_neighbors is 0 and decode_self is true, the NeighborEncoder will work as a regular FeedforwardAutoencoder.

Parameters:

layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.
n_neighbors (int) – the number of nearest neighbors to be considered
decode_self (bool) – specifies whether a point itself should also be decoded (default: False)
batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)
dropout (float) – Set the amount of dropout you want to use (default: None)
activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)
decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing the data point itself (class is FullyConnectedBlock). Only used if decode_self is true.

Type:: FullyConnectedBlock

neighbor_decoders

list containing one decoder network (class is FullyConnectedBlock) for each nearest neighbor

Type:: list

fitted

boolean value indicating whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

Examples

>>> from clustpy.data import create_subspace_data
>>> from clustpy.deep import get_dataloader
>>> from scipy.spatial.distance import pdist, squareform

>>> X, L = create_subspace_data(1500, subspace_features=(3, 50), random_state=1)
>>> n_neighbors = 3

>>> dist_matrix = squareform(pdist(X))
>>> neighbor_ids = np.argsort(dist_matrix, axis=1)
>>> neighbors = [X[neighbor_ids[:, 1 + i]] for i in range(n_neighbors)]
>>> # Alternatively: neighbors = get_neighbors_batchwise(X, n_neighbors)

>>> dataloader = get_dataloader(X, 256, True, additional_inputs=neighbors)
>>> neighbor_encoder = NeighborEncoder(layers=[X.shape[1], 512, 256, 10], n_neighbors=n_neighbors, decode_self=False)
>>> neighbor_encoder.fit(dataloader=dataloader, n_epochs=5, lr=1e-3)

References

Yeh, Chin-Chia Michael, et al. “Representation Learning by Reconstructing Neighborhoods.” arXiv preprint arXiv:1811.01557 (2018).

decode(embedded: Tensor) → Tensor[source]

Apply the decoder function of each neighbor network to embedded. Returns a (n_neighbors x batch_size x dimensionality) tensor if decode_self is false, else a (n_neighbors + 1 x batch_size x dimensionality) tensor

Parameters:: embedded (torch.Tensor) – embedded data point, can also be a mini-batch of embedded points
Returns:: decoded_neighbors – returns the reconstructions of the embedded sample concerning its neighbor decoders
Return type:: torch.Tensor

fit(n_epochs: int, optimizer_params: dict, batch_size: int = 128, dataloader: ~torch.utils.data.dataloader.DataLoader = None, evalloader: ~torch.utils.data.dataloader.DataLoader = None, optimizer_class: ~torch.optim.optimizer.Optimizer = <class 'torch.optim.adam.Adam'>, ssl_loss_fn: ~torch.nn.modules.loss._Loss = MSELoss(), patience: int = 5, scheduler: <module 'torch.optim.lr_scheduler' from '/home/docs/checkouts/readthedocs.org/user_builds/clustpy/envs/v0.0.2/lib/python3.12/site-packages/torch/optim/lr_scheduler.py'> = None, scheduler_params: dict = None, corruption_fn: ~collections.abc.Callable = None, model_path: str = None) → NeighborEncoder[source]

Trains the NeighborEncoder in place. Equal to fit function of the FeedforwardAutoencoder but does only work with a dataloader (not with a regular data array). This is because the dataloader must contain the nearest neighbors of each point at the positions 2, 3, ….

Parameters:

n_epochs (int) – number of epochs for training
optimizer_params (dict) – parameters of the optimizer, includes the learning rate
batch_size (int) – size of the data batches (default: 128)
dataloader (torch.utils.data.DataLoader) – dataloader to be used for training (default: default=None)
evalloader (torch.utils.data.DataLoader) – dataloader to be used for evaluation, early stopping and learning rate scheduling if scheduler=torch.optim.lr_scheduler.ReduceLROnPlateau (default: None)
optimizer_class (torch.optim.Optimizer) – optimizer to be used (default: torch.optim.Adam)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss (default: torch.nn.MSELoss())
patience (int) – patience parameter for EarlyStopping (default: 5)
scheduler (torch.optim.lr_scheduler) – learning rate scheduler that should be used. If torch.optim.lr_scheduler.ReduceLROnPlateau is used then the behaviour is matched by providing the validation_loss calculated based on samples from evalloader (default: None)
scheduler_params (dict) – dictionary of the parameters of the scheduler object (default: None)
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)
model_path (str) – if specified will save the trained model to the location. If evalloader is used, then only the best model w.r.t. evaluation loss is saved (default: None)

Returns:

self – this instance of the NeighborEncoder

Return type:

NeighborEncoder

loss(batch: list, ssl_loss_fn: ~torch.nn.modules.loss._Loss, device: ~torch.device, corruption_fn: ~collections.abc.Callable = None) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)[source]

Calculate the loss of a single batch of data. Corresponds to the sum of losses concerning each neighbor. batch must contain the data object at the first position and the neighbors at the following positions.

Parameters:

batch (list) – the different parts of a dataloader (id, samples, 1-nearest-neighbor, 2-nearest-neighbor, …)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss
device (torch.device) – device to be trained on
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)

Returns:

loss – the sum of the reconstruction losses of the input sample, the embedded input sample, the reconstructions of the embedded sample concerning its neighbor decoders

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor)

class clustpy.deep.neural_networks.StackedAutoencoder(layers: list, batch_norm: bool = False, dropout: float = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_output_fn: ~torch.nn.modules.module.Module = None, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None)[source]

Bases: FeedforwardAutoencoder

A stacked autoencoder. Regarding its architecture, it corresponds to a standard FeedforwardAutoencoder but uses a different training strategy. First, each layer is trained separately in a greedy manner, referred to as layer-wise training. Afterward, all layers are trained at once to finetune the autoencoder.

Parameters:

layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the embedding dimension. Note that in case of a StackedAutoencoder the decoder requires the reversed structure of the encoder.
batch_norm (bool) – Set True if you want to use torch.nn.BatchNorm1d (default: False)
dropout (float) – Set the amount of dropout you want to use (default: None)
activation_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: None)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

fitted

indicates whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

References

E.g.: Bengio, Yoshua, et al. “Greedy layer-wise training of deep networks.” Advances in neural information processing systems 19 (2006).

or

Vincent, Pascal, et al. “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.” Journal of machine learning research 11.12 (2010).

fit(n_epochs_per_layer: int = 20, n_epochs: int = 100, optimizer_params: dict = None, batch_size: int = 128, data: ~numpy.ndarray | ~torch.Tensor = None, data_eval: ~numpy.ndarray | ~torch.Tensor = None, dataloader: ~torch.utils.data.dataloader.DataLoader = None, evalloader: ~torch.utils.data.dataloader.DataLoader = None, optimizer_class: ~torch.optim.optimizer.Optimizer = <class 'torch.optim.adam.Adam'>, ssl_loss_fn: ~torch.nn.modules.loss._Loss = MSELoss(), patience: int = 5, scheduler: <module 'torch.optim.lr_scheduler' from '/home/docs/checkouts/readthedocs.org/user_builds/clustpy/envs/v0.0.2/lib/python3.12/site-packages/torch/optim/lr_scheduler.py'> = None, scheduler_params: dict = {}, corruption_fn: ~collections.abc.Callable = None, model_path: str = None) → StackedAutoencoder[source]

Trains the autoencoder in place. First, a greedy layer-wise training is performed. Afterward, the weights are finetuned by training all layer simultaneously.

Parameters:

n_epochs_per_layer (int) – number of epochs for training each layer separately (default: 20)
n_epochs (int) – number of epochs for the final finetuning (default: 100)
optimizer_params (dict) – parameters of the optimizer, includes the learning rate (default: {“lr”: 1e-3})
batch_size (int) – size of the data batches (default: 128)
data (np.ndarray | torch.Tensor) – train data set. If data is passed then dataloader can remain empty (default: None)
data_eval (np.ndarray | torch.Tensor) – evaluation data set. If data_eval is passed then evalloader can remain empty. Only used for finetuning (default: None)
dataloader (torch.utils.data.DataLoader) – dataloader to be used for training (default: default=None)
evalloader (torch.utils.data.DataLoader) – dataloader to be used for evaluation, early stopping and learning rate scheduling if scheduler=torch.optim.lr_scheduler.ReduceLROnPlateau. Only used for finetuning (default: None)
optimizer_class (torch.optim.Optimizer) – optimizer to be used (default: torch.optim.Adam)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss (default: torch.nn.MSELoss())
patience (int) – patience parameter for EarlyStopping. Only used for finetuning (default: 5)
scheduler (torch.optim.lr_scheduler) – learning rate scheduler that should be used. If torch.optim.lr_scheduler.ReduceLROnPlateau is used then the behaviour is matched by providing the validation_loss calculated based on samples from evalloader. Only used for finetuning (default: None)
scheduler_params (dict) – dictionary of the parameters of the scheduler object. Only used for finetuning (default: {})
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)
model_path (str) – if specified will save the trained model to the location. If evalloader is used, then only the best model w.r.t. evaluation loss is saved (default: None)

Returns:

self – this instance of the autoencoder

Return type:

StackedAutoencoder

Raises:

ValueError – data cannot be None if dataloader is None:
ValueError – evalloader cannot be None if scheduler=torch.optim.lr_scheduler.ReduceLROnPlateau:

layerwise_training(n_epochs_per_layer: int = 20, optimizer_params: dict = None, batch_size: int = 128, data: ~numpy.ndarray | ~torch.Tensor = None, dataloader: ~torch.utils.data.dataloader.DataLoader = None, optimizer_class: ~torch.optim.optimizer.Optimizer = <class 'torch.optim.adam.Adam'>, ssl_loss_fn: ~torch.nn.modules.loss._Loss = MSELoss(), corruption_fn: ~collections.abc.Callable = None) → StackedAutoencoder[source]

Trains the autoencoder in a greedy layer-wise fashion.

Parameters:

n_epochs_per_layer (int) – number of epochs for training each layer separately (default: 20)
optimizer_params (dict) – parameters of the optimizer, includes the learning rate (default: {“lr”: 1e-3})
batch_size (int) – size of the data batches (default: 128)
data (np.ndarray | torch.Tensor) – train data set. If data is passed then dataloader can remain empty (default: None)
dataloader (torch.utils.data.DataLoader) – dataloader to be used for training (default: default=None)
optimizer_class (torch.optim.Optimizer) – optimizer to be used (default: torch.optim.Adam)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss (default: torch.nn.MSELoss())
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)

Returns:

self – this instance of the autoencoder

Return type:

StackedAutoencoder

Raises:

ValueError – data cannot be None if dataloader is None:

class clustpy.deep.neural_networks.VariationalAutoencoder(layers: list, batch_norm: bool = False, dropout: float = None, activation_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.LeakyReLU'>, bias: bool = True, decoder_layers: list = None, decoder_output_fn: ~torch.nn.modules.module.Module = <class 'torch.nn.modules.activation.Sigmoid'>, work_on_copy: bool = True, random_state: ~numpy.random.mtrand.RandomState | int = None)[source]

Bases: FeedforwardAutoencoder

A variational autoencoder (VAE).

Parameters:

layers (list) – list of the different layer sizes from input to embedding, e.g. an example architecture for MNIST [784, 512, 256, 10], where 784 is the input dimension and 10 the dimension of the mean and variance value in the central layer. If decoder_layers are not specified then the decoder is symmetric and goes in the same order from embedding to input.
batch_norm (bool) – set True if you want to use torch.nn.BatchNorm1d (default: False)
dropout (float) – set the amount of dropout you want to use (default: None)
activation (torch.nn.Module) – activation function from torch.nn, set the activation function for the hidden layers, if None then it will be linear (default: torch.nn.LeakyReLU)
bias (bool) – set False if you do not want to use a bias term in the linear layers (default: True)
decoder_layers (list) – list of different layer sizes from embedding to output of the decoder. If set to None, will be symmetric to layers (default: None)
decoder_output_fn (torch.nn.Module) – activation function from torch.nn, set the activation function for the decoder output layer, if None then it will be linear. E.g. set to torch.nn.Sigmoid if you want to scale the decoder output between 0 and 1 (default: torch.nn.Sigmoid)
work_on_copy (bool) – If set to true, deep clustering algorithms will optimize a copy of the autoencoder and not the autoencoder itself. Ensures that the same autoencoder can be used by multiple deep clustering algorithms. As copies of this object are created, the memory requirement increases (default: True)
random_state (np.random.RandomState | int) – use a fixed random state to get a repeatable solution. Can also be of type int (default: None)

encoder

encoder part of the autoencoder, responsible for embedding data points (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

decoder

decoder part of the autoencoder, responsible for reconstructing data points from the embedding (class is FullyConnectedBlock)

Type:: FullyConnectedBlock

mean

mean value of the central layer

Type:: torch.nn.Linear

log_variance

logarithmic variance of the central layer (use logarithm of variance - numerical purposes)

Type:: torch.nn.Linear

fitted

boolean value indicating whether the autoencoder is already fitted

Type:: bool

work_on_copy

indicates whether deep clustering algorithms should work on a copy of the original autoencoder

Type:: bool

References

Kingma, Diederik P., and Max Welling. “Auto-encoding variational Bayes.” Int. Conf. on Learning Representations.

encode(x: ~torch.Tensor) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>)[source]

Apply the encoder function to x. Overwrites function from FeedforwardAutoencoder.

Parameters:: x (torch.Tensor) – input data point, can also be a mini-batch of points
Returns:: tuple – mean value of the central VAE layer, logarithmic variance value of the central VAE layer (use logarithm of variance - numerical purposes)
Return type:: (torch.Tensor, torch.Tensor)

forward(x: ~torch.Tensor) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)[source]

Applies both the encode and decode function. The forward function is automatically called if we call self(x). Overwrites function from FeedforwardAutoencoder.

Parameters:: x (torch.Tensor) – input data point, can also be a mini-batch of embedded points
Returns:: tuple – sampling using q_mean and q_logvar, mean value of the central VAE layer, logarithmic variance value of the central VAE layer (use logarithm of variance - numerical purposes), the reconstruction of the data point
Return type:: (torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor)

loss(batch: list, ssl_loss_fn: ~torch.nn.modules.loss._Loss, device: ~torch.device, corruption_fn: ~collections.abc.Callable = None, beta: float = 1) -> (<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>)[source]

Calculate the loss of a single batch of data.

Parameters:

batch (list) – the different parts of a dataloader (id, samples, …)
ssl_loss_fn (torch.nn.modules.loss._Loss) – self-supervised learning (ssl) loss function for training the network, e.g. reconstruction loss
device (torch.device) – device to be trained on
corruption_fn (Callable) – Can be used to corrupt the input data, e.g., when using a denoising autoencoder. Note that the function must match the data and the data loaders. For example, if the data is normalized, this may have to be taken into account in the corruption function - e.g. in case of salt and pepper noise (default: None)
beta (float) – weighting of the KL loss (default: 1)

Returns:

total_loss – the ssl loss of the input sample, the sampling, the reconstruction of the data point

Return type:

(torch.Tensor, torch.Tensor, torch.Tensor)